Modified responsibility sensitivity safety model
18 claims: 4 independent, 14 dependent
- 1プロセッサ回路に、ホスト車両のナビゲーション目標を達成するための計画された運転動作を取得する段階と、前記ホスト車両の周囲の環境を表すセンサデータをセンサデバイスから受信する段階と、前記環境内を移動している目標車両と、前記目標車両の現在の速度とを、前記センサデータから識別する段階と、前記目標車両の前記現在の速度と、前記目標車両の最大減速度とに基づいて、前記目標車両が完全停止に達するまでの停止距離を計算する段階と、前記目標車両が前記完全停止に達するまでの予測された軌道を計算する段階と、前記運転動作に対応する、前記ホスト車両の計画された軌道を計算する段階と、前記ホスト車両の前記計画された軌道と、前記目標車両の前記予測された軌道との交点を識別する段階と、安全要件を満たすための前記ホスト車両のブレーキ動作を決定する段階であって、前記ブレーキ動作は、前記ホスト車両の最大制動量よりも小さい、少なくとも最小制動量を適用し、前記安全要件は、前記ホスト車両が完全停止になるまでの停止距離であって前記目標車両の前記停止距離よりも小さい停止距離を含む、決定する段階と、前記ホスト車両の前記計画された軌道と、前記目標車両の前記予測された軌道との前記交点を識別することに応じて、前記ホスト車両に前記ブレーキ動作を適用させる段階であって、前記ブレーキ動作は、前記ホスト車両の前記計画された軌道を、変更された軌道に変更するために、前記ホスト車両を減速させ、前記ホスト車両の前記変更された軌道が前記目標車両の前記予測された軌道と交差しなくなるまで、前記ホスト車両に前記ブレーキ動作を適用させる段階と、を実行させるプログラム。
- 2前記目標車両の前記最大減速度は、前記目標車両の最大制動能力から提供される、請求項1に記載のプログラム。
- 3前記ブレーキ動作は、前記ホスト車両が、前記ホスト車両の前記最大制動量を使用せずに停止することを可能にする、請求項1または2に記載のプログラム。
- 4前記計画された運転動作は、前記ホスト車両の前記計画された軌道を継続することを含む、請求項1から3のいずれか一項に記載のプログラム。
- 5前記プロセッサ回路に、前記安全要件を満たすための、前記ホスト車両の操舵動作を決定する段階と、前記ホスト車両の前記計画された軌道を横方向に変更するために、前記ホスト車両に前記操舵動作を適用させる段階と、をさらに実行させる、請求項1から4のいずれか一項に記載のプログラム。
- 6前記計画された運転動作は、合流またはレーン変更操作の一部である、請求項1から5のいずれか一項に記載のプログラム。
- 7前記計画された運転動作は、追走操作の一部である、請求項1から6のいずれか一項に記載のプログラム。
- 8前記センサデバイスはカメラを含み、前記センサデータは少なくとも一つの画像を含む、請求項1から7のいずれか一項に記載のプログラム。
- 9前記センサデバイスは、レーダシステムまたはライダシステムの少なくとも一つを含み、前記センサデータは、前記レーダシステムまたは前記ライダシステムの前記少なくとも一つからのデータを含む、請求項1から8のいずれか一項に記載のプログラム。
- 10センサデバイスからセンサデータを受信するための少なくとも一つのインタフェースであって、前記センサデータはホスト車両の周囲の環境を表す、インタフェースと、前記ホスト車両のナビゲーション目標を達成するための計画された運転動作を取得し、前記環境内を移動している目標車両と、前記目標車両の現在の速度とを、前記センサデータから識別し、前記目標車両の前記現在の速度と、前記目標車両の最大減速度とに基づいて、前記目標車両が完全停止に達するまでの停止距離を計算し、前記目標車両が前記完全停止に達するまでの予測された軌道を計算し、前記運転動作に対応する、前記ホスト車両の計画された軌道を計算し、前記ホスト車両の前記計画された軌道と、前記目標車両の前記予測された軌道との交点を識別し、安全要件を満たすための前記ホスト車両のブレーキ動作を決定し、前記ブレーキ動作は、前記ホスト車両の最大制動量よりも小さい、少なくとも最小制動量を適用し、前記安全要件は、前記ホスト車両が完全停止になるまでの停止距離であって前記目標車両の前記停止距離よりも小さい停止距離を含み、前記ホスト車両の前記計画された軌道と、前記目標車両の前記予測された軌道との前記交点を識別することに応じて、前記ホスト車両に前記ブレーキ動作を適用させ、前記ブレーキ動作は、前記ホスト車両の前記計画された軌道を、変更された軌道に変更するために、前記ホスト車両を減速させ、前記ホスト車両の前記変更された軌道が前記目標車両の前記予測された軌道と交差しなくなるまで、前記ホスト車両に前記ブレーキ動作を適用させる、ための動作を実行するように構成される少なくとも一つのプロセッサと、を備える計算装置。
- 11前記目標車両の前記最大減速度は、前記目標車両の最大制動能力から提供される、請求項10に記載の計算装置。
- 12前記ブレーキ動作は、前記ホスト車両が、前記ホスト車両の前記最大制動量を使用せずに停止することを可能にする、請求項10または11に記載の計算装置。
- 13前記計画された運転動作は、前記ホスト車両の前記計画された軌道を継続することを含む、請求項10から12のいずれか一項に記載の計算装置。
- 14前記少なくとも一つのプロセッサは、前記安全要件を満たすための、前記ホスト車両の操舵動作を決定し、前記ホスト車両の前記計画された軌道を横方向に変更するために、前記ホスト車両に前記操舵動作を適用させる、ための動作を実行するようにさらに構成される、請求項10から13のいずれか一項に記載の計算装置。
- 15前記計画された運転動作は、合流またはレーン変更操作の一部である、請求項10から14のいずれか一項に記載の計算装置。
- 16前記計画された運転動作は、追走操作の一部である、請求項10から15のいずれか一項に記載の計算装置。
- 17前記センサデバイスはカメラを含み、前記センサデータは少なくとも一つの画像を含む、請求項10から16のいずれか一項に記載の計算装置。
- 18前記センサデバイスは、レーダシステムまたはライダシステムの少なくとも一つを含み、前記センサデータは、前記レーダシステムまたは前記ライダシステムの前記少なくとも一つからのデータを含む、請求項10から17のいずれか一項に記載の計算装置。
Independent claims18
836 paragraphs, as filed
Cross-reference to related applications [0001] This application is based on U.S. Provisional Patent Application No. 62/645,479, filed on March 20, 2018, and U.S. Provisional Patent Application No. 62/646,579, filed on March 22, 2018. , U.S. Provisional Patent Application No. 62/718,554, filed on August 14, 2018, U.S. Provisional Patent Application No. 62/724,355, filed on August 29, 2018, and U.S. Provisional Patent Application No. 62/724,355, filed on November 28, 2018. It claims the benefit of priority of U.S. Provisional Patent Application No. 62/772,366, filed on December 11, 2018, and U.S. Provisional Patent Application No. 62/777,914, filed on December 11, 2018. All of the above applications are incorporated herein by reference in their entirety.
BACKGROUND OF THE INVENTION [0002] This disclosure relates generally to autonomous vehicle navigation. Additionally, the present disclosure relates to systems and methods for navigating within potential accident liability constraints.
Background information [0003] As technology continues to evolve, the goal of fully autonomous vehicles capable of navigating the road becomes a reality. An autonomous vehicle may need to consider various factors and based on those factors may make appropriate decisions to reach the intended destination safely and accurately. For example, autonomous vehicles may need to process and interpret visual information (e.g., information captured from cameras), information from radar or lidar, as well as information from other sources (e.g., GPS devices, speed sensors, Information obtained from accelerometers, suspension sensors, etc.) may also be used. At the same time, in order to navigate to a destination, an autonomous vehicle must identify its position within a particular road (e.g., a particular lane in a multi-lane road), navigate alongside other vehicles, and avoid obstacles. The need to avoid objects and pedestrians, observe traffic lights and signs, proceed from one road to another at appropriate intersections or interchanges, and respond to any other situations that arise or develop during the operation of the vehicle. It is also possible. Additionally, navigation systems may have to comply with certain imposed constraints. In some cases, those constraints may relate to interactions between the host vehicle and one or more other objects, such as other vehicles or pedestrians. In other cases, the constraints may relate to liability rules to follow when performing one or more navigation operations for the host vehicle.
<p>[0004] In the field of autonomous driving, there are two important considerations for viable autonomous vehicle systems. The first consideration is the standardization of safety assurance, including the requirements that all autonomous vehicles must meet to ensure safety, and how those requirements can be verified. The second consideration is scalability, as engineering solutions that cause costs to rise do not scale to millions of cars and may not scale well to widespread adoption of autonomous vehicles or even less. This is because even less widespread adoption may be hindered. Therefore, there is a need for an interpretable mathematical model for safety assurance and the design of a system that is scalable to millions of vehicles while complying with safety assurance requirements.</p>
<p>Overview [0005] Embodiments according to the present disclosure provide systems and methods for autonomous vehicle navigation. Disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, according to embodiments of the present disclosure, the disclosed system may include one, two, or more cameras that monitor the environment of the vehicle. The disclosed system may provide navigation responses based on, for example, analysis of images captured by one or more of the cameras. The navigation response may also consider other data, including, for example, global positioning (GPS) data, sensor data (eg, from accelerometers, speed sensors, suspension sensors, etc.), and/or other map data.</p><p>[0006] In one embodiment, a system for navigating a host vehicle may include at least one processing device programmed to receive at least one image representative of the host vehicle's environment. At least one image may be received from an image capture device. The at least one processing device may be programmed to determine a planned navigation operation to achieve a navigation goal of the host vehicle based on the at least one driving policy. The at least one processing device analyzes the at least one image to identify a target vehicle in the host vehicle's environment and to identify the relationship between the host vehicle and the target vehicle that will occur if the planned navigation operation is performed. It may be further programmed to determine the distance of the next state between. The at least one processing device may be programmed to determine a maximum braking capability of the host vehicle, a maximum acceleration capability of the host vehicle, and a current speed of the host vehicle. The at least one processing device may also be programmed to determine a current stopping distance of the host vehicle based on the host vehicle's current maximum braking capability, the host vehicle's current maximum acceleration capability, and the host vehicle's current speed. . The at least one processing device may be further programmed to determine a current speed of the target vehicle and assume a maximum braking capability of the target vehicle based on the at least one recognized characteristic of the target vehicle. The at least one processing device determines that the determined current stopping distance of the host vehicle is a distance traveled of the target vehicle determined based on a current speed of the target vehicle and an assumed maximum braking capability of the target vehicle. It may also be programmed to perform a planned navigation operation if the distance is less than the distance of the next state plus the distance of the next state.</p><p>[0007] In one embodiment, a system for navigating a host vehicle may include at least one processing device. The at least one processing device may be programmed to receive at least one image representative of the host vehicle's environment from the image capture device. The at least one processing device may also be programmed to determine planned navigation operations to achieve the host vehicle's navigation goals. The planned navigation operation may be based on at least one driving policy. The at least one processing device may be programmed to analyze the at least one image to identify a target vehicle within the environment of the host vehicle. The at least one processing device may be further programmed to determine a next state distance between the host vehicle and the target vehicle that will occur if the planned navigation operation is performed. At least one processing device may be programmed to determine a current speed of the host vehicle and a current speed of the target vehicle. The at least one processing device may be programmed to assume a maximum braking rate capability of the target vehicle based on the at least one recognized characteristic of the target vehicle. The at least one processing device is configured to determine the current speed of the target vehicle and the assumed current speed of the target vehicle for the determined current speed of the host vehicle and at a predetermined submaximal braking rate that is less than the maximum braking rate capability of the host vehicle. If the host vehicle can be stopped within a stopping distance of the host vehicle that is less than the distance traveled by the target vehicle, determined based on the maximum braking rate capability determined, plus the distance of the determined next state, the planned may be further programmed to perform the specified navigation operations.</p><p>[0008] In one embodiment, a system for navigating a host vehicle may include at least one processing device. The at least one processing device may be programmed to receive at least one image representative of the host vehicle's environment from the image capture device. The at least one processing device may also be programmed to determine planned navigation operations to achieve the host vehicle's navigation goals. The planned navigation operation may be based on at least one driving policy. The at least one processing device may be programmed to analyze the at least one image to identify a target vehicle within the environment of the host vehicle. The at least one processing device may be further programmed to determine a next state distance between the host vehicle and the target vehicle that will occur if the planned navigation operation is performed. At least one processing device may be programmed to determine a current speed of the host vehicle. The at least one processing device may be programmed to determine a current speed of the target vehicle and to assume a maximum braking rate capability of the target vehicle based on the at least one recognized characteristic of the target vehicle. The at least one processing device is configured to determine, for the determined current speed of the host vehicle and for the predetermined braking rate profile, a target vehicle that is determined based on the current speed of the target vehicle and an assumed maximum braking rate capability of the target vehicle. may be programmed to perform the planned navigation operation if the host vehicle may be stopped within a stopping distance of the host vehicle that is less than the distance traveled by the vehicle plus the determined next state distance; The predetermined braking rate profile progressively increases from a submaximal braking rate to a maximum braking rate of the host vehicle.</p><p>[0009] In one embodiment, a system for braking a host vehicle may include at least one processing device programmed to perform one or more operations. The at least one processing device may be programmed to receive output from the at least one sensor representative of the environment of the host vehicle. The at least one processing device may be further programmed to detect a target vehicle within the environment of the host vehicle based on the output. The at least one processing device may be programmed to determine a current speed of the host vehicle and a current distance between the host vehicle and the target vehicle. Based on at least the current speed of the host vehicle and the current distance between the host vehicle and the target vehicle, the at least one processor may be programmed to determine whether a braking condition exists. If it is determined that a braking condition exists, the at least one processor controls the host vehicle according to a predetermined braking profile that includes a segment that begins at a submaximal braking rate of the host vehicle and that increases progressively up to a maximum braking rate of the host vehicle. It may be programmed to apply braking devices associated with the vehicle.</p><p>[0010] In one embodiment, an autonomous system for selectively replacing control of a host vehicle by a human driver may include at least one processing device. The at least one processing device is configured to receive at least one image representative of the host vehicle's environment from the image capture device and to detect at least one obstacle in the host vehicle's environment based on analysis of the at least one image. can be programmed to The at least one processing device may be programmed to monitor driver input to at least one of throttle control, brake control, or steering control associated with the host vehicle. The at least one processing device may also be programmed to determine whether the driver input will cause the host vehicle to navigate within a proximity buffer to the at least one obstacle. If the at least one processing device determines that the driver input will not cause the host vehicle to navigate within a proximity buffer to the at least one obstacle, the at least one processing device determines that the driver input will not cause the host vehicle to navigate within a proximity buffer to the at least one obstacle. may be programmed to enable corresponding changes in one or more operational control systems to occur. If the at least one processing device determines that the driver input results in the host vehicle navigating within a proximity buffer to at least one obstacle, the at least one processing device determines that the driver input causes the host vehicle to navigate within a proximity buffer with respect to at least one obstacle. may be programmed to prevent corresponding changes in one or more operational control systems from occurring.</p><p>[0011] In one embodiment, a navigation system for navigating an autonomous host vehicle according to at least one navigation goal of the host vehicle may include at least one processor. At least one processor may be programmed to receive sensor output from one or more sensors indicative of at least one aspect of host vehicle operation relative to the host vehicle's environment. The sensor output is arranged at a first point in time that is after the data acquisition point at which the measurement or data acquisition on which the sensor output is based is obtained and before a second point in time at which the sensor output is received by the at least one processor. can be generated at any point in time. at least one processor based at least in part on the received sensor output and an estimate of how at least one aspect of the operation of the host vehicle will change over a time interval between the data acquisition point and the predicted operation point; and may be programmed to generate a prediction of at least one aspect of host vehicle operation for a predicted operation time. The at least one processor determines a planned navigation behavior of the host vehicle based at least in part on the at least one navigation objective of the host vehicle and based on the generated prediction of at least one aspect of the host vehicle's behavior. can be programmed to do so. The at least one processor may be further configured to generate navigation commands to implement at least a portion of the planned navigation operation. The at least one processor may be programmed to provide navigation commands to at least one actuation system of the host vehicle. The navigation command is transmitted at least at a third point in time that is later than the second point in time and that is prior to or substantially the same as the point in time in which the at least one actuation system component responds to the received command. One actuation system may be provided to receive navigation commands. In some embodiments, the motion prediction time is after the data acquisition time and is before or equal to the actuation time.</p><p>[0012] According to other disclosed embodiments, a non-transitory computer-readable storage medium is executable by at least one processing device and performs any of the steps and/or methods described herein. It may store program instructions for execution.</p><p>[0013] The general description above and the detailed description below are merely exemplary and explanatory and are not limitations on the scope of the claims.</p><p>BRIEF DESCRIPTION OF THE DRAWINGS [0014] The accompanying drawings, which are incorporated into and constitute a part of this disclosure, illustrate various disclosed embodiments.</p>
<figref num="1">[0015] FIG. 1 is a diagrammatic representation of an example system in accordance with disclosed embodiments.</figref><figref num="2A">[0016] FIG. 2 is a side view representation of an example vehicle including a system according to disclosed embodiments.</figref><figref num="2B">[0017] FIG. 2A is a top view representation of the vehicle and system shown in FIG. 2A according to disclosed embodiments.</figref><figref num="2C">[0018] FIG. 3 is a top view representation of another embodiment of a vehicle that includes a system according to disclosed embodiments.</figref><figref num="2D">[0019] FIG. 4 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments.</figref><figref num="2E">[0020] FIG. 4 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments.</figref><figref num="2F">[0021] FIG. 1 is a diagrammatic representation of an example vehicle control system in accordance with disclosed embodiments.</figref><figref num="3A">[0022] FIG. 2 is a diagrammatic representation of the interior of a vehicle including a rearview mirror and a user interface of a vehicle imaging system according to disclosed embodiments.</figref><figref num="3B">[0023] FIG. 3 is an illustration of an example camera mount configured to be positioned behind a rearview mirror and opposite a vehicle windshield, according to a disclosed embodiment.</figref><figref num="3C">[0024] FIG. 3B is a view of the camera mount shown in FIG. 3B from a different perspective, according to a disclosed embodiment.</figref><figref num="3D">[0025] FIG. 3 is an illustration of an example camera mount configured to be positioned behind a rearview mirror and opposite a vehicle windshield, according to a disclosed embodiment.</figref><figref num="4">[0026] FIG. 3 is an example block diagram of a memory configured to store instructions for performing one or more operations in accordance with disclosed embodiments.</figref><figref num="5A">[0027] FIG. 3 is a flowchart illustrating an example process for generating one or more navigation responses based on monocular image analysis, according to disclosed embodiments.</figref><figref num="5B">[0028] FIG. 3 is a flowchart illustrating an example process for detecting one or more vehicles and/or pedestrians in a set of images, according to disclosed embodiments.</figref><figref num="5C">[0029] FIG. 3 is a flowchart illustrating an example process for detecting road markings and/or lane geometry information in a set of images, according to disclosed embodiments.</figref><figref num="5D">[0030] FIG. 3 is a flowchart illustrating an example process for detecting traffic lights in a set of images, according to disclosed embodiments.</figref><figref num="5E">[0031] FIG. 3 is a flowchart of an example process for generating one or more navigation responses based on a vehicle path, according to disclosed embodiments.</figref><figref num="5F">[0032] FIG. 3 is a flowchart illustrating an example process for determining whether a preceding vehicle is changing lanes, according to disclosed embodiments.</figref><figref num="6">[0033] FIG. 3 is a flowchart illustrating an example process for generating one or more navigation responses based on stereo image analysis, according to disclosed embodiments.</figref><figref num="7">[0034] FIG. 3 is a flowchart illustrating an example process for generating one or more navigation responses based on analysis of three sets of images, according to disclosed embodiments.</figref><figref num="8">[0035] FIG. 2 is a block diagram representation of modules that may be implemented by one or more specifically programmed processing devices of a navigation system for an autonomous vehicle, according to disclosed embodiments.</figref><figref num="9">[0036] FIG. 4 is a graph of navigation options, according to disclosed embodiments.</figref><figref num="10">[0037] FIG. 4 is a graph of navigation options, according to disclosed embodiments.</figref><figref num="11A">[0038] FIG. 5 illustrates a schematic diagram of navigation options for a host vehicle within a merge area, according to disclosed embodiments.</figref><figref num="11B">[0038] FIG. 5 illustrates a schematic diagram of navigation options for a host vehicle within a merge area, according to disclosed embodiments.</figref><figref num="11C">[0038] FIG. 5 illustrates a schematic diagram of navigation options for a host vehicle within a merge area, according to disclosed embodiments.</figref><figref num="11D">[0039] FIG. 4 illustrates a diagrammatic representation of a double confluence scenario according to disclosed embodiments.</figref><figref num="11E">[0040] FIG. 4 illustrates a graph of potentially useful options in a double confluence scenario according to disclosed embodiments.</figref><figref num="12">[0041] FIG. 4 illustrates an illustration of a representative image captured of a host vehicle's environment with potential navigation constraints, in accordance with disclosed embodiments.</figref><figref num="13">[0042] FIG. 4 illustrates a flowchart of an algorithm for navigating a vehicle, according to disclosed embodiments.</figref><figref num="14">[0043] FIG. 4 illustrates a flowchart of an algorithm for navigating a vehicle, according to disclosed embodiments.</figref><figref num="15">[0044] FIG. 4 illustrates a flowchart of an algorithm for navigating a vehicle, according to disclosed embodiments.</figref><figref num="16">[0045] FIG. 5 illustrates a flowchart of an algorithm for navigating a vehicle, according to disclosed embodiments.</figref><figref num="17A">[0046] FIG. 4 illustrates an illustration of a host vehicle navigating into a roundabout, according to disclosed embodiments.</figref><figref num="17B">[0046] FIG. 4 illustrates an illustration of a host vehicle navigating into a roundabout, according to disclosed embodiments.</figref><figref num="18">[0047] FIG. 4 illustrates a flowchart of an algorithm for navigating a vehicle, according to disclosed embodiments.</figref><figref num="19">[0048] FIG. 4 illustrates an example of a host vehicle traveling on a multi-lane highway, according to disclosed embodiments.</figref><figref num="20A">[0049] FIG. 4 illustrates an example of a vehicle cutting in front of another vehicle, according to disclosed embodiments.</figref><figref num="20B">[0049] FIG. 4 illustrates an example of a vehicle cutting in front of another vehicle, according to disclosed embodiments.</figref><figref num="21">[0050] FIG. 4 illustrates an example of a vehicle following another vehicle, according to disclosed embodiments.</figref><figref num="22">[0051] FIG. 4 illustrates an example of a vehicle exiting a parking lot and merging onto a potentially busy road, in accordance with disclosed embodiments.</figref><figref num="23">[0052] FIG. 2 illustrates a vehicle traveling on a road, according to disclosed embodiments.</figref><figref num="24A">[0053] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="24B">[0053] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="24C">[0053] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="24D">[0053] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="25">[0054] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="26">[0055] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="27">[0056] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="28A">[0057] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="28B">[0057] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="29A">[0058] FIG. 7 illustrates an example of a failure in an interrupt scenario, according to disclosed embodiments.</figref><figref num="29B">[0058] FIG. 7 illustrates an example of a failure in an interrupt scenario, according to disclosed embodiments.</figref><figref num="30A">[0059] FIG. 7 illustrates an example of a failure in an interrupt scenario, according to disclosed embodiments.</figref><figref num="30B">[0059] FIG. 7 illustrates an example of a failure in an interrupt scenario, according to disclosed embodiments.</figref><figref num="31A">[0060] FIG. 7 illustrates an example of a mistake in a drifting scenario, according to disclosed embodiments.</figref><figref num="31B">[0060] FIG. 7 illustrates an example of a mistake in a drifting scenario, according to disclosed embodiments.</figref><figref num="31C">[0060] FIG. 7 illustrates an example of a mistake in a drifting scenario, according to disclosed embodiments.</figref><figref num="31D">[0060] FIG. 7 illustrates an example of a mistake in a drifting scenario, according to disclosed embodiments.</figref><figref num="32A">[0061] FIG. 4 illustrates an example of a mistake in a two-way traffic scenario, according to disclosed embodiments.</figref><figref num="32B">[0061] FIG. 4 illustrates an example of a mistake in a two-way traffic scenario, according to disclosed embodiments.</figref><figref num="33A">[0062] FIG. 4 illustrates an example of a mistake in a two-way traffic scenario, according to disclosed embodiments.</figref><figref num="33B">[0062] FIG. 4 illustrates an example of a mistake in a two-way traffic scenario, according to disclosed embodiments.</figref><figref num="34A">[0063] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="34B">[0063] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="35A">[0064] FIG. 4 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="35B">[0064] FIG. 4 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="36A">[0065] FIG. 7 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="36B">[0065] FIG. 7 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="37A">[0066] FIG. 7 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="37B">[0066] FIG. 7 illustrates an example of a fault in a route priority scenario, in accordance with disclosed embodiments.</figref><figref num="38A">[0067] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="38B">[0067] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="39A">[0068] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="39B">[0068] FIG. 7 illustrates an example of a fault in a route priority scenario, according to disclosed embodiments.</figref><figref num="40A">[0069] FIG. 4 illustrates an example of a fault in a traffic light scenario, according to disclosed embodiments.</figref><figref num="40B">[0069] FIG. 4 illustrates an example of a fault in a traffic light scenario, according to disclosed embodiments.</figref><figref num="41A">[0070] FIG. 7 illustrates an example of a fault in a traffic light scenario, according to disclosed embodiments.</figref><figref num="41B">[0070] FIG. 7 illustrates an example of a fault in a traffic light scenario, according to disclosed embodiments.</figref><figref num="42A">[0071] FIG. 7 illustrates an example of a mistake in a traffic light scenario, according to disclosed embodiments.</figref><figref num="42B">[0071] FIG. 7 illustrates an example of a mistake in a traffic light scenario, according to disclosed embodiments.</figref><figref num="43A">[0072] FIG. 2 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="43B">[0072] FIG. 2 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="43C">[0072] FIG. 2 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="44A">[0073] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, according to disclosed embodiments.</figref><figref num="44B">[0073] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, according to disclosed embodiments.</figref><figref num="44C">[0073] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, according to disclosed embodiments.</figref><figref num="45A">[0074] FIG. 3 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="45B">[0074] FIG. 3 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="45C">[0074] FIG. 3 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="46A">[0075] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="46B">[0075] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="46C">[0075] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="46D">[0075] FIG. 4 illustrates an example vulnerable road user (VRU) scenario, in accordance with disclosed embodiments.</figref><figref num="47A">[0076] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="47B">[0076] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="48">[0077] FIG. 7 is a flowchart illustrating an example process for navigating a host vehicle, according to disclosed embodiments.</figref><figref num="49A">[0078] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="49B">[0078] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="49C">[0078] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="49D">[0078] FIG. 4 illustrates an example scenario in which a vehicle follows another vehicle, according to disclosed embodiments.</figref><figref num="50">[0079] FIG. 3 is a flowchart illustrating an example process for applying brakes to a host vehicle, according to disclosed embodiments.</figref><figref num="51">[0080] FIG. 7 is a flowchart illustrating an example process for navigating a host vehicle, according to disclosed embodiments.</figref><figref num="52A">[0081] FIG. 4 illustrates an example of a proximity buffer for a host vehicle, according to disclosed embodiments.</figref><figref num="52B">[0081] FIG. 4 illustrates an example of a proximity buffer for a host vehicle, according to disclosed embodiments.</figref><figref num="52C">[0081] FIG. 4 illustrates an example of a proximity buffer for a host vehicle, according to disclosed embodiments.</figref><figref num="52D">[0081] FIG. 4 illustrates an example of a proximity buffer for a host vehicle, according to disclosed embodiments.</figref><figref num="53A">[0082] FIG. 4 illustrates an example scenario involving a proximal buffer, according to disclosed embodiments.</figref><figref num="53B">[0082] FIG. 4 illustrates an example scenario involving a proximal buffer, according to disclosed embodiments.</figref><figref num="54A">[0083] FIG. 4 illustrates an example scenario involving a proximal buffer, according to disclosed embodiments.</figref><figref num="54B">[0083] FIG. 4 illustrates an example scenario involving a proximal buffer, according to disclosed embodiments.</figref><figref num="55">[0084] FIG. 4 illustrates a flowchart for selectively replacing control of a host vehicle by a human driver, according to disclosed embodiments.</figref><figref num="56">[0085] FIG. 3 is a flowchart illustrating an example process for navigating a host vehicle, according to disclosed embodiments.</figref><figref num="57A">[0086] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="57B">[0086] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="57C">[0086] FIG. 4 illustrates an example scenario according to disclosed embodiments.</figref><figref num="58">[0087] FIG. 7 is a flowchart illustrating an example process for navigating a host vehicle, according to disclosed embodiments.</figref>
DETAILED DESCRIPTION [0088] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. Although several example embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substitution, reordering, deletion, or addition of steps in the disclosed methods. Can be changed. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the following claims.
[0089] Autonomous Vehicle Overview [0090] As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle that is capable of performing at least one navigation change without driver input. "Navigation change" refers to one or more changes in steering, braking, or acceleration/deceleration of a vehicle. To be autonomous, a vehicle does not need to be fully automatic (eg, fully capable of operation without a driver or driver input). Rather, autonomous vehicles include vehicles that can operate under the control of a driver during certain periods of time and without driver control during other periods of time. Autonomous vehicles may control only some aspects of vehicle navigation, such as steering (e.g., to maintain vehicle course between vehicle lane constraints), or may only control some aspects of vehicle navigation under certain circumstances (rather than under all circumstances). It may also include vehicles that control such steering operations, but may leave other aspects (eg, braking or braking under certain circumstances) to the driver. In some cases, an autonomous vehicle may handle some or all aspects of braking, speed control, and/or steering of the vehicle.
[0091] Since human drivers typically rely on visual cues and observations to control their vehicles, the transportation infrastructure is built accordingly, with lane markings, traffic signs and traffic lights providing visual information to drivers. designed to provide. In view of these design features of the transportation infrastructure, autonomous vehicles may include cameras and processing units that analyze visual information captured from the vehicle's environment. Visual information may include, for example, images representing transportation infrastructure components (e.g. lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g. other vehicles, pedestrians, debris, etc.) that are observable by the driver. may include. Furthermore, the motor vehicle may also use stored information when navigating, such as information that provides a model of the vehicle's environment. For example, the vehicle uses GPS data, sensor data (e.g. from accelerometers, speed sensors, suspension sensors, etc.) and/or other map data to relate to the vehicle's environment while the vehicle is driving. The vehicle (and other vehicles) can use the information to locate itself in the model. Some vehicles may also be able to communicate between vehicles, share information, change peer vehicles of danger or changes in the vehicle's surroundings, etc.
[0092] System overview [0093] FIG. 1 is a block diagram representation of a system 100 according to the disclosed example embodiments. System 100 may include various components depending on particular implementation requirements. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. . Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, image acquisition unit 120 may include any number of image acquisition devices and components depending on the requirements of a particular application. In some embodiments, image acquisition unit 120 includes one or more image capture devices (e.g., cameras, CCDs, any other type of image sensor). System 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition unit 120. For example, data interface 128 may include one or more optional wired and/or wireless links for transmitting image data acquired by image acquisition unit 120 to processing unit 110.
[0094] The wireless transceiver 172 is configured to exchange transmissions with one or more networks (e.g., cellular, the Internet, etc.) via a wireless interface through the use of radio frequencies, infrared frequencies, magnetic fields, or electric fields. May include one or more devices. Wireless transceiver 172 may transmit and/or receive data using any known standard (eg, Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmission may include communication from the host vehicle to one or more remotely located servers. Such transmission may include communication between the host vehicle and one or more locations within the host vehicle's environment (e.g., to facilitate coordination of the host vehicle's navigation in consideration of or with a target vehicle within the host vehicle's environment). It may also include communications (unidirectional or bidirectional) with the target vehicle, as well as broadcast transmissions to unspecified recipients in the vicinity of the transmitting vehicle.
[0095] Both application processor 180 and image processor 190 may include various types of hardware-based processing devices. For example, one or both of application processor 180 and image processor 190 may include a microprocessor, preprocessor (such as an image preprocessor), graphics processor, central processing unit (CPU), support circuitry, digital signal processor, integrated circuit, memory, etc. or any other type of device suitable for running applications and processing and analyzing images. In some embodiments, application processor 180 and/or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. For example, a variety of processing devices are available, including processors available from manufacturers such as Intel(R), AMD(R), etc., and a variety of architectures (e.g., x86 processors, ARM(R), etc.). ) may be included.
[0096] In some embodiments, application processor 180 and/or image processor 190 may include any EyeQ series processor available from Mobileye®. These processor designs include multiple processing units, each with local memory and instruction sets. Such a processor may include a video input for receiving image data from multiple image sensors, and may also include video output functionality. In one example, EyeQ2® uses 90nm-micron technology operating at 332MHz. The EyeQ2® architecture consists of two floating point hyperthreaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Vision Compute Engines (VCEs), and three vector microcode processors (VMPs). ), Denali 64-bit mobile DDR controller, 128-bit internal audio interconnect, dual 16-bit video input and 18-bit video output controllers, 16-channel DMA and several peripherals. MIPS34K The CPU manages 5 VCEs, 3 VMPs and DMAs, a second MIPS34K CPU and multi-channel DMA as well as other peripherals. Five VCEs, three VMPs and a MIPS34K CPU can perform intensive vision computations required by multi-function bundle applications. In another example, the disclosed embodiments may use the EyeQ3®, which is a third generation processor and is six times more powerful than the EyeQ2®. In other examples, EyeQ4(R) and/or EyeQ5(R) may be used in the disclosed embodiments. Of course, newer or future EyeQ processing devices may be used with the disclosed embodiments.
[0097] Any of the processing devices disclosed herein can be configured to perform particular functions. Configuring a processing device, such as any of the EyeQ processors or other controllers or microprocessors described, to perform certain functions may include programming computer-executable instructions to be executed during operation of the processing device. may include providing those instructions to a processing device. In some embodiments, configuring the processing device may include programming architectural instructions directly into the processing device. In other embodiments, configuring the processing device may include storing executable instructions on memory accessible to the processing device during operation. For example, a processing device may access memory during operation to retrieve and execute stored instructions. In any event, the processing device configured to perform the sensing, image analysis and/or navigation functions disclosed herein may be a dedicated hardware-based device that controls multiple hardware-based components of the host vehicle. represents the system of
[0098] Although FIG. 1 shows two separate processing devices included in processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to accomplish the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Furthermore, in some embodiments, system 100 may include one or more processing units 110 without other components such as image acquisition unit 120.
[0099] Processing unit 110 may include various types of devices. For example, processing unit 110 may include various devices such as a controller, image preprocessor, central processing unit (CPU), support circuitry, digital signal processor, integrated circuit, memory, or any other type of device that processes and analyzes images. may include. The image preprocessor may include a video processor that captures, digitizes, and processes images from the image sensor. A CPU may include any number of microcontrollers or microprocessors. Support circuitry may be any number of circuits commonly known in the art, including cache, power supply, clock, and input/output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include a database and image processing software. Memory may include any number of random access memory, read-only memory, flash memory, disk drives, optical storage, tape storage, removable storage, and other types of storage. In one example, the memory may be separate from processing unit 110. In another example, memory may be integrated into processing unit 110.
[0100] Each memory 140, 150 may contain software instructions that, when executed by a processor (eg, application processor 180 and/or image processor 190), may control the operation of various aspects of system 100. These memory units may include various database and image processing software as well as trained systems such as neural networks or deep neural networks. The memory unit may include random access memory, read-only memory, flash memory, disk drives, optical storage, tape storage, removable storage, and/or any other type of storage. In some embodiments, memory units 140, 150 may be separate from application processor 180 and/or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and/or image processor 190.
[0101] Location sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, location sensor 130 may include a GPS receiver. Such receivers can determine a user's location and velocity by processing signals broadcast by Global Positioning System satellites. Location information from location sensor 130 may be provided to application processor 180 and/or image processor 190.
[0102] In some embodiments, system 100 may include components such as a speed sensor (eg, a speedometer) to measure the speed of vehicle 200. System 100 may also include one or more accelerometers (single-axis or multi-axis) to measure acceleration of vehicle 200 along one or more axes.
[0103] The memory units 140, 150 may include data organized in a database or any other format that indicates the locations of known landmarks. Sensory information of the environment (images, radar signals, depth information by lidar or stereo processing of two or more images, etc.) is processed together with position information such as GPS coordinates and vehicle self-motion to identify known landmarks. The vehicle's current position can be determined and the vehicle's position can be refined. Certain aspects of this technology are included in the location technology known as REM, sold by the assignee of this application.
[0104] User interface 170 may include any device suitable for providing information or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices, including, for example, a touch screen, microphone, keyboard, pointer device, trackwheel, camera, knob, buttons, and the like. Using such input devices, a user can type instructions or information, provide voice commands, use buttons, pointers or eye tracking features, or any other suitable way to communicate information to the system 100. Through techniques, it may be possible to provide information input or commands to the system 100 by selecting on-screen menu options.
[0105] User interface 170 includes one or more processing devices configured to provide information to or receive information from a user and process that information for use by, for example, application processor 180. can be provided. In some embodiments, such a processing device includes instructions for recognizing and tracking eye movements, receiving and interpreting voice commands, recognizing and/or recognizing touches and/or gestures made on a touch screen. Instructions that interpret, respond to keyboard input or menu selections, etc. may be executed. In some embodiments, user interface 170 may include a display, a speaker, a haptic device, and/or any other device that provides output information to a user.
[0106] Map database 160 may include any type of database that stores map data useful to system 100. In some embodiments, the map database 160 stores data related to the location of various items in a reference coordinate system, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc. may be included. Map database 160 may store not only the locations of such items, but also descriptors associated with those items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively or additionally, map database 160, or portions thereof, may be located remotely with respect to other components of system 100 (eg, processing unit 110). In such embodiments, information from map database 160 may be downloaded to a network via a wired or wireless data connection (eg, via a cellular network and/or the Internet, etc.). In some cases, map database 160 may store sparse data models that include polynomial representations of specific road features (eg, lane markings) or target trajectories of the host vehicle. Map database 160 may also include stored representations of various recognized landmarks that may be used to determine or update the host vehicle's known position relative to the target trajectory. Landmark representations may include data fields such as landmark type and landmark location, among other potential identifiers.
[0107] Image capture devices 122, 124 and 126 may each include any type of device suitable for capturing at least one image from an environment. Additionally, any number of image capture devices may be used to obtain images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, even four or more image capture devices. Image capture devices 122, 124, and 126 are further described below with reference to FIGS. 2B-2E.
[0108] One or more cameras (eg, image capture devices 122, 124, and 126) may be part of a sensing block included on the vehicle. Various other sensors may be included in the sensing block, and any or all of the sensors may be utilized to develop a sensed navigation state of the vehicle. In addition to cameras (forward-facing, side-facing, rear-facing, etc.), other sensors such as radar, lidar, acoustic sensors, etc. may be included in the sensing block. Additionally, the sensing block may include one or more components configured to communicate, send and receive information related to the vehicle's environment. For example, such components include wireless transceivers (such as RF) that may receive sensor-based information or any other type of information related to the environment of the host vehicle from a source located remotely with respect to the host vehicle. obtain. Such information may include sensor output information or related information received from vehicle systems other than the host vehicle. In some embodiments, such information may include information received from remote computing devices, centralized servers, and the like. Furthermore, cameras can take many different configurations, such as single camera unit, multiple cameras, camera clusters, long FOV, short FOV, wide angle, fisheye, etc.
[0109] System 100 or various components of system 100 may be incorporated into a variety of different platforms. In some embodiments, system 100 may be included in vehicle 200, as shown in FIG. 2A. For example, vehicle 200 may include processing unit 110 and any other components of system 100, as described above with respect to FIG. In some embodiments, vehicle 200 may include only a single image capture device (e.g., a camera), whereas in other embodiments, such as the embodiments discussed in connection with FIGS. image capture devices are available. For example, as shown in FIG. 2A, either of the image capture devices 122 and 124 of the vehicle 200 may be part of an ADAS (Advanced Driver Assistance Systems) imaging set.
[0110] The image capture device included in vehicle 200 as part of image acquisition unit 120 may be located in any suitable location. In some embodiments, the image capture device 122 may be placed near the rearview mirror, as shown in FIGS. 2A-2E and 3A-3C. This position may provide a line of sight similar to that of the driver of vehicle 200 and may assist the driver in determining what is and is not visible. Although the image capture device 122 may be located at any location near the rearview mirror, positioning the image capture device 122 on the driver's side of the mirror further assists in capturing images representative of the driver's field of view and/or line of sight. It is possible.
[0111] Other positions for the image capture device of image acquisition unit 120 may also be used. For example, image capture device 124 may be placed on or in the bumper of vehicle 200. Such a position may be particularly suitable for image capture devices with a wide field of view. The line of sight of an image capture device placed in the bumper can be different than the driver's line of sight, so the bumper image capture device and the driver are not always looking at the same object. Image capture devices (eg, image capture devices 122, 124, and 126) may also be located at other locations. For example, the image capture device may be located on one or both side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the side of vehicle 200, and mounted on any window of vehicle 200. , may be positioned at the rear or at the front, and may be mounted at or near the front and/or rear lights of the vehicle 200.
[0112] In addition to the image capture device, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be integrated into a vehicle's engine control unit (ECU) or included in vehicle 200 separately from the ECU. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver, and vehicle 200 may also include a map database 160 and memory units 140 and 150.
[0113] As mentioned above, wireless transceiver 172 may receive data and/or via one or more networks (eg, a cellular network, the Internet, etc.). For example, wireless transceiver 172 may upload data collected by system 100 to one or more servers and download data from one or more servers. Via wireless transceiver 172, system 100 may receive updates to data stored in map database 160, memory 140, and/or memory 150, for example, periodically or on demand. Similarly, wireless transceiver 172 may receive any data from system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130, other sensors or vehicle control systems, etc.) and/or Any data processed by processing unit 110 may be uploaded to one or more servers.
[0114] System 100 may upload data to a server (eg, cloud) based on privacy level settings. For example, the system 100 may implement privacy level settings that regulate or limit the type of data (including metadata) that may uniquely identify a vehicle and/or a driver/owner of the vehicle that is sent to the server. Such settings may be set by a user via wireless transceiver 172, initialized by factory default settings, or set by data received by wireless transceiver 172, for example.
[0115] In some embodiments, the system 100 may upload data according to a "high" privacy level, and under the settings the system 100 may upload data according to a "high" privacy level, and under the settings the system 100 may (e.g., location information related to the route, captured images, etc.). For example, when uploading data in accordance with a "high" privacy level, the system 100 does not include the vehicle identification number (VIN) or the name of the driver or owner of the vehicle, but instead includes only information related to the captured images and/or route. Data such as location information can be transmitted.
[0116] Other privacy levels are also contemplated. For example, system 100 may send data to a server according to a "medium" privacy level, and a "high" privacy level, such as a make and/or model of a vehicle and/or vehicle type (e.g., passenger car, sport utility vehicle, truck, etc.). May contain additional information not included below the level. In some embodiments, system 100 may upload data according to a "low" privacy level. Under a "low" privacy level setting, the system 100 uploads sufficient data to uniquely identify a particular vehicle, owner/driver and/or part or all of the route traveled by the vehicle, and may contain information. Such "low" privacy level data may include, for example, the VIN, driver/owner name, starting point of the vehicle before departure, intended destination of the vehicle, vehicle make and/or model, vehicle type, etc. may include one or more.
[0117] FIG. 2A is a side view representation of an example vehicle imaging system according to disclosed embodiments. FIG. 2B is a top view representation of the embodiment shown in FIG. 2A. As shown in FIG. 2B, the disclosed embodiments include a first image capture device 122 positioned proximate a rearview mirror and/or proximate a driver of vehicle 200 and a bumper region of vehicle 200 (e.g., a bumper region A vehicle 200 may be shown including within its body a system 100 having a processing unit 110 and a second image capture device 124 positioned on or in the bumper area (one of the vehicle 210).
[0118] As shown in FIG. 2C, both image capture devices 122 and 124 may be positioned near the rearview mirror of vehicle 200 and/or near the driver. Additionally, although two image capture devices 122 and 124 are shown in FIGS. 2B and 2C, it is understood that other embodiments may include more than two image capture devices. For example, in the embodiment shown in FIGS. 2D and 2E, a first image capture device 122, a second image capture device 124, and a third image capture device 126 are included in the system 100 of the vehicle 200.
[0119] As shown in FIG. 2D, image capture device 122 may be positioned near the rearview mirror of vehicle 200 and/or near the driver, and image capture devices 124 and 126 may be positioned in the bumper area of vehicle 200 (e.g. , one of the bumper regions 210). Also, as shown in FIG. 2E, image capture devices 122, 124, and 126 may be positioned near the rearview mirror and/or near the driver seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices may be positioned at any suitable location within and/or on vehicle 200.
[0120] It should be understood that the disclosed embodiments are not limited to vehicles, but are applicable in other situations as well. It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200, but may be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.
[0121] First image capture device 122 may include any suitable type of image capture device. Image capture device 122 may include an optical axis. In one example, image capture device 122 is an Aptina M9V024 with a global shutter. May include a WVGA sensor. In other embodiments, image capture device 122 may provide a resolution of 1280 x 960 pixels and may include a rolling shutter. Image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included to provide the desired focal length and field of view of the image capture device, for example. In some embodiments, image capture device 122 may be associated with a 6mm lens or a 12mm lens. In some embodiments, image capture device 122 may be configured to capture an image having a desired field of view (FOV) 202, as shown in FIG. 2D. For example, the image capture device 122 may be configured to have a typical FOV, such as within a range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or a degree greater than the 52 degree FOV. Alternatively, image capture device 122 may be configured to have a narrow FOV ranging from 23 to 40 degrees, such as a 28 degree FOV or a 36 degree FOV. Additionally, image capture device 122 may be configured to have a wide FOV ranging from 100 to 180 degrees. In some embodiments, image capture device 122 may include a wide-angle bumper camera or a bumper camera with up to a 180 degree FOV. In some embodiments, image capture device 122 may be a 7.2M pixel image capture device with an aspect ratio of approximately 2:1 (eg, H×V=3800×1900 pixels) with a horizontal FOV of approximately 100 degrees. Such image capture devices may be used in place of three-dimensional image capture device configurations. Due to large lens distortion, the vertical FOV of such image capture devices can be much lower than 50 degrees in implementations where the image capture device uses radially symmetric lenses. For example, such lenses are not radially symmetrical, thereby allowing a horizontal FOV of 100 degrees and a vertical FOV of greater than 50 degrees.
[0122] First image capture device 122 may acquire a plurality of first images of a scene associated with vehicle 200. The plurality of first images may each be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may include multiple pixels.
[0123] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. Scan rate may refer to the rate at which an image sensor can acquire image data associated with each pixel included in a particular scan line.
[0124] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be utilized with a rolling shutter, such that each pixel within a row is read one at a time, and the scanning of the row is advanced row by row until the entire image frame is captured. . In some embodiments, rows may be captured sequentially from top to bottom for frames.
[0125] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high resolution imager, with greater than 5M pixels. , more than 7M pixels, more than 10M pixels, or more.
[0126] With the use of a rolling shutter, pixels in different rows may be exposed and captured at different times, which may cause skew and other image artifacts in the captured image frame. On the other hand, if image capture device 122 is configured to operate with a global or synchronous shutter, all pixels may be exposed during a common exposure period for the same amount of time. As a result, the image data in a frame collected from a system that utilizes a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a particular time. Conversely, when applying a rolling shutter, each row in the frame is exposed and the data is captured at different times. Therefore, moving objects may appear distorted in image capture devices with rolling shutters. This phenomenon will be explained in more detail below.
[0127] Second image capture device 124 and third image capture device 126 may be any type of image capture device. Like first image capture device 122, each of image capture devices 124 and 126 may include an optical axis. In one embodiment, each of image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of image capture devices 124 and 126 may include a rolling shutter. Like image capture device 122, image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture devices 124 and 126 are the same as the FOV associated with image capture device 122 (such as FOV 202) or provide a narrow FOV (such as FOV 204 and 206). It is possible. For example, image capture devices 124 and 126 may have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less than 20 degrees.
[0128] Image capture devices 124 and 126 may acquire a plurality of second and third images of a scene associated with vehicle 200. Each of the plurality of second and third images may be acquired as a series of second and third image scanlines, which may be captured using a rolling shutter. Each scan line or row may have multiple pixels. Image capture devices 124 and 126 may have second and third scan rates associated with the acquisition of each image scan line included in the second and third series.
[0129] Each image capture device 122, 124, and 126 may be positioned in any suitable location and in any suitable orientation relative to vehicle 200. The relative positions of image capture devices 122, 124, and 126 may be selected to assist in fusing together information obtained from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (FOV 204) is the FOV associated with image capture device 122 (such as FOV 202) and the FOV associated with image capture device 126 (such as FOV 206). may overlap partially or completely.
[0130] Image capture devices 122, 124, and 126 may be positioned on vehicle 200 at any suitable relative height. In one example, there may be a height difference between image capture devices 122, 124, and 126, and the height difference may provide sufficient disparity information to enable stereoscopic analysis. For example, as shown in FIG. 2A, two image capture devices 122 and 124 are at different heights. There may also be lateral displacement differences between image capture devices 122, 124 and 126, providing additional parallax information for stereoscopic analysis by processing unit 110, for example. The lateral displacement difference may be denoted by dx, as shown in FIGS. 2C and 2D. In some embodiments, anterior displacement or posterior displacement (eg, range displacement) may exist between image capture devices 122, 124, 126. For example, image capture device 122 may be positioned 0.5 to 2 meters or more behind image capture device 124 and/or image capture device 126. With this type of displacement, one of the image capture devices may be able to cover potential blind spots of the other image capture device.
[0131] Image capture device 122 may have any suitable resolution capability (e.g., number of pixels associated with the image sensor), and the resolution of the image sensor associated with image capture device 122 is the same as that of image capture device 124 and The resolution of the image sensor associated with 126 may be higher, lower, or the same. In some embodiments, the image sensors associated with image capture device 122 and/or image capture devices 124 and 126 have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution. obtain.
[0132] The frame rate (eg, the rate at which an image capture device acquires a set of pixel data for one image frame before moving on to capture pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rate associated with image capture devices 122, 124, and 126 may depend on various factors that may affect the timing of the frame rate. For example, one or more of the image capture devices 122, 124, and 126 may be used before or after the acquisition of image data associated with one or more pixels of an image sensor within the image capture devices 122, 124, and/or 126. may include an imposed selectable pixel delay period. Generally, image data corresponding to each pixel may be acquired according to the device's clock rate (eg, one pixel per clock cycle). Additionally, in embodiments that include a rolling shutter, one or more of image capture devices 122, 124, and 126 may be configured to capture image data associated with pixel rows of image sensors within image capture devices 122, 124, and/or 126. It may include a selectable horizontal blanking period imposed before or after acquisition. Additionally, one or more of the image capture devices 122, 124, and/or 126 may have a selectable vertical blanking that is imposed before or after the acquisition of image data associated with the image frames of the image capture devices 122, 124, and 126. may include a period of time.
[0133] These timing controls may allow the frame rates associated with image capture devices 122, 124, and 126 to be synchronized even if each image capture device has a different line scan rate. Additionally, as discussed in more detail below, these selectable timing controls, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.), allow the field of view of image capture device 122 to It may be possible to synchronize image capture from areas where the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even if the FOV of image capture device 122 overlaps with the FOV of one or more of image capture devices 124 and 126.
[0134] Frame rate timing at image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensor. For example, if one device contains an image sensor with a resolution of 640x480 and the other device contains an image sensor with a resolution of 1280x960, assuming the line scan rates of both devices are similar, then the higher resolution The more time it takes to acquire a frame of image data from a sensor.
[0135] Another factor that can affect the timing of image data acquisition at image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquisition of a row of image data from image sensors included in image capture devices 122, 124, and 126 requires some minimum amount of time. Assuming no pixel delay periods are added, this minimum amount of time to acquire a line of image data will be related to the maximum line scan rate of the particular device. A device that provides a higher maximum line rate has the potential to provide a higher frame rate than a device that has a lower maximum line rate. In some embodiments, one or both of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and/or 126 may be 1.25 times, 1.5 times, 1.75 times, or more than 2 times the maximum line scan rate of image capture device 122.
[0136] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate that is less than or equal to its maximum line scan rate. The system may be configured such that one or both of image capture devices 124 and 126 operate at a line scan rate equal to the line scan rate of image capture device 122. In other examples, the system is configured such that the line scan rate of image capture device 124 and/or 126 can be 1.25 times, 1.5 times, 1.75 times, or more than 2 times the line scan rate of image capture device 122. obtain.
[0137] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, these image capture devices may include cameras with different fields of view (FOV) and focal lengths. The field of view of image capture devices 122, 124, and 126 may include, for example, any desired area of the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 capture image data from an environment in front of vehicle 200, an environment behind vehicle 200, an environment on either side of vehicle 200, or a combination thereof. may be configured to obtain.
[0138] Additionally, the focal length associated with each image capture device 122, 124, and/or 126 may be selectable such that each device captures images of objects at a desired distance range from the vehicle 200 ( e.g. by inclusion of appropriate lenses, etc.). For example, in some embodiments, image capture devices 122, 124, and 126 may capture images of nearby objects within several meters of the vehicle. Image capture devices 122, 124, 126 may also be configured to capture images of objects at greater distances from the vehicle (eg, 25 m, 50 m, 100 m, 150 m or more). Additionally, the focal lengths of image capture devices 122, 124, and 126 are such that one image capture device (e.g., image capture device 122) can capture images of objects that are relatively close to the vehicle (e.g., within 10 m or within 20 m). while other image capture devices (e.g., image capture devices 124 and 126) may capture images of objects that are further away from vehicle 200 (e.g., greater than 20 m, greater than 50 m, greater than 100 m, greater than 150 m, etc.). You can choose to do so.
[0139] According to some embodiments, the FOV of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have a 140 degree FOV for image capture devices 122, 124, and 126, which may be used specifically to capture images of areas in the vicinity of vehicle 200. For example, image capture device 122 may be used to capture images of the right or left area of vehicle 200, and in such embodiments, image capture device 122 may have a wide FOV (e.g., at least 140 degrees). There are some things that are desirable.
[0140] The field of view associated with each of image capture devices 122, 124 and 126 may depend on the respective focal length. For example, as the focal length increases, the corresponding field of view decreases.
[0141] Image capture devices 122, 124 and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV of 23-46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV of 26-52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and/or image capture device 126 may vary from 1.5 to 2.0. In other embodiments, this ratio may vary from 1.25 to 2.25.
[0142] System 100 may be configured such that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and/or image capture device 126. In some embodiments, the system 100 provides a system in which the field of view of image capture devices 124 and 126 is, for example, within the field of view of image capture device 122 (e.g., smaller than the field of view of image capture device 122); may be configured to share a common center with. In other embodiments, image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partially overlapping FOVs. In some embodiments, the field of view of image capture devices 122, 124, and 126 may be such that the center of image capture device 124 and/or 126, which has a narrower FOV, is located in the lower half of the field of view of device 122, which has a wider FOV. It can be aligned as follows.
[0143] FIG. 2F is a diagrammatic representation of an example vehicle control system in accordance with disclosed embodiments. As shown in FIG. 2F, vehicle 200 may include a throttle system 220, a brake system 230, and a steering system 240. System 100 connects one or more of throttle system 220, brake system 230, and steering system 240 via one or more data links (e.g., any wired and/or wireless links or links that transmit data). Inputs (eg, control signals) may be provided. For example, based on analysis of images acquired by image capture devices 122, 124, and/or 126, system 100 sends control signals to navigate vehicle 200 to one of throttle system 220, brake system 230, and steering system 240. or more than one (eg, by causing acceleration, turns, lane shifts, etc.). Additionally, the system 100 may input inputs indicating operating conditions of the vehicle 200 (e.g., speed, whether the vehicle 200 is braking and/or turning, etc.) to one of the throttle system 220, the brake system 230, and the steering system 24. It may be received from one or more. Further details are provided below in connection with FIGS. 4-7.
[0144] As shown in FIG. 3A, vehicle 200 may also include a user interface 170 for interacting with a driver or occupant of vehicle 200. For example, a user interface 170 within a vehicle application may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or occupant of vehicle 200 may use a steering wheel (e.g., located on or near the steering column of vehicle 200, including, for example, a turn signal handle), a button (e.g., located on the steering wheel of vehicle 200), etc. can also interact with system 100. In some embodiments, microphone 350 may be positioned adjacent rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be positioned near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (eg, a vehicle audio system speaker). For example, system 100 may provide various notifications (eg, alerts) via speaker 360.
[0145] FIGS. 3B-3D illustrate an exemplary camera mount configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and opposite a vehicle windshield, according to disclosed embodiments. 370 diagram. As shown in FIG. 3B, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind a glare shield 380, which may be in direct contact with the windshield and may include a film and/or composition of anti-reflective material. For example, glareshield 380 may be positioned to be aligned opposite a windshield with a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 may be positioned behind a glare shield 380, for example, as shown in FIG. 3D. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. FIG. 3C is a view of the camera mount 370 shown in FIG. 3B from the front.
[0146] Many variations and/or modifications to the above-disclosed embodiments may be made, as will be appreciated by those skilled in the art with the benefit of this disclosure. For example, not all components are essential to the operation of system 100. Furthermore, any components may be placed in any suitable portion of system 100, and components may be rearranged into various configurations while still providing the functionality of the disclosed embodiments. Accordingly, the configurations described above are examples, and regardless of the configurations described above, the system 100 may provide a wide range of capabilities for analyzing the surroundings of the vehicle 200 and navigating the vehicle 200 in response to the analysis.
[0147] As discussed in more detail below, in accordance with various disclosed embodiments, system 100 may provide various features related to autonomous driving and/or driver assistance technologies. For example, system 100 may analyze image data, location data (eg, GPS location information), map data, speed data, and/or data from sensors included in vehicle 200. System 100 may collect data for analysis from, for example, image acquisition unit 120, position sensor 130, and other sensors. Additionally, system 100 may analyze the collected data to determine whether vehicle 200 should take a particular action and then automatically take the determined action without human intervention. For example, when vehicle 200 navigates without human participation, system 100 may automatically control the brakes, acceleration, and/or steering of vehicle 200 (e.g., send control signals to throttle system 220, brake system 230, and (by transmitting to one or more of the steering systems 240). Additionally, system 100 may analyze the collected data and issue warnings and/or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by system 100 are provided below.
[0148] Forward-facing multi-imaging system [0149] As mentioned above, system 100 may provide driving assistance functionality using a multi-camera system. A multi-camera system may use one or more cameras facing toward the front of the vehicle. In other embodiments, a multi-camera system may include one or more cameras facing the side of the vehicle or the rear of the vehicle. In one embodiment, for example, system 100 may use a two-camera imaging system, where a first camera and a second camera (e.g., image capture devices 122 and 124) are connected to a vehicle (e.g., vehicle 200). ) may be positioned at the front and/or side of the Other camera configurations are consistent with the disclosed embodiments, and the configurations disclosed herein are examples. For example, system 100 may include any number (eg, one, two, three, four, five, six, seven, eight, etc.) camera configurations. Additionally, system 100 may include "clusters" of cameras. For example, the cluster of cameras (including any suitable number of cameras, e.g., one, four, eight, etc.) can be forward-facing with respect to the vehicle, or facing in any other direction. (e.g. facing backwards, sideways, diagonally, etc.). Accordingly, system 100 may include multiple clusters of cameras, with each cluster oriented in a particular direction to capture images from a particular region of the vehicle's environment.
[0150] The first camera may have a field of view that is larger, smaller, or partially overlaps the field of view of the second camera. Further, the first camera may be connected to the first image processor to perform monocular image analysis of images provided by the first camera, and the second camera may be connected to the second image processor. may perform monocular image analysis of the images provided by the second camera. The outputs (eg, processed information) of the first and second image processors may be combined. In some embodiments, a second image processor may receive images from both the first camera and the second camera to perform stereoscopic analysis. In another embodiment, system 100 may use a three-camera imaging system, where each camera has a different field of view. Such a system may thus make decisions based on information derived from objects at various distances both in front of and to the sides of the vehicle. Reference to monocular image analysis may refer to cases in which the image analysis is performed based on images captured from a single viewpoint (eg, a single camera). Stereo image analysis may refer to when image analysis is performed based on two or more images captured with one or more of the image capture parameters changed. For example, captured images suitable for performing stereoscopic image analysis include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, and images captured with disparity information. It may include captured images, etc.
[0151] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (such as 34 degrees or other value selected from a range of approximately 20 to 45 degrees) and image capture device 124 may provide a wide field of view (such as 34 degrees or other value selected from a range of approximately 20 to 45 degrees). For example, the image capture device 126 may provide a medium field of view (e.g., 46 degrees or other values selected from a range of about 35 to about 60 degrees); other values) may be provided. In some embodiments, image capture device 126 may operate as a primary or primary camera. Image capture devices 122-126 may be positioned substantially side by side (eg, 6 cm apart) behind rearview mirror 310. Additionally, in some embodiments, one or more of the image capture devices 122-126 may be mounted behind a glare shield 380 flush with the windshield of the vehicle 200, as described above. Such a shield may operate to minimize the impact of any reflections from the interior of the vehicle on the image capture devices 122-126.
[0152] In another embodiment, as described above in connection with FIGS. 3B and 3C, the wide field of view camera (e.g., image capture device 124 in the above example) is a narrow main field of view camera (e.g., the image capture device 124 in the above example). capture devices 122 and 126). This configuration may provide free line of sight from the wide field of view camera. To reduce reflections, the camera may be mounted near the windshield of the vehicle 200 and may include a polarizer to attenuate reflected light.
[0153] A three-camera system may offer certain performance characteristics. For example, some embodiments may include the ability to verify object detections by one camera based on detections from another camera. In the three-camera configuration described above, processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as described above), each processing device including one of the image capture devices 122-126. The present invention is directed to the processing of images captured by one or more.
[0154] In a three-camera system, the first processing device may receive images from both the main camera and the narrow-field camera, and performs vision processing for the narrow-FOV camera to detect other vehicles, pedestrians, lanes, etc. Marks, traffic signs, traffic lights and other road objects can be detected. Additionally, the first processing device may calculate the pixel mismatch between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of the vehicle 200. The first processing device may then combine the 3D reconstruction with 3D information calculated based on 3D map data or information from another camera.
[0155] The second processing device may receive images from the primary camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device may calculate camera displacements, and based on the displacements, calculate pixel mismatches between successive images to create a 3D reconstruction (eg, structure from motion) of the scene. The second processing device may send the structure from motion based on the 3D reconstruction to the first processing device and combine the structure from motion with the stereoscopic 3D image.
[0156] A third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the image and identify moving objects in the image, such as vehicles changing lanes, pedestrians, etc.
[0157] In some embodiments, having image-based information streams independently captured and processed may provide an opportunity to provide redundancy in the system. Such redundancy may include, for example, verifying information obtained by capturing and processing image information from at least a second image capture device using a first image capture device and images processed from that device. and/or may be supplemented.
[0158] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) in providing navigation assistance to vehicle 200, and a third image capture device (e.g., image capture devices 122 and 124). Capture device 126) may be used to provide redundancy and verify analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereoscopic analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100. Images may be provided to provide redundancy and verification of information obtained based on captured images from image capture device 123 and/or image capture device 124. That is, image capture device 126 (and corresponding processing device) provides redundant subsystems that provide checks to the analyzes derived from image capture devices 122 and 124 (e.g., provide an automatic emergency braking (AEB) system). (to do so). Further, in some embodiments, the received data is based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers external to the vehicle, etc.). can compensate for redundancy and verification.
[0159] Those skilled in the art will recognize that the above camera configuration, camera arrangement, number of cameras, camera positions, etc. are merely examples. These components and the like described with respect to the overall system may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of multi-camera systems to provide driver assistance and/or autonomous vehicle functionality follow below.
[0160] FIG. 4 is an example functional block diagram of memory 140 and/or 150 that may be stored/programmed with instructions to perform one or more operations in accordance with disclosed embodiments. Although reference is made below to memory 140, those skilled in the art will recognize that instructions can be stored in memory 140 and/or 150.
[0161] As shown in FIG. 4, memory 140 may store a monocular image analysis module 402, a stereoscopic image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Additionally, application processor 180 and/or image processor 190 may execute instructions stored in any of the modules 402-408 included in memory 140. Those skilled in the art will appreciate that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps of the following processes may be performed by one or more processing devices.
[0162] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, cause the image capture devices 122, 124, and 126 to Perform monocular image analysis of the acquired image set. In some embodiments, processing unit 110 may combine information from the image set with additional sensory information (eg, information from radar) to perform monocular image analysis. As described in connection with FIGS. 5A-5D below, the monocular image analysis module 402 is configured to analyze information associated with lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and the vehicle's environment. The method may include instructions for detecting a set of features within a set of images, such as any other feature. Based on the analysis, system 100 may cause one or more navigation responses in vehicle 200, such as turns, lane shifts, and acceleration changes, as discussed below in connection with navigation response module 408 (e.g., (via processing unit 110).
[0163] In one embodiment, monocular image analysis module 402 includes instructions for performing monocular image analysis of a set of images acquired by one of image capture devices 122, 124, and 126 when executed by processing unit 110. (computer vision software, etc.). In some embodiments, processing unit 110 may combine information from the image set with additional sensory information (eg, information from radar, lidar, etc.) to perform monocular image analysis. As described below in connection with FIGS. 5A-5D, the monocular image analysis module 402 analyzes information related to lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and the environment of the vehicle. It may include instructions for detecting a set of features within a set of images, such as any other features. Based on the analysis, system 100 determines one or more navigation responses, such as turns, lane shifts, acceleration changes, etc., as discussed below in connection with determining navigation responses (e.g., by processing unit 110). may occur in vehicle 200.
[0164] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, cause the image capture device selected from the image capture devices 122, 124, and 126 to be performing a stereoscopic image analysis of the first and second sets of images obtained by the combination of; In some embodiments, processing unit 110 may combine information from the first and second sets of images with additional sensory information (eg, information from radar) to perform stereoscopic image analysis. For example, stereoscopic image analysis module 404 may provide instructions for performing stereoscopic image analysis based on a first set of images acquired by image capture device 124 and a second set of images acquired by image capture device 126. may be included. As described below in connection with FIG. 6, the stereoscopic image analysis module 404 analyzes the first and second sets of lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and hazards. May include instructions for detecting a set of features within an image. Based on the analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, such as turns, lane shifts, and acceleration changes, as described below in connection with navigation response module 408. Additionally, in some embodiments, stereo image analysis module 404 may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.
[0165] In one embodiment, velocity and acceleration module 406 analyzes data received from one or more computational and electromechanical devices within vehicle 200 that are configured to cause changes in the velocity and/or acceleration of vehicle 200. Software configured to be stored may be stored. For example, processing unit 110 executes instructions associated with velocity and acceleration module 406 to determine a target for vehicle 200 based on data derived from execution of monocular image analysis module 402 and/or stereoscopic image analysis module 404. Speed can be calculated. Such data may include, for example, target position, speed and/or acceleration, position and/or speed of vehicle 200 relative to nearby vehicles, pedestrians or road objects, and position information of vehicle 200 relative to lane markings on the road. obtain. In addition, processing unit 110 may be based on sensory input (e.g., information from radar) and input from other systems of vehicle 200, such as throttle system 220, brake system 230, and/or steering system 240 of vehicle 200. Then, the target speed of vehicle 200 can be calculated. Based on the calculated target speed, processing unit 110 sends electronic signals to throttle system 220, brake system 230 and/or steering system 240 of vehicle 200 to physically weaken the brakes of vehicle 200, for example, or Or a change in speed and/or acceleration may be triggered by depressing the accelerator.
[0166] In one embodiment, navigation response module 408 is executable by processing unit 110 and determines a desired navigation response based on data derived from execution of monocular image analysis module 402 and/or stereo image analysis module 404. may store software to do so. Such data may include location and velocity information associated with nearby vehicles, pedestrians, and road objects, target location information for vehicle 200, and the like. Further, in some embodiments, the navigation response is one detected from map data, a predetermined location of vehicle 200 and/or vehicle 200, and execution of monocular image analysis module 402 and/or stereoscopic image analysis module 404. or may be based (partially or fully) on relative velocity or relative acceleration between objects. Navigation response module 408 determines the desired response based on sensory input (e.g., information from radar) and input from other systems of vehicle 200, such as throttle system 220, brake system 230, and steering system 240 of vehicle 200. Navigation responses can also be determined. Based on the desired navigation response, processing unit 110 sends electronic signals to vehicle 200's throttle system 220, brake system 230, and steering system 240 to, for example, turn the steering wheel of vehicle 200 and perform a predetermined angle of rotation. Achieving this may trigger the desired navigation response. In some embodiments, processing unit 110 uses the output of navigation response module 408 (e.g., the desired navigation response) as an input to the execution of velocity and acceleration module 406 to calculate the velocity change of vehicle 200. It is possible.
[0167] Additionally, any of the modules disclosed herein (e.g., modules 402, 404, and 406) may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems. can do.
[0168] FIG. 5A is a flowchart illustrating an example process 500A for generating one or more navigation responses based on monocular image analysis, according to disclosed embodiments. At step 510, processing unit 110 may receive a plurality of images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera (such as an image capture device 122 having a field of view 202) included in the image acquisition unit 120 captures a plurality of images of an area in front of the vehicle 200 (or, for example, on the side or rear of the vehicle) and connects the data connection ( (eg, digital, wired, USB, wireless, Bluetooth, etc.) to processing unit 110. Processing unit 110 may execute monocular image analysis module 402 to analyze the plurality of images in step 520, as described in further detail in connection with FIGS. 5B-5D below. By performing the analysis, processing unit 110 may detect sets of features within the set of images such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, and traffic lights.
[0169] Processing unit 110 may also execute monocular image analysis module 402 in step 520 to detect various road hazards, such as truck tire parts, fallen road signs, loose cargo, and small animals. can. The structure, shape, size and color of road hazards can vary, making detection of such hazards more difficult. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera movement between successive image frames and calculate pixel mismatch between frames to construct a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and hazards present on the road surface.
[0170] In step 530, processing unit 110 executes navigation response module 408 to provide one or more navigation responses to vehicle 200 based on the analysis performed in step 520 and the techniques described above in connection with FIG. A navigation response may be generated. Navigation responses may include, for example, turns, lane shifts, acceleration changes, and the like. In some embodiments, processing unit 110 may use data derived from execution of velocity and acceleration module 406 to generate one or more navigation responses. Additionally, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof. For example, processing unit 110 may cause vehicle 200 to cross one lane and then, for example, accelerate, by, for example, sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to apply the brakes and simultaneously shift lanes, for example, by simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.
[0171] FIG. 5B is a flowchart illustrating an example process 500B for detecting one or more vehicles and/or pedestrians in a set of images, according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500B. At step 540, processing unit 110 may identify a set of candidate objects representing vehicles and/or pedestrians that may be present. For example, processing unit 110 scans one or more images, compares the images to one or more predetermined patterns, and detects objects of interest (e.g., vehicles, pedestrians, or parts thereof) within each image. may identify possible locations that may include. A predetermined pattern may be specified to achieve a low rate of "false hits" and a low rate of "misses." For example, processing unit 110 may identify candidate objects as potential vehicles or pedestrians using a low similarity threshold to a predetermined pattern. By doing so, processing unit 110 may be able to reduce the probability of missing (eg, not identifying) candidate objects representing vehicles or pedestrians.
[0172] At step 542, processing unit 110 may filter the set of candidate objects to exclude certain candidates (eg, irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various characteristics associated with object types stored in a database (eg, a database stored in memory 140). The characteristics may include the object's shape, size, texture, position (eg, relative to vehicle 200), and the like. Accordingly, processing unit 110 may use one or more sets of criteria to reject spurious candidates from the set of candidate objects.
[0173] At step 544, processing unit 110 may analyze the plurality of image frames to identify whether objects in the set of candidate images represent vehicles and/or pedestrians. For example, processing unit 110 may track detected candidate objects over successive frames and accumulate frame-by-frame data (eg, size, location relative to vehicle 200, etc.) associated with the detected objects. Additionally, processing unit 110 may estimate parameters of the detected object and compare frame-by-frame position data of the object with the predicted position.
[0174] At step 546, processing unit 110 may construct a set of measurements of the detected object. Such measurements may include, for example, position, velocity, and acceleration values (relative to vehicle 200) associated with the detected object. In some embodiments, processing unit 110 includes estimation techniques that use a set of time-based observations, such as a Kalman filter or linear quadratic estimation (LQE), and/or different object types (e.g., cars, trucks, pedestrians, etc.). Measurements can be constructed based on modeling data available on bicycles, road signs, etc.). The Kalman filter may be based on a measure of the scale of the object, where the scale measure is proportional to the time to collision (eg, the amount of time it takes the vehicle 200 to reach the object). Accordingly, by performing steps 540-546, processing unit 110 identifies vehicles and pedestrians that appear within the set of captured images, and information associated with the vehicles and pedestrians (e.g., location, speed, size). can be derived. Based on the identified and derived information, processing unit 110 may cause one or more navigation responses at vehicle 200, as described above in connection with FIG. 5A.
[0175] In step 548, processing unit 110 performs optical flow analysis of the one or more images to reduce the probability of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. It is possible. Optical flow analysis may refer, for example, to analyzing movement patterns for the vehicle 200 in one or more images associated with other vehicles and pedestrians, separate from road surface motion. Processing unit 110 may calculate the movement of the candidate object by observing different positions of the object across multiple image frames captured at different times. Processing unit 110 may use the position and time values as inputs to a mathematical model to calculate movement of the candidate object. Accordingly, optical flow analysis may provide another method of detecting vehicles and pedestrians in the vicinity of vehicle 200. Processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy in detecting vehicles and pedestrians and increase reliability of system 100.
[0176] FIG. 5C is a flowchart illustrating an example process 500C for detecting road markings and/or lane geometry information in a set of images, according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500C. At step 550, processing unit 110 may detect the set of objects by scanning one or more images. To detect lane mark divisions, lane geometry information, and other relevant road markings, processing unit 110 filters the set of objects that are determined to be irrelevant (e.g., small holes, small rocks, etc.). can be excluded. In step 552, processing unit 110 may group together the segments detected in step 550 that belong to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, to represent the detected partitions.
[0177] At step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, processing unit 110 may create a projection of the detection segment from the image plane to the real world plane. The projection may be characterized using a third order polynomial with coefficients corresponding to physical properties such as position, slope, curvature and curvature derivative of the detected road. In generating the projection, processing unit 110 may take into account road surface changes and pitch and roll rates associated with vehicle 200. In addition, processing unit 110 may model road height by analyzing the location and motion cues present on the road surface. Further, processing unit 110 may estimate pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.
[0178] At step 556, processing unit 110 may perform multi-frame analysis, for example, by tracking the detection segment over successive image frames and accumulating frame-by-frame data associated with the detection segment. If processing unit 110 performs a multi-frame analysis, the set of measurements constructed in step 554 may become more reliable and may be associated with increasingly higher degrees of confidence. Thus, by performing steps 550-556, processing unit 110 may identify road markings that appear within the set of captured images and derive lane geometry information. Based on the identified and derived information, processing unit 110 may cause one or more navigation responses at vehicle 200, as described above in connection with FIG. 5A.
[0179] At step 558, processing unit 110 may further develop a safety model of vehicle 200 in the vehicle's surroundings by considering additional information sources. Processing unit 110 may use the safety model to define situations in which system 100 may safely perform autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 uses the location and movement of other vehicles, detected road edges and barriers, and/or map data (such as data from map database 160). It is possible to consider the general road shape description extracted from . By considering additional information sources, processing unit 110 may provide redundancy in detecting road markings and lane geometry, increasing the reliability of system 100.
[0180] FIG. 5D is a flowchart illustrating an example process 500D for detecting traffic lights in a set of images, according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500D. At step 560, processing unit 110 may scan the set of images and identify objects appearing at locations within the images that are likely to include traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects excluding objects that are unlikely to correspond to traffic lights. Filtering may be performed based on various characteristics associated with traffic lights, such as shape, size, texture, and location (eg, relative to vehicle 200). Such characteristics may be based on many examples of traffic lights and traffic control signals and may be stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on a set of candidate objects reflecting possible traffic lights. For example, processing unit 110 may track a candidate object over successive image frames, estimate the real-world position of the candidate object, and filter out objects that are moving (unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects and identify relative positions of detected colors represented within potential traffic lights.
[0181] At step 562, processing unit 110 may analyze the geometry of the intersection. The analysis includes (i) the number of lanes detected on each side of the vehicle 200, (ii) marks detected on the road (such as arrow marks), and (iii) map data (such as data from the map database 160). can be based on any combination of intersection descriptions. Processing unit 110 may use information derived from execution of monocular analysis module 402 to perform the analysis. In addition, processing unit 110 may identify correspondences between the traffic lights detected in step 560 and lanes appearing in the vicinity of vehicle 200.
[0182] As the vehicle 200 approaches the intersection, the processing unit 110 may update the analyzed intersection geometry and confidence values associated with the detected traffic lights at step 564. For example, the number of traffic lights that are estimated to appear at an intersection compared to the number that actually appear at the intersection can affect the confidence level. Therefore, based on the reliability, processing unit 110 may delegate control to the driver of vehicle 200 to improve the safety situation. By performing steps 560-564, processing unit 110 may identify traffic lights that appear within the set of captured images and analyze intersection geometry information. Based on the identification and analysis, processing unit 110 may generate one or more navigation responses at vehicle 200, as described above in connection with FIG. 5A.
[0183] FIG. 5E is a flowchart of an example process 500E for generating one or more navigation responses at vehicle 200 based on a vehicle path, according to disclosed embodiments. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. A vehicle path may be represented using a set of points with coordinates (x,y), and the distance di between two points within the set of points may be in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as left and right road polynomials. The processing unit 110 calculates the geometric midpoint between the two polynomials and, if there is a predetermined offset (an offset of 0 may correspond to driving in the center of the lane), the processing unit 110 calculates the geometric midpoint between the two polynomials, each point included in the resulting vehicle path may be offset by an offset). The offset may be in a direction perpendicular to the division between any two points in the vehicle path. In another embodiment, processing unit 110 uses one polynomial and the estimated lane width to size each point in the vehicle path by half the estimated lane width plus a predetermined offset (e.g., smart lane offset). It can be offset.
[0184] At step 572, processing unit 110 may update the vehicle route constructed at step 570. The processing unit 110 determines that the distance dk between two points in the set of points representing the vehicle route is the distance d described above.<sub>i</sub>The vehicle path constructed at 570 may be reconstructed using a higher resolution so that it is shorter than . For example, distance d<sub>k</sub>can range from 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may result in a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path). .
[0185] In step 574, processing unit 110 calculates the look-ahead point ((x<sub>l</sub>,z<sub>l</sub>) can be specified, expressed in coordinates as ). Processing unit 110 may extract look-ahead points from the cumulative distance vector S, and may associate look-ahead distances and look-ahead times with the look-ahead points. The look-ahead distance may have a lower limit range of 10-20 meters and may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of vehicle 200 decreases, the lookahead distance may also decrease (eg, until a lower limit is reached). The look-ahead time, which can range from 0.5 to 1.5 seconds, can be inversely proportional to the gain of one or more control loops associated with producing a navigation response in vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop may depend on the bandwidth of the yaw rate loop, the steering actuator loop, the vehicle lateral dynamics, etc. Therefore, the higher the gain of the advancing error tracking control loop, the shorter the look ahead time.
[0186] At step 576, processing unit 110 may determine a heading error and yaw rate command based on the look-ahead points identified at step 574. The processing unit 110 calculates the arctangent of the look-ahead point, e.g. arctan(x<sub>l</sub>/z<sub>l</sub>) can identify progress errors. Processing unit 110 may determine the yaw rate command as the product of the heading error and the high level control gain. The high level control gain may be equal to (2/lookahead time) if the lookahead distance is not at the lower limit. If the lookahead distance is the lower limit, the high level control gain may be equal to (2*speed of vehicle 200/lookahead distance).
[0187] FIG. 5F is a flowchart illustrating an example process 500F for determining whether a preceding vehicle is changing lanes, according to disclosed embodiments. At step 580, processing unit 110 may identify navigation information associated with a preceding vehicle (eg, a vehicle traveling in front of vehicle 200). For example, processing unit 110 may determine the position, velocity (eg, direction and velocity), and/or acceleration of a preceding vehicle using the techniques described above in connection with FIGS. 5A and 5B. Processing unit 110 uses the techniques described above in connection with FIG. It is also possible to specify a set of points that describe
[0188] At step 582, processing unit 110 may analyze the navigation information identified at step 580. In one embodiment, processing unit 110 may calculate the distance (eg, along the trail) between the snail trail and the road polynomial. If the difference in this distance along the trail exceeds a predetermined threshold (e.g. 0.1-0.2 meters on a straight road, 0.3-0.4 meters on a gently curved road, 0.5-0.6 meters on a sharply curved road), the processing unit 110 It can be determined that there is a high possibility that the preceding vehicle is changing lanes. If multiple vehicles are detected to be traveling in front of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trail of another vehicle is likely to be changing lanes. Processing unit 110 may further compare the curvature of the snail trail (associated with the preceding vehicle) to the expected curvature of the road segment on which the preceding vehicle is traveling. The expected curvature may be extracted from map data (eg, data from map database 160), road polynomials, snail trails of other vehicles, prior knowledge about the road, and the like. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the preceding vehicle is likely changing lanes.
[0189] In another embodiment, processing unit 110 may compare the instantaneous position of the preceding vehicle to a look-ahead point (associated with vehicle 200) over a particular period of time (eg, 0.5-1.5 seconds). The difference in distance and the cumulative sum of the differences between the instantaneous position of the preceding vehicle and the look-ahead point during a particular time period is determined by a predetermined threshold (e.g. 0.3-0.4 meters on a straight road and 0.7-0.8 on a gently curved road). (1.3 to 1.7 meters on roads with sharp curves), the processing unit 110 may determine that there is a high possibility that the preceding vehicle is changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature is calculated: (δ<sub>z</sub><sup>2</sup>+δ<sub>x</sub><sup>2</sup>)/2/(δ<sub>x</sub>), where σ<sub>x</sub>represents the horizontal movement distance, and σz represents the vertical movement distance. If the difference between the lateral travel distance and the expected curvature exceeds a predetermined threshold (eg, 500-700 meters), processing unit 110 may determine that the preceding vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the preceding vehicle. If the position of the preceding vehicle obscures the road polynomial (eg, the preceding vehicle overlaps the road polynomial), processing unit 110 may determine that the preceding vehicle is likely changing lanes. If the position of the preceding vehicle is such that another vehicle is detected in front of the preceding vehicle and the snail trails of the two vehicles are not parallel, the processing unit 110 determines that the (closer) preceding vehicle is changing lanes. It can be judged that there is a high possibility that this is the case.
[0190] At step 584, processing unit 110 may determine whether the preceding vehicle 200 is changing lanes based on the analysis performed at step 582. For example, processing unit 110 may make its determination based on a weighted average of the individual analyzes performed in step 582. Under such a scheme, for example, a determination by processing unit 110 that the preceding vehicle is likely to be changing lanes, based on a particular type of analysis, may be assigned the value "1" ("0" indicates that it is unlikely that the vehicle in front is changing lanes). Different analyzes performed in step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyzes and weights. Additionally, in some embodiments, the analysis may utilize a trained system (e.g., a machine learning or deep learning system) that can, for example, determine the location of the vehicle based on images captured at the current location. It is possible to estimate future routes.
[0191] FIG. 6 is a flowchart illustrating an example process 600 for generating one or more navigation responses based on stereo image analysis, according to disclosed embodiments. At step 610, processing unit 110 may receive first and second plurality of images via data interface 128. For example, a camera (such as an image capture device 122 and 124 having a field of view 202 and 204) included in the image acquisition unit 120 captures a first and second plurality of images of an area in front of the vehicle 200 and a digital connection ( (eg, USB, wireless, Bluetooth, etc.) to processing unit 110. In some embodiments, processing unit 110 may receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0192] At step 620, processing unit 110 executes stereoscopic image analysis module 404 to perform stereoscopic image analysis of the first and second plurality of images to create a 3D map of the road in front of the vehicle and to create a 3D map of the road in front of the vehicle. Features in images such as marks, vehicles, pedestrians, road signs, highway exit ramps, traffic lights and road hazards may be detected. Stereo image analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D. For example, processing unit 110 executes stereoscopic image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road markings, traffic lights, road hazards, etc.) in the first and second plurality of images. may filter out a subset of candidate objects based on various criteria, perform multi-frame analysis, construct measurements, and identify confidence levels of remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images rather than information from only one set of images. For example, processing unit 110 may analyze differences in pixel level data (or other data subsets from among the two streams of captured images) of candidate objects that appear in both the first and second plurality of images. As another example, the processing unit 110 may detect that an object appears in one of the plurality of images but not the other, or that there may be other objects appearing in the two image streams. The candidate object's position and/or velocity (eg, relative to vehicle 200) may be estimated for the difference in . For example, the position, velocity, and/or acceleration for vehicle 200 may be determined based on the trajectory, position, movement characteristics, etc. of features associated with objects that appear in one or both of the image streams.
[0193] In step 630, processing unit 110 executes navigation response module 408 to determine one or more navigation responses in vehicle 200 based on the analysis performed in step 620 and the techniques described above in connection with FIG. Navigation movements may occur. Navigation responses may include, for example, turns, lane shifts, acceleration changes, speed changes, braking, and the like. In some embodiments, processing unit 110 may use data derived from execution of velocity and acceleration module 406 to generate one or more navigation responses. Further, multiple navigation responses may be performed simultaneously, sequentially, or any combination thereof.
[0194] FIG. 7 is a flowchart illustrating an example process 700 for generating one or more navigation responses based on analysis of three sets of images, according to disclosed embodiments. At step 710, processing unit 110 may receive a first, second, and third plurality of images via data interface 128. For example, the cameras included in the image acquisition unit 120 (such as image capture devices 122, 124 and 126 having fields of view 202, 204 and 206) are arranged in the first, second and second positions in the front and/or side area of the vehicle 200. 3 images may be captured and transmitted to processing unit 110 via a digital connection (eg, USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first, second, and third plurality of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface that communicates data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0195] At step 720, processing unit 110 analyzes the first, second, and third plurality of images to include images of lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards, etc. can detect features within. The analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D and FIG. 6. For example, processing unit 110 may perform monocular image analysis on each of the first, second, and third plurality of images (e.g., related to performing monocular image analysis module 402 and FIGS. 5A-5D). (based on the steps mentioned above). Alternatively, the processing unit 110 performs stereoscopic image analysis on the first and second plurality of images, the second and third plurality of images and/or the first and third plurality of images. (e.g., via execution of stereo image analysis module 404 and based on the steps described above in connection with FIG. 6). Processed information corresponding to analysis of the first, second and/or third plurality of images may be combined. In some embodiments, processing unit 110 may perform a combination of monocular and stereo image analysis. For example, processing unit 110 performs monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and performs stereoscopic image analysis on the second and third plurality of images. (eg, via execution of the stereo image analysis module 404). The configuration of image capture devices 122, 124 and 126 - including their respective positions and fields of view 202, 204 and 206 - can affect the type of analysis performed on the first, second and third plurality of images. . The disclosed embodiments are not limited to the particular configuration of image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.
[0196] In some embodiments, processing unit 110 may perform tests on system 100 based on the images acquired and analyzed in steps 710 and 720. Such testing may provide an indicator of the overall performance of system 100 with a particular configuration of image capture devices 122, 124, and 126. For example, processing unit 110 may identify a rate of "false hits" (eg, when system 100 incorrectly determines the presence of a vehicle or pedestrian) and "misses."
[0197] In step 730, processing unit 110 causes one or more navigation responses in vehicle 200 based on information derived from two of the first, second and third plurality of images. obtain. The choice between the first, second and third plurality of images may depend on various factors such as, for example, the number, type and size of objects detected in each of the plurality of images. The processing unit 110 determines the quality and resolution of the image, the effective field of view reflected in the image, the number of captured frames, and the extent to which the object or objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the object The selection can be made based on the proportion of such frames that appear in each frame, etc.).
[0198] In some embodiments, processing unit 110 determines the degree to which information derived from one image source is consistent with information derived from other image sources. Information derived from two of the third plurality of images may be selected. For example, processing unit 110 combines processed information (whether monocular, stereoscopic, or any combination of the two) derived from each of image capture devices 122, 124, and 126. to provide visual indicators (e.g., lane markings, detected vehicles and/or their locations and/or routes, detected traffic lights, etc.) that are consistent across images captured from each of image capture devices 122, 124, and 126. can be specified. Processing unit 110 may also filter out information that is inconsistent across captured images (eg, vehicles changing lanes, lane models showing vehicles too close to vehicle 200, etc.). Accordingly, processing unit 110 may select derived information from two of the first, second and third plurality of images based on the identification of consistent and inconsistent information.
[0199] Navigation responses may include, for example, turns, lane shifts, and acceleration changes. Processing unit 110 may generate one or more navigation responses based on the analysis performed in step 720 and the techniques described above in connection with FIG. 4. Processing unit 110 may also use data derived from execution of velocity and acceleration module 406 to generate one or more navigation responses. In some embodiments, the processing unit 110 determines the relative position, relative velocity, and/or relative acceleration between the vehicle 200 and the object detected in any of the first, second, and third plurality of images. One or more navigation responses may be generated based on the navigation response. Multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.
[0200] Reinforcement learning and trained navigation system [0201] The following sections describe whether a vehicle's autonomous control is fully autonomous (self-driving vehicle) or partially autonomous (e.g., one or more drivers assisting a system or function). Discusses autonomous driving along with systems and methods for achieving autonomous control of vehicles. As shown in FIG. 8, the autonomous driving task can be divided into three main modules, including a detection module 801, a driving policy module 803, and a control module 805. In some embodiments, modules 801, 803, and 805 may be stored within memory unit 140 and/or memory unit 150 of system 100, and/or modules 801, 803, and 805 (or portions thereof) may be stored within memory unit 140 and/or memory unit 150 of system 100. It may be stored remotely from system 100 (eg, stored in a server accessible to system 100, eg, by wireless transceiver 172). Additionally, any of the modules disclosed herein (e.g., modules 801, 803, and 805) may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems. can.
[0202] Sensing module 801, which may be implemented using processing unit 110, may handle various tasks related to sensing navigation conditions within the host vehicle's environment. Such tasks may rely on input from various sensors and sensing systems associated with the host vehicle. Those inputs may include images or image streams from one or more onboard cameras, GPS location information, accelerometer output, user feedback, user input to one or more user interface devices, radar, lidar, etc. may be included. Collect and analyze sensing, which may include data from cameras and/or any other available sensors along with map information, into a "sensing state" that describes the information extracted from the scene within the host vehicle's environment. It can be expressed systematically. Sensing conditions include target vehicles, lane markings, pedestrians, traffic lights, road geometry, lane shapes, obstacles, distance to other objects/vehicles, relative speed, among other potential sensing information. It may specifically include sensing information related to relative acceleration. Supervised machine learning may be performed to create sensing status outputs based on sensing data provided to sensing module 801. The output of the sensing module may represent a sensed navigation "state" of the host vehicle, which may be sent to the driving policy module 803.
[0203] Although sensing conditions may be developed based on image data received from one or more cameras or image sensors associated with the host vehicle, sensing conditions used for navigation may be developed using any suitable sensor. Or it can be developed using a combination of sensors. In some embodiments, sensing conditions may be developed without using captured image data. Indeed, any of the navigation principles described herein may be applicable to sensing states developed based on captured image data as well as sensing states developed using other non-image-based sensors. The sensing condition may also be determined by a source external to the host vehicle. For example, the sensed state may be based on information received from a source remote from the host vehicle (e.g., shared from other vehicles, shared from a central server, or information related to the host vehicle's navigation state). (based on shared sensor information, processed state information, etc. from any other source).
[0204] Driving policy module 803, described in more detail below and which may be implemented using processing unit 110, determines one or more navigation actions to be taken by the host vehicle in response to the sensed navigation conditions. A desired driving policy can be implemented to If there are no other agents (eg, target vehicles or pedestrians) in the host vehicle's environment, the sensed conditions input to the driving policy module 803 can be processed in a relatively simple manner. This task becomes more complex if the sensing condition requires negotiation with one or more other agents. The techniques used to generate the output of driving policy module 803 may include reinforcement learning (discussed in more detail below). The output of the driving policy module 803 may include at least one navigation action for the host vehicle, including, among the potential desired navigation actions, a desired acceleration (which may lead to an updated speed of the host vehicle); It may specifically include the desired yaw rate, desired trajectory of the host vehicle.
[0205] Based on the output from the driving policy module 803, the control module 805, which may also be implemented using the processing unit 110, provides control for one or more actuators or controlled devices associated with the host vehicle. Instructions can be developed. Such actuators and devices may include an accelerator, one or more steering controls, brakes, signal transmitters, displays, or any other actuators or devices that can be controlled as part of navigation operations associated with the host vehicle. Aspects of control theory may be used to generate the output of control module 805. To implement the desired navigation goals or requirements of driving policy module 803, control module 805 may be responsible for developing and outputting instructions to controllable components of the host vehicle.
[0206] Returning to the driving policy module 803, in some embodiments, the driving policy module 803 can be implemented using a trained system trained by reinforcement learning. In other embodiments, the driving policy module 803 may be implemented without machine learning methods by using specified algorithms to "manually" address various scenarios that may arise during autonomous navigation. However, while feasible, such approaches may result in driving policies that are too simple and may lack the flexibility of trained systems based on machine learning. A trained system may be better equipped to handle complex navigational situations and may be better able to determine whether a taxi is parked, stopping to pick up or drop off a passenger. , better determine whether a pedestrian is about to cross the road in front of the host vehicle, better balance unexpected behavior of other drivers against self-defense, and reduce the risk of target vehicles and/or pedestrians. better navigate congested roads, including when to interrupt certain navigation rules or augment others; It is possible to better predict whether obstacles will appear from behind. Trained systems based on reinforcement learning may be better equipped to deal with continuous and high-dimensional state spaces as well as continuous motion spaces.
[0207] Training the system using reinforcement learning may include learning driving policies to map sensing conditions to navigation actions. The driving policy is a function π:SA, where S is a set of states,<math num="1"><img file="JP7367255B2_D0001.tif" /></math>is the motion space (eg, desired velocity, acceleration, yaw command, etc.). The state space is S=S<sub>S</sub>×S<sub>p</sub>and S<sub>S</sub>is the detected state and Sp is additional information to the state saved by the policy. It works in discrete time intervals, allowing us to observe the current state stS at time t, and apply a policy to achieve the desired behavior a<sub>t</sub>=π(s<sub>t</sub>) can be obtained.
[0208] The system can be trained by exposing it to various navigation conditions, having the system apply a policy, and rewarding it (based on a reward function designed to reward the desired navigation behavior). . Based on the reward feedback, the system can "learn" the policy and become trained in producing the desired navigation behavior. For example, a learning system can use the current state s<sub>t</sub>Observe S and set the policy<math num="2"><img file="JP7367255B2_D0002.tif" /></math>works based on a<sub>t</sub>A can be determined. Based on the determined behavior (and the implementation of that behavior), the environment moves to the next state s for observation by the learning system.<sub>t+1</sub>Move to S. For each behavior developed in response to the observed state, the feedback to the learning system is the reward signal r<sub>1</sub>, r2, ....
[0209] The goal of reinforcement learning (RL) is to find the policy π. At time t, state s<sub>t</sub>and works a<sub>t</sub>A reward function r that measures the immediate quality of doing<sub>t</sub>It is usually assumed that there is. However, at time t the action a<sub>t</sub>Doing so affects the environment and thus the value of future states. As a result, when deciding which action to perform, one should not only consider current rewards, but also future rewards. In some cases, the system determines that you will realize more reward in the future if you take the lower-reward option here, even if the particular behavior is associated with a lower reward than the alternative available. Even the system should do that operation. To formalize this, the policy π and the initial state s are<math num="3"><img file="JP7367255B2_D0003.tif" /></math>Observe that the agent causes a distribution over state s<sub>0</sub>If we start from =s and follow policy π from there, then the vector (r1,...,r<sub>T</sub>) is the probability of reward r<sub>1</sub>,...,r<sub>T</sub>is the probability of observing . The value of the initial state s can be determined by the following equation.
[0210]
<math num="4"><img file="JP7367255B2_D0004.tif" /></math>
[0211] Instead of limiting the target period to T, the following equation can be determined for some fixed γ(0,1) by discounting future rewards.
[0212]
<math num="5"><img file="JP7367255B2_D0005.tif" /></math>
[0213] In any case, the optimal policy is [0214]
<math num="6"><img file="JP7367255B2_D0006.tif" /></math> [0215], and the expected value spans the initial state s.
[0216] There are several possible methodologies for training a driving policy system. For example, the system can use imitation techniques (e.g., behavior cloning) that learn from state/action pairs, where actions are selected by a superior agent (e.g., a human) depending on a particular observed state. It is something. Assume that a human driver is observed. This observation allows us to use (s<sub>t</sub>,a<sub>t</sub>) of the form (s<sub>t</sub>is the state and a<sub>t</sub>(is the behavior of human drivers) can be obtained, observed, and used. For example, π(s<sub>t</sub>)a<sub>t</sub>The policy π can be learned using supervised learning such that . This approach has many potential advantages. First, there is no need to define a reward function. Second, the learning is supervised and occurs offline (no agent needs to be applied within the learning process). The disadvantage of this method is that different human drivers and even the same human driver are not deterministic in their policy choices. Therefore, ||π(s<sub>t</sub>)-a<sub>t</sub>It is often impossible to learn functions for which || is very small. Furthermore, even small errors may accumulate over time to produce large errors.
[0217] Another technique that may be used is policy-based learning. Here, the policy can be expressed in parametric form and directly optimized using a suitable optimization technique (eg, stochastic gradient descent). This method is<math num="7"><img file="JP7367255B2_D0007.tif" /></math>It is to directly solve the problem given by . Of course, there are many ways to solve this problem. One advantage of this method is that it directly addresses the problem and therefore often yields good practical results. One potential disadvantage is that this technique often requires "on-policy" training, i.e. learning π is an iterative process, where a less than perfect policy πj is obtained, the following policy π<sub>j</sub>To construct π<sub>j</sub>must interact with the environment while acting based on
[0218] The system can also be trained by value-based learning (Q-function or V-function learning). Assume that we can learn a good approximation for the optimal value function V*. An optimal policy can be constructed (eg, by using the Bellman equation). Some versions of value-based learning can be conducted offline (referred to as "off-policy" training). Some disadvantages of value-based approaches can be attributed to the fact that this approach relies heavily on Markov assumptions and the required approximation of complex functions (it is more difficult to approximate the value function than to directly approximate the policy). ).
[0219] Another technique may include model-based learning and planning (learning the probabilities of state transitions and solving an optimization problem to find the optimal V). A combination of these techniques can also be used to train a learning system. In this method, the dynamics of the process, i.e. (s<sub>t</sub>,a<sub>t</sub>) and the next state s<sub>t+1</sub>It is possible to learn a function that yields a distribution over Once this function is learned, an optimization problem can be solved to find the policy π whose value is optimal. This is called a "plan." One advantage of this method is that the learning part is supervised and the triplet (s<sub>t</sub>,a<sub>t</sub>,s<sub>t+1</sub>) can be applied offline by observing the Similar to the "imitation" approach, one disadvantage of this approach may be that small errors in the learning process may accumulate and result in poorly functioning policies.
[0220] Another approach to training the driving policy module 803 may include decomposing the driving policy function into semantically significant components. Doing so requires manual implementation of parts of the policy that can guarantee security, as well as adaptability to many scenarios, human-like balance between defensive/aggressive behavior and other Other parts of the policy can be implemented using reinforcement learning methods that may enable human-like negotiation with drivers. From a technical point of view, reinforcement learning methods can combine several methodologies to provide a tractable training procedure in which most of the training can be performed using recorded data or self-built simulators.
[0221] In some embodiments, training of driving policy module 803 may utilize a "choice" mechanism. To illustrate it, consider a simple scenario of a driving policy for a two-lane highway. In direct RL method, the state is<math num="8"><img file="JP7367255B2_D0008.tif" /></math>The first component of π(s) is the desired acceleration command and the second component of π(s) is the yaw rate. With the modified approach, the following policies can be constructed:
[0222] Automatic cruise control (ACC) policy, o<sub>ACC</sub>:SA: This policy always outputs a yaw rate of 0 and changes only the speed to ensure smooth and accident-free driving.
[0223] ACC+Left policy, o<sub>L</sub>:SA: The vertical command of this policy is the same as the ACC command. Yaw rate is a simple implementation of centering the vehicle toward the center of the left lane while ensuring safe lateral movement (eg, not moving to the left if there is a car on the left).
[0224] ACC+Right policy, o<sub>R</sub>:SA:o<sub>L</sub>, but the vehicle may be centered towards the center of the right lane.
[0225] These policies can be called "options." Depending on these "alternatives", we can learn a policy for selecting alternatives, πo:SO, where O is the set of available alternatives. In one case, O={o<sub>ACC</sub>,o<sub>L</sub>,o<sub>R</sub>} holds true. For all s,<math num="9"><img file="JP7367255B2_D0009.tif" /></math>By setting , the option selector policy πo determines the actual policy, π:SA.
[0226] In reality, the policy function can be decomposed into a graph of options 901 as shown in FIG. FIG. 10 shows another example of a graph 1000 of options. The graph of choices may represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). There is a special node called the root node 903 of the graph. This node has no input nodes. The decision process starts at the root node and traverses this graph until it reaches a "leaf" node, which refers to a node that has no output decision line. As shown in FIG. 9, leaf nodes may include nodes 905, 907, and 909, for example. Upon encountering a leaf node, driving policy module 803 may output acceleration and steering commands related to the desired navigational behavior associated with the leaf node.
[0227] For example, internal nodes such as nodes 911, 913, 915, etc. may provide implementation of a policy that selects a child among its available choices. The set of available children of an internal node includes all of the nodes that are related to the particular internal node by decision lines. For example, the internal node 913 shown as "Merge" in FIG. 9 has three child nodes 909, 915, and 917 ("Stay," "Pass on the right," and "Pass on the left," respectively) connected to node 913 by a decision line. including.
[0228] Flexibility of the decision-making system can be achieved by allowing nodes to adjust their position within the hierarchy of the graph of choices. For example, any of the nodes may be allowed to declare themselves "critical." Each node may implement a function "is critical" that outputs "true" if the node is within the critical section of its policy implementation. For example, the node responsible for the takeover may declare itself critical during the operation. This may impose a constraint on the set of available children of node u, such that all such children are children of node u and are designated as critical from v to a leaf node. It may include all nodes v for which there is a path passing through the node. Such an approach may, on the one hand, allow the declaration of a desired path on the graph at each time step, whereas on the other hand, it may, among other things, preserve the stability of the policy while critical parts of the policy are being implemented. I can do it.
[0229] By defining a graph of options, the problem of learning the driving policy π:SA can be decomposed into the problem of defining a policy for each node of the graph, and the policies at internal nodes can be should be selected from among the child nodes. For some of the nodes, individual policies can be implemented manually (e.g., by an if-then algorithm that specifies a set of actions depending on the observed state), while for others, reinforcement learning Policies can be implemented using trained systems built by The choice between a manual approach or a trained/learned approach may depend on the safety aspects associated with the task and its relative simplicity. The graph of choices may be constructed in such a way that some of the nodes may be easily implemented while other nodes may depend on the trained model. Such an approach can ensure safe operation of the system.
[0230] The discussion below provides further details regarding the role of the selection graph of FIG. 9 within the driving policy module 803. As discussed above, the input to the driving policy module is, for example, a "sensing state" that outlines the environmental map obtained from the available sensors. The output of the driving policy module 803 is a set of desires (optionally accompanied by a set of hard constraints) that define a trajectory as a solution to the optimization problem.
[0231] As mentioned above, the graph of choices represents a hierarchical set of decisions organized as a DAG. There is a special node called the "root" of the graph. The root node is the only node that does not have an input edge (eg, a decision line). The decision process starts at the root node and traverses the graph until it reaches a "leaf" node, ie, a node that has no output edges. Each internal node should implement a policy to choose one child from among its available children. Every leaf node should implement a policy that defines a set of desires (eg, a set of navigation goals for the host vehicle) based on the entire path from the root to the leaf. Together with the set of hard constraints defined directly based on the sensed conditions, the set of desires establishes an optimization problem whose solution is the trajectory of the vehicle. Hard constraints can be used to further increase the safety of the system, and aspirations can be used to provide driving comfort and human-like driving behavior of the system. The trajectory provided as a solution to the optimization problem therefore defines the commands that should be given to the steering, brake and/or engine actuators to achieve the trajectory.
[0232] Returning to FIG. 9, a graph of options 901 is a graph of options for a two-lane highway that includes a merging lane (meaning that at some point a third lane merges into the right or left lane of the highway). represents. Root node 903 first determines whether the host vehicle is in a simple road scenario or approaching a merge scenario. This is an example of a determination that can be made based on the detection state. The simple road node 911 includes three child nodes: a stay node 909, a left overtake node 917, and a right overtake node 915. Staying refers to a situation where the host vehicle wants to continue traveling in the same lane. The nodes that stay are leaf nodes (no output edges/lines). Therefore, the staying nodes define the desire set. The first desire defined by this node may include, for example, a desired lateral position as close as possible to the center of the current travel lane. There may also be a desire to navigate smoothly (eg, within a predefined or permissible acceleration maximum). The staying node can also define how the host vehicle reacts to other vehicles. For example, a lingering node can examine detected target vehicles and assign a semantic meaning to each that can be translated into components of a trajectory.
[0233] Various semantic meanings can be assigned to target vehicles within the host vehicle's environment. For example, in some embodiments, the semantic meaning may include any of the following indications: 1) Unrelated: Indicates that the detected vehicle in the scene is currently unrelated; 2) Adjacent Lane: Detected. Indicates that the vehicle is in an adjacent lane and that an appropriate offset should be maintained for that vehicle (the exact offset can be computed in an optimization problem that constructs a trajectory given a desire and hard constraints). 3) Give way: The host vehicle may (among others 4) Proceed: if the host vehicle determines that the target vehicle is likely to cut into the host vehicle's lane; 5) Pursuit: The host vehicle wishes to follow this target vehicle and maintain smooth driving; 6) Left/Right Overtake: This is It means that the host vehicle wants to initiate a lane change to the left lane or the right lane. The left overtaking node 917 and the right overtaking node 915 are internal nodes that have not yet defined a desire.
[0234] The next node in the graph of choices 901 is a gap selection node 919. This node may be responsible for selecting the gap between two target vehicles in a particular target lane that the host vehicle wants to enter. By selecting a node of the form IDj, for some value of j, the host vehicle enters a leaf that specifies its desire regarding the trajectory optimization problem, e.g. the operation the host vehicle wants to perform in order to reach the selected gap. reach. Such maneuvering may include first accelerating/braking within the current lane and proceeding to the target lane at the appropriate time to enter the selected gap. If the gap selection node 919 cannot find a suitable gap, it returns to the center of the current lane and proceeds to the abort node 921 which defines the desire to cancel the overtaking.
[0235] Returning to the merge node 913, as the host vehicle approaches the merge, the host vehicle has several options that may depend on the particular situation. For example, as shown in FIG. 11A, host vehicle 1105 is moving along a two-lane road without detecting any other target vehicles within the main lane or merge lane 1111 of the two-lane road. In this situation, upon reaching the merge node 913, the driving policy module 803 may select a node 909 to stay at. That is, if a target vehicle is not detected as merging onto the road, it may be desirable to remain within one's current lane.
[0236] In FIG. 11B, the situation is slightly different. Here, the host vehicle 1105 detects one or more target vehicles 1107 entering the main road 1112 from the merging lane 1111. In this situation, when the driving policy module 803 encounters the merging node 913, the driving policy module 803 may decide to initiate a left-hand overtaking maneuver to avoid the merging situation.
[0237] In FIG. 11C, host vehicle 1105 encounters one or more target vehicles 1107 entering main road 1112 from merge lane 1111. Host vehicle 1105 also detects target vehicle 1109 moving in a lane adjacent to the host vehicle's lane. The host vehicle also detects one or more target vehicles 1110 traveling in the same lane as the host vehicle 1105. In this situation, driving policy module 803 may decide to adjust the speed of host vehicle 1105 to yield to target vehicle 1107 and proceed in front of target vehicle 1115. This may be accomplished, for example, by going to gap selection node 919, which selects the gap between ID0 (vehicle 1107) and ID1 (vehicle 1115) as the appropriate merge gap. In that case, the appropriate gap in the merging situation defines the objective of the trajectory planner optimization problem.
[0238] As discussed above, a node in a graph of choices can declare itself to be "critical," and such a declaration ensures that the chosen option passes through the critical node. obtain. Formally, each node can implement the function IsCritical. After making a forward pass from root to leaf on the graph of choices and solving the trajectory planner optimization problem, we can make a backward pass from leaf to root. Along this backward path we can call the IsCritical function of every node in the path and save a list of all critical nodes. The driving policy module 803 may be required to select a path from the root node to the leaf node that passes through all significant nodes in the forward path corresponding to the next time period.
[0239] Figures 11A-11C can be used to illustrate the potential benefits of this approach. For example, in a situation where an overtaking operation is initiated and the driving policy module 803 reaches the leaf corresponding to IDk, it is not desirable to select the stay node 909, for example, if the host vehicle is in the middle of an overtaking operation. To avoid this jumpiness, IDj nodes can designate themselves as critical. During the operation, the success of the trajectory planner can be monitored, and the function IsCritical returns a "true" value if the overtaking operation proceeds as intended. This approach may ensure that the overtaking operation continues within the next time frame (rather than jumping to another potentially inconsistent operation before completing the originally selected operation). On the other hand, the function IsCritical can return a "false" value if monitoring of the operation indicates that the selected operation is not proceeding as intended or if the operation is no longer necessary or possible. This may allow the gap selection node to select a different gap within the next time frame, or to abort the overtaking operation altogether. This technique, on the one hand, may allow the declaration of a desired path on the graph of choices at each time step, while on the other hand, it promotes stability of the policy during critical parts of the execution. I can support you.
[0240] Hard constraints, which are discussed in more detail below, can be distinguished from navigational desires. For example, strict constraints can ensure safe driving by applying an additional filtering layer of planned navigation operations. Rather than by using a trained system built on reinforcement learning, the hard constraints involved can be determined from the sensing state, which can be manually programmed and defined. However, in some embodiments, the trained system may learn applicable hard constraints to be applied and followed. Such techniques may facilitate driving policy module 803 to arrive at a selected behavior that is already compliant with the applicable hard constraints, so that it can be later modified to comply with the applicable hard constraints. Selected operations that may require .selected operations may be reduced or eliminated. Nevertheless, as a redundant safety measure, hard constraints may be applied to the output of the driving policy module 803 even if the driving policy module 803 is trained to consider predetermined hard constraints.
[0241] There are many examples of potential hard constraints. For example, hard constraints may be defined in relation to road edge guardrails. Under no circumstances will the host vehicle be allowed to cross the guardrail. Such rules impose strict lateral constraints on the trajectory of the host vehicle. Another example of a hard constraint may include a bump in the road (eg, a speed control bump), which may cause a hard constraint on driving speed before or while crossing the bump. Hard constraints can be considered to be safety-critical and can therefore be defined manually, rather than relying solely on a trained system learning the constraints during training.
[0242] In contrast to hard constraints, an aspirational goal may be to enable or achieve comfortable driving. As discussed above, one example of a desire may include a goal of positioning the host vehicle at a lateral position within the lane that corresponds to the center of the host vehicle's lane. Another desire may include the ID of the gap to enter. The host vehicle does not need to be strictly in the center of the lane, but instead the desire to be as close to the center of the lane as possible makes it easier for the host vehicle to move to the center of the lane even if it deviates from the center of the lane. Please note that we can guarantee that Aspirations do not have to prioritize safety. In some embodiments, the desire may require negotiation with other drivers and pedestrians. One approach to building desires may utilize a graph of choices, and the policies implemented within at least some nodes of the graph may be based on reinforcement learning.
[0243] For nodes in the graph of choices 901 or 1000 that are implemented as nodes that are trained based on learning, the training process may include decomposing the problem into a supervised learning phase and a reinforcement learning phase. In the supervised learning phase,<math num="10"><img file="JP7367255B2_D0010.tif" /></math>So that (s<sub>t</sub>,a<sub>t</sub>)from<math num="11"><img file="JP7367255B2_D0011.tif" /></math>A differentiable mapping to can be learned. This can be similar to "model-based" reinforcement learning. But in the forward loop of the network,<math num="12"><img file="JP7367255B2_D0012.tif" /></math>s<sub>t+1</sub>can be replaced by the actual value of , thereby eliminating the problem of error accumulation.
<math num="13"><img file="JP7367255B2_D0013.tif" /></math>The role of prediction is to convey messages from the future to past behavior. In this sense, the algorithm can be a combination of "model-based" reinforcement learning and "policy-based learning".
[0244] An important element that may be provided in some scenarios is a differentiable path from future losses/rewards back to decisions about actions. In the structure of a graph of choices, implementations of choices that include safety constraints are typically not differentiable. To overcome this problem, the selection of children at the nodes of the learned policy can be probabilistic. That is, a node can output a probability vector p, which assigns a probability to be used in selecting each of the children of a particular node. Suppose a node has k children and a<sup>(1)</sup>,...,a<sup>(k)</sup>Let be the motion of the path from each child to the leaf. Therefore, the resulting predicted behavior is<math num="14"><img file="JP7367255B2_D0014.tif" /></math>, which may yield a differentiable path from the action to p. In fact, the action a is a with respect to i~p<sup>(i)</sup>You can choose to be a and<math num="15"><img file="JP7367255B2_D0015.tif" /></math>The difference can be called additive noise.
[0245] s<sub>t</sub>,a<sub>t</sub>Given,<math num="16"><img file="JP7367255B2_D0016.tif" /></math>Supervised learning can be used with real data to train the . A simulator can be used to train the node's policies. Fine-tuning of the policy can later be achieved using actual data. Two concepts can make simulations more realistic. First, an initial policy can be constructed using a "behavior cloning" paradigm that uses imitation and uses large real-world datasets. In some cases, the resulting agent may be suitable. In other cases, the resulting agent forms at least a very good initial policy for other agents on the road. Second, self-play can be used to augment training using our proprietary policies. For example, a policy can be trained based on a simulator given an initial implementation of other agents (vehicles/pedestrians) that may be encountered. Some of the other agents can be replaced with new policies and the process can be repeated. As a result, the policy can continue to improve as it has to respond to a wider variety of other agents with different levels of sophistication.
[0246] Additionally, in some embodiments, the system may implement a multi-agent approach. For example, the system can consider data from various sources and/or images captured from multiple angles. Additionally, consideration may be given to predicting events that do not directly involve the host vehicle, but may have an impact on the host vehicle, and may even result in unpredictable situations involving other vehicles. (e.g., the radar can "see through" unavoidable events that affect the preceding and host vehicles, as well as a high likelihood of such events), some disclosed embodiments may can result in savings.
[0247] Trained Systems with Imposed Navigation Constraints [0248] In the context of autonomous driving, a critical concern is how to ensure that the learned policies of the trained navigation network are safe. It is. In some embodiments, constraints may be used to train the driving policy system so that the actions selected by the trained system may already take into account applicable safety constraints. In addition, some embodiments provide an additional layer of safety by subjecting selected operations of the trained system to one or more strict constraints implicated by a particular sensed scene within the host vehicle's environment. can do. Such an approach may ensure that operations performed by the host vehicle are limited to those determined to meet applicable safety constraints.
[0249] At its core, a navigation system may include a learning algorithm based on a policy function that maps observed conditions to one or more desired actions. In some implementations, the learning algorithm is a deep learning algorithm. The desired actions may include at least one action that is expected to maximize the expected reward for the vehicle. In some cases, the actual action taken by the vehicle may correspond to one of the desired actions, but in other cases, the actual action taken may be related to the observed state, one or more desired actions. and non-learning strict constraints (eg, safety constraints) imposed on the learning navigation engine. These constraints may include non-driving zones surrounding various types of detected objects (e.g., target vehicles, pedestrians, static objects on or in the road, moving objects on or in the road, guardrails, etc.) . In some cases, the size of the area may vary based on the detected motion (eg, speed and/or direction) of the detected object. Other constraints include maximum travel speed when passing within the pedestrian impact zone, maximum deceleration (to address target vehicle spacing behind the host vehicle), and enforcement at detected crosswalks or railroad crossings. This may include suspension, etc.
[0250] Hard constraints used with systems trained by machine learning can provide a degree of safety in autonomous driving that may exceed the degree of safety that can be obtained based on the output of the trained system alone. For example, a machine learning system can be trained using a desired set of constraints as a training guideline, such that the trained system is configured with the limits of the applicable navigation constraints and detects behavior that abides by such constraints. can be selected depending on the navigation state. However, the trained system still has some flexibility in choosing the navigational behavior, so there is at least a possibility that the behavior selected by the trained system does not strictly follow the relevant navigational constraints. There are several possible situations. Therefore, in order to require that the selected actions strictly follow the relevant navigation constraints, we use non-machine learning components that ensure strict enforcement of the relevant navigation constraints outside of the learning/training framework, and train You can combine, compare, filter, adjust, modify, etc. the outputs of already used systems.
[0251] The following discussion discusses trained systems and the potential benefits (particularly from a safety perspective) that can be gained from combining trained systems with algorithmic components that are outside of the training/learning framework. Showing further details regarding. As discussed earlier, reinforcement learning objectives with policies can be optimized by stochastic gradient ascent. The objective (e.g. expected reward) is<math num="17"><img file="JP7367255B2_D0017.tif" /></math>It can be defined as
[0252] In machine learning scenarios, objectives that include expected values can be used. However, such objectives, while not being bound by navigational constraints, may not return behavior that is strictly bound by those constraints. For example, for a trajectory that represents a rare "turn" event (e.g., an accident) that should be avoided,<math num="18"><img file="JP7367255B2_D0018.tif" /></math>holds, and for the remaining orbits,<math num="19"><img file="JP7367255B2_D0019.tif" /></math>Considering the reward function that holds, one goal of the learning system could be to learn to perform overtaking maneuvers. Normally, on an accident-free trajectory,<math num="20"><img file="JP7367255B2_D0020.tif" /></math>rewards smooth overtaking without any problems and penalizes staying in the lane and thus within the range [-1,1] without completing the overtaking. sequence<math num="21"><img file="JP7367255B2_D0021.tif" /></math>If represents an accident, reward-r should give a penalty high enough to prevent such occurrence. The question is what value of r should be to ensure accident-free driving.
[0253]
<math num="22"><img file="JP7367255B2_D0022.tif" /></math>The effect of the accident on is the additive term -p<sup>r</sup>Note that , and p is the probability mass of the trajectory with the accident event. If this term is negligible, i.e. p<<1/r, then the learning system performs overtaking operations more frequently than a policy that is more defensive, at the expense of some overtaking operations not completing successfully. To achieve success, one can prioritize an accident-making policy (or adopt a generally careless driving policy). In other words, if the probability of an accident is less than or equal to p, r must be set so that r>>1/p. Make p extremely small (for example, p=10<sup>-9</sup>degree) may be desirable. Therefore, r should be large. In the policy gradient,<math num="23"><img file="JP7367255B2_D0023.tif" /></math>The gradient of can be estimated. The following lemma shows that the random variable<math num="24"><img file="JP7367255B2_D0024.tif" /></math>variance exceeds r with r>>1/p<math num="25"><img file="JP7367255B2_D0025.tif" /></math>This shows that the size increases with the increase in size. Therefore, it may be difficult to estimate the objective, and even more difficult to estimate its slope.
[0254] Lemma: π<sub>o</sub>Let be a policy, p and r be scalars, so that with probability p,<math num="26"><img file="JP7367255B2_D0026.tif" /></math>is obtained, and with probability 1-p,<math num="27"><img file="JP7367255B2_D0027.tif" /></math>is obtained. Therefore, the following formula [0255]
<math num="28"><img file="JP7367255B2_D0028.tif" /></math> [0256] holds, and the last approximation applies when r1/p.
[0257] This explanation is based on the format<math num="29"><img file="JP7367255B2_D0029.tif" /></math>We show that the object may not be able to guarantee functional safety without causing distribution problems. Baseline subtraction methods to reduce variance may not provide sufficient treatment for the problem, because the problem is<math num="30"><img file="JP7367255B2_D0030.tif" /></math>This is because from a high variance of , the estimate shifts to an equally high variance of the baseline constant, which suffers equally from numerical instability. Furthermore, if the probability of an accident is p, then on average at least 1/p sequences should be sampled before obtaining an accident event. this is,<math num="31"><img file="JP7367255B2_D0031.tif" /></math>implies a lower bound of 1/p samples of the sequence for a learning algorithm that tries to minimize . A solution to this problem can be found in the architectural design described herein rather than through numerical tuning techniques. The approach here is based on the idea that strict constraints should be injected outside the learning framework. In other words, the policy function can be decomposed into a learnable part and an unlearnable part. Formally, the policy function is<math num="32"><img file="JP7367255B2_D0032.tif" /></math>can be configured as,<math num="33"><img file="JP7367255B2_D0033.tif" /></math>maps an (agnostic) state space to a set of desires (e.g., desired navigational goals, etc.) while π<sup>(T)</sup>maps desires into trajectories (which can determine how the car should move in the short term). function<math num="34"><img file="JP7367255B2_D0034.tif" /></math>factors such as driving comfort and which other cars to overtake, which other cars to yield to, and the desired position of the host vehicle within the host vehicle's lane. Responsible for making strategic decisions. The mapping from detected navigation states to desires is a policy that can be learned from experience by maximizing the expected reward.<math num="35"><img file="JP7367255B2_D0035.tif" /></math>It is.
<math num="36"><img file="JP7367255B2_D0036.tif" /></math>The desire created by can be transformed into a cost function over a driving trajectory. A non-function π is learned by finding a cost-minimizing trajectory subject to hard constraints on functional safety.<sup>(T)</sup>can be carried out. This disassembly can provide driving comfort and at the same time guarantee functional safety.
[0258] The double-merging navigation situation shown in FIG. 11D provides an example that further illustrates these concepts. In a double merge, vehicles arrive at the merge area 1130 from both the left and right sides. From each side, it can be determined whether a vehicle, such as vehicle 1133 or vehicle 1135, merges into a lane on the opposite side of merge area 1130. Successfully performing double merging in congested traffic can require significant negotiation skill and experience, and heuristics by enumerating all possible trajectories that all agents in the scene can take. It can be difficult to implement in a brute force manner or in a brute force manner. In this double merging example, a desired set D suitable for the double merging operation can be defined. D is the Cartesian product of the following pairs D= [0,v<sub>max</sub>]×Lx{g,t,o}, where [0,v<sub>max</sub>] is the desired target speed of the host vehicle, L={1,1.5,2,2.5,3,3.5,4} is the desired lateral position in lane units, and the integer is the center of the lane. , the fractions indicate the lane boundaries, and {g,t,o} are the classification labels assigned to each of the other n vehicles. If the host vehicle should give way to another vehicle, it can assign a 'g' to the other vehicle, and if the host vehicle should get the way to another vehicle, it can assign a 't' to the other vehicle. or if the host vehicle should maintain an offset distance from the other vehicle, the other vehicle can be assigned an 'o'.
[0259] Below is the set of desires (v,l,c<sub>1</sub>,...,c<sub>n</sub>)D can be transformed into a cost function over driving trajectories. The driving trajectory is (x<sub>1</sub>,y<sub>1</sub>),...,(x<sub>k</sub>,y<sub>k</sub>) and (x<sub>i</sub>,y<sub>i</sub>) is the (lateral, longitudinal) position of the host vehicle (in egocentric units) at time τ·i. In some experiments, τ=0.1 seconds and k=10. Of course, other values can also be selected. The costs assigned to the trajectory may include a weighted sum of the individual costs assigned to the desired speed, lateral position, and labels assigned to each of the other n vehicles.
[0260] Desired velocity v [0,v<sub>max</sub>], the cost of the trajectory relative to its velocity<math num="37"><img file="JP7367255B2_D0037.tif" /></math>It is.
[0261] Given the desired lateral position lL, the cost associated with the desired lateral position<math num="38"><img file="JP7367255B2_D0038.tif" /></math> [0262] where dist(x,y,l) is the distance from the point (x,y) to the lane position l. For any other vehicle, with respect to costs attributable to other vehicles, (x'<sub>1</sub>,y'<sub>1</sub>),...,(x'<sub>k</sub>,y'<sub>k</sub>) can represent other vehicles in egocentric units of the host vehicle, and i is (x<sub>i</sub>,y<sub>i</sub>) and (x'<sub>j</sub>,y'<sub>j</sub>) can be the earliest point that exists such that the distance between j is small. If there is no such point, i can be set as i=. If another vehicle is classified as "yielding", it may be desirable that τi > τj + 0.5, which means that the host vehicle will be able to give way at least 0.5 seconds after the other vehicle reaches the intersection of the trajectories. It means reaching the same point. A possible formula to convert the above constraints into costs is [τ(ji)+0.5]+.
[0263] Similarly, if another car is classified as "taking the road", it may be desirable that τj>τi+0.5, which translates to a cost [τ(ij)+0.5]+ can be done. If another car is classified as "offset", it may be desirable that i=, meaning that the host vehicle's trajectory and the offset vehicle's trajectory do not intersect. This condition can be translated into cost by imposing a penalty on the distance between trajectories.
[0264] Assigning weights to each of these costs may provide a single objective function π(T) for the trajectory planner. Costs can be added to promote smooth driving. Hard constraints can be added to the objective to ensure functional safety of the trajectory. For example, (x<sub>i</sub>,y<sub>i</sub>) may be prohibited from leaving the road and (x<sub>i</sub>,y<sub>i</sub>) is an arbitrary trajectory point (x'<sub>j</sub>,y'<sub>j</sub>) about (x'<sub>j</sub>,y'<sub>j</sub>) may be prohibited from approaching.
[0265] In summary, the policy πθ can be decomposed into a mapping from an agnostic state to a set of desires and a mapping from desires to actual trajectories. The latter mapping can be performed by solving an optimization problem that is not based on learning and whose cost depends on the desire and whose strict constraints can guarantee the functional safety of the policy.
[0266] The discussion below describes the mapping from agnostic states to desire sets. As explained above, in order to comply with functional safety, systems that rely solely on reinforcement learning must<math num="39"><img file="JP7367255B2_D0039.tif" /></math>may face large and intractable variances in terms of This result shows that by using policy gradient iterations, we can map the (agnostic) state space to a set of desires, followed by a mapping to an actual trajectory that does not involve the system being trained based on machine learning. This can be avoided by breaking down the problem into
[0267] Decisions can be further decomposed into semantically significant components for various reasons. For example, the size of D may be large and even continuous. In the double confluence scenario described above with respect to Figure 11D, D= [0,v<sub>max</sub>]×Lx{g,t,o}<sup>n</sup>) holds true. In addition, the gradient estimator has the term<math num="40"><img file="JP7367255B2_D0040.tif" /></math>may include. In this formula, the variance may increase with time period T. In some cases, the value of T may be approximately 250, which may be high enough to create significant variance. Assuming the sampling rate is in the range of 10 Hz and the merging area 1130 is 100 meters, preparation for merging may begin approximately 300 meters before the merging area. If the host vehicle moves at 16 meters/second (approximately 60 km/h), the value of T for an episode may be approximately 250.
[0268] Returning to the concept of a graph of options, a graph of options that may represent the double confluence scenario shown in FIG. 11D is shown in FIG. 11E. As discussed above, a graph of choices may represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). There may be a special node in the graph called a "root" node 1140, which is the only node that does not have an input edge (eg, a decision line). The decision process may start at the root node and traverse the graph until it reaches a "leaf" node, ie, a node that has no output edges. Each internal node can implement a policy function that selects one child from among its available children. There may be a default mapping from the set of traversals on the graph of choices to the set of desires D. In other words, crossing choices on the graph can be automatically converted into desires within D. Given a node v in the graph, the parameter vector θ<sub>v</sub>may specify a policy for selecting children of v. θ is all θ<sub>v</sub>, then at each node v θ<sub>v</sub>By traversing the graph from root to leaf, selecting child nodes, using the policy defined by<math num="41"><img file="JP7367255B2_D0041.tif" /></math>can be determined.
[0269] The double merging option graph 1139 of FIG. 11E shows whether the host vehicle is within a merging region (e.g., region 1130 of FIG. 11D), or alternatively, the host vehicle is approaching a merging region and is not in a possible merging region. The root node 1140 can first determine whether provision is necessary. In either case, the host vehicle may need to decide whether to change lanes (eg, to the left or right) or stay within the current lane. If the host vehicle decides to change lanes, the host vehicle must continue and determine (e.g., at the "go" node 1142) whether conditions are suitable for performing a lane change operation. could be. If it is not possible to change lanes, the host vehicle heads toward the desired lane (e.g., at node 1144 as part of a negotiation with a vehicle in the desired lane) by aiming to be on a lane mark. You can try to "push through" by Alternatively, the host vehicle may choose to "stay" within the same lane (eg, at node 1146). This process can determine the lateral position of the host vehicle in a natural way. for example.
[0270] This may allow the desired lateral position to be determined in a natural way. For example, if a host vehicle changes lanes from lane 2 to lane 3, the "go" node could set the desired lateral position to 3, and the "stay" node could set the desired lateral position to 2. and the "push forward" node can set the desired lateral position to 2.5. The host vehicle can then decide whether to keep the "same" speed (node 1148), "accelerate" (node 1150), or "slow down" (node 1152). The host vehicle can then enter a "chain" structure 1154 that looks at other vehicles and sets their semantic meaning to the values in the set {g,t,o}. This process may set desires for other vehicles. The parameters of all nodes in this chain may be shared (similar to a recurrent neural network).
[0271] A potential benefit of the option is the interpretability of the results. Another potential benefit is that the decomposable structure of set D can be exploited, thus allowing the policy at each node to be chosen among a small number of possibilities. Additionally, the structure may allow reducing the variance of the policy gradient estimator.
[0272] As discussed above, the episode length in a double confluence scenario may be approximately T=250 steps. This value (or any other suitable value depending on the particular navigation scenario) may allow sufficient time to acknowledge the consequences of the host vehicle's actions (e.g., the host vehicle may change lanes in preparation for merging). If you decide to do so, the host vehicle will only recognize the benefit after successfully completing the merge). On the other hand, driving dynamics require the host vehicle to make decisions at a sufficiently fast frequency (eg 10Hz in the above case).
[0273] The graph of choices may allow the effective value of T to be reduced in at least two ways. First, given a high-level decision, rewards can be established for low-level decisions while taking into account shorter episodes. For example, if the host vehicle has already selected the "change lane" and "go" nodes, a policy for assigning semantic meanings to the vehicle can be learned by watching 2-3 second episodes (T is 20~30 instead of 250). Second, for high-level decisions (such as whether to change lanes or stay within the same lane), the host vehicle may not need to make decisions every 0.1 seconds. Alternatively, the host vehicle may be able to make decisions less frequently (eg, every second), or may implement an "end of choice" function, and the slope may be calculated only after each end of a choice. In either case, the effective value of T may be an order of magnitude smaller than its original value. Overall, the estimator at every node can depend on the value of T an order of magnitude smaller than the original 250 steps, which can immediately change to a smaller variance.
[0274] As discussed above, strict constraints can promote safer driving, and there can be several different types of constraints. For example, static hard constraints can be defined directly from the sensing state. These may include deceleration bumps, speed limits, road curvatures, intersections, etc. in the host vehicle's environment, which may imply one or more constraints on the vehicle's speed, heading, acceleration, braking (deceleration), etc. Static hard constraints can also include semantic free space, where the host vehicle is, for example, prohibited from going outside of free space and from navigating too close to physical barriers. Ru. Static hard constraints can also restrict (e.g., prohibit) operations that do not follow various aspects of the vehicle's kinematic motion; for example, static hard constraints can prevent the host vehicle from rolling over. , can be used to prohibit maneuvers that could lead to slipping, or otherwise losing control.
[0275] Hard constraints may also be related to vehicles. For example, constraints may be used that require a vehicle to maintain a longitudinal distance of at least 1 meter to other vehicles and a lateral distance of at least 0.5 meter from other vehicles. Constraints may also be applied to avoid keeping the host vehicle on a collision course with one or more other vehicles. For example, time τ may be a measure of time based on a particular scene. The predicted trajectory of the host vehicle and one or more other vehicles from the current time to time τ may be considered. If the two trajectories intersect,<math num="42"><img file="JP7367255B2_D0042.tif" /></math>may represent the time at which vehicle i reaches and leaves the intersection. That is, each car reaches a point when the first part of the car passes the intersection, and a certain amount of time is required until the last part of the car passes the intersection. This time separates arrival time from leaving time.
<math num="43"><img file="JP7367255B2_D0043.tif" /></math>(i.e. the arrival time of vehicle 1 is less than the arrival time of vehicle 2), it is desired to ensure that vehicle 1 leaves the intersection before vehicle 2 arrives. Otherwise a collision will occur. Therefore,<math num="44"><img file="JP7367255B2_D0044.tif" /></math>We can implement strict constraints such that . Additionally, obtain an additional safety margin by including a buffer time (e.g., 0.5 seconds or another suitable value) in the constraint to ensure that vehicle 1 and vehicle 2 do not hit each other by a minimum amount. I can do it. The hard constraints related to the predicted intersecting trajectories of the two vehicles are:<math num="45"><img file="JP7367255B2_D0045.tif" /></math>It can be expressed as
[0276] The time τ for tracking the trajectory of the host vehicle and one or more other vehicles may be different. However, in intersection scenarios where speeds may be low, τ can be longer and τ can be defined such that the host vehicle enters and exits the intersection in less than τ seconds.
[0277] Of course, applying hard constraints to vehicle trajectories requires that those vehicle trajectories be predicted. For the host vehicle, predicting the trajectory can be relatively simple because the host vehicle generally already knows and actually plans the intended trajectory at any given time. For other vehicles, predicting their trajectories may not be so easy. In other vehicles, the baseline calculations for determining the predicted trajectory are captured, for example, by one or more cameras and/or other sensors (radar, lidar, acoustic, etc.) on board the host vehicle. It may depend on the current speed and heading of other vehicles determined based on analysis of the image stream.
[0278] However, there may be some exceptions that simplify the problem or at least provide more confidence in the predicted trajectory for another vehicle. For example, with respect to structured roads where there may be lane instructions and yield rules, the trajectory of other vehicles may be based at least in part on the position of other vehicles relative to the lane and the applicable road. It can be based on the yield rule. Therefore, in some situations, given the observed lane structure, vehicles in adjacent lanes may be expected to adhere to the lane boundaries. That is, the host vehicle will not be allowed to drive into the adjacent lane unless there is observed evidence that the vehicle in the adjacent lane will cut into the host vehicle's lane (e.g., traffic lights, strong lateral movement, movement across lane boundaries). It can be assumed that vehicles in lanes of 1 and 2 remain within their own lanes.
[0279] Other circumstances may also give hints about the expected trajectory of other vehicles. For example, at stop signs, traffic lights, roundabouts, etc., where the host vehicle may have the right of way, it can be assumed that other vehicles will protect that right of way. Therefore, unless there is evidence that a rule has been broken, it can be assumed that other vehicles will proceed along a trajectory that respects the right of way that the host vehicle has.
[0280] Hard constraints may also be applied with respect to pedestrians within the host vehicle's environment. For example, a buffer distance for pedestrians may be established such that the host vehicle is prohibited from navigating any closer than a specified buffer distance for any observed pedestrian. The pedestrian buffer distance may be any suitable distance. In some embodiments, the buffer distance may be at least 1 meter for observed pedestrians.
[0281] Similar to the vehicle situation, strict constraints can also be applied to the relative motion between the pedestrian and the host vehicle. For example, the trajectory of a pedestrian (based on heading and speed) can be monitored against the predicted trajectory of the host vehicle. Given a particular pedestrian's trajectory, for every point p on the trajectory, t(p) may represent the time it takes the pedestrian to reach point p. In order to maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) (with a sufficient time difference such that the host vehicle passes in front of the pedestrian with a distance difference of at least 1 meter) must be greater than the time for the host vehicle to reach point p, or (e.g., if the host vehicle brakes and yields to a pedestrian) t(p) must be the time for the host vehicle to reach point p. must be below. Furthermore, in the latter example, the host vehicle should arrive at point p sufficiently later than the pedestrian so that the host vehicle can pass behind the pedestrian and maintain the required buffer distance of at least 1 meter. may be required by strict constraints. Of course, there may be exceptions to the strict pedestrian restriction. For example, if the host vehicle has the right of way or is very slow and there is no observed basis for the pedestrian refusing to yield to or navigating towards the host vehicle. , the strict constraints on pedestrians can be relaxed (eg to a narrower buffer of at least 0.75 meters or 0.50 meters).
[0282] In some examples, constraints may be relaxed if it is determined that not all can be met. For example, in situations where the road is too narrow to allow the desired distance (eg, 0.5 meters) from both curves or from a curve and a parked vehicle, one or more of the constraints may be relaxed if there are mitigating reasons. For example, if there are no pedestrians (or other objects) on the sidewalk, you can proceed slowly 0.1 meters from the curve. In some embodiments, constraints may be relaxed if doing so improves the user experience. For example, to avoid potholes, constraints may be relaxed to allow a vehicle to navigate closer to lane edges, curves, or pedestrians than would normally be allowed. Additionally, when determining which constraints to relax, in some embodiments, the one or more constraints that are decided to relax are those that are deemed to have the least adverse effect on safety. . For example, constraints on how close a vehicle can move to a curve or concrete barrier may be relaxed before constraints dealing with proximity to other vehicles are relaxed. In some embodiments, pedestrian constraints may be the last to be relaxed, or may never be relaxed in some circumstances.
[0283] FIG. 12 shows an example of a scene that may be captured and analyzed during host vehicle navigation. For example, the host vehicle may receive a plurality of images representative of the host vehicle's environment from a camera associated with the host vehicle (e.g., at least one of image capture device 122, image capture device 124, and image capture device 126). A navigation system (eg, system 100) may be included. The scene shown in FIG. 12 is one example of an image that may be captured at time t from the environment of a host vehicle moving in lane 1210 along predicted trajectory 1212. The navigation system includes at least one processing device (e.g., (including either an EyeQ processor or other devices). In particular, at least one processing device may implement a sensing module 801, an operational policy module 803, and a control module 805 shown in FIG. The sensing module 801 may be responsible for collecting and outputting image information collected from cameras and providing that information in the form of identified navigation states to a driving policy module 803, which may perform supervised learning. A trained navigation system can be constructed that is trained by machine learning methods such as or reinforcement learning. Based on the navigation state information provided by the sensing module 801 to the driving policy module 803, the driving policy module 803 determines whether the host vehicle Desired navigation operations can be generated to perform.
[0284] In some embodiments, at least one processing device can directly convert desired navigational movements into navigational commands, using, for example, control module 805. However, other embodiments apply strict constraints to test the desired navigation behavior provided by the driving policy module 803 against various predetermined navigation constraints that may be involved by the scene and the desired navigation behavior. be able to. For example, if driving policy module 803 outputs a desired navigation behavior that causes the host vehicle to follow trajectory 1212, that navigation behavior is subject to one or more hard constraints related to various aspects of the host vehicle's environment. Can be tested. For example, the captured image 1201 may determine curves 1213, pedestrians 1215, target vehicles 1217, and static objects (eg, an overturned box) that are present in the scene. Each of these may be associated with one or more hard constraints. For example, curve 1213 may be associated with a static constraint that prevents the host vehicle from navigating into or beyond the curve onto sidewalk 1214. Curve 1213 includes a predetermined distance (e.g., a buffer zone) extending along the curve and separated from the curve (e.g., 0.1 meters, 0.25 meters, 0.5 meters, 1 meter, etc.) that defines a non-navigable area for the host vehicle. It may also be related to the road barrier envelope that defines the road barrier envelope. Of course, static constraints may be associated with other types of roadside boundaries (e.g. guardrails, concrete columns, cones, pylons, any other type of roadside barrier).
[0285] It should be noted that distance and ranging can be determined by any suitable method. For example, in some embodiments distance information may be provided by an onboard radar and/or lidar system. Alternatively or additionally, distance information may be derived by analyzing one or more images captured from the host vehicle's environment. For example, the number of pixels of the recognized object represented in the image can be determined and compared to the known field of view and focal length geometry of the image capture device to determine scale and distance. For example, velocity and acceleration can be determined by observing the change in scale between objects from image to image over a known time interval. This analysis may indicate how fast the object is moving away from or approaching the host vehicle, as well as the direction of movement toward or away from the host vehicle. By analyzing the change in position of the object's X coordinate from one image to another over a known time period, the traversal velocity can be determined.
[0286] Pedestrians 1215 may be associated with a pedestrian envelope that defines a buffer zone 1216. In some cases, the imposed strict constraints may prohibit the host vehicle from navigating within a distance of 1 meter (in any direction relative to the pedestrian) from the pedestrian 1215. Pedestrians 1215 may also define the location of pedestrian impact zones 1220. This zone of influence may be associated with constraints that limit the speed of the host vehicle within the zone of influence. The affected zone can extend from pedestrian 1215 to 5 meters, 10 meters, 20 meters, etc. Different speed limits can be associated with each class of impact zone. For example, within an area of 1 meter to 5 meters from a pedestrian 1215, the host vehicle may drive at a first speed (e.g., 10 mph, 20 mph, etc.) that may be less than the speed limit within the pedestrian impact zone extending from 5 meters to 10 meters. ) may be limited to. Any grading for the various stages of the impact zone can be used. In some embodiments, the first stage can be narrower than 1 meter to 5 meters, and may only span 1 meter to 2 meters. In other embodiments, the first stage of the impact zone may extend from 1 meter (boundary of the non-navigated zone around the pedestrian) to a distance of at least 10 meters. The second stage may then extend from 10 meters to at least about 20 meters. The second stage may be associated with a maximum travel speed of the host vehicle that exceeds the maximum travel speed associated with the first stage of the pedestrian impact zone.
[0287] One or more static object constraints may also be implicated by the detected scene within the host vehicle's environment. For example, in image 1201, at least one processing device may detect a static object, such as a box 1219, in a road. The detected static objects may include various objects such as at least one of a tree, a pole, a road sign, an object in the road, etc. One or more predefined navigation constraints may be associated with the detected static object. For example, such a constraint may include a static object envelope that defines a buffer area around an object within which navigation of the host vehicle may be prohibited. At least a portion of the buffer area may extend a predetermined distance from an edge of the detected static object. For example, in the scene represented by image 1201, a buffer zone of at least 0.1 meters, 0.25 meters, 0.5 meters or more may be associated with box 1219 so that the host vehicle is protected from collisions with detected static objects. Pass to the right or left of the box by at least some distance (eg, the distance of the buffer zone) to avoid it.
[0288] The predefined hard constraints may also include one or more target vehicle constraints. For example, target vehicle 1217 may be detected within image 1201. One or more hard constraints may be used to ensure that the host vehicle does not collide with target vehicle 1217. In some cases, a target vehicle envelope may be associated with a single buffer zone distance. For example, a buffer zone may be defined by a distance of 1 meter surrounding the target vehicle in all directions. The buffer zone may define an area extending at least one meter from the target vehicle into which the host vehicle is prohibited to navigate.
[0289] However, the envelope surrounding target vehicle 1217 need not be defined by a fixed buffer distance. In some cases, the predetermined strict constraints associated with the target vehicle (or any other movable object detected within the host vehicle's environment) may depend on the host vehicle's orientation relative to the detected target vehicle. For example, in some cases the distance of the longitudinal buffer zone (e.g. extending from the target vehicle to the front or rear of the host vehicle, such as when the host vehicle is driving toward the target vehicle) is at least 1 meter. It can be. (For example, if the host vehicle is moving in the same or opposite direction as the target vehicle, such that the side of the host vehicle passes directly beside the side of the target vehicle, etc.) The distance of the lateral buffer zone (extending to the sides of) may be at least 0.5 meter.
[0290] As explained above, other constraints may also be involved by detecting target vehicles or pedestrians within the host vehicle's environment. For example, one can consider the predicted trajectories of a host vehicle and a target vehicle 1217, and the hard constraint is that when the two trajectories intersect (e.g., at intersection point 1230),<math num="46"><img file="JP7367255B2_D0046.tif" /></math>The host vehicle is Vehicle 1 and the target vehicle 1217 is Vehicle 2. Similarly, the trajectory of the pedestrian 1215 (based on heading and speed) can be monitored relative to the predicted trajectory of the host vehicle. Given a particular pedestrian's trajectory, for every point p on the trajectory, t(p) represents the time it takes for the pedestrian to reach point p (ie, point 1231 in Figure 12). In order to maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) (with a sufficient time difference such that the host vehicle passes in front of the pedestrian with a distance difference of at least 1 meter) must be greater than the time for the host vehicle to reach point p, or (e.g., if the host vehicle brakes and yields to a pedestrian) t(p) must be the time for the host vehicle to reach point p. must be below. Furthermore, in the latter example, the host vehicle should arrive at point p sufficiently later than the pedestrian so that the host vehicle can pass behind the pedestrian and maintain the required buffer distance of at least 1 meter. strict constraints require.
[0291] Other hard constraints can also be used. For example, in at least some cases the maximum deceleration rate of the host vehicle may be used. This maximum deceleration rate may be determined based on the detected distance to a target vehicle following the host vehicle (eg, using images collected from a rear-facing camera). Hard constraints may include mandatory stops at detected crosswalks or railroad crossings or other applicable constraints.
[0292] If analysis of the scene within the host vehicle's environment indicates that one or more predefined navigation constraints may be involved, those constraints may be applied to one or more planned navigation movements of the host vehicle. can be imposed. For example, if the analysis of the scene results in driving policy module 803 returning a desired navigation behavior, that desired navigation behavior may be tested against one or more participating constraints. If the desired navigation behavior is determined to violate any aspect of the involved constraints (e.g., when the predefined hard constraints require that the host vehicle remain at least 1.0 meters from the pedestrian 1215), the desired navigation behavior carries the host vehicle within 0.7 meters of the pedestrian 1215), at least one modification to the desired navigation behavior may be made based on one or more predefined navigation constraints. Adjusting the desired navigation behavior in this manner may result in the host vehicle's actual navigation behavior complying with constraints imposed by the particular scene detected within the host vehicle's environment.
[0293] After determining the actual navigation behavior of the host vehicle, performing the navigation behavior by causing an adjustment of at least one navigation actuator of the host vehicle in response to the determined actual navigation behavior of the host vehicle. I can do it. The navigation actuator may include at least one of the host vehicle's steering mechanism, brakes, or accelerator.
[0294] Prioritized Constraints [0295] As discussed above, various strict constraints may be used with the navigation system to ensure safe operation of the host vehicle. The constraints may include, among other things, a minimum safe driving distance with respect to pedestrians, target vehicles, road barriers or detected objects, maximum travel speed or maximum deceleration rate of the host vehicle when passing within the zone of influence of the detected pedestrian. . These constraints can be imposed by a trained system that is trained based on machine learning (supervised, reinforcement or a combination thereof) (e.g. directly on the expected situations occurring within the scene of the host vehicle's environment). Even untrained systems (using coping algorithms) can be useful.
[0296] In any case, there can be a hierarchy of constraints. In other words, some navigation constraints may take precedence over other navigation constraints. Therefore, if a situation arises where no navigation action is available that would satisfy all the constraints involved, the navigation system can determine the available navigation action that first fulfills the highest priority constraints. . For example, the system can cause a vehicle to avoid a pedestrian first, even if navigation to avoid the pedestrian would result in a collision with another vehicle or object detected in the roadway. In another example, the system may cause the vehicle to drive around a curve to avoid a pedestrian.
[0297] FIG. 13 shows a flowchart illustrating an algorithm for implementing a hierarchy of participating constraints determined based on an analysis of a scene within a host vehicle's environment. For example, in step 1301, at least one processor associated with a navigation system (eg, an EyeQ processor, etc.) may receive a plurality of images representative of the host vehicle's environment from an onboard camera of the host vehicle. At step 1303, navigation conditions associated with the host vehicle may be determined by analyzing one or more images representing scenes of the host vehicle's environment. For example, the navigation state may include, among other characteristics of the scene, the host vehicle is moving along a two-lane road 1210, as shown in Figure 12, and the target vehicle 1217 is proceeding through an intersection in front of the host vehicle. It may specifically indicate that a pedestrian 1215 is waiting to cross the road on which the host vehicle is traveling, and that there is an object 1219 in front of the host vehicle's lane.
[0298] At step 1305, one or more navigation constraints contributed by the host vehicle's navigation state may be determined. For example, the at least one processing device analyzes a scene in the host vehicle's environment represented by one or more captured images, and then includes objects, vehicles, pedestrians, etc. that are recognized by the image analysis of the captured images. One or more navigation constraints can be determined. In some embodiments, the at least one processing device can determine at least a first default navigation constraint and a second default navigation constraint that are implicated by the navigation state, the first default navigation constraint may be different from the second default navigation constraint. For example, a first navigation constraint may relate to one or more target vehicles detected within the host vehicle's environment, and a second navigation constraint may relate to a pedestrian detected within the host vehicle's environment. can be related to.
[0299] At step 1307, the at least one processing device may determine priorities associated with the constraints identified at step 1305. In the illustrated example, the second predetermined navigation constraint related to pedestrians may have a higher priority than the first predetermined navigation constraint related to the target vehicle. Although the priority associated with a navigation constraint may be determined or assigned based on various factors, in some embodiments the priority of a navigation constraint is determined or assigned based on its relative priority from a safety perspective. can be related to importance. For example, while it may be important to follow or satisfy all implemented navigational constraints in as many situations as possible, some constraints may be associated with stronger safety risks than others. , and therefore can be assigned higher priority. For example, a navigation constraint that requires the host vehicle to maintain a distance of at least 1 meter from pedestrians may have higher priority than a constraint that requires the host vehicle to maintain a distance of at least 1 meter from the target vehicle. . This may be because a collision with a pedestrian may have more serious consequences than a collision with another vehicle. Similarly, maintaining separation between the host vehicle and the target vehicle may prevent the host vehicle from avoiding boxes in the road, driving below a certain speed over deceleration bumps, or maximizing the host vehicle's occupants. It may have higher priority than constraints requiring exposure below an acceleration level.
[0300] The driving policy module 803 is designed to maximize safety by satisfying the navigational constraints implicated by a particular scene or navigational situation, but does not physically satisfy all the constraints implicated by the situation. It can be impossible. In such a situation, the priority of each participating constraint may be used to determine which of the participating constraints should be satisfied first, as shown in step 1309. Continuing with the above example, in a situation where it is not possible to satisfy both the pedestrian clearance constraint and the target vehicle clearance constraint, but only one of the constraints can be satisfied, the pedestrian clearance constraint A higher priority may result in the constraints being met before attempting to maintain clearance to the target vehicle. Accordingly, under normal circumstances, if both the first predefined navigation constraint and the second predefined navigation constraint can be satisfied, as shown in step 1311, at least one processing device A first navigation operation for the host vehicle that satisfies both the first predetermined navigation constraint and the second predetermined navigation constraint may be determined based on the determined navigation state. However, in other situations where it is not possible to satisfy all the constraints involved, as shown in step 1313, if both the first default navigation constraint and the second default navigation constraint cannot be satisfied, at least One processing device satisfies the second default navigation constraint (i.e. the higher priority constraint) based on the identified navigation state, but satisfies the first default navigation constraint (i.e. the higher priority constraint) A second navigation operation may be determined for the host vehicle that does not satisfy the host vehicle (also having a lower priority).
[0301] Next, in step 1315, to implement the determined navigation operation for the host vehicle, the at least one processing device performs the determined first navigation operation for the host vehicle or the determined navigation operation for the host vehicle. An adjustment of at least one navigation actuator of the host vehicle can be caused in response to the second navigation operation. Similar to the previous example, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.
[0302] Constraint relaxation [0303] As discussed above, navigation constraints can be imposed for safety. The constraints may include, among other things, a minimum safe driving distance with respect to pedestrians, target vehicles, road barriers or detected objects, maximum travel speed or maximum deceleration rate of the host vehicle when passing within the zone of influence of the detected pedestrian. . These constraints can be imposed within a learning or non-learning navigation system. In certain circumstances these constraints can be relaxed. For example, if the host vehicle slows down or stops near a pedestrian and moves slowly to signal an intention to pass by the pedestrian, the pedestrian's reaction can be detected from the captured images. If the pedestrian's response is to stay still or stop moving (and/or if eye contact with the pedestrian is detected), the navigation system's intention to pass by the pedestrian is determined by the pedestrian. can be understood to have been recognized. In such situations, the system relaxes one or more of the default constraints and enforces less stringent constraints (e.g. within 0.5 meters of the pedestrian rather than within the more strict 1 meter boundary). (enables the vehicle to navigate).
[0304] FIG. 14 shows a flowchart for implementing control of a host vehicle based on relaxing one or more navigation constraints. At step 1401, at least one processing device may receive a plurality of images representative of the host vehicle's environment from a camera associated with the host vehicle. Analyzing the images at step 1403 may enable identifying navigation conditions associated with the host vehicle. At step 1405, the at least one processor may determine navigation constraints associated with a navigation state of the host vehicle. The navigation constraints may include a first default navigation constraint that is committed by at least one aspect of the navigation state. At step 1407, analyzing the plurality of images may determine the presence of at least one navigation constraint mitigating factor.
[0305] A navigational constraint relaxation factor may include any suitable indicator that one or more navigational constraints may be suspended, changed, or otherwise relaxed in at least one aspect. In some embodiments, at least one navigation constraint mitigation factor may include a determination (based on image analysis) that the pedestrian's eyes are looking in the direction of the host vehicle. In that case, it can be more safely assumed that the pedestrian is aware of the host vehicle. As a result, there may be greater confidence that the pedestrian will not be involved in unexpected movements that would cause the pedestrian to move into the path of the host vehicle. Other constraint relaxation factors can also be used. For example, the at least one navigation constraint mitigating factor may be a pedestrian who is determined to be not moving (e.g., who is estimated to be unlikely to enter the path of the host vehicle) or whose movement is determined to be slowing down. may include pedestrians. Navigation constraint mitigation factors can also include more complex movements, such as a pedestrian who is determined not to be moving when the host vehicle stops and then resumes movement. In such situations, it can be assumed that the pedestrian understands that the host vehicle has the right of way, and the pedestrian who stops can indicate his or her intention to yield to the host vehicle. . Other circumstances that may cause one or more constraints to be relaxed include the type of curb (e.g., low curbs or gentle slopes may allow for a relaxation of the distance constraint), pedestrians or other objects on the sidewalk. A vehicle without its engine running may have a relaxed distance, or a situation in which the pedestrian is deviated from the area into which the host vehicle is traveling and/or the pedestrian is distanced. Including the situation.
[0306] Upon identifying the presence of a navigation constraint mitigating factor (eg, at step 1407), a second navigation constraint may be determined or developed in response to detecting the constraint mitigating factor. This second navigation constraint may be different from the first navigation constraint and may include at least one relaxed characteristic compared to the first navigation constraint. The second navigation constraint may include a newly generated constraint based on the first constraint, the newly generated constraint including at least one modification that relaxes the first constraint in at least one respect. include. Alternatively, the second constraint may constitute a predetermined constraint that is less stringent in at least one respect than the first navigation constraint. In some embodiments, this second constraint may be reserved for use only in situations where constraint mitigating factors are identified within the host vehicle's environment. Whether the second constraint is newly generated or selected from a set of predetermined constraints that are fully or partially available, the more stringent The application of the second navigation constraint instead of the first navigation constraint can be referred to as constraint relaxation and can be realized in step 1409.
[0307] If at least one constraint relaxing factor is detected in step 1407 and the at least one constraint is relaxed in step 1409, a navigation behavior for the host vehicle may be determined in step 1411. Navigation operations for the host vehicle can be based on the identified navigation condition and can satisfy the second navigation constraint. At step 1413, the navigation operation may be performed by causing an adjustment of at least one navigation actuator of the host vehicle in response to the determined navigation operation.
[0308] As discussed above, the use of navigation constraints and relaxed navigation constraints can be applied to trained navigation systems (e.g., by machine learning) or untrained navigation systems (e.g., depending on a particular navigation state). systems that are programmed to respond with predetermined actions). When using a trained navigation system, the availability of relaxed navigation constraints for a particular navigation situation may represent a mode switch from a trained system response to an untrained system response. For example, the trained navigation network may determine an original navigation behavior for the host vehicle based on the first navigation constraint. However, the actions performed by the vehicle may be different from the navigation actions that satisfy the first navigation constraint. Rather, the actions taken may satisfy a second, more relaxed navigation constraint (e.g., in response to detecting certain conditions in the host vehicle's environment, such as the presence of a navigation constraint relaxing factor). ) can be a behavior developed by an untrained system.
[0309] There are many examples of navigation constraints that may be relaxed in response to detecting constraint relaxing factors within the host vehicle's environment. For example, if the default navigation constraints include a buffer area associated with a detected pedestrian, and at least a portion of the buffer area extends a predetermined distance from the detected pedestrian, then A relaxed navigation constraint (recalled from memory from a set or generated as a relaxed version of an existing constraint) may include a different or modified buffer area. For example, the different or modified buffer area may have a shorter distance to the detected pedestrian than the original or unmodified buffer area. As a result, if an appropriate constraint relaxation factor is detected in the host vehicle's environment, the host vehicle may be allowed to take into account the relaxed constraints and navigate closer to the detected pedestrian.
[0310] Relaxed characteristics of the navigational constraints may include a reduced width of a buffer area associated with at least one pedestrian as described above. However, the characteristics that are mitigated may also include a reduced width of a buffer area associated with the target vehicle, a detected object, a roadside barrier, or any other object detected within the host vehicle's environment.
[0311] The at least one relaxed property may also include other types of modifications in navigation constraint properties. For example, the relaxed characteristics may include a speed increase associated with at least one predefined navigation constraint. The relaxed characteristics may also include an increase in the maximum allowable deceleration/acceleration associated with at least one predefined navigation constraint.
[0312] As noted above, constraints may be relaxed in certain situations, while navigation constraints may be augmented in other situations. For example, in some situations, the navigation system may determine that conditions warrant augmenting the normal set of navigation constraints. Such augmentation may include adding new constraints to the set of predefined constraints or adjusting one or more aspects of the predefined constraints. This addition or adjustment may result in more careful navigation to the predefined set of constraints applicable under normal operating conditions. Conditions that may justify increased constraints may include sensor failure, adverse environmental conditions (rain, snow, fog, or other conditions associated with reduced visibility or reduced vehicle traction), and the like.
[0313] FIG. 15 shows a flowchart for implementing control of a host vehicle based on augmenting one or more navigation constraints. At step 1501, at least one processing device may receive a plurality of images representative of the host vehicle's environment from a camera associated with the host vehicle. Analyzing the images at step 1503 may enable identifying navigation conditions associated with the host vehicle. At step 1505, the at least one processor may determine navigation constraints associated with a navigation state of the host vehicle. The navigation constraints may include a first default navigation constraint that is involved by at least one aspect of the navigation state. At step 1507, analyzing the plurality of images may determine the presence of at least one navigation constraint augmentation factor.
[0314] The navigational constraints involved may include any of the navigational constraints discussed above (eg, with respect to FIG. 12) or any other suitable navigational constraints. A navigation constraint augmentation factor may include any indicator that one or more navigation constraints may be supplemented/augmented in at least one aspect. Supplementation or augmentation of navigation constraints can be done on a set-by-set basis (e.g., by adding new navigation constraints to a given set of constraints) or on a constraint-by-constraint basis (e.g., by adding new navigation constraints to a given set of constraints). modifying a particular constraint such that it is more restrictive than the original, or adding a new constraint that corresponds to a given constraint, the new constraint being more restrictive than the corresponding constraint in at least one aspect restrictive). Additionally or alternatively, supplementing or augmenting navigation constraints may refer to selecting among a predetermined set of constraints based on a hierarchy. For example, an augmented set of constraints may be provided for selection based on whether navigation augmentation factors are detected within or for the host vehicle's environment. Under normal conditions, where no augmentation factors are detected, the navigation constraints involved can be derived from constraints applicable under normal conditions. On the other hand, if one or more constraint augmentation factors are detected, the constraints involved can be derived from augmented constraints generated or predetermined for the one or more augmentation factors. . The augmented constraint may be more restrictive in at least one aspect than a corresponding constraint that would be applicable under normal conditions.
[0315] In some embodiments, the at least one navigation constraint augmentation factor includes detecting (e.g., based on image analysis) that there is ice, snow, or water on a road surface in the host vehicle's environment. may be included. This determination can be made, for example, by areas with higher reflectance than expected on a dry road (e.g. indicating ice or water on the road), white areas on the road surface indicating the presence of snow, or longitudinal areas on the road. Shadows on the road consistent with the presence of grooves (e.g. tire tracks in the snow), water droplets or small particles of ice/snow on the windshield of the host vehicle, or other signs of water or ice/snow on the road surface. It may be based on detecting any suitable indicator.
[0316] The at least one navigation constraint augmentation factor may also include detecting small particles on the outer surface of the host vehicle's windshield. Such small particles can impair the image quality of one or more image capture devices associated with the host vehicle. Although described with respect to the host vehicle's windshield in relation to a camera mounted on the back of the host vehicle's windshield, other surfaces associated with the host vehicle (e.g., camera lens or lens cover, headlight lens, rear window Detecting small particles on the tail light lens or any other surface of the host vehicle that is visible to the image capture device (or detected by a sensor) may also indicate that there is a navigation constraint augmentation factor.
[0317] Navigation constraint augmentation factors may also be detected as characteristics of one or more image acquisition devices. For example, a detected decrease in the image quality of one or more images captured by an image capture device (eg, a camera) associated with the host vehicle may also constitute a navigation constraint augmentation factor. The reduction in image quality may be associated with a hardware failure or partial hardware failure associated with the image capture device or an assembly associated with the image capture device. Such image quality degradation may also be caused by environmental conditions. For example, the presence of smoke, fog, rain, snow, etc. in the air surrounding the host vehicle may contribute to reduced image quality regarding possible roads, pedestrians, target vehicles, etc. within the host vehicle's environment.
[0318] Navigation constraint augmentation factors may also relate to other aspects of the host vehicle. For example, in some situations, the navigation constraint augmentation factor may include a detected failure or partial failure of a system or sensor associated with the host vehicle. Such augmentation factors may include, for example, speed sensors, GPS receivers, accelerometers, cameras associated with the host vehicle that may affect the host vehicle's ability to navigate in relation to navigation constraints associated with the host vehicle's navigation state. , detecting failures or partial failures of radar, lidar, brakes, tires or any other system.
[0319] If it is identified (eg, at step 1507) that a navigation constraint augmentation factor exists, a second navigation constraint may be determined or developed in response to detecting the constraint augmentation factor. This second navigation constraint can be different from the first navigation constraint and can include at least one characteristic that is augmented with respect to the first navigation constraint. Detecting a constraint-enhancing factor within the host vehicle's environment or associated with the host vehicle indicates that at least one navigational capability of the host vehicle may be reduced compared to normal operating conditions. The second navigation constraint may be more restrictive than the first navigation constraint in order to obtain. Such reduced performance may be caused by reduced road traction (e.g. ice, snow or water on the road, reduced tire pressure, etc.), impaired visibility (e.g. rain, snow, dust reducing captured image quality, etc.). , smoke, fog, etc.), impaired detection capabilities (e.g., sensor failure or partial failure, degraded sensor performance, etc.), or the host vehicle's ability to navigate in response to the detected navigation conditions. may include any decrease in
[0320] If at least one constraint augmentation factor is detected in step 1507 and at least one constraint is augmented in step 1509, a navigation behavior for the host vehicle may be determined in step 1511. Navigation operations for the host vehicle can be based on the identified navigation state and can satisfy a second navigation (ie, augmented) constraint. At step 1513, the navigation operation may be performed by causing an adjustment of at least one navigation actuator of the host vehicle in response to the determined navigation operation.
[0321] As discussed above, the use of navigation constraints and augmented navigation constraints can be applied to trained navigation systems (e.g., by machine learning) or to untrained navigation systems (e.g., depending on a particular navigation state). (a system programmed to respond with an action). When using a trained navigation system, the availability of augmented navigation constraints for a particular navigation situation may represent a mode switch from a trained system response to an untrained system response. For example, the trained navigation network may determine an original navigation behavior for the host vehicle based on the first navigation constraint. However, the actions performed by the vehicle may be different from the navigation actions that satisfy the first navigation constraint. Rather, the actions taken may satisfy the augmented second navigation constraint (e.g., in response to detecting a particular condition in the host vehicle's environment, such as the presence of a navigation constraint augmentation factor). It can be a behavior developed by an untrained system.
[0322] There are many examples of navigation constraints that may be generated, supplemented, or augmented in response to detecting constraint augmentation factors within the host vehicle's environment. For example, if the default navigation constraints include a buffer zone associated with a detected pedestrian, object, vehicle, etc., and at least a portion of the buffer zone extends a predetermined distance from the detected pedestrian/object/vehicle; Augmented navigation constraints (newly developed, recalled from memory from a predetermined set, or generated as an augmented version of existing constraints) may include different or modified buffer areas. For example, the different or modified buffer area may have a longer distance for the detected pedestrian/object/vehicle than the original or unmodified buffer area for the detected pedestrian/object/vehicle. As a result, if an appropriate constraint augmentation factor is detected in the host vehicle's environment or in relation to the host vehicle, the augmented constraint is taken into account and the detected pedestrian/object/vehicle is navigated further away. The host vehicle may be forced to do so.
[0323] The at least one augmented property may also include other types of modifications in navigation constraint properties. For example, the characteristics that are augmented may include speed reductions associated with at least one predefined navigation constraint. The augmented characteristics may also include a reduction in maximum allowable deceleration/acceleration associated with at least one predefined navigation constraint.
[0324] Navigation based on long-term planning [0325] In some embodiments, the disclosed navigation system is not only capable of responding to detected navigation conditions within the host vehicle's environment, but also determining one or more navigation actions based on long-term planning. For example, the system may consider the potential impact on future navigation states of one or more navigation operations available as options for navigating with respect to the detected navigation state. Considering the effects of available actions on future conditions may allow the navigation system to determine navigation actions based not only on currently detected navigation conditions, but also on long-term plans. Navigation using long-term planning techniques may be particularly applicable when one or more reward functions are used by the navigation system as a technique for selecting navigation actions among available options. Potential rewards can be analyzed with respect to available navigation actions that can be taken in response to the detected current navigation state of the host vehicle. However, it is also possible to analyze the potential rewards in relation to actions that can be taken in response to predicted future navigation states resulting from the available actions for the current navigation state. As a result, in some cases disclosing even though the navigation action chosen may not result in the highest reward of the available actions that can be taken depending on the current navigation state. The navigation system may select navigation operations depending on detected navigation conditions. This, among other things, triggers one or more potential navigational actions that provide a higher reward than either the selected action or, in some cases, the available actions for the current navigational state. This may be the case when the system determines that the selected action may result in a navigation state. This principle can be more simply expressed as performing less advantageous actions now to yield higher-reward alternatives in the future. Accordingly, the disclosed navigation system capable of long-term planning can select the next best short-term action when long-term predictions indicate that a short-term loss in reward may result in a long-term increase in reward.
[0326] In general, autonomous driving applications can involve a series of planning problems in which the navigation system can determine immediate actions to optimize long-term objectives. For example, if a vehicle is faced with a merging situation at a roundabout, the navigation system can determine an immediate acceleration or braking command to begin navigation into the roundabout. Immediate actions for navigation conditions detected at a roundabout may include acceleration or braking commands depending on the detected condition, but the long-term objective is to successfully merge, and the long-term The effect is the success/failure of the merge. Planning problems can be addressed by decomposing the problem into two phases. First, supervised learning can be applied to predict the near future based on the present (assuming the predictors are differentiable with respect to the current representation). Second, a regression neural network can be used to model the agent's complete trajectory, with unaccounted factors being modeled as (additional) input nodes. This may enable solutions to long-term planning problems to be found using supervised learning methods and direct optimization on recurrent neural networks. Such techniques can also enable robust policy learning by incorporating adversarial elements to the environment.
[0327] Two of the most fundamental elements of an autonomous driving system are sensing and planning. Sensing deals with finding a compact representation of the current state of the environment, whereas planning deals with deciding which actions to take to optimize future objectives. Supervised machine learning methods are useful for solving detection problems. Machine learning algorithmic frameworks can also be used for the planning part, particularly reinforcement learning (RL) frameworks as described above.
[0328] RL can be performed in a series of consecutive rounds. In round t, the planner (aka agent or driving policy module 803) creates a state s representing the agent as well as the environment.<sub>t</sub>S can be observed. The planner then performs the action a<sub>t</sub>A should be determined. After performing an action, the agent receives an immediate reward<math num="47"><img file="JP7367255B2_D0047.tif" /></math>and new status s<sub>t+1</sub>will be moved to As an example, the host vehicle may include an adaptive cruise control (ACC) system in which the vehicle accelerates/brakes autonomously to maintain a sufficient distance to the vehicle in front while maintaining smooth driving. should be implemented. The state is<math num="48"><img file="JP7367255B2_D0048.tif" /></math>can be modeled as x<sub>t</sub>is the distance to the preceding vehicle, and v<sub>t</sub>is the speed of the host vehicle relative to the speed of the preceding vehicle. motion<math num="49"><img file="JP7367255B2_D0049.tif" /></math>is an acceleration command (a<sub>t</sub>If <0 holds, the host vehicle decelerates). The reward is (reflects the smoothness of driving)|a<sub>t</sub>| and (reflecting that the host vehicle maintains a safe distance from the preceding vehicle)<sub>t</sub>can be a function that depends on . The planner's goal is to maximize cumulative compensation (possibly up to a discounted sum of period or future compensation). To do that, the planner can make use of the policy π:SA that maps states to actions.
[0329] Supervised learning (SL) can be seen as a special case of RL, in which supervised learning involves learning from some distribution over S<sub>t</sub>is sampled and the reward function is r<sub>t</sub>=-l(a<sub>t</sub>,y<sub>t</sub>), where l is the loss function and the learning side is the (possibly noisy) value of the optimal action to take given state st.<sub>t</sub>Observe the value of . There may be some differences between the general RL model and the specific case of SL, and those differences can make the general RL problem more difficult.
[0330] In some SL situations, the actions (or predictions) made by the learner may have no impact on the environment. In other words, s<sub>t+1</sub>and a<sub>t</sub>are independent. This may have two important implications. First, in SL, sample(s<sub>1</sub>,y<sub>1</sub>),...,(s<sub>m</sub>,y<sub>m</sub>) can be collected in advance, and only then can one begin searching for policies (or predictors) that have good accuracy for the sample. In contrast, in RL, state s<sub>t+1</sub>typically depends on the action taken (and even the previous state), which in turn depends on the policy used to generate the action. This links the data generation process to the policy learning process. Second, in SL, operations do not affect the environment, so a<sub>t</sub>The contribution of the selection of is local. Specifically, a<sub>t</sub>affects only the value of the immediate reward. In contrast, in RL, actions taken in round t can have long-term effects on reward values in future rounds.
[0331] In SL, the shape of the reward r<sub>t</sub>=-l(a<sub>t</sub>,y<sub>t</sub>) with the "correct" solution y<sub>t</sub>knowledge of a<sub>t</sub>can provide complete knowledge of the rewards regarding all possible options for a<sub>t</sub>It may be possible to calculate the derivative of the reward with respect to In contrast, in RL, the "one-shot" value of a reward can be all that can be observed for a particular selection of actions performed. This can be called "bandit" feedback. In RL-based systems, if only "bandit" feedback is available, this may result in long-term This is one of the most important reasons why "exploration" is necessary as part of any navigation plan.
[0332] Many RL algorithms rely, at least in part, on mathematically explicit models of Markov Decision Processes (MDPs). The Markov assumption is s<sub>t</sub>and a<sub>t</sub>given s<sub>t+1</sub>The distribution of is completely determined. This yields a closed form of the cumulative reward of a given policy in terms of a stationary distribution over the states of the MDP. A stationary distribution of policies can be expressed as a solution to a linear programming problem. This involves 1) optimization with respect to the main problem, which can be called policy search, and 2) whose variables are value functions V<sup>π</sup>It provides two groups of algorithms called optimization for dual problems. The value function determines the expected cumulative reward if the MDP starts from the initial state s and from there actions are selected according to π. The relevant quantity is the state-action value function Qπ(s,a), which assumes starting from state s, an action a chosen immediately, and an action chosen from there according to π, and the cumulative reward Determine. This Q-function may lead to the characterization of the optimal policy (using the Bellman equation). Specifically, this Q-function allows us to show that the optimal policy is a deterministic function from S to A (in fact, the optimal policy is a "greedy" policy with respect to the optimal Q-function). ).
[0333] One potential advantage of the MDP model is that it allows the future to be coupled to the present using the Q function. For example, given that the host vehicle is currently in state s, and Q<sup>π</sup>The value of (s,a) may indicate the effect of performing action a on the future. Therefore, the Q-function provides a local measure of the quality of action a, which can make the RL problem more similar to the SL scenario.
[0334] Many RL algorithms approximate the V or Q function in some way. Value-iteration algorithms, such as Q-learning algorithms, can take advantage of the fact that the V-function and Q-function of the optimal policy can be fixed points of some operators obtained from the Bellman equation. Actor-critic policy iterative algorithms aim to learn policies in an iterative manner, where at iteration t, the ``critic'' is<math num="50"><img file="JP7367255B2_D0050.tif" /></math>Based on this estimation, the "actor" improves the policy.
[0335] Despite the mathematical simplicity of MDP and the convenience of switching to a Q-function representation, this approach may have some limitations. For example, in some cases the concept of approximation of a Markov behavioral state may be all that can be found. Moreover, state transitions may depend not only on the actions of the agent, but also on the actions of other players in the environment. For example, in the ACC example above, the dynamics of the autonomous vehicle may be Markovian, but the next state may depend on the behavior of other car drivers, which is not necessarily Markovian. One possible solution to this problem is to use a partially observed MDP, in which the Markov states are present but only visible according to the hidden states. It is assumed that the observations are distributed.
[0336] A more direct approach could consider a game-theoretic generalization of MDP (eg, a stochastic game framework). In fact, algorithms for MDP can be generalized to multi-agent games (eg, minimax Q-learning or Nash Q-learning). Other techniques may include explicit modeling of other players and vanishing regret learning algorithms. Learning within a multi-agent setting can be more complex than learning within a single-agent setting.
[0337] A second limitation of the Q-function representation can arise from departing from the tabular setting. A tabular setting is when the number of states and actions is small, so that Q can be represented as a table of |S| rows and |A| columns. However, if the natural representations of S and A involve Euclidean space, and the state and action spaces are discretized, then the number of states/actions can be exponential in scale. In such cases, it may not be practical to adopt a tabular setting. Alternatively, the Q-function may be approximated by some function from a parametric hypothesis class (eg, a neural network of a particular architecture). For example, a deep Q network (DQN) learning algorithm can be used. In DQN, the state space may be continuous, but the operation space may remain a small discrete set. Approaches to accommodate continuous motion spaces are possible, but they may rely on approximating the Q-function. In any case, the Q-function can be complex and sensitive to noise, and therefore difficult to learn.
[0338] A different approach may be to use a regression neural network (RNN) to address the RL problem. In some cases, RNNs can be combined with the concepts of multi-agent games and robustness to adversarial environments from game theory. Furthermore, this approach may not explicitly rely on Markov assumptions.
[0339] In the following, a technique for navigation by prediction-based planning will be described in more detail. In this method, the state space S is<math num="51"><img file="JP7367255B2_D0051.tif" /></math>, and the state space A is<math num="52"><img file="JP7367255B2_D0052.tif" /></math>can be assumed to be a subset of . This may be a natural representation in many applications. As mentioned above, there may be two main differences between RL and SL, namely: (1) past behavior affects future rewards, so (2) the "bandit" nature of the reward may obscure the dependence between (state, behavior) and reward, which may lead to learning This can complicate the process.
[0340] As a first step in this approach, it can be observed that there are interesting problems where the bandit nature of the reward is not an issue. For example, reward values for ACC applications (discussed in more detail below) may be differentiable with respect to current state and action. In fact, even if the rewards are given in a "bandit" style,<math num="53"><img file="JP7367255B2_D0053.tif" /></math>A differentiable function such that<math num="54"><img file="JP7367255B2_D0054.tif" /></math>The problem of learning can be a relatively simple SL problem (e.g., a one-dimensional regression problem). Therefore, the first step in this method is to create a function that is differentiable with respect to s and a.<math num="55"><img file="JP7367255B2_D0055.tif" /></math>or the instance vector is<math num="56"><img file="JP7367255B2_D0056.tif" /></math>and the target scalar is r<sub>t</sub>a differentiable function that minimizes at least some regression loss over the samples with<math num="57"><img file="JP7367255B2_D0057.tif" /></math>could be by using a regression learning algorithm to learn. In some situations, elements of exploration can be used to create a training set.
[0341] Similar concepts can be used to address connections between the past and the future. for example,<math num="58"><img file="JP7367255B2_D0058.tif" /></math>Differentiable function such that<math num="59"><img file="JP7367255B2_D0059.tif" /></math>Assume that is learnable. Learning such a function can be characterized as an SL problem.
<math num="60"><img file="JP7367255B2_D0060.tif" /></math>can be considered as a predictive factor for the near future. The policy that maps S to A can then be written using the parametric function πθ:SA. Representing πθ as a neural network can allow the representation of episodes of running an agent T rounds using a recurrent neural network (RNN), where the next state is<math num="61"><img file="JP7367255B2_D0061.tif" /></math>is defined as here,<math num="62"><img file="JP7367255B2_D0062.tif" /></math>can be determined by the environment and can represent unpredictable aspects of the near future. s<sub>t+1</sub>s<sub>t</sub>and a<sub>t</sub>may enable a link between future reward values and past behavior. policy function π<sub>θ</sub>The parameter vector of can be learned by backpropagation on the resulting RNN. With explicit probabilistic assumptions v<sub>t</sub>Note that there is no need to impose Specifically, there is no need for Markov-related requirements. Alternatively, a regression network may be utilized to propagate "enough" information between the past and the future. Intuitively,<math num="63"><img file="JP7367255B2_D0063.tif" /></math>can describe a predictable part of the near future, while v<sub>t</sub>may represent unpredictable aspects that may be caused by the behavior of other players in the environment. The learning system should learn policies that are robust to the behavior of other players. ||v<sub>t</sub>If || is large, the connections between past actions and future reward values may be too noisy to learn meaningful policies. Articulating the dynamics of a system in a transparent manner can allow past knowledge to be incorporated more easily. For example, past knowledge<math num="64"><img file="JP7367255B2_D0064.tif" /></math>The problem of determining can be simplified.
[0342] As discussed above, the learning system can benefit from robustness to hostile environments, such as the host vehicle's environment, which may include other drivers that may behave in unexpected ways. v<sub>t</sub>In models that do not impose stochastic assumptions on v<sub>t</sub>One can consider an environment in which , is selected in an adversarial manner. In some cases μ<sub>t</sub>limits can be placed on otherwise an adversary could make the planning problem difficult or even impossible. One natural constraint is ||μ<sub>t</sub>It may be possible to require that || be bounded by a constraint.
[0343] Robustness to hostile environments may be useful in autonomous driving applications. μ in an adversarial way<sub>t</sub>Choosing can even accelerate the learning process, as it can focus the learning system towards a robust optimal policy. A simple game can be used to explain this concept. The condition is<math num="65"><img file="JP7367255B2_D0065.tif" /></math>And the operation is<math num="66"><img file="JP7367255B2_D0066.tif" /></math>and the immediate loss function is 0.1|a<sub>t</sub>|+[|s<sub>t</sub>|-2]+ and [x]<sub>+</sub>=max{x,0} is a ReLU (Regularized Linear Unit) function. The next state is s<sub>t+1</sub>=s<sub>t</sub>+a<sub>t</sub>+v<sub>t</sub>and v<sub>t</sub>[-0.5,0.5] is chosen for the environment in an adversarial way. Here we can write the optimal policy for a two-layer network with ReLU:a<sub>t</sub>=-[s<sub>t</sub>-1.5]<sub>+</sub>+[-s<sub>t</sub>-1.5]<sub>+</sub>. |s<sub>t</sub>Recognize that when |(1.5,2], the optimal action may have a larger immediate loss than the action a=0. Therefore, the system can plan for the future and It is not necessary to rely only on the loss of .The differential of the loss with respect to at is 0.1sign(a<sub>t</sub>) and s<sub>t</sub>The derivative with respect to 1[|s<sub>t</sub>|>2]sign(s<sub>t</sub>). s<sub>t</sub>In the situation (1.5,2], v<sub>t</sub>The adversarial selection of v<sub>t</sub>=0.5, so a<sub>t</sub>>1.5-s<sub>t</sub>There can be a non-zero loss on round t+1 whenever . In such a case, the derivative of the loss is a<sub>t</sub>can be directly backpropagated. Therefore v<sub>t</sub>The adversarial selection of a<sub>t</sub>The navigation system can facilitate obtaining non-zero backpropagation messages when the selection of is suboptimal. Such a relationship is based on the expectation that the current action will provide an opportunity for a more optimal action that will yield a higher reward in the future (even if that action results in suboptimal reward or even loss). It may encourage selection of actions.
[0344] Such an approach can be applied to virtually all possible navigation situations. In the following, one example will be described, namely a method applied to adaptive cruise control (ACC). In an ACC problem, the host vehicle may attempt to maintain a sufficient distance to the target vehicle in front (eg, 1.5 seconds to the target vehicle). Another goal may be to run as smoothly as possible while maintaining the desired clearance. A model representing this situation can be defined as follows. The state space is<math num="67"><img file="JP7367255B2_D0067.tif" /></math>and the operating space is<math num="68"><img file="JP7367255B2_D0068.tif" /></math>It is. The first coordinate of the state is the speed of the target vehicle, the second coordinate is the speed of the host vehicle, and the last coordinate is the distance between the host vehicle and the target vehicle (e.g. on a curve in the road). (along the host vehicle's position minus the target's position). The action that the host vehicle should perform is acceleration, a<sub>t</sub>It can be shown as The quantity τ may indicate the time difference between successive rounds. τ can be set to any suitable amount, but in one example τ can be 0.1 seconds. position s<sub>t</sub>teeth,<math num="69"><img file="JP7367255B2_D0069.tif" /></math>The (unknown) acceleration of the target vehicle can be expressed as<math num="70"><img file="JP7367255B2_D0070.tif" /></math>It can be shown as
[0345] The complete dynamics of the system can be described by:
<math num="71"><img file="JP7367255B2_D0071.tif" /></math>
[0346] This can be written as the sum of two vectors.
<math num="72"><img file="JP7367255B2_D0072.tif" /></math>
[0347] The first vector is the predictable part and the second vector is the unpredictable part. The reward for round t is determined as follows.
<math num="73"><img file="JP7367255B2_D0073.tif" /></math>
[0348] The first term can result in a penalty for non-zero acceleration, thus encouraging smooth driving. The second term is the target car x<sub>t</sub>distance to and desired distance<math num="74"><img file="JP7367255B2_D0074.tif" /></math>This is defined as the maximum value between a distance of 1 meter and a braking distance of 1.5 seconds. In some cases this ratio may be exactly 1, but as long as this ratio is in the range [0.7,1.3] the policy can get away without any penalty and it will achieve smooth operation. The host vehicle can be allowed to experience some slack in navigation, a characteristic that can be important in navigation.
[0349] Implementing the techniques outlined above, the host vehicle's navigation system may select an action in response to an observed condition (e.g., by action of the driving policy module 803 within the processing unit 110 of the navigation system). I can do it. The actions selected are not based on an analysis of only the rewards associated with the response actions available with respect to the sensed navigational state, but rather with respect to future states, potential actions and potential actions in response to future states. It can also be based on reviewing and analyzing remuneration.
[0350] FIG. 16 shows an algorithmic approach to navigation based on detection and long-term planning. For example, in step 1601, at least one processing device 110 of a navigation system for a host vehicle may receive a plurality of images. These images may capture scenes representative of the host vehicle's environment and may be provided by any of the image capture devices described above (eg, cameras, sensors, etc.). Analyzing one or more of these images at step 1603 may enable at least one processing device 110 (as described above) to identify a current navigation state associated with the host vehicle.
[0351] At steps 1605, 1607 and 1609, various potential navigation actions may be determined depending on the detected navigation conditions. (e.g., to complete a merge, to smoothly follow a preceding vehicle, to overtake a target vehicle, to avoid objects in the road, to slow down for a detected stop sign, to cut in, etc.) These potential navigation actions (e.g., the available navigation actions from the first (up to the Nth navigation operation) can be determined based on the sensing conditions and the long-term goals of the navigation system.
[0352] For each potential navigational action determined, the system may determine an expected reward. Expected rewards can be determined according to any of the techniques described above and can include an analysis of specific potential behaviors for one or more reward functions. Expected rewards 1606, 1608, and 1610 may be determined for each of the (eg, first, second, and Nth) potential navigation actions determined in steps 1605, 1607, and 1609, respectively.
[0353] In some cases, the host vehicle's navigation system selects among the potential actions available based on the values associated with expected rewards 1606, 1608, and 1610 (or any other type of indicator of expected rewards). You can make a selection from. For example, in some situations the action that results in the highest expected reward may be selected.
[0354] Among other instances where the navigation system engages in long-term planning to determine navigation actions for the host vehicle, the system may not select the potential action that yields the highest expected reward. Rather, the system can look to the future and analyze whether there may be an opportunity to realize a higher reward later if a lower reward action is selected depending on the current navigation state. For example, future states may be determined for any or all of the potential actions determined in steps 1605, 1607, and 1609. Each future state determined in steps 1613, 1615, and 1617 is based on the current navigation state modified by each potential action (e.g., the potential action determined in steps 1605, 1607, and 1609). May represent future navigation states that are expected to occur.
[0355] For each of the future states predicted in steps 1613, 1615 and 1617, one or more future actions (as navigational options available depending on the determined future state) are determined. can be evaluated. Steps 1619, 1621 and 1623, e.g., develop expected reward values or any other type of indicators (e.g., based on one or more reward functions) associated with one or more of the future actions. be able to. The expected reward associated with one or more future actions is assessed by comparing the values of the reward functions associated with each future action or by comparing any other indicators associated with the expected reward. be able to.
[0356] In step 1625, the navigation system for the host vehicle determines whether (e.g., The host based on comparing the expected reward also based on the expected reward determined as a result of the available future potential actions depending on the predicted future state (determined in steps 1613, 1615 and 1617). Navigation behavior for the vehicle can be selected. The selection at step 1625 may be based on the analysis of options and rewards performed at steps 1619, 1621, and 1623.
[0357] The selection of navigational actions at step 1625 may be based solely on comparing expected rewards associated with future action options. In this case, the navigation system may select an action for the current state based solely on comparing the expected reward resulting from the action for potential future navigation states. For example, the system may select the potential action identified in steps 1605, 1607, or 1609 that is associated with the highest future reward value as determined by the analysis in steps 1619, 1621, and 1623.
[0358] The selection of the navigation action at step 1625 may be based solely on comparing current action options (as described above). In this situation, the navigation system may select the potential action identified in step 1605, 1607 or 1609 that is associated with the highest expected reward 1606, 1608 or 1610. This selection may be made with little or no consideration of future navigational conditions or future expected rewards for available navigational actions depending on expected future navigational conditions.
[0359] On the other hand, in some cases, the selection of the navigation action at step 1625 may be based on comparing the expected rewards associated with both future action options and current action options. This may actually be one of the principles of navigation based on long-term planning. In order to realize potentially higher rewards depending on subsequent navigational actions that are expected to be available depending on future navigational states, the expected rewards for future actions are analyzed and applied to the current navigational state. It can then be determined whether either may be justified in selecting the lower reward action accordingly. As an example, the value or other indicator of expected reward 1606 may indicate the highest expected reward among rewards 1606, 1608, and 1610. On the other hand, expected reward 1608 may indicate the lowest expected reward among rewards 1606, 1608, and 1610. Rather than simply selecting the potential action determined in step 1605 (i.e., the action that yields the highest expected reward 1606), in making the navigation action selection in step 1625, the future state, potential future Analysis of behavior and future rewards can be used. In one example, the reward identified in step 1621 (in response to the at least one future action for the future state determined in step 1615 based on the second potential action determined in step 1607) is the expected reward 1606. It may be determined that there is a higher probability than Based on this comparison, select the second potential action determined in step 1607 over the first potential action determined in step 1605, even though the expected reward 1606 is higher than the expected reward 1608. be able to. In one example, the potential navigation action determined in step 1605 may include merging in front of the detected target vehicle, while the potential navigation action determined in step 1607 may include merging in front of the detected target vehicle. This may include merging. Although the expected reward 1606 of merging in front of the target vehicle may be higher than the expected reward 1608 associated with merging behind the target vehicle, merging behind the target vehicle may be higher than the expected reward 1606, 1608 or it may be determined that a future state is likely to result in an action option that provides an even higher potential reward than other rewards based on the actions available depending on the current sensed navigational state. be.
[0360] Making the selection among the potential actions in step 1625 may include any of the expected rewards (or any other measure or indicator of the benefit associated with one potential action over another). can be based on appropriate comparisons. In some cases, as explained above, the second potential action is predicted to result in at least one future action associated with a higher expected reward than the reward associated with the first potential action. , the second potential action may be selected in preference to the first potential action. In other cases, more complex comparisons can be used. For example, a reward associated with a choice of actions depending on a predicted future state may be compared to a plurality of expected rewards associated with potential actions to be determined.
[0361] In some scenarios, at least one of the future actions results in a higher reward than any of the expected rewards as a result of the potential actions for the current state (e.g., expected rewards 1606, 1608, 1610, etc.) If expected, actions based on predicted future states and expected rewards may influence the selection of potential actions for the current state. In some cases, the expected reward associated with the potential action for the current state detected as well as the potential The future action option that yields the highest expected reward (among the expected rewards associated with the future action options) can be used. That is, after identifying the future behavior option that yields the highest expected reward (or reward above a predetermined threshold, etc.), the potential for future states associated with the identified future behavior that yields the highest expected reward is identified. Actions may be selected at step 1625.
[0362] In other instances, the selection of available actions can be made based on a determined difference between expected rewards. For example, if the difference between the expected reward associated with the future action determined in step 1621 and the expected reward 1606 exceeds the difference between the expected reward 1608 and the expected reward 1606 (assuming a + sign difference), then in step 1607 A second potential action may be selected to be determined. In another example, the difference between the expected reward associated with the future action determined in step 1621 and the expected reward associated with the future action determined in step 1619 exceeds the difference between expected reward 1608 and expected reward 1606. If so, the second potential action determined in step 1607 may be selected.
[0363] Several examples have been described for selecting among potential actions for the current navigation state. However, any other suitable comparison technique or criterion may be used to select actions available with a long-term plan based on an analysis of actions and rewards over predicted future states. In addition, Figure 16 shows that in a long-term planning analysis there are two layers (e.g., a first layer that considers the rewards that result from potential actions relative to the current state and a first layer that considers the rewards that result from potential actions relative to the current state; A second layer (considering the resulting rewards) is shown, but an analysis based on even more layers may also be possible. For example, instead of basing long-term planning analysis on one or two layers, there are three or four layers of analysis when making choices among the potential actions available depending on the current navigation state. Or even more layers can be used.
[0364] After making the selection among the potential actions in response to the sensed navigation condition, in step 1627 the at least one processor controls at least one of the host vehicle's navigation actuators in response to the selected potential navigation action. Can cause one adjustment. A navigation actuator may include any suitable device for controlling at least one aspect of a host vehicle. For example, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.
[0365] Navigation based on inferred aggressiveness of others [0366] The target vehicle can be monitored by analyzing the acquired image stream to determine indicators of driving aggressiveness. Here, aggressiveness is described as a qualitative or quantitative parameter, but other characteristics may be used, namely the perceived level of attention (latent driver deficiencies, distractions - cell phones, falling asleep, etc.). . In some cases, the target vehicle can be considered to have a defensive posture, and in some cases, the target vehicle can be considered to have a more aggressive posture. Navigation behavior can be selected or developed based on indicators of aggression. For example, in some cases, relative speed, relative acceleration, increase in relative acceleration, pursuit distance, etc. relative to the host vehicle may be tracked to determine whether the target vehicle is aggressive or defensive. For example, the host vehicle may be inclined to yield to the target vehicle if the target vehicle is determined to have a level of aggressiveness above a threshold. Determining the level of aggression of the target vehicle based on the determined behavior of the target vehicle with respect to one or more obstacles in the route or near the target vehicle (e.g., preceding vehicles, obstacles in the road, traffic lights, etc.) You can also.
[0367] As an introduction to this concept, we describe an example experiment involving a host vehicle merging into a roundabout, where the navigation goal is to exit through the roundabout. This situation may begin with the host vehicle reaching the roundabout entrance and may end with the host vehicle reaching the roundabout exit (eg, a second exit). Success is measured based on whether the host vehicle maintains a safe distance from all other vehicles at all times, whether the host vehicle completes the route as quickly as possible, and whether the host vehicle follows a policy of smooth acceleration. can do. In this illustration, NT target vehicles may be randomly placed on a roundabout. To model the confusion between adversarial and typical behavior, the target vehicle can be modeled with an "aggressive" driving policy with probability p, so that the host vehicle attempts to merge in front of the target vehicle. When attempting, the aggressive target vehicle accelerates. With probability 1-p, the target vehicle can be modeled with a "defensive" driving policy, such that the target vehicle slows down and merges with the host vehicle. In this experiment, p=0.5, and no information about the type of other drivers needs to be given to the host vehicle's navigation system. Other driver types may be randomly selected at the beginning of the episode.
[0368] The navigation state can be expressed as the speed and position of the host vehicle (agent) and the position, speed and acceleration of the target vehicle. Continuing to observe the target's acceleration can be important to distinguish between aggressive and defensive drivers based on current conditions. All target vehicles may move on a one-dimensional curve that outlines the route of the roundabout. The host vehicle can move on its own one-dimensional curve that intersects the target vehicle's curve at the merging point, which is the origin of both curves. To model reasonable driving, the absolute value of all vehicle accelerations can be capped by a constant. Speed can also pass ReLU, as driving in the opposite direction is not allowed. Note that by not allowing driving in the opposite direction, the agent cannot repent of its past actions, so long-term planning may be required.
[0369] As explained above, the next state s<sub>t+1</sub>is the predictable part<math num="75"><img file="JP7367255B2_D0075.tif" /></math>and the unpredictable part v<sub>t</sub>It can be decomposed into the sum of Expression<math num="76"><img file="JP7367255B2_D0076.tif" /></math>can represent the dynamics of the vehicle's position and velocity (which can be clearly defined in a differentiable way), while v<sub>t</sub>may represent the acceleration of the target vehicle.
<math num="77"><img file="JP7367255B2_D0077.tif" /></math>It can be proved that can be expressed as a combination of ReLU functions on an affine transformation, so s<sub>t</sub>and is differentiable with respect to at. vector v<sub>t</sub>may be defined by the simulator in a non-differentiable manner and may implement the offensive behavior of some targets and the defensive behavior of other targets. Two frames from such a simulator are shown in Figures 17A and 17B. In this example experiment, host vehicle 1701 learned to decelerate when it reached the entrance of a roundabout. Host vehicle 1701 also learns to yield to aggressive vehicles (e.g., vehicles 1703 and 1705) and proceed safely when merging in front of defensive vehicles (e.g., vehicles 1706, 1708, and 1710). did. In the example shown in FIGS. 17A and 17B, the type of target vehicle is not given to the navigation system of host vehicle 1701. Rather, whether a particular vehicle is determined to be offensive or defensive is determined by inference based on, for example, the observed position and acceleration of the target vehicle. In FIG. 17A, based on position, velocity, and/or relative acceleration, host vehicle 1701 may determine that vehicle 1703 has aggressive tendencies, and therefore host vehicle 1701 merges in front of target vehicle 1703. Instead of attempting to do so, the driver can stop and wait for the target vehicle 1703 to pass. However, in FIG. 17B, the target vehicle 1710, which is moving behind vehicle 1703, exhibits defensive tendencies (again based on the observed position, velocity, and/or relative acceleration of vehicle 1710). 1701 has been recognized and has therefore successfully merged in front of target vehicle 1710 and behind target vehicle 1703.
[0370] FIG. 18 shows a flowchart representing an example algorithm for navigating a host vehicle based on the predicted aggressiveness of other vehicles. In the example of FIG. 18, a level of aggressiveness associated with at least one target vehicle can be inferred based on the observed behavior of the target vehicle toward objects in the target vehicle's environment. For example, in step 1801, at least one processing device of the host vehicle's navigation system (eg, processing device 110) may receive a plurality of images representative of the host vehicle's environment from a camera associated with the host vehicle. At step 1803, analyzing one or more of the received images may enable at least one processor to identify a target vehicle (eg, vehicle 1703) within the environment of host vehicle 1701. At step 1805, analyzing one or more of the received images may enable the at least one processing device to identify at least one obstacle to the target vehicle within the host vehicle's environment. The object may be debris in the road, a stop light/traffic light, a pedestrian, another vehicle (e.g. a vehicle moving in front of the target vehicle, a parked vehicle, etc.), a box in the road, a road barrier, a curve, or the host vehicle. may include any other type of object that may be encountered within the environment. In step 1807, analyzing one or more of the received images enables the at least one processing device to determine at least one navigation characteristic of the target vehicle relative to the at least one identified obstacle for the target vehicle. It can be done.
[0371] Various navigation characteristics can be used to infer the level of aggressiveness of the detected target vehicle in order to develop an appropriate navigation response to the target vehicle. For example, such navigation characteristics may include the relative acceleration between the target vehicle and at least one identified obstacle, the distance of the target vehicle from the obstacle (e.g., the following distance of the target vehicle behind another vehicle) and/or the relative speed between the target vehicle and the obstacle, etc.
[0372] In some embodiments, navigation characteristics of the target vehicle can be determined based on outputs from sensors (eg, radar, speed sensor, GPS, etc.) associated with the host vehicle. However, in some cases, navigation characteristics of the target vehicle may be determined based partially or completely on analyzing images of the host vehicle's environment. For example, image analysis techniques described by way of example in US Pat. No. 9,168,868, mentioned above and incorporated herein by reference, can be used to recognize a target vehicle within a host vehicle's environment. Monitoring the location of the target vehicle in the captured images over a period of time and/or the location in the captured images of one or more features associated with the target vehicle (e.g., taillights, headlights, bumpers, wheels, etc.) Monitoring involves determining the relative distance, velocity, and/or acceleration between the target vehicle and the host vehicle or between the target vehicle and one or more other objects in the host vehicle's environment. can be made possible.
[0373] The level of aggressiveness of the identified target vehicle may be inferred from any suitable observed navigation characteristics or any combination of observed navigation characteristics of the target vehicle. For example, a determination of aggressiveness can be made based on any observed characteristic and one or more predetermined threshold levels or any other suitable qualitative or quantitative analysis. In some embodiments, a target vehicle may be considered aggressive if the target vehicle is observed to follow the host vehicle or another vehicle at a distance less than a predetermined aggressive distance threshold. . On the other hand, a target vehicle that is observed to follow the host vehicle or another vehicle at a distance that exceeds a predetermined offensive distance threshold can be considered self-defensive. The predetermined offensive distance threshold need not be the same as the predetermined defensive distance threshold. Additionally, either or both of the predetermined offensive distance threshold and the predetermined defensive distance threshold may include a range rather than a clear boundary. Furthermore, neither the predetermined offensive distance threshold nor the predetermined defensive distance threshold need be fixed. Rather, these values or ranges may shift over time, and different thresholds/ranges of thresholds may be applied based on the observed characteristics of the target vehicle. For example, the applied threshold may depend on one or more other characteristics of the target vehicle. Higher relative velocities and/or accelerations observed may justify application of larger thresholds/threshold ranges. Conversely, lower relative velocities and/or accelerations, including zero relative velocities and/or accelerations, may justify application of smaller distance thresholds/threshold ranges when making offensive/defensive inferences.
[0374] Offensive/defensive inferences may also be based on relative velocity and/or relative acceleration thresholds. A target vehicle may be considered aggressive if its observed relative speed and/or its relative acceleration with respect to another vehicle exceeds a predetermined level or range. A target vehicle may be considered self-defensive if its observed relative speed and/or its relative acceleration with respect to another vehicle is below a predetermined level or range.
[0375] Although an offensive/defensive decision may be made based solely on any observed navigational characteristics, the decision may also depend on any combination of observed characteristics. For example, as noted above, in some cases a target vehicle may be considered aggressive solely based on being observed following another vehicle at a distance below a certain threshold or range. . However, in other cases, following another vehicle by less than a predetermined amount (which may be the same as or different from the threshold that would apply if the decision was based on distance only) and following another vehicle by more than a predetermined amount or range. A target vehicle can be considered aggressive if it has speed and/or relative acceleration. Similarly, a target vehicle may be considered self-defensive solely based on being observed following another vehicle at a distance that exceeds a certain threshold or range. However, in other cases, following another vehicle by more than a predetermined amount (which may be the same as or different from the threshold that would apply if the decision was based on distance only) and following another vehicle by less than a predetermined amount or range. A target vehicle can be considered defensive if it has a relative velocity and/or relative acceleration. For example, if the vehicle exceeds an acceleration or deceleration of 0.5G (e.g. jerk 5m/s3), if the vehicle has a lateral acceleration of 0.5G on a lane change or curve, the system 100 when a vehicle changes lanes and gives way to another vehicle with a deceleration of more than 0.3G or a jerk of 3m/s3, and/or without the vehicle stopping. If you change 2 lanes, you can do it offensively/defensively.
[0376] It is to be understood that reference to an amount above a range can indicate that the amount exceeds or falls within all associated values of the range. Similarly, reference to an amount below a range can indicate that the amount is less than or inclusive of all values associated with the range. Additionally, although the examples described for making offensive/defensive inferences are described in terms of distance, relative acceleration, and relative velocity, any other suitable quantities may be used. For example, any indirect indicators of time to collision or distance, acceleration and/or speed of the target vehicle for which calculations can be used can be used. Although the example above focuses on the target vehicle against other vehicles, offensive/defensive reasoning can also be applied to the target vehicle against any other type of obstacle (e.g. pedestrians, road barriers, traffic lights, debris, etc.). It should also be noted that this can be done by observing the navigation characteristics of
[0377] Returning to the example shown in FIGS. 17A and 17B, as host vehicle 1701 approaches a roundabout, a navigation system including at least one processing device thereof may receive an image stream from a camera associated with the host vehicle. . Based on one or more analyzes of the received images, any of the target vehicles 1703, 1705, 1706, 1708, and 1710 can be identified. Additionally, the navigation system may analyze one or more navigation characteristics of the identified target vehicle. The navigation system may recognize that the gap between target vehicles 1703 and 1705 represents a first opportunity for potential merging into the roundabout. The navigation system may analyze the target vehicle 1703 to determine indicators of aggressiveness associated with the target vehicle 1703. If target vehicle 1703 is deemed aggressive, the host vehicle's navigation system may decide to yield to vehicle 1703 rather than merge in front of vehicle 1703. On the other hand, if the target vehicle 1703 is deemed to be defensive, the host vehicle's navigation system may attempt to complete the merging operation in front of the vehicle 1703.
[0378] When the host vehicle 1701 reaches the roundabout, at least one processing device of the navigation system may analyze the captured images to determine navigation characteristics associated with the target vehicle 1703. For example, based on the images, it may be determined that vehicle 1703 is following vehicle 1705 at a distance that provides sufficient clearance for host vehicle 1701 to safely enter. In fact, it may be determined that vehicle 1703 is following vehicle 1705 at a distance that exceeds the offensive distance threshold, and therefore, based on this information, the host vehicle's navigation system may decide to pursue target vehicle 1703 in self-defense. may be inclined to identify as a target. However, in some situations, multiple navigational characteristics of the target vehicle may be analyzed when making offensive/defensive decisions as discussed above. Expanding the analysis, the host vehicle's navigation system determines that target vehicle 1703 is following target vehicle 1705 at a non-aggressive distance, but that vehicle 1703 exceeds one or more thresholds associated with aggressive behavior. It may be determined that the vehicle 1705 has a relative velocity and/or relative acceleration relative to the vehicle 1705. In fact, host vehicle 1701 may determine that target vehicle 1703 is accelerating relative to vehicle 1705, closing the gap between vehicles 1703 and 1705. Based on further analysis of relative velocities, accelerations, and distances (as well as the speed at which the gap between vehicles 1703 and 1705 is closing), host vehicle 1701 determines that target vehicle 1703 is behaving aggressively. can do. Therefore, although there may be sufficient clearance for the host vehicle to safely navigate, host vehicle 1701 anticipates that merging in front of target vehicle 1703 will result in a vehicle navigating aggressively directly behind the host vehicle. can do. Additionally, if host vehicle 1701 merges in front of vehicle 1703, target vehicle 1703 may continue to accelerate toward host vehicle 1701 or have a non-zero relative velocity based on behavior observed by image analysis or other sensor output. may be expected to proceed towards host vehicle 1701 at . Such a situation may be undesirable from a safety perspective and may also result in discomfort for the host vehicle's passengers. For that reason, as shown in FIG. 17B, host vehicle 1701 yields to vehicle 1703 and locates vehicle 1710 behind vehicle 1703 and which is deemed to be defensive based on one or more analyzes of its navigational characteristics. You may decide to merge into the roundabout ahead.
[0379] Returning to FIG. 18, in step 1809, at least one processing device of the host vehicle's navigation system determines a navigation characteristic for the host vehicle based on the at least one identified navigation characteristic of the target vehicle relative to the identified obstacle. A navigation action (eg, merging in front of vehicle 1710 and behind vehicle 1703) may be determined. To perform the navigation operation (step 1811), the at least one processing device can cause an adjustment of at least one navigation actuator of the host vehicle in response to the determined navigation operation. For example, the brakes can be applied to give way to vehicle 1703 in FIG. 17A, and the acceleration can be applied along with steering the host vehicle's wheels to force the host vehicle into a roundabout behind vehicle 1703, as shown in FIG. 17B. You can step on it.
[0380] As explained in the examples above, the host vehicle's navigation may be based on the target vehicle's navigation characteristics relative to another vehicle or object. Additionally, the host vehicle's navigation may be based solely on the target vehicle's navigation characteristics without specific reference to another vehicle or object. For example, in step 1807 of FIG. 18, analyzing multiple images captured from the host vehicle's environment determines at least one navigation characteristic of the identified target vehicle that is indicative of a level of aggressiveness associated with the target vehicle. It may be possible to do so. Navigation characteristics may include speed, acceleration, etc. that do not need to be referenced to another object or target vehicle to make offensive/defensive decisions. For example, an observed acceleration and/or velocity associated with the target vehicle that exceeds a predetermined threshold or falls within or exceeds a certain range of values may indicate aggressive behavior. Conversely, observed accelerations and/or velocities associated with the target vehicle that are below a predetermined threshold or fall within or exceed a certain range of values may indicate defensive behavior.
[0381] Of course, in some instances, observed navigational characteristics (e.g., position, distance, acceleration, etc.) may be referenced relative to the host vehicle to make offensive/defensive decisions. . For example, observed navigational characteristics of a target vehicle that indicate the level of aggressiveness associated with the target vehicle include an increase in the relative acceleration between the target vehicle and the host vehicle, the following distance of the target vehicle behind the host vehicle, and the target vehicle's pursuit distance behind the host vehicle. This may include the relative speed between the vehicle and the host vehicle, etc.
[0382] Navigation Based on Accident Liability Constraints [0383] As explained in the sections above, planned navigation operations can be tested against predetermined constraints to ensure compliance with particular regulations. . In some embodiments, this concept can be extended to consideration of potential accident liability. As discussed below, the primary goal of autonomous navigation is safety. Absolute safety may not be possible (e.g. because at least a given host vehicle under autonomous control cannot control other vehicles around it, only its own movements) Therefore, the use of potential accident liability as a consideration in autonomous navigation and as a constraint on planned behavior means that a particular autonomous vehicle will not be able to perform any behavior that is considered dangerous (e.g., potential accident liability This may help ensure that the vehicle does not perform any actions that may result in The desired level of accident avoidance (e.g., 10<sup>-9</sup>(less than) can be achieved.
[0384] The challenges posed by modern approaches to autonomous driving are the lack of safety guarantees (or at least the inability to provide the desired level of safety), as well as the lack of scalability. We consider the problem of ensuring multi-agent safe driving. Acceptable safety levels are paramount to the acceptance of autonomous vehicles, as society is unlikely to tolerate road deaths caused by machines. The goal may be to have no accidents at all, but this may not be possible since we can envisage situations in which multiple agents are generally involved in an accident and accidents occur only due to the negligence of other agents. There is. For example, as shown in FIG. 19, host vehicle 1901 is traveling on a multi-lane expressway, and host vehicle 1901 can control its own movements with respect to target vehicles 1903, 1905, 1907, and 1909; cannot control the movement of target vehicles surrounding the target vehicle. As a result, host vehicle 1901 may not be able to avoid an accident with at least one of the target vehicles, for example, if vehicle 1905 suddenly cuts into the host vehicle's lane on a collision course with the host vehicle. To address this difficulty, the typical response of autonomous vehicle experts is to use data-driven methods, where safety validation becomes more rigorous as data over more and more miles is collected. be.
[0385] However, to understand the problematic nature of data-driven approaches to safety, the fatality rate caused by accidents per hour of (human) driving is 10<sup>-6</sup>First consider what is known to be . For society to accept machines replacing humans in driving tasks, the fatality rate must be in the triple digits, or 10 per hour.<sup>-9</sup>It is reasonable to assume that the probability should be reduced to . This estimate is similar to the assumed fatality rate of airbags and aviation standards. For example, 10<sup>-9</sup>is the probability that a wing will spontaneously detach from an aircraft in mid-air. However, attempts to ensure safety using data-driven approaches that provide more confidence in proportion to accumulated mileage are impractical. 10 per hour of driving<sup>-9</sup>The amount of data required to guarantee a lethality rate of 10<sup>9</sup>time data), which is approximately 30 billion miles. Moreover, a multi-agent system interacts with its environment (unless a realistic simulator is available that emulates real human driving with all its richness and complexity such as reckless driving). The problem of verifying this is even more difficult than creating a secure autonomous vehicle agent)) It is highly likely that it cannot be verified offline. Any change to the planning and control software requires the same amount of new data collection, which is clearly unmanageable and impractical. Moreover, developing systems with data always faces a lack of interpretability and explainability of the actions performed, and when autonomous vehicles (AVs) cause fatal accidents, it is difficult to understand why. No. As a result, model-based approaches to safety are needed, but existing "functional safety" and ASIL requirements in the automotive industry are not designed to address multi-agent environments.
[0386] The second major challenge in developing safe driving models for autonomous vehicles is the need for scalability. The underlying premise of AVs goes beyond "making the world a better place" and is instead based on the premise that driverless mobility can be sustained at lower costs than with a driver. This premise is always tied to the concept of scalability, in the sense of supporting mass production of AVs (in the millions) and, more importantly, the trivial incremental costs of making them operational in new cities. . Therefore, the cost of computation and sensing is significant, and if AVs are mass produced, the cost of validation and the ability to drive "anywhere" rather than a few selected cities will also be necessary to sustain business. It is a requirement.
[0387] Issues with current approaches are centered around three axes: (i) the required "computation density," (ii) the way high-definition maps are defined and created, and (iii) the required specifications of the sensor. It's a brute force way of thinking. Brute-force methods work against scalability, as infinite on-board computation becomes ubiquitous, the cost of building and maintaining HD maps becomes trivial and scalable, and novel and ultra-advanced sensors are developed to match the grade of the vehicle. Shift the burden to a future product that will be commercialized at a trivial cost. A future in which any of the above materializes does indeed make sense, but gaining all of the above influences is likely to be a low-probability event. Therefore, it provides a formal model that integrates safety and scalability within an AV program that is both socially acceptable and scalable in the sense of supporting millions of vehicles operating anywhere in the developed world. There is a need to.
[0388] The disclosed embodiments can provide targeted safety levels (and even exceed safety goals) and can scale to systems containing millions (or more) of autonomous vehicles. Represents a solution. In terms of safety, a model called ``Responsibility Sensitive Safety'' (RSS) has been introduced, which formalizes that the concept of ``accidental negligence'' is interpretable and explainable, and that Incorporate a sense of responsibility. The RSS definition is agnostic by the way it is implemented, which is a key feature in furthering the goal of creating a convincing global safety model. RSS is motivated by the idea that agents play asymmetric roles in accidents (as seen in Figure 19); in such accidents, typically only one of the agents is the cause of the accident, and therefore Take responsible. The RSS model also includes a formal treatment of "driving carefully" under limited sensing conditions where not all agents are visible at all times (e.g. due to occlusion). One main goal of the RSS model is to ensure that agents never cause accidents that are their "fault" or are their fault. Models can only be useful with efficient policies that comply with RSS (eg, functions that map "sensing states" to actions). For example, actions that seem harmless at the moment can lead to catastrophe in the distant future (the ``butterfly effect''). RSS can help construct a set of local constraints for the short-term future that can guarantee (or at least virtually guarantee) that future accidents will not occur as a result of the host vehicle's actions.
[0389] Another contribution revolves around the introduction of a "semantic" language composed of units, measurements and operational spaces and the specification of how these are incorporated into AV planning, sensing and operation. Ru. To see what the semantics is, consider in this context how a person taking a driving course is instructed to think about "driving policy." These instructions are non-geometric ('travel 13.7 meters at current speed, then 0.8 m/s<sup>2</sup>). Rather, these instructions are of a semantic nature ("follow the car in front" or "pass the car on the left"). The typical language of human driving policy concerns longitudinal and lateral targets rather than geometric units of acceleration vectors. A formal semantic language is connected to the complexity of planning calculations that do not grow exponentially with time and number of agents, connected to the way safety and comfort interact, and that sensing calculations are connected to the way safety and comfort interact. It can be useful in several ways, connected to the methods defined, and connected to the specification of sensor modalities and how they interact within the fusion methodology. The fusion method system (based on semantic languages) is 10<sup>5</sup>While only offline validation is performed over a dataset of hours-long driving data, the RSS model performs a<sup>-9</sup>can be guaranteed to achieve a lethality rate of
[0390] For example, in a reinforcement learning setting, regardless of the time horizon used for planning, the number of trajectories to examine at any given time is constrained by a Q function (e.g. A function that evaluates the long-term quality of performing an action aA when the agent is in state sS.Given such a Q-function, the natural choice of actions is the one with the highest quality, π(s)= argmax<sub>a</sub>can be determined by choosing Q(s,a). The signal-to-noise ratio within this space is likely to be high, allowing effective machine learning methods to successfully model the Q-function. In the case of detection calculations, semantics may make it possible to distinguish between mistakes that affect safety and those that affect driving comfort. Define a PAC model for sensing (Probabilistically Approximately Correct (PAC), to borrow Valiant's PAC learning terminology) that is tied to a Q-function and is RSS compliant but allows optimization of driving comfort Demonstrate how to incorporate measurement errors into your planning in a way that Other standard error measures, such as error relative to a global coordinate system, may not conform to the PAC detection model, so the semantic language may be important to the success of this characteristic aspect of the model. In addition, semantic languages can be constructed using low-bandwidth sensing data and thus can be an important enabler for defining HD maps that can be constructed by crowdsourcing and support scalability. .
[0391] In summary, the disclosed embodiments may include a formal model that spans the key elements of an AV: sensing, planning, and operation. This model may facilitate ensuring that there are no incidents of responsibility on the part of the AV from a planning perspective. Furthermore, due to the PAC detection model, even with detection errors, the described fusion methodology may require only a very reasonable size offline data set to comply with the described safety model. Furthermore, this model can combine security and scalability through a semantic language, thereby providing a complete methodology for secure and scalable AV. Finally, it should be noted that developing an approved safety model adopted by industry and regulatory agencies may be a necessary condition for AV success.
[0392] The RSS model can generally follow the classic sense-plan-move robot control method. The sensing system may be responsible for understanding the current state of the host vehicle's environment. The planning part, which can be called a "driving policy" and can be implemented as a set of hard-coded instructions by a trained system (e.g., a neural network) or a combination of available options to achieve a driving goal. It may be responsible for determining what is the next best move from a perspective (for example, how to move from the left lane to the right lane to exit the highway). The operating part is responsible for carrying out the plan (e.g. one or more actuators and one or more controller system). The embodiments described below mainly focus on the sensing part and the planning part.
[0393] Accidents can originate from sensing or planning errors. The plan is a multi-agent endeavor since there are other road users (human and machine) that react to the AV's actions. The described RSS model is specifically designed to address security for the planning part. This can be called multi-agent security. In statistical methods, estimation of the probability of planning error can be done "on-line". That is, each time the software is updated, billions of miles must be driven on the new version to yield an acceptable level of estimation of the frequency of planning errors. This is clearly not practicable. Alternatively, the RSS model could give a 100% guarantee (or virtually 100% guarantee) that the planning module will not make mistakes for which the AV is at fault (the concept of "fault" is formally defined). . The RSS model may also provide an efficient means for its validation that does not rely on online testing.
[0394] Errors in the sensing system are less likely to be verified because detection can be independent of vehicle motion and therefore the probability of severe detection errors can be verified using "offline" data. It can be easy. But 10<sup>9</sup>Even collecting offline data of driving over time is difficult. As part of the description of the disclosed sensing system, we have described a fusion approach that can be validated using significantly smaller amounts of data.
[0395] The described RSS system may also be scalable to millions of vehicles. For example, the described semantic driving policies and applied safety constraints can meet sensing and mapping requirements that are scalable to millions of vehicles even with today's technology.
[0396] A fundamental component of such a system is a safety definition, which is the minimum standard that an AV system may need to adhere to. The following technical lemma states that statistical methods for AV system verification are infeasible, even for testing simple claims such as "The system causes N accidents per hour." is shown. This implies that model-based safety definitions are the only viable tools for validating AV systems. Lemma 1 Let X be a probability space and let A be Pr(A)=p<sub>1</sub>Let it be an event that holds <0.1. From X<math num="78"><img file="JP7367255B2_D0078.tif" /></math>sample iid sample,<math num="79"><img file="JP7367255B2_D0079.tif" /></math>shall be. Therefore, Pr(Z=0)e<sup>-2</sup>holds true.
Proof Inequality 1-xe<sup>-2x</sup>(proved for completeness in Appendix A.1) to obtain the following equation:
<math num="80"><img file="JP7367255B2_D0080.tif" /></math>Series 1 AV system AV<sub>1</sub>is a small but insufficient probability p<sub>1</sub>Assume that an accident occurs. 1/p<sub>1</sub>Any deterministic verification procedure given a sample is AV<sub>1</sub>Another AV system AV that does not cause any accidents<sub>0</sub>and do not differentiate with a certain probability.
[0397] To obtain perspective over typical values of such probabilities, 10 per hour<sup>-9</sup>If you want an accident probability of 10 for a particular AV system,<sup>-8</sup>Assume that we only provide the probability of . Even if the system is 10<sup>8</sup>Even if you get the time running, there is a certain probability that the verification process will not be able to show that the system is unsafe.
[0398] Finally, note that this challenge concerns disabling a single, specific, and dangerous AV system. A complete solution cannot be considered as a single system, as new versions, bug fixes and updates are required. Each change, even a single line of code change, creates a new system from the verifier's perspective. Therefore, a statistically validated solution must do so online over new samples after every small modification or change to account for shifts in the distribution of states observed and reached by the new system. Obtaining such a huge number of samples repeatedly and systematically (with a certain probability that the system cannot be verified) is not feasible.
[0399] Furthermore, any statistical claim must be formalized for measurement. Claiming statistical properties over the number of accidents a system causes is significantly weaker than claiming that ``the system drives in a safe manner.'' To say that, we need to formally define what safety is.
[0400] Absolute safety is impossible [0401] An action a performed by car c is absolutely safe if it is possible at some point in the future that no accident will occur after that action. It can be considered as For example, by observing a simple driving scenario as shown in Figure 19, it can be seen that it is impossible to achieve absolute safety. From the perspective of vehicle 1901, there is no action that can guarantee that surrounding cars will not collide with it. It is also impossible to solve the problem by prohibiting autonomous vehicles from being in such situations. All highways with more than two lanes will suffer from this problem at some point, so forbidding this scenario would mean having to stay in a garage. These implications seem disappointing at first glance. Nothing is absolutely safe. However, such requirements for absolute safety as set out above may be too stringent, as evidenced by the fact that human drivers do not adhere to absolute safety requirements. Rather, humans act according to a concept of safety that depends on responsibility.
[0402] Responsibility-Sensitive Safety (RSS) [0403] An important aspect missing from the concept of absolute safety is the asymmetry of most accidents, i.e., there is no responsibility for the crash and therefore no fault. That is usually one of the drivers. In the example of FIG. 19, for example, if a car 1909 on the left suddenly rear-ends, the car 1901 in the center is not at fault. To formalize the fact considering that the central car 1901 is not at fault, the behavior of AV 1901 staying within its own lane can be considered safe. To do this, we will explain the formal concept of ``accidental negligence'' or accident responsibility, which can serve as a premise for safe driving techniques.
[0404] As an example, two cars driving at the same speed in tandem along a straight road c<sub>f</sub>,c<sub>r</sub>Consider a simple case of . The car in front c<sub>f</sub>Suppose that the driver suddenly applies the brakes due to an obstacle that appears on the road and is able to avoid it. Unfortunately, c<sub>f</sub>I did not keep a sufficient distance from c<sub>r</sub>c without being able to react in time<sub>f</sub>collides with the rear of the vehicle. Negligence is c<sub>r</sub>It is clear that it is the responsibility of the car behind to maintain a safe distance from the car in front and be prepared for unexpected but reasonable braking.
[0405] Next, we will consider a broader set of scenarios, such as driving on a multi-lane road where cars can freely change lanes, cutting into other cars' routes, and driving at various speeds. To simplify the explanation below, we will assume a straight road on a plane with the horizontal and vertical axes being the x and y axes, respectively. This can be achieved under mild conditions by defining homomorphisms between real curved roads and straight roads. In addition, we consider discrete space-time. The definition may facilitate distinguishing between two intuitively different sets of cases: simple ones where no significant lateral manipulation takes place and more complex ones with lateral movements.
[0406] Definition 1 (vehicle zone) The zone of car c is the range [c<sub>x</sub>,l<sub>eft</sub>,c<sub>x,right</sub>]×[±], where c<sub>x,left</sub>,c<sub>x,right</sub>are the position of the leftmost corner of c and the position of the rightmost corner of c.
[0407] Definition 2 (Interruption) At time t-1, car c1 (for example, car 2003 in FIGS. 20A and 20B) does not intersect the zone of car c0 (for example, car 2001 in FIGS. 20A and 20B), and If they intersect at t, then at time t the car c<sub>1</sub>is car c<sub>0</sub>break into the zone.
[0408] A further distinction can be made between the anterior/posterior parts of the zone. The term "direction of interruption" may refer to moving in the direction of the boundary of the associated zone. These definitions can define cases involving lateral movement. In simple cases where such occurrences do not occur, such as the simple case of one car tailgating another car, a safe longitudinal distance may be established.
[0409] Definition 3 (Safe longitudinal distance) Car c<sub>r</sub>(car 2103) and c<sub>r</sub>Another car within the forward zone of c<sub>f</sub>(car 2105) is safe with respect to reaction time p and c<sub>f</sub>Any brake command performed by a, |a|<a<sub>max,brake</sub>About, c<sub>r</sub>If applies its maximum brake from time p to a complete stop, then c<sub>r</sub>is c<sub>f</sub>Do not collide with
[0410] Lemma 2 below shows that c<sub>r</sub>,c<sub>f</sub>velocity, response time p and maximum acceleration α<sub>max,brake</sub>Calculate d accordingly. p is also α<sub>max,brake</sub>is also a constant and should be determined by rules to some valid value. In a further example, the response time p and the maximum acceleration α<sub>max,brake</sub>can be configured for a particular vehicle or type of vehicle, or adapted depending on measurements or input parameters regarding vehicle conditions, road conditions, user (e.g. driver or passenger) preferences, etc. /can be adjusted.
[0411] Lemma 2 Let cr be the vehicle behind cf on the vertical axis. Maximum brake command and maximum acceleration command a<sub>max,brake</sub>,a<sub>max, accel</sub>and c<sub>r</sub>Let the reaction time of ρ be ρ. The longitudinal speed of the car is υ<sub>r</sub>,υ<sub>f</sub>and their length is l<sub>f</sub>, l<sub>r</sub>shall be. υ<sub>p,max</sub>=υ<sub>r</sub>+ρ.a<sub>max, accel</sub>Established,<math num="81"><img file="JP7367255B2_D0081.tif" /></math>It is determined that L=(l<sub>r</sub>+l<sub>f</sub>)/2. Therefore, c<sub>r</sub>The minimum safe longitudinal distance for is given by the following formula:
<math num="82"><img file="JP7367255B2_D0082.tif" /></math>
[0412] Proof Let the distance at time t be d<sub>t</sub>shall be. To prevent accidents, d for all t.<sub>t</sub>>I have to get L. d<sub>min</sub>To construct d<sub>0</sub>We need to find the tightest required lower bound for . Obviously, d<sub>0</sub>must be at least L. Unless the two cars stop after T ρ seconds, the speed of the car in front is υ<sub>f</sub>-Ta<sub>max,brake</sub>On the other hand, the speed of cr is υ<sub>ρ,max</sub>-(T-ρ)a<sub>max, accel</sub>An upper limit is set by So the distance between these cars after T seconds is<math num="83"><img file="JP7367255B2_D0083.tif" /></math>A lower limit is set by .
[0413] T<sub>r</sub>is c<sub>r</sub>is the time to come to a complete stop (speed of 0) and T<sub>f</sub>Note that is the time at which the other vehicle comes to a complete stop. a<sub>max,brake</sub>(T<sub>r</sub>-T<sub>f</sub>)=υ<sub>ρ,max</sub>-υ<sub>f</sub>+ρa<sub>max,brake</sub>holds, so T<sub>r</sub>T<sub>f</sub>If d<sub>0</sub>Note that it is sufficient to require >L. T<sub>r</sub>>T<sub>f</sub>In this case, the following equation holds true.
<math num="84"><img file="JP7367255B2_D0084.tif" /></math>d<sub>Tr</sub>The proof is completed by requesting >L and rearranging the terms.
[0414] Finally, we define a comparison operator that allows comparison with some concepts of "margin". When comparing lengths, velocities, etc., it is necessary to accept very similar quantities as "equal."
Definition 4 (μ comparison) A μ comparison of two numbers a and b is a>μb if a>b+μ, a<μb if a<b-μ, and |ab| If μ, a=μb.
[0415] The following comparisons (argmin, argmax, etc.) are μ comparisons for some appropriate μ. car c<sub>1</sub>,c<sub>2</sub>Assume that an accident has occurred between the two. Define the relevant moments that need to be investigated in order to consider who is at fault for the accident. This is the point before the accident and intuitively the "point of no return" after which nothing can be done to prevent the accident.
[0416] Definition 5 (Point of Fault) The point of fault for an accident is: - at the earliest point before the accident, when there was an intersection between one of the cars and the other car zone, and - the longitudinal distance was unsafe. be.
[0417] There is clearly such a point at the moment of the accident, since both conditions apply. The point of fault can be divided into two distinct categories: - the one at which the interruption also occurs, i.e. it is the first moment of intersection of one car with the zone of another car; The crossing is at a dangerous distance.
- No interruption occurs, i.e. there is already an intersection with the zone at a safe longitudinal distance and that distance has turned dangerous at the time of the fault.
[0418] Definition 6 (μ-Losing due to lateral velocity) Car c<sub>1</sub>,c<sub>2</sub>Assume that an interrupt occurs between. If its lateral velocity with respect to the direction of the interrupt is μ faster than the lateral velocity of c2, we say that c1 is μ-Lose in lateral velocity.
[0419] It should be noted that the direction of velocity is important. For example, a speed of -1,1 (both cars collide with each other) means they are evenly matched, but a speed of 1,1+μ/2 means that the car has a positive direction relative to the other car. There is a mistake. Intuitively, this definition allows a car that rear-ends another car laterally very quickly to be at fault.
[0420] Definition 7 (depending on lateral position (μ<sub>1</sub>,μ<sub>2</sub>)-win) car c<sub>1</sub>,c<sub>2</sub>Assume that an interrupt occurs between. Its lateral position relative to the center of the interrupt lane (the center closest to the zone associated with the interrupt) is (in absolute value) μ<sub>1</sub>smaller than c<sub>2</sub>μ than that of<sub>2</sub>If small, c<sub>1</sub>is the lateral position (μ<sub>1</sub>,μ<sub>2</sub>) - You can say you win.
[0421] Intuitively, very close to the center of the lane (μ<sub>1</sub>) much more than other cars (μ<sub>2</sub>), the car will not be held at fault.
[0422] Definition 8 (negligence) car c<sub>1</sub>,c<sub>2</sub>Fault or responsibility for an accident between vehicles is a function of the conditions at the time of fault and is determined as follows: If the fault is not at the time of the cut-in, then the fault lies with the car behind.
- If the time of fault is also the time of interruption, then for one of the cars, without loss of generality, the following two conditions for some predetermined μ for c1, namely: - do not lose in lateral velocity - lateral position Unless it's the case that you win, the fault lies with both cars.
[0423] In this case, c<sub>1</sub>is forgiven. In other words, if a dangerous cut-in occurs, both cars are at fault unless one of the cars is (significantly) laterally faster and (significantly) closer to the center of the lane. This captures the desired behavior of keeping a safe distance when following a car, and only cutting in at a safe distance when cutting into a zone of cars simply traveling in its own lane. As discussed further below, automatic controller-based systems for adhering to the above safety guidelines should not cause overly defensive driving.
[0424] Dealing with Limited Sensing [0425] After considering the highway example, a second example now addresses the problem of limited sensing. A very common human reaction when blamed for an accident falls into the "but I couldn't see it" category. That is true in many cases. Sometimes due to unconscious decisions to focus on different parts of the road, sometimes due to inattention, and sometimes due to physical constraints (it is impossible to see a pedestrian hidden behind a parked vehicle), humans Detection ability is limited. Of those human constraints, advanced automatic detection systems are only affected by the latter, along with the fact that computers can never be careless, and 360° views of the road mean that computers can outperform human detection abilities. cause. Returning to the "but I didn't see it" example, the appropriate answer is "I should have been more careful." To formalize what is prudent with respect to limited detection, consider the scenario shown in Figure 22. car 2201(c<sub>0</sub>) is exiting the parking lot and joining a (possibly) busy road, but his view is blocked by building 2203, so he cannot see if there are any cars on the road. Assume that this is a narrow city street with a speed limit of 30 km/h. The human driver's behavior is to slowly merge onto the road to gain more field of view until the detection limit is no longer present. We should define a moment of significance, the first time an occluded object is revealed for the first time, and after the object is revealed, treat it like any other detectable object.
[0426] Definition 9 (Exposure Point) The exposure point of an object is the first point in time when it can be seen.
[0427] Definition 10 (Negligent due to unreasonable speed) At the time of exposure or thereafter, the vehicle c<sub>1</sub>(car 2205) has speed υ>υ<sub>limit</sub>It is running at c<sub>0</sub>Assume that the vehicle was not traveling at that speed. In this case, the negligence is c<sub>1</sub>Only in c<sub>1</sub>can be said to be at fault due to unreasonable speed.
[0428] This extension<sub>0</sub>allows the person to exit the parking lot safely. We have modified our previous definition of liability-sensitive safety (using road conditions and speed limits plus a reasonable margin) into dynamic υ<sub>limit</sub>In the worst case illustrated, all that is required is c<sub>1</sub>gaυ<sub>limit</sub>It only checks whether the interrupt is at a safe vertical distance, assuming no more than . Intuitively, this encourages c0 to drive slower and further away from cover, thereby slowly expanding its field of view and later allowing it to safely merge onto the road.
[0429] We extend the definition of accident liability to this basic case of limited detection, and a set of extensions may address similar cases. What kind of objects can be hidden from view (a potentially fast car cannot be hidden between two cars parked close together, but a pedestrian can) and What is the worst case operation that the target can perform (the pedestrian's υ<sub>limit</sub>is the car's υ<sub>limit</sub>The simple assumption that the amount of exposure (much less than A more complex example in an urban scenario can be obtained from the scenario of a pedestrian, possibly hidden by a parked vehicle. Accidental negligence regarding accidents with pedestrians can be determined.
[0430] Definition 11 (Fault in an accident with a pedestrian) There are three faults in an accident with a pedestrian: - A pedestrian collides with the side of the car, and the lateral speed of the car is in the direction of the collision..The speed of the pedestrian at the time of exposure or after that is υ<sub>limit</sub>The fault is always with the car, unless one of the following applies: the car exceeded the limit, or the car came to a complete stop.
[0431] Irregularly, if the car does not rear-end the pedestrian faster than μ but the pedestrian collides with the side of the car, or the car is stationary, or the pedestrian is not necessarily in the direction of the collision. The car is not at fault only if it was traveling superhumanly fast in some direction.
[0432] Although the described system may not guarantee absolute safety, it may result in scenarios where very few (if any) accidents occur between autonomous vehicles. For example, accidents could be eliminated if all cars (and other road users) could safely verify that they are not at fault for an accident as a result of their actions. By definition, there is at least one culpable vehicle for every accident. Therefore, if no car performs any actions that could be responsible (according to the RSS model above) for the resulting accident, then the accident should not have occurred at all, making statistical methods unwieldy and impractical. Provides the type of absolute or near-absolute security that is sought.
[0433] Not all roads have simple structures. Some, such as intersections and roundabouts, involve more complex situations with various right-of-way rules. Not all hidden objects are cars or pedestrians; bicycles and motorbikes, all legal road users, should be considered. The principles introduced in this section can be extended to these additional cases.
[0434] Efficiently Verified Conditions for Responsibility-Sensitive Security [0435] This section discusses implementation aspects of RSS. At the outset, it should be noted that the actions taken here can have a butterfly effect that triggers a chain of events for an accident after, for example, 10 minutes of driving. A ``brute force'' approach that examines all possible future outcomes is not only impractical, but likely impossible. To overcome this challenge, we next describe the above liability-sensitive security definitions along with a computationally efficient method to verify them.
[0436] Computationally Feasible Safety Verification.
[0437] The main mathematical tool for computationally feasible verification is "induction." To prove a claim by induction, we start by proving the claim on simple cases, and then each step of induction extends the proof to more complex cases. To demonstrate how this induction tool can be useful for safety verification, we will use another car c<sub>f</sub>car chasing c<sub>r</sub>Consider again the simple example of (Figure 21). The following constraints are c<sub>r</sub>can be applied to the following policies. parable c<sub>f</sub>ga-a<sub>max</sub>Even if we add a deceleration of , the resulting c at the next time step<sub>r</sub>and c<sub>f</sub>The policy can choose any acceleration command at each time step t such that the distance between is at least a safe longitudinal distance (as defined by Definition 3 and Lemma 2). If no such behavior exists, c<sub>r</sub>is deceleration-a<sub>max</sub>must be applied. The following lemma shows, by induction, that any policy subject to the above constraints c<sub>f</sub>Prove that you will never cause an accident.
[0438] Lemma 3 Under the assumption given in Definition 3, c<sub>r</sub>If the policy of follows the constraints given above, then c<sub>r</sub>is c<sub>f</sub>Never cause an accident.
[0439] Proof The proof is by induction. Regarding the basics of induction, we start from an initial state where the distance between the two vehicles is safe (according to Lemma 2). The steps of induction are as follows. c at a certain time t<sub>r</sub>and c<sub>f</sub>Consider the distance between. (c<sub>f</sub>There is no problem if there is a movement that results in a safe distance (even if the maximum deceleration is achieved). If all movements cannot guarantee a safe distance, let t'<t be the maximum time of any movement that is not maximum deceleration. By the assumption of induction, at time t'+1, we were at a safe distance, and from there we decelerated as much as possible. Therefore, by the definition of a safe distance, there are no collisions from time t' to the present, which concludes the proof.
[0440] The above example illustrates a more general concept, i.e. in extreme cases c<sub>r</sub>done by c<sub>r</sub>Demonstrate that there is some emergency operation that will return the device to a "safe condition". It should be noted that the constraints on the policy described above depend on only one time step in the future and can therefore be verified in a computationally efficient manner.
[0441] To generalize the concept of sufficient local properties for RSS, we first define a default emergency policy (DEP) and refer to it as a building block for defining local properties of behavior execution, which we refer to as "prudent". use. Therefore, it is shown that taking only prudent commands is sufficient for RSS.
[0442] Definition 12 (Default Emergency Policy) Default Emergency Policy (DEP) is to apply maximum braking force and maximum heading change towards zero heading for the lane. The maximum braking force and heading changes are derived from the physical parameters of the vehicle (possibly also from weather and road conditions). Definition 13 (Safe state) State s is safe if performing DEP starting from state s does not cause an accident for which the person is at fault. We define a command as prudent if it results in a safe condition, as in the simple case of a car following another car. Here, definition 14 (cautious command) state s<sub>0</sub>Assume that The next state s for a set of possible commands A that other vehicles may perform here<sub>1</sub>Command a is prudent if is safe. The above definition relies on the worst case commands in set A that other vehicles can perform. Build set A based on maximum braking/acceleration and reasonable upper limits for lateral movement.
[0443] The following theory proves, again by induction, that if you issue only prudent commands, there will be no accidents that are your fault. Theorem 1 Assume that c is in a safe state at time 0, and that for every time step c issues only prudent commands, and if there are no prudent commands at any time step, then c applies DEP. Therefore, c never causes an accident for which he is at fault. Proof by induction. The basis for induction comes from the steps from defining a safe state and defining a prudent command.
[0444] One benefit of this technique is that it may not be necessary to look infinitely into the future, as one can quickly return to a safe state and safely proceed from there. Furthermore, given that you can plan again at t+1, and therefore perform DEP there if necessary, rather than any possible long-term plan that might be in your head (that plan cannot be changed at t+1), ), you should only examine the command you are giving at time t. This transparent model then allows for the incorporation of learning components into the system when verified at runtime. Finally, this local validation implies a complete future RSS, which is our desired goal. An impediment to implementation is that another agent<sub>brake</sub>The definition of prudence includes all feasible trajectories up to t<sub>brake</sub>Even so, it's a huge amount of space. To address this issue, we next develop an efficiently computational method to verify prudence and thus RSS in a scalable manner.
[0445] Verification of Efficient Prudence [0446] The first view is that while we are performing DEP, a particular vehicle may carry out commands from Group A that cause an accident for which we are at fault.<math num="85"><img file="JP7367255B2_D0085.tif" /></math>A state is unsafe if and only if Therefore,<math num="86"><img file="JP7367255B2_D0086.tif" /></math>In a scene and general case where there is a single target vehicle, denoted by , the procedure can be performed sequentially for each of the other vehicles in the scene.
[0447] When considering a single target vehicle, all in set A cause an accident for which c is at fault.<math num="87"><img file="JP7367255B2_D0087.tif" /></math>Shown by<math num="88"><img file="JP7367255B2_D0088.tif" /></math>Action a is indiscreet if and only if there is a sequence of commands for . As we have already proven, at time 0,<math num="89"><img file="JP7367255B2_D0089.tif" /></math>If it is true that is within the forward zone of c, there is a simple way to check the prudence of a, i.e. if<math num="90"><img file="JP7367255B2_D0090.tif" /></math>We only need to verify that even if the maximum brake is applied for one time step (we perform a), the resulting longitudinal distance remains safe. The following lemma gives sufficient conditions for prudence in more complicated cases where lateral operations should also be considered.
[0448] Lemma 4 At time T=0,<math num="91"><img file="JP7367255B2_D0091.tif" /></math>Suppose that is not in the forward zone of c. For all T(0,t<sub>brake</sub>], if there are no dangerous interruptions for which c is at fault, then a is prudent.
[0449] Proof that a is not careful, i.e. causes an accident where c is at fault.<math num="92"><img file="JP7367255B2_D0092.tif" /></math>Assume that there exists. There must be a time T of the interruption before the accident. First, assume that T>0. If this interrupt was at a safe vertical distance, DEP would be -a<sub>max</sub>Due to the deceleration of , and carried out based on the definition of a safe longitudinal distance, no accident is possible that is our fault (assuming here that the reaction time ρ exceeds the time resolution of the step). If the interrupt was unsafe, by the assumption of the lemma, c is not at fault, and therefore c is not at fault for the accident.
[0450] Finally, if T<0, by the assumption of the lemma, at the moment of the interrupt,<math num="93"><img file="JP7367255B2_D0093.tif" /></math>was in the rear zone of c. By induction, c has only made safe interruptions in the past, so either the interruption was safe or c was at fault. In either case, c is not at fault for the current accident.
[0451] In light of Lemma 4, the question remains to check whether there can be a dangerous interrupt where c is at fault. An efficient algorithm for checking the possibility of a dangerous interrupt at time t is presented below. To verify the entire trajectory, time interval [0,t<sub>brake</sub>] and apply the algorithm to all time intervals (with slightly larger values of p within the safe distance definition to ensure that the discretization is not compromised).
<math num="94"><img file="JP7367255B2_D0094.tif" /></math>The length of the diagonal of the minimum rectangle bounding is<math num="95"><img file="JP7367255B2_D0095.tif" /></math>shall be. Time t [0,t<sub>brake</sub>], such that c is the vertical "span" of c at time t<sub>length</sub>(t);<math num="96"><img file="JP7367255B2_D0096.tif" /></math>shall be. c<sub>width</sub>Define [t] in a similar manner,<math num="97"><img file="JP7367255B2_D0097.tif" /></math>holds true.
<math num="98"><img file="JP7367255B2_D0098.tif" /></math>
[0452] The following theorem proves the correctness of the above algorithm.
[0453] Theorem 2 If Algorithm 1 returns "unfeasible", there cannot be a dangerous interruption at time t for which the own vehicle is at fault. To prove this theorem, we utilize the following important lemma that proves the validity of the two components of Algorithm 1. Start with vertical feasibility.
[0454] Lemma 5 Under the notation of Algorithm 1, if the procedure for examining vertical feasibility is concluded by returning "unfeasible", then there is no dangerous interruption in which the own vehicle is at fault at time t. I don't get it.
[0455] Proof Ignoring the lateral aspects of interrupt operations, c and<math num="99"><img file="JP7367255B2_D0099.tif" /></math>Examine the mere possibility that the vertical distance between position<math num="100"><img file="JP7367255B2_D0100.tif" /></math>is at time t,<math num="101"><img file="JP7367255B2_D0101.tif" /></math>It is clear to bound the position that can be obtained by .
<math num="102"><img file="JP7367255B2_D0102.tif" /></math>By holding, it can be obtained that any obtainable vertical dangerous distance is L.
<math num="103"><img file="JP7367255B2_D0103.tif" /></math>Assume that a<sub>y,min</sub>,a<sub>y,max</sub>Using the acceleration command bounded by<math num="104"><img file="JP7367255B2_D0104.tif" /></math><math num="105"><img file="JP7367255B2_D0105.tif" /></math>Assume by the law of contradiction that the critical longitudinal position and velocity given by are obtained.
<math num="106"><img file="JP7367255B2_D0106.tif" /></math>By definition,<math num="107"><img file="JP7367255B2_D0107.tif" /></math>is obtained, and therefore the following distance is larger, i.e.<math num="108"><img file="JP7367255B2_D0108.tif" /></math>holds true.
<math num="109"><img file="JP7367255B2_D0109.tif" /></math>is (y[t],υ<sub>y</sub>[t]), so by the definition of longitudinal danger, the resulting speed<math num="110"><img file="JP7367255B2_D0110.tif" /></math>teeth,<math num="111"><img file="JP7367255B2_D0111.tif" /></math>It has to be smaller than . However, to achieve a slower speed, throughout the time window [0,t],<math num="112"><img file="JP7367255B2_D0112.tif" /></math>is a<sub>y,min</sub>You must use an average acceleration of less than a<sub>y,min</sub>,a<sub>y,max</sub>It is clear that this contradicts what can be obtained using the command bounded by .
<math num="113"><img file="JP7367255B2_D0113.tif" /></math>In the case of , the proof is completed by considering symmetric arguments.
[0456] Next is the feasibility in the lateral direction.
[0457] Lemma 6 Under the notation of Algorithm 1, if the procedure for checking lateral feasibility concludes by returning "unfeasible", then there is no dangerous interruption in which the own vehicle is at fault at time t. I don't get it.
[0458] Proof First, assume x[t]=0,<math num="114"><img file="JP7367255B2_D0114.tif" /></math>and v<sub>x</sub>With respect to [t], it is clear that generality is not lost by simple changes in coordinates and consideration of relative velocities. Furthermore, by similar arguments,<math num="115"><img file="JP7367255B2_D0115.tif" /></math>It can be easily extended to cases where .
[0459] In our case, the position of the car involved in the interruption is (x[t],x[t]-W) due to some lateral position that affects the fault (μ1,μ2) - Note that it implies a win trait. μ- to "tie" the lateral velocity (if c does not win in lateral position (μ1, μ2), this may be sufficient to hold c at fault), or Under the assumption that c is at fault in order to μ-win (this is necessary to hold c at fault if c wins in lateral position (μ1, μ2)-wins), vx[t ] At time t, according to our assumption that<math num="116"><img file="JP7367255B2_D0116.tif" /></math>It is obtained that the maximum lateral velocity that can be used is 0. final position<math num="117"><img file="JP7367255B2_D0117.tif" /></math>with a<sub>x,max</sub>using the lateral acceleration bounded by<math num="118"><img file="JP7367255B2_D0118.tif" /></math>starting from<math num="119"><img file="JP7367255B2_D0119.tif" /></math>It remains to check if there is an operation ending in . In words, it is an interrupt that ends with a desired lateral velocity of zero.
[0460] Algorithm definition<math num="120"><img file="JP7367255B2_D0120.tif" /></math>I would like you to remember that. t<sub>top</sub>Assume <0. This is the maximum lateral acceleration, i.e.<math num="121"><img file="JP7367255B2_D0121.tif" /></math>When using<math num="122"><img file="JP7367255B2_D0122.tif" /></math>This implies that the time required for t to reach a lateral velocity of 0 is less than t. This implies that there are no operations that it can perform to reach the desired speed in time, and therefore no problematic operations. Therefore, this procedure is t<sub>top</sub>If <0 returns ``unfeasible'', there is no realizability of the dangerous interrupt for which c is actually at fault.
[0461]t<sub>top</sub>Consider the case >0. Here, the procedure is x<sub>max</sub><-W returns "impossible".
<math num="123"><img file="JP7367255B2_D0123.tif" /></math>in the time range [0,t], denoted by and parameterized by a<math num="124"><img file="JP7367255B2_D0124.tif" /></math>Consider a group of lateral velocity profiles for . In a similar way to that used within the algorithm,<math num="125"><img file="JP7367255B2_D0125.tif" /></math>is determined for each a. All of<math num="126"><img file="JP7367255B2_D0126.tif" /></math>About, t<sub>top</sub>Note that (a)>0 holds. Next, the velocity profile ua for all time points t'0[0,t] is determined as follows.
<math num="127"><img file="JP7367255B2_D0127.tif" /></math>
[0462] First, u<sub>a</sub>is a constraint<math num="128"><img file="JP7367255B2_D0128.tif" /></math>It can be seen that it is satisfied. Second, u<sub>a</sub>We can calculate the distance traveled while using , since this distance corresponds to the integral of a piecewise linear function. reach position<math num="129"><img file="JP7367255B2_D0129.tif" /></math>and x defined in the algorithm<sub>max</sub>is exactly,<math num="130"><img file="JP7367255B2_D0130.tif" /></math>Please note that. Third, it can be seen that the distance traveled increases monotonically with a and is not bounded. Therefore, any desired final position<math num="131"><img file="JP7367255B2_D0131.tif" /></math>about,<math num="132"><img file="JP7367255B2_D0132.tif" /></math>There exists a value of a for which . Specifically, for x=x[t]-W, such a value exists and is denoted by acut.
[0463] x defined in the algorithm<sub>max</sub>Since <x[t]-W, a<sub>cut</sub>>a<sub>x,max</sub>I would like to be recognized that I can obtain this. This is insufficient to show that no valid operation can result in a position x[t]-W, which is only implied for members of the U group. Then all velocity profiles that obtain the final position of x[t]-W even outside of U are at least a<sub>cut</sub>must be used on the road, such acceleration values invalidate their velocity profiles, thus completing the proof.
[0464] Some velocity profiles u are bounded constraints<math num="133"><img file="JP7367255B2_D0133.tif" /></math>Assume that the following is satisfied. Furthermore, the velocity profile u is at some final position.<math num="134"><img file="JP7367255B2_D0134.tif" /></math>Suppose we obtain . In this way, the following equation is obtained.
<math num="135"><img file="JP7367255B2_D0135.tif" /></math>
[0465] For all τ,<math num="136"><img file="JP7367255B2_D0136.tif" /></math>Assume that holds true. in particular,<math num="137"><img file="JP7367255B2_D0137.tif" /></math>holds true. From the mean value theorem,<math num="138"><img file="JP7367255B2_D0138.tif" /></math>ζ0[0,t<sub>top</sub>(a<sub>cut</sub>)] exists and a<sub>x,max</sub>implies the infeasibility of u because it uses an acceleration greater than (i.e., u', the derivative of velocity).
[0466] Next,<math num="139"><img file="JP7367255B2_D0139.tif" /></math>Assume that does not apply to all τ. In that case,<math num="140"><img file="JP7367255B2_D0140.tif" /></math>By holding,<math num="141"><img file="JP7367255B2_D0141.tif" /></math>There must be a point where Such point τ<sub>large</sub>is [0,t<sub>top</sub>(a<sub>cut</sub>)], then an acceleration that is too large is used, ζ0[0,τ<sub>large</sub>]. This point is [t<sub>top</sub>(a<sub>cut</sub>),t], -a<sub>x,max</sub>The point at which acceleration values less than ζ0[τ<sub>large,t</sub>] is given a similar argument and concludes the proof. With the above lemma, the proof of Theorem 2 is immediate.
[0467] Safety Verification - Coverage [0468] In the same manner as we deal with observed objects, we deal with hidden objects, provided that prudence implies that there is never an accident for which we are at fault. An extension of prudence can be defined using a theorem similar to Theorem 1.
[0469] Definition 15 (Caution towards hidden objects) If the object is exposed at t+1, the Default Emergency Policy (DEP) is ordered at t+1, and there is no accident for which we are at fault, then at time t The commands given are cautious about hidden objects.
[0470] Lemma 7 If you issue only prudent commands for hidden and non-hidden objects, you will never have an accident that is your fault.
[0471] Proof Suppose that in a state where the exposure time is t't, an accident for which the person is at fault occurs at time t. The assumption of prudence allowed the command given at t'-1 to command DEP at t' without being at fault for the accident. Since there was an accident for which we were at fault, we clearly did not order DEP at time t'. However, since we were safe from objects that were not hidden after t', the commands we gave were safe, and there were no accidents where we were at fault.
[0472] Once again, we provide an efficient method for verifying caution against worst-case assumptions regarding hidden objects, allowing RSS to be both feasible and scalable.
[0473] Driving Policy [0474] Driving policy is the process from sensing states (descriptions of the world surrounding the human being) to driving commands (e.g., commands include where the car should be and what speed it should be 1 second from now. is the lateral and longitudinal acceleration of the next second). Driving commands are communicated to the controller, which aims to actually move the car to the desired position/speed.
[0475] The previous section described proposed constraints on commands issued by formal safety models and driving policies that ensure safety. Safety constraints are designed for extreme cases. By and large, you don't even want to need these constraints, you want to build a driving policy that results in a comfortable ride. The focus of this section is on how to build efficient driving policies, specifically driving policies that require computational resources that can be scaled to millions of cars. Currently, this explanation does not address the question of how to obtain a sensing state, but rather assumes an ideal sensing state that faithfully represents the world surrounding humans without any restrictions. The impact of inaccurate sensing conditions on driving policy will be discussed in a later section.
[0476] Clarify the problem of defining driving policies in the reinforcement learning (RL) language discussed in the above section. During each iteration of RL, the agent describes the world s<sub>t</sub>Observe the states denoted by a and map the states to actions based on a policy function π.<sub>t</sub>You should choose the behavior shown in . As a result of their own actions and other factors beyond their control (such as the actions of other agents), the state of the world is s<sub>t+1</sub>Changes to (state, action) sequence<math num="142"><img file="JP7367255B2_D0142.tif" /></math>Indicated by Every policy induces a probability function over a (state, action) sequence. This probability function can be influenced by the actions that the agent performs, but it also depends on the environment (specifically how other agents behave). Let Pπ denote the probability over a (state, action) sequence caused by π. The quality of the policy is<math num="143"><img file="JP7367255B2_D0143.tif" /></math>It is determined that<math num="144"><img file="JP7367255B2_D0144.tif" /></math>is the sequence<math num="145"><img file="JP7367255B2_D0145.tif" /></math>It is a reward function that measures how good a person is. In most cases,<math num="146"><img file="JP7367255B2_D0146.tif" /></math>teeth,<math num="147"><img file="JP7367255B2_D0147.tif" /></math>, where ρ(s,a) is an instantaneous reward function that measures the immediate quality of being in state s and performing action a. For simplicity, we will proceed with this simpler case.
[0477] In order to clarify the problem of driving policy in the RL language mentioned above, some representation of the road and the position, speed and acceleration of not only the own vehicle but also other road users.<sub>t</sub>shall be. Let at be the horizontal and vertical acceleration command. next state s<sub>t+1</sub>depends on how at and other agents behave. The instantaneous reward ρ(st,at) is the relative position/velocity/acceleration with respect to other cars, the difference between our speed and the desired speed, whether we follow the desired route, whether our acceleration is comfortable, etc. may depend on.
[0478] One challenge in determining which action a policy should take at time t comes from having to estimate the long-term effect of this action on reward. For example, in relation to a driving policy, the action taken at time t now seems like a good action (i.e. the reward value ρ(s<sub>t</sub>,a<sub>t</sub>) may be good), but it may lead to an accident after 5 seconds (i.e., the reward value after 5 seconds will be disastrous). Therefore, it is necessary to estimate the long-term quality of performing action a when the agent is in state s. This is often called the Q function, ie Q(s,a) should reflect the long-term quality of performing action a at time s. Given such a Q-function, the natural choice of motion is the one with the highest quality, π(s)=argmax<sub>a</sub>The problem is to choose Q(s,a).
[0479] The problem at hand is how to define Q and how to efficiently evaluate Q. s<sub>t+1</sub>is (s<sub>t</sub>,a<sub>t</sub>), i.e. s<sub>t+1</sub>=f(s<sub>t</sub>,a<sub>t</sub>) I would like to first simplify the (completely unrealistic) assumption that Anyone familiar with Markov Decision Processes (MDP) knows that this assumption is even stronger than the Markov assumption of MDP (i.e., (s<sub>t</sub>,a<sub>t</sub>) given s<sub>t+1</sub>is conditionally independent of the past). As mentioned in [5], even the Markov assumption is insufficient for multi-agent scenarios such as driving, and we therefore relax the assumption later.
[0480] Under this simplifying assumption, s<sub>t</sub>Given a sequence of T-step decisions (a<sub>t</sub>,...,a<sub>t+T</sub>), the future state (s<sub>t+1</sub>,...,s<sub>t+T+1</sub>) and reward values can be calculated accurately. All these reward values, e.g. their sum<math num="148"><img file="JP7367255B2_D0148.tif" /></math>Summarizing into a single number by taking , we can define Q(s,a) as follows.
<math num="149"><img file="JP7367255B2_D0149.tif" /></math>
[0481] That is, Q(s,a) is the best future that we can hope for if we are in state s and immediately perform action a.
[0482] Explain how Q can be calculated. The first idea is to define the set A of possible actions as a finite set.<math num="150"><img file="JP7367255B2_D0150.tif" /></math>, and simply traverse all motion sequences in the discretized set. Then the discrete motion sequence<math num="151"><img file="JP7367255B2_D0151.tif" /></math>The runtime is governed by the number of
<math num="152"><img file="JP7367255B2_D0152.tif" /></math>If represents a lateral acceleration of 10 and a longitudinal acceleration of 10, we obtain a possibility of 100T, which becomes unrealizable even for small T values. Although heuristics exist to speed up the search (eg, coarse-grained search), this brute-force approach requires tremendous computational power.
[0483] The parameter T, often referred to as the "planning horizon," controls the natural trade-off between computation time and quality of evaluation, with the larger T being On the other hand, a large T increases the computation time exponentially, although their evaluation is better (because it examines its effects more explicitly into the future). To understand why a large T-value may be necessary, consider a scenario where you are 200 meters before a freeway exit and should take that exit. If the period of interest is long enough, the cumulative reward indicates whether the exit lane has been reached at some time τ between t and t+T. On the other hand, with short time horizons, even if you perform the correct immediate action, you do not know whether it will eventually lead you to the exit lane.
[0484] Another technique is<math num="153"><img file="JP7367255B2_D0153.tif" /></math>We perform offline calculations to construct an approximation of Q, denoted by , during online execution of the policy, without explicitly rolling out the future.<math num="154"><img file="JP7367255B2_D0154.tif" /></math>attempts to use as an approximation to Q. One way to construct such an approximation is to discretize both the motion domain and the state domain. These discretized sets are<math num="155"><img file="JP7367255B2_D0155.tif" /></math><math num="156"><img file="JP7367255B2_D0156.tif" /></math>Indicated by Offline calculations are available for all<math num="157"><img file="JP7367255B2_D0157.tif" /></math>The value of Q(s,a) can be evaluated for . Then all<math num="158"><img file="JP7367255B2_D0158.tif" /></math>Regarding,<math num="159"><img file="JP7367255B2_D0159.tif" /></math>of<math num="160"><img file="JP7367255B2_D0160.tif" /></math>Q(s,a). Furthermore, based on Bellman's pioneering work [2,3], and based on dynamic programming procedures (e.g. value iteration algorithm),<math num="161"><img file="JP7367255B2_D0161.tif" /></math>Q(s,a) can be calculated for each time, and under our assumption, the total runtime is<math num="162"><img file="JP7367255B2_D0162.tif" /></math>That's about it. The main problem with this method is that in any reasonable approximation<math num="163"><img file="JP7367255B2_D0163.tif" /></math>is extremely large (due to the problem of the number of dimensions). In fact, the sensing state should represent six parameters for all other relevant vehicles in the scene: longitudinal position, lateral position, velocity and acceleration. Even if we discretize each dimension to only 10 values (a very coarse discretization), there are 6 dimensions, so 106 states are required to describe a single car, and k cars 106k states are required to describe . this is,<math num="164"><img file="JP7367255B2_D0164.tif" /></math>This incurs unrealistic memory requirements to store the values of Q for all (s,a) in .
[0485] One approach to dealing with this dimensionality problem is to use linear functions over manually determined features, such as linear functions or deep neural networks, that come from a restricted family of functions (often referred to as hypothesis classes). It is to limit Q. For example, consider a deep neural network that approximates Q in the context of playing an Atari game. Provided that the family of functions approximating Q can be evaluated efficiently, this provides a resource efficient solution. However, there are some disadvantages to this approach. First, we do not know whether the family of functions chosen contains a good approximation to the desired Q-function. Second, even if such a function existed, it is unclear whether existing algorithms could learn it efficiently. So far, there are not many success stories of learning Q-functions for complex multi-agent problems such as those encountered in driving. There are several theoretical reasons why this task is difficult. As mentioned regarding the Markov assumption, there are problems with the underlying existing methods. However, as explained below, a more serious problem is the very small signal-to-noise ratio due to the temporal resolution of decision-making.
[0486] Consider a simple scenario where a vehicle needs to change lanes to exit a freeway exit with 200 meters remaining and there is currently nothing on the road. The best decision is to start changing lanes. A decision can be made every 0.1 seconds, so at the moment t Q(s<sub>t</sub>The best value of ,a) should be for the motion a that corresponds to a slight lateral acceleration to the right. Consider the motion a' corresponding to zero lateral acceleration. There is only a very small difference in changing lanes here or changing after 0.1 seconds, so Q(s<sub>t</sub>,a) and Q(s<sub>t</sub>,a') are almost the same. In other words, the advantage of choosing a over a' is very small. On the other hand, we are using a function approximation with respect to Q, and the state s<sub>t</sub>Because there is noise when measuring Q, our approximation to the Q value is likely to be noisy. This causes a very small signal-to-noise ratio, which leads to very slow learning, especially in probabilistic learning algorithms frequently used in the approximation class of neural networks. However, as mentioned, this problem is not a property of any particular class of function approximations, but rather is specific to the definition of the Q function.
[0487] In summary, the available techniques can be roughly divided into two groups. The first group are brute force techniques that involve searching over many sequences of operations or discretizing the sensed state domain and keeping large tables in memory. This approach can yield a very accurate approximation of Q, but requires infinite resources either in terms of computation time or in terms of memory. The second group are resource-efficient approaches that either search for short sequences of motion or apply a functional approximation to Q. In both cases, we are faced with obtaining a low-precision approximation of Q, which can lead to low-quality decisions.
[0488] The techniques described herein are resource efficient and accurate, and allow us to move away from geometric behavior and construct a Q-function that should fit into the semantic behavior space described in the next subsection. include.
[0489] Semantic Approach [0490] As a basis for the semantic approach to disclosure, consider a young person who has just obtained a driver's license. The young man's father sits next to him and gives him instructions on ``driving policy.'' These instructions are non-geometric ('travel 13.7 meters at current speed, then 0.8 m/s<sup>2</sup>). Rather, these instructions are of a semantic nature ("Follow the car in front" or "Quickly pass that car on the left"). We formalize a semantic language for such instructions and use it as a semantic operating space. Next, we define the Q-function over the semantic motion space. We show that semantic operations can have very long time horizons, which allows us to estimate Q(s,a) without planning many future semantic operations. The total number of semantic operations remains small. This allows obtaining an accurate estimate of the Q function while still remaining resource efficient. Moreover, as shown later, we combine learning techniques to further improve the quality function without facing small signal-to-noise ratios due to significant differences between various semantic operations.
[0491] Next, we define the semantic action space. The main idea is to define horizontal and vertical goals and the level of attack intensity to achieve those goals. The lateral goal is a desired location within the lane's coordinate system (eg, "The goal is to be in the center of lane number 2"). Vertical goals are of three types. The first is the position and speed relative to other vehicles (e.g. "The goal is to be behind car number 3 at the same speed as car number 3 and at a distance of 2 seconds from car number 3") . The second is a speed target (for example, ``I will drive at a speed that is 110% of the allowable speed for this road''). The third is a speed constraint at a specific location (e.g. "zero speed at the stop line" when reaching an intersection or "speed up to 60kmh at a specific location on the curve" when passing through a sharp curve). "). In a third option, a "velocity profile" (several discrete points on the route and the desired speed at each of those points) can be applied instead. A reasonable number of lateral targets is bounded by 16=4×4 (4 positions in a maximum of 4 relevant lanes). A reasonable number of longitudinal targets of the first type is bounded by 8×2×3=48 (8 relevant cars in front or behind and 3 relevant distances). A reasonable number of absolute speed targets is 10, and a reasonable upper limit on the number of speed constraints is 2. To accomplish a given lateral or longitudinal goal, it is necessary to apply an acceleration followed by a deceleration (or vice versa). Aggressiveness to achieve a goal is the maximum acceleration/deceleration (in absolute value) to achieve the goal. Using kinematic calculations with a defined goal and aggressiveness, a closed formula for implementing the goal is obtained. The only part left is to decide on the combination between horizontal and vertical goals (e.g. "start with horizontal goals and right in the middle start applying vertical goals as well") That's true. A set of 5 mixing times and 3 aggressiveness levels seems more than sufficient. Overall, we obtained a semantic motion space whose size is approximately 104.
[0492] It is worth mentioning that the variable time required to satisfy these semantic operations is not the same as the frequency of the decision-making process. To be responsive to a dynamic world, at high frequency, our implementation should make decisions every 100ms. In contrast, each such decision is based on constructing a trajectory that satisfies some semantic operation with a much longer time horizon (eg, 10 seconds). We use longer time horizons to facilitate better evaluation of short-term prefixes of trajectories. In the next subsection, we will explain the evaluation of semantic operations, but first we will argue that semantic operations induce a sufficient search space.
[0493] As discussed above, the semantic operational space is a subset of all possible geometric curves whose size (in T) is exponentially smaller than enumerating all possible geometric curves. cause. The first immediate question is whether this smaller set of search space short-term prefixes contains all the geometric commands that we want to use. This is actually sufficient in the following sense. If there are no other agents on the road, there is no reason to make any changes except to set vertical targets and/or absolute acceleration commands and/or speed constraints for specific locations. If the road includes other agents, you may want to negotiate the right of way with other agents. In this case, it is sufficient to set vertical goals for other agents. Although the exact implementation of these goals may change over time, the short-term prefixes do not change much. Therefore, a very good coverage of relevant short-term geometrical commands is obtained.
[0494] Construction of evaluation function for semantic actions [0495] A set of semantic actions represented by AS was defined. We need a method to select the best aS0AS given the current state s. To address this issue, we follow an approach similar to the choice mechanism in [6]. The basic idea is that a<sup>S</sup>is to think of it as a meta-action (or option). For each meta-action selection, a geometric trajectory (s<sub>1</sub>,a<sub>1</sub>),...,(s<sub>T</sub>,a<sub>T</sub>). To do that, of course we need to know how other agents will react to our actions, but we currently have s for some known deterministic functions f.<sub>t+1</sub>=f(s<sub>t</sub>,a<sub>t</sub>) continues to use the (unrealistic) assumption that holds true. As a good approximation to the quality of performing the semantic action aS when in state s1,<math num="165"><img file="JP7367255B2_D0165.tif" /></math>can be used.
[0496] This approach can result in a strong driving policy. However, in some situations a more sophisticated quality function may be required. For example, suppose you have to go to the exit lane and you are following a slower truck in front of the exit lane. One semantic option is to continue driving slowly behind the truck. Another semantic option is to pass the truck in the hope that you can later get back into the exit lane and exit in time. The quality measures described earlier do not take into account what happens after overtaking the truck, so they choose the second semantic behavior even if there is enough time to overtake and return to the exit lane. Never. Machine learning may facilitate building better estimates of semantic behavior that take into account more than immediate semantic behavior. As discussed above, learning the Q-function over immediate geometric motion is problematic due to low signal-to-noise ratio (lack of advantage). This is not a problem when considering semantic operations, because there are large differences between performing various semantic operations, and because there are many semantic time periods (how many (taking into account the semantic behavior of) are very small (often less than 3).
[0497] Another potential advantage of applying machine learning is for generalization, i.e., the ability to fully evaluate all roads by manual investigation of road characteristics, which may involve some trial and error. Functions can be set. However, is it possible to automatically apply it to all roads? Here, in order to generalize to unknown roads, the machine learning method discussed above can be trained based on a wide variety of road types. The semantic operations space according to the disclosed embodiments can enable potential benefits, namely that semantic operations contain information about long time horizons and thus improve their quality while remaining resource efficient. A very accurate evaluation can be obtained.
[0498] Dynamics of other agents [0499] So far, s<sub>t+1</sub>s<sub>t</sub>and a<sub>t</sub>We have used the assumption that it is a deterministic function. As highlighted earlier, this assumption is not completely realistic since our actions influence the behavior of other road users. It takes into account reactions on the part of other agents to its actions (for example, when performing a safe cut-in, it is assumed that the car behind itself adjusts its speed to avoid hitting us from behind). , it is unrealistic to assume that all of the dynamics of other agents are modeled. The solution to this problem is to frequently reapply decisions, thereby constantly adapting our policies to parts of the environment beyond our modeling. In a sense, this can be thought of as a Markovization of the world at every step.
[0500] Detection [0501] This section describes detection states, which are descriptions of relevant information of a scene and form inputs to the driving policy module. Generally, sensing conditions include static objects and dynamic objects. Static objects are information about lanes, physical road divisions, speed constraints, right-of-way constraints, and obstructions (eg, fences blocking relevant parts of a merging road). Dynamic objects can be vehicles (e.g. bounding box, speed, acceleration), pedestrians (bounding box, speed, acceleration), traffic lights, dynamic road delimitations (e.g. cones in a construction zone), temporary traffic signs. and police actions and other obstacles on the road (eg animals, mattresses fallen from trucks, etc.).
[0502] One cannot expect to obtain an accurate sensing state s with any reasonable sensor configuration. Instead, there are sensing systems that consider the raw sensor and mapping data, denoted by x0X, and take x to create the appropriate sensing conditions.
[0503] Definition 16 (Detection System) Let S denote the area of the detection state, and let X denote the area of raw sensor and mapping data. The detection system functions<math num="166"><img file="JP7367255B2_D0166.tif" /></math>It is.
[0504]
<math num="167"><img file="JP7367255B2_D0167.tif" /></math>It is important to understand when to accept s as a reasonable approximation to s. A final way to answer this question is by examining the implications of this approximation for the performance of our driving policy in general and safety in particular. Following our safety-comfort distinction, we again distinguish between detection errors that lead to dangerous behavior and detection errors that affect aspects of ride comfort. Before going into details, the types of errors that a sensing system may make may include the following:
.False negative: The detection system misses the object .False positive: The detection system shows an object that is "not there" .Inaccurate measurement: The detection system correctly detects the object but incorrectly determines its location or velocity Inferring/inaccurate semantics: The detection system correctly detects the object but incorrectly interprets its semantic meaning, e.g. the color of a traffic light.
[0505] Comfort [0506] Recall that for a semantic action a, given that the current sensing state is s, we used Q(s,a) to denote the evaluation of a. . Our policy is the operation π(s)=argmax<sub>a</sub>Choose Q(s,a). instead of s,<math num="168"><img file="JP7367255B2_D0168.tif" /></math>, the selected semantic behavior is<math num="169"><img file="JP7367255B2_D0169.tif" /></math>become.
<math num="170"><img file="JP7367255B2_D0170.tif" /></math>If holds true,<math num="171"><img file="JP7367255B2_D0171.tif" /></math>should be accepted as a good approximation to s. However, for the true state s<math num="172"><img file="JP7367255B2_D0172.tif" /></math>The quality of is approximately optimal for some parameter ε, i.e.<math num="173"><img file="JP7367255B2_D0173.tif" /></math>As long as it holds true,<math num="174"><img file="JP7367255B2_D0174.tif" /></math>There is no problem if you choose. in this case,<math num="175"><img file="JP7367255B2_D0175.tif" /></math>We say that is εaccurate for Q. Naturally, one cannot expect a sensing system to always be εaccurate. Therefore, we also accept that the detection system will fail with some small probability δ. In this case, (borrowing Valiant's PAC learning terminology)<math num="176"><img file="JP7367255B2_D0176.tif" /></math>We say that is Probably (wp of at least 1-δ), Approximately (up to ε), Correct or PAC for short.
[0507] Several (ε, δ) pairs can be used to evaluate various aspects of the system. For example, three thresholds ε are used to represent minor mistakes, moderate mistakes and critical mistakes.<sub>1</sub><ε<sub>2</sub><ε<sub>3</sub>, and set different values of δ for each of them. This results in the following definition.
[0508] Definition 17 (PAC detection system) The pair of (accuracy, reliability) is defined as ((ε<sub>1</sub>,δ<sub>1</sub>),...,(ε<sub>k</sub>,δ<sub>k</sub>)), the sensing state region is S, the raw sensor and mapping data region is X, and the distribution over X×S is D. Let the motion space be A, let the quality function be Q:S×A|, and π:SA is π(s)argmax<sub>a</sub>Assume that Q(s,a) holds. For all i{1,...,k},<math num="177"><img file="JP7367255B2_D0177.tif" /></math>If the detection system<math num="178"><img file="JP7367255B2_D0178.tif" /></math>is probabilistically approximately true (PAC) with respect to the above parameters.
[0509] Here, the definition depends on the distribution D over X×S. It is important to emphasize that this distribution is constructed by recording data of many human drivers, not by following specific policies of our autonomous vehicles. While the former seems more appropriate, the latter requires online verification which makes the development of detection systems impractical. Since the impact of any reasonable policy on D is small, a sufficient distribution can be constructed by applying simple data augmentation techniques, followed by offline validation at every major update of the detection system. This definition is<math num="179"><img file="JP7367255B2_D0179.tif" /></math>Provide sufficient but not necessary conditions for a comfortable ride. This is not necessary because it ignores the important fact that short-term wrong decisions have little effect on ride comfort. For example, suppose there is a vehicle 100 meters away and that vehicle is slower than the host vehicle. The best decision is now to start accelerating weakly. If the detection system misses the vehicle but detects it the next time (100 milliseconds later), the difference between the two trips is not noticeable. To simplify the presentation, we ignored this issue and called for stronger conditions. Conformance to the multiframe PAC definition is conceptually simple, but more technical.
[0510] Next, a design principle obtained from the above PAC definition is derived. Recall that we have described several types of detection errors. For errors of the false negative, false positive and imprecise semantics types, the error is for an unrelated object (e.g. a left turn signal when going straight) or is captured by the δ part of the definition. Either. Therefore, we will focus on the types of mistakes that frequently occur: "inaccurate measurements".
[0511] Somewhat surprisingly, the common approach of measuring the accuracy of a sensing system by self-accuracy (i.e., by measuring the accuracy of the position of all objects relative to the host vehicle) does not guarantee a PAC sensing system. Indicates that the results are insufficient. Then, we propose another method to guarantee the PAC detection system and show how to obtain it efficiently. Let's start with some additional definitions.
[0512] For each object о in the scene, p(о),<math num="180"><img file="JP7367255B2_D0180.tif" /></math>s, the position of о in the coordinate system of the host vehicle by each of s,<math num="181"><img file="JP7367255B2_D0181.tif" /></math>shall be. Note that the distance between о and the host vehicle is ||p||.
<math num="182"><img file="JP7367255B2_D0182.tif" /></math>The additional error of<math num="183"><img file="JP7367255B2_D0183.tif" /></math>It is. for the distance between о and the host vehicle.<math num="184"><img file="JP7367255B2_D0184.tif" /></math>The relative error of is the additive error divided by ||p(о)||, i.e.<math num="185"><img file="JP7367255B2_D0185.tif" /></math>It is.
[0513] It is not realistic to require that the additional error be small for distant objects. In fact, consider о to be a vehicle at a distance of 150 meters from the host vehicle and ε to be of medium size, e.g. ε=0.1. To obtain additional accuracy, this means that the vehicle's position should be known to an accuracy of 10 cm. This is impractical for affordable sensors. On the other hand, relative accuracy requires estimating the position to 10%, which corresponds to an accuracy of 15 m. This is possible (as explained below).
[0514] For all оО, p(о) and<math num="186"><img file="JP7367255B2_D0186.tif" /></math>If the (relative) error between is at most ε, then the sensing system<math num="187"><img file="JP7367255B2_D0187.tif" /></math>positions the set of objects O in an εego-accurate manner. The following example demonstrates that εego-accurate sensing conditions for all reasonable Qs do not guarantee a PAC sensing system. In fact, consider a scenario where the host vehicle is traveling at a speed of 30 m/s and there is a stopped vehicle 150 meters ahead of it. If the vehicle is in its own lane and there is no option to change lanes without delay, it must begin to decelerate immediately at a rate of at least 3 m/s2 (otherwise it may not stop in time or (must be strongly decelerated). On the other hand, if the vehicle is on the shoulder of the road, there is no need to apply strong decelerations. p(о) is one of these cases, while<math num="188"><img file="JP7367255B2_D0188.tif" /></math>is another case, and suppose there is a difference of 5 meters between these two positions. Therefore,<math num="189"><img file="JP7367255B2_D0189.tif" /></math>The relative error of is as follows.
<math num="190"><img file="JP7367255B2_D0190.tif" /></math>
[0515] That is, the sensing system may be εego-accurate for relatively small ε values (less than 3.5% error), and also needs to be braked hard for any reasonable Q-function. The value of Q is completely different because you are confusing the situation with a situation where you do not need to apply the brakes hard.
[0516] The above example shows that εego-accuracy does not guarantee that our sensing system is a PAC. Q determines whether the PAC detection system has sufficient other characteristics. We describe a set of Q functions that have simple positioning properties that guarantee a PAC detection system. The problem with εego-accuracy is that it can lead to semantic mistakes, and in the example above,<math num="191"><img file="JP7367255B2_D0191.tif" /></math>Even when ε<3.5% and εego-accurate, it failed to assign the vehicle to the correct lane. To solve this problem, we use semantic units related to lateral position.
[0517] Definition 18 (Semantic unit) The lane center is a simple natural curve, i.e. a differentiable injective mapping l:[a,b]|3, for all at<sub>1</sub><t<sub>2</sub>For b, length<math num="192"><img file="JP7367255B2_D0192.tif" /></math>ist<sub>2</sub>-t<sub>1</sub>be equivalent to. The width of the lane is the function w:[a,b]|<sub>+</sub>It is. Point x| on the curve<sup>3</sup>The projection of is the point on the curve closest to x, i.e. t<sub>x</sub>=argmin<sub>t[a,b]</sub>The point l(t) about ||l(t)-x||<sub>x</sub>). The semantic vertical position of x with respect to the lane, is t<sub>x</sub>, and the semantic lateral position of x with respect to the lane is l(t<sub>x</sub>)/w(t<sub>x</sub>). Semantic velocity and acceleration are defined as the above first and second derivatives.
[0518] As with geometric units, we use relative error in terms of semantic vertical distance, and while the true distance is p(о),<math num="193"><img file="JP7367255B2_D0193.tif" /></math>is about some object<math num="194"><img file="JP7367255B2_D0194.tif" /></math>If we cause a semantic vertical distance of , then the relative error is<math num="195"><img file="JP7367255B2_D0195.tif" /></math>(The maximum in the denominator is the case where objects have approximately the same vertical distance (e.g., dealing with neighboring cars on different lanes). Because the semantic vertical distance is small, we add an additional error to them. This results in the following definition:
[0519] Definition 19 (Semantic unit error) Assume that the lane is l and the semantic vertical distance of the host vehicle to the lane is 0. a certain point<math num="196"><img file="JP7367255B2_D0196.tif" /></math>and the semantic horizontal and vertical distances to that point with respect to the lane are p<sub>lat</sub>(x),p<sub>lon</sub>(x). Approximate measurements<math num="197"><img file="JP7367255B2_D0197.tif" /></math><math num="198"><img file="JP7367255B2_D0198.tif" /></math>shall be. for x<math num="199"><img file="JP7367255B2_D0199.tif" /></math>The distance between and p is determined as follows<math num="200"><img file="JP7367255B2_D0200.tif" /></math>
[0520] Distances for lateral velocity and longitudinal velocity are similarly determined. Equipped with the above definition, we are ready to define the properties of Q and the corresponding sufficient conditions for the PAC detection system.
[0521] Definition 20 (Semantically-Lipschitz Q) All a, s,<math num="201"><img file="JP7367255B2_D0201.tif" /></math>about,<math num="202"><img file="JP7367255B2_D0202.tif" /></math>If holds, then the Q function is L-semantically-Lipschitz, and<math num="203"><img file="JP7367255B2_D0203.tif" /></math>, p is s with respect to the object о,<math num="204"><img file="JP7367255B2_D0204.tif" /></math>This is a measurement caused by
[0522] The following is obtained as a direct system.
[0523] Lemma 8 If Q is L-semantically-Lipschitz and the detection system<math num="205"><img file="JP7367255B2_D0205.tif" /></math>But with probability at least 1-δ,<math num="206"><img file="JP7367255B2_D0206.tif" /></math>If we generate semantic measurements such that we get<math num="207"><img file="JP7367255B2_D0207.tif" /></math>is a PAC sensing system with parameters 0, δ.
[0524] Safety [0525] This section discusses the possibility of detecting errors that can cause undesired behavior. As mentioned earlier, the policy is provably safe in the sense that it does not cause accidents for which the host AV is at fault. Such accidents can still occur due to hardware failures (eg failure of all sensors or flat tire on the highway), software failures (serious bugs on part of the module) or detection errors. Our ultimate goal is for the probability of such an event to be extremely small (10 such accidents per hour).<sup>-9</sup>probability). To put this value into perspective, the average number of hours drivers in the United States spend on the road (in 2016) is less than 300 hours. So the expectation is that you would have to live 3.3 million years to experience an accident resulting from one of these types of events.
[0526] Let us first define what is a safety-related detection error. Our policy is the value of a that maximizes Q(s,a), i.e. π(s)=argmax<sub>a</sub>Recall that we choose Q(s,a) step by step. Safety is guaranteed by setting Q(s,a)=- for all unwary actions a (see Definition 14). Therefore, the first type of safety-critical sensing error is when our sensing system causes us to choose an unsafe action. Formally,<math num="208"><img file="JP7367255B2_D0208.tif" /></math>decision by<math num="209"><img file="JP7367255B2_D0209.tif" /></math>year,<math num="210"><img file="JP7367255B2_D0210.tif" /></math>If holds true,<math num="211"><img file="JP7367255B2_D0211.tif" /></math>This can lead to serious safety mistakes. The second type of safety-critical detection error is when all operations<math num="212"><img file="JP7367255B2_D0212.tif" /></math>is unsafe according to s and standard emergency policies (e.g. applying the brakes harder) must be applied, while safe actions according to s, i.e. max<sub>a</sub>This is the case when Q(s,a)>-. This can be dangerous if you are going fast and there are cars behind you. This kind of mistake is called a safety-critical ghost.
[0527] Generally, safety-critical errors are caused by false negatives, while safety-critical ghosts are caused by false positives. Such errors can sometimes be caused by grossly inaccurate measurements, but in most cases our comfort goals are far removed from the boundaries of our definition of safety. Therefore, moderate measurement errors are unlikely to cause safety-critical errors. The probability of a safety-critical mistake is very small, e.g. 10 per hour.<sup>-9</sup>How can we ensure that it is less than From Lemma 1, without making any further assumptions, 10<sup>9</sup>There is a need to examine this system based on longer runs than hours. This is unrealistic (or at least extremely difficult); this number of hours corresponds to logging 3.3 million car drives over one year. Moreover, constructing a system that achieves such high accuracy is a major challenge. A solution to both system design and verification challenges is to utilize several subsystems, each of which is designed independently and relies on different techniques to improve individual accuracy. These systems are integrated in a way that ensures that
[0528] s<sub>1</sub>,s<sub>2</sub>,s<sub>3</sub>Suppose we want to build three subsystems (extension to more than four is easy): Each subsystem should receive a and output safe/danger. An operation that a majority of the subsystems (two in this case) accept as safe is considered safe. If no action is considered safe by at least two subsystems, the default emergency policy is applied. The performance of this fusion scheme is analyzed based on the definitions below as follows.
[0529] Definition 21 (One side c-approximate independent)<math num="213"><img file="JP7367255B2_D0213.tif" /></math>holds, then the two Bernoulli random variables r<sub>1</sub>, r<sub>2</sub>is called one side c-approximate independent.
[0530] Regarding i0{1,2,3},<math num="214"><img file="JP7367255B2_D0214.tif" /></math><math num="215"><img file="JP7367255B2_D0215.tif" /></math>Let denote Bernoulli random variables that respectively indicate whether subsystem i makes a safety-critical mistake/ghost. Similarly, e<sup>m</sup>,e<sup>g</sup>indicates a safety-critical mistake/ghost in the fusion system. For any pair ij, the random variable<math num="216"><img file="JP7367255B2_D0216.tif" /></math><math num="217"><img file="JP7367255B2_D0217.tif" /></math>is one sided c-approximate independent, and the same thing is<math num="218"><img file="JP7367255B2_D0218.tif" /></math><math num="219"><img file="JP7367255B2_D0219.tif" /></math>We use the assumption that the same applies to Before explaining why this assumption is reasonable, I would like to first analyze its implications. e<sup>m</sup>The probability of<math num="220"><img file="JP7367255B2_D0220.tif" /></math>can be bounded by
[0531] Therefore, all subsystems are<math num="221"><img file="JP7367255B2_D0221.tif" /></math>If you have<math num="222"><img file="JP7367255B2_D0222.tif" /></math>holds true. Exactly the same differential applies to safety-critical ghosting mistakes. Therefore, by applying the coupling limit, we conclude as follows.
[0532] Corollary 2 For any pair ij, the random variable<math num="223"><img file="JP7367255B2_D0223.tif" /></math><math num="224"><img file="JP7367255B2_D0224.tif" /></math>is one sided c-approximate independent, and the same thing is<math num="225"><img file="JP7367255B2_D0225.tif" /></math><math num="226"><img file="JP7367255B2_D0226.tif" /></math>Assume that this also applies to Furthermore, for every i,<math num="227"><img file="JP7367255B2_D0227.tif" /></math>as well as<math num="228"><img file="JP7367255B2_D0228.tif" /></math>Assume that holds true. Therefore, the following equation holds.
<math num="229"><img file="JP7367255B2_D0229.tif" /></math>
[0533] This system allows using significantly smaller data sets to validate the sensing system. For example, 10<sup>-9</sup>If you want to achieve a probability of a safety-critical mistake of 10<sup>9</sup>instead of taking example of degree 10<sup>5</sup>It is sufficient to take the example of degrees and test each system separately.
[0534] There may be pairs of sensors that result in uncorrelated errors. For example, in contrast to cameras, which are affected by adverse weather conditions but are less likely to be affected by metal objects, radars perform well within adverse weather conditions, but may be affected by irrelevant metal objects. It may not work. At first glance, the camera and lidar have common sources of error, for example both are affected by foggy weather, heavy rain or snow. However, cameras and lidar have different types of errors; cameras may miss objects due to bad weather, and lidar may detect ghosts due to reflections from particles in the air. Since we have distinguished the two types of errors, independence of approximations is likely to still apply.
[0535] Our definition of a safety-critical ghost requires all operations to be hazardous with at least two sensors. It is unlikely to occur even in difficult conditions (eg dense fog). The reason is that in such situations, systems affected by difficult conditions (e.g. the rider) can declare high speeds and lateral maneuvers as dangerous movements, making it extremely self-defensive. This is because the driver will be ordered to drive safely. As a result, the host AV travels very slowly, in which case even if an emergency stop is required, it is not dangerous due to the low speed of travel. Our definition thus results in the adaptation of driving style to road conditions.
[0536] Building a Scalable Sensing System [0537] The requirements for a sensing system in terms of both comfort and safety have been described. Next, we will explain a method for constructing a detection system that is scalable and meets these requirements. There are three main components of the detection system. The first component is the long range 360 degree coverage of the scene based on the camera. The three main advantages of cameras are (1) high resolution, (2) texture, and (3) price. Low price allows for scalable systems. Textures allow for understanding the semantics of the scene, including lane markings, traffic lights, pedestrian intent, and more. High resolution allows long-range detection. Furthermore, detecting lane marks and objects within the same region allows for superior semantic lateral accuracy. The two main disadvantages of cameras are (1) the information is 2D, making longitudinal distance estimation difficult, and (2) sensitivity to lighting conditions (low sun or bad weather). We overcome these difficulties using the following two components of our system.
[0538] A second component of the system determines a precise position along the target trajectory based on the position (e.g., in the image) of recognized landmarks identified in the host vehicle's environment. REM (Road Experience It is a semantic high-definition mapping technology called ``Management''. A common geometric approach to creating maps is to record a 3D point cloud (obtained by lidar) within the map creation process, and then match existing lidar points to points in the map. will give you a location on the map. This approach has several disadvantages. First, this method requires a large amount of memory per kilometer of mapping data, as many points need to be stored. This necessitates an expensive communications infrastructure. Second, not all cars may be equipped with lidar sensors, so the maps are updated very infrequently. Road changes may occur (construction zones, hazards) and the "time to reality" of lidar-based mapping solutions is long. In contrast, REM follows a semantics-based approach. The idea is to utilize a large number of vehicles equipped with cameras and software to detect semantically significant objects in the scene (lane marks, curves, poles, traffic lights, etc.). Today, many new cars are equipped with ADAS systems that can be used to create crowd-sourced maps. Since the processing is done on the vehicle side, only a small amount of semantic data should be communicated to the cloud. This allows maps to be updated very frequently in a scalable manner. In addition, autonomous vehicles can receive mapping data of small size over existing communication platforms (cellular networks). Finally, very accurate positioning on the map can be obtained based on cameras without the need for expensive lidar.
[0539] REM can be used for several purposes. First, REM gives you foresight about the static nature of the road (you can plan ahead on how to exit the highway). Second, REM provides another accurate source of all static information, which together with camera detection results in a robust view of the static part of the world. Third, REM solves the problem of lifting the 2D information of the image plane into the 3D world as follows. The map describes all of the lanes as curves in a 3D world. Localization of the own vehicle on the map allows all objects on the road to be trivially lifted from the image plane to their 3D position. This results in a positioning system that follows accuracy in semantic units. The third component of the system may be a complementary radar and lidar system. These systems can serve two purposes. First, these systems can provide extremely high levels of accuracy to augment safety. Secondly, these systems can give direct measurements on speed and distance, which further improves riding comfort.
[0540] The following section contains technical lemmas and some implementation considerations for RSS systems.
[0541] Lemma 9 For all x0 [0,0.1], 1-x>e<sup>-2x</sup>applies.
Proof f(x)=1-xe<sup>-2x</sup>shall be. Our goal is to show that 0 for x0 [0,0.1]. Note that f(0)=0, so it is sufficient to obtain f(x)0 within the above range. Specifically, f'(x)=-1+2e<sup>-2x</sup>holds true. Clearly f'(0)=1 and it is monotonically decreasing, so it is sufficient to verify that f'(0.1)>0, which can be easily done numerically ( f'(0.1)0.637).
[0542] Efficient Verification of Discretion - Hidden Objects [0543] As with non-hidden objects, it is possible to check whether providing the current command, and subsequently giving the DEP, is RSS . Therefore, assuming that the exposure time is 1, t<sub>brake</sub>unroll the future until then command DEP, which by definition is sufficient for prudence. Assuming the worst case regarding hidden objects, all t'0 [0,t<sub>brake</sub>] to find out whether an at-fault accident can occur. Use some of the worst case operations and safe distancing rules. We use an occluding object-based approach to find points of interest, i.e., compute the worst case for each occluding object. This is a definitive efficiency-driven approach; for example, a pedestrian can hide in many positions behind a car and perform many maneuvers, but a single worst-case position and There are operations that pedestrians can perform.
[0544] Next, consider the more complex case of hidden pedestrians. Consider shielded areas behind parked vehicles. The closest point within the occlusion area and the front/side of our car c can be found, for example, by their geometric properties (triangle, rectangle). Formally, the occluded region can be considered as a combination of a small number of convex regions of simple shape, each of which can be treated separately. Furthermore, a pedestrian will move from the shielded area (v<sub>limit</sub>(under the constraints of ), it is possible to rear-end the front of the car. The maximum distance a pedestrian can travel is v<sub>limit</sub>- Use being t' to obtain a simple confirmation regarding the possibility of hitting the front. Regarding a side collision, if and only if our lateral velocity exceeds μ in the direction of the collision, the path is v<sub>limit</sub>.If it is shorter than t', I would like to point out that we are responsible. What we disclose is an algorithm for verifying prudence toward hidden pedestrians, which is shown here in free pseudocode. The critical part, which is determining whether there is a possibility of an at-fault accident with a pedestrian hidden by a vehicle, is done in the simple manner described above.
<math num="230"><img file="JP7367255B2_D0230.tif" /></math>
[0545] Regarding the problem of validating simulators [0546] As discussed above, the safety of multi-agents can be difficult to statistically verify, since such verification is performed in an "online" manner. This is because it should be done. Some may argue that by building a simulator of the driving environment, driving policies can be tested in a ``laboratory.'' However, verifying that a simulator faithfully represents reality is just as difficult as verifying the policy itself. To understand why this is the case, applying the driving policy π to the simulator is<math num="231"><img file="JP7367255B2_D0231.tif" /></math>, the probability of an accident in the real world is p,<math num="232"><img file="JP7367255B2_D0232.tif" /></math>Assume that the simulator is verified in the sense that . (0 is 10<sup>-9</sup>). Next, the driving policy is replaced with π'. Ten<sup>-8</sup>Suppose that with probability π' performs a strange behavior that confuses a human driver and leads to an accident. Consistent with the simulator's very good ability to estimate the performance of the original policy π, this strange behavior may (and is more likely) not been modeled within the simulator. . This proves that even if the simulator shows that it reflects the reality of a driving policy π, it does not guarantee that it reflects the reality of another driving policy.
[0547] Lane-Based Coordinate System [0548] One simplifying assumption that can be made within the definition of RSS is that the road is composed of adjacent straight lanes of constant width. The distinction between horizontal and vertical axes as well as the ordering of vertical positions can play an important role in RSS. Furthermore, the definition of those directions is explicitly based on the shape of the lane. Transforming from a plane (global) position to a lane-based coordinate system reduces the problem to the original "constant width straight lane" case.
[0549] Assume that the center of the lane is a smooth directed curve r on the plane, and that all of its segments denoted by r(1),...,r(k) are straight lines or arcs. Note that the smoothness of a curve implies that no pair of consecutive pieces can be a straight line. Formally, the curve has a "longitudinal" parameter Y0[Y<sub>min</sub>,Y<sub>max</sub>]| in the plane, i.e. the curve is r:[Y<sub>min</sub>,Y<sub>max</sub>It is a function of the form ]|2. Continuous lane width function w:[Y that maps vertical position Y to positive lane width values<sub>min</sub>,Y<sub>max</sub>]→|<sub>+</sub>Define. For each Y, from the smoothness of r, we can determine the unit normal vector to the curve at position Y, denoted by r(Y). We naturally define the subset of points on the plane that lie within the lane as follows.
R={r(Y)+αw(Y)r(Y)|Y0[Y<sub>min</sub>,Y<sub>max</sub>],α0[±1/2]}
[0550] Informally, our goal is to construct a transformation φ of R such that for the two cars on the lane their "logical ordering" is preserved, i.e. on the curve. c<sub>r</sub>is c<sub>f</sub>If it is "behind" φ(c<sub>r</sub>)y<φ(c<sub>f</sub>)<sub>y</sub>holds true. c<sub>l</sub>is on the curve c<sub>r</sub>φ(c<sub>l</sub>)x<φ(c<sub>r</sub>)<sub>x</sub>holds true. Similar to RSS, associate the y-axis with the "vertical" axis and the x-axis with the "horizontal" axis.
[0551] To determine φ, use the assumption that if r(i) is an arc of radius ρ for all i, then the width of the lane throughout r(i) is ρ/2 . Note that this assumption applies to any realistic road. This assumption means that for all (x',y')0R, there is a unique pair such that (x',y')=r(Y')+α'w(Y')r(Y'). Y'0[Y<sub>min</sub>,Y<sub>max</sub>],α'0[±1/2] trivially implies that it exists. Next, φ:R|<sup>2</sup>can be defined as φ(x',y')=(Y',α'), and (Y',α') is (x',y')=r(Y')+α It is a unique value that satisfies 'w(Y')r(Y').
[0552] This definition captures the concept of "lateral maneuvering" within the lane's coordinate system. For example, consider a widened lane with cars driving directly above one of the lane boundaries (see Figure 23). The widening of the lane 2301 means that the car 2303 moves away from the center of the lane and therefore has a lateral speed with respect to the lane. However, this does not mean that the car 2303 performs a lateral maneuver. The definition of φ(x',y')x=α', i.e. the lateral distance to the center of the lane in units of w(Y'), indicates that the lane boundaries have a fixed lateral position of It implies having. Therefore, a vehicle that does not leave one of the lane boundaries is not considered to make any lateral movement. Finally, it turns out that φ is homomorphic. The term lane-based coordinate system refers to φ(R)=[Y<sub>min</sub>,Y<sub>max</sub>Used when discussing ]x[±1/2]. In this way, a reduction from the general lane geometry to a linear longitudinal/abscissa coordinate system was obtained.
[0553] Extension of RSS to General Road Structures [0554] This section provides a complete definition of RSS that applies to all road structures. This section deals with the definition of RSS, rather than how to efficiently ensure that policies comply with RSS. To capture arbitrary situations where multiple lane geometries exist, such as intersections, we next introduce the concept of route priority.
[0555] The second generalization deals with two-way roads where there may be two cars traveling in opposite directions. In this case, the already established definition of RSS remains valid with a minor generalization of "safe distance" to oncoming traffic. Controlled intersections (using traffic lights to direct traffic flow) can be fully addressed by route right-of-way and two-way street concepts. Unstructured roads without a clear definition of route (eg, parking lots) can also be addressed with RSS. RSS is still valid in this case; the only modification required is the method for defining the virtual route and for assigning each car to (possibly several) routes.
[0556] Route Right-of-Way [0557] To address scenarios in which one scene has several different road geometries that overlap within a particular area, we next introduce the concept of route right-of-way. . The examples shown in Figures 24A-D include roundabouts, intersections, and merging onto expressways. A method for converting a general lane geometry to a lane-based one has been described, along with a consistent meaning for the vertical and horizontal axes. Next, we address the scenario where there are multiple routes with different road geometries. As a result, when two vehicles reach an overlapping area, both vehicles will cut into the front zone of the other vehicle. This phenomenon cannot occur if the two routes have the same geometry (as in the case of two adjacent freeway lanes). Broadly speaking, the principle of root priority is that root r<sub>1</sub>, r<sub>2</sub>overlap and r<sub>1</sub>is r<sub>2</sub>If you prefer r<sub>2</sub>Enter the forward zone of a vehicle coming from r<sub>1</sub>Vehicles coming from are not considered to be cutting in.
[0558] To explain this concept formally, the fault of an accident depends on geometric properties derived from the lane coordinate system, and on worst-case assumptions that also depend on the lane coordinate system. I would like you to remember that. r the route that defines the road structure.<sub>1</sub>,...,r<sub>k</sub>shall be. As a simple example, consider the merging scenario shown in Figure 24A. two cars 2401(c<sub>1</sub>) and 2402(c<sub>2</sub>) is root r<sub>1</sub>, r<sub>2</sub>are running, and r<sub>1</sub>Assume that is the preferred route. For example, r<sub>1</sub>is the highway lane and r<sub>2</sub>Assume that is a merging lane. Since a coordinate system based on the route is determined for each route, the first consideration is that any operation can be considered within the coordinate system of any route. For example, r<sub>2</sub>If we use the coordinate system of r<sub>1</sub>Traveling in a straight line towards r<sub>2</sub>It appears to be merging to the left. One approach to the definition of RSS is that if and only if i0{1,2}, then the operation r<sub>i</sub>Each of the vehicles may be able to perform the maneuver if it is safe for them to do so. However, this is the route that performs the convergence r<sub>2</sub>It implies that c1 should be very conservative, traveling on the preferred route for c<sub>2</sub>is able to travel accurately on the route and thus win in lateral position. In this case, it is unnatural because cars on the highway have the right of way. To overcome this problem, certain areas are demarcated to define the priority of the route, and only parts of the route are considered relevant for safety.
[0559] Definition 22 (Accident liability regarding route right of way) r<sub>1</sub>, r<sub>2</sub>Suppose that are two overlapping . r<sub>1</sub>r within the vertical interval [b,e] (Fig. 25) of the coordinate system of<sub>1</sub>is r<sub>2</sub>r to express priority over<sub>1</sub>>[b,e]r<sub>2</sub>use. root r<sub>1</sub>, r<sub>2</sub>car driving above c<sub>1</sub>and c<sub>2</sub>Assume that an accident occurs between For i0{1,2}, r<sub>i</sub>If we consider the coordinate system of b<sub>i</sub>Let {1,2} indicate. The fault in the accident is as follows:.r<sub>1</sub>>[b,e]r<sub>2</sub>and r<sub>1</sub>When one of the cars is at fault for r<sub>1</sub>If the vertical axis of the system is within the interval [b,e], then the error is b<sub>1</sub>by.
.Otherwise, the negligence is b<sub>1</sub>b<sub>2</sub>by.
[0560] To illustrate this definition, consider again the example of merging onto a highway. The lines marked "b" and "e" in Figure 25 are r<sub>1</sub>>[b,e]r<sub>2</sub>Indicates the values of b and e. It thus allows cars to travel naturally on the highway, with the implication that merging cars must be safe for cars traveling on the highway. Specifically, car 2401 c<sub>1</sub>If the car drives in the middle of the priority lane with no lateral speed, then the car 2401 c<sub>1</sub>is a car driving in a non-priority lane 2402 c<sub>2</sub>Regarding the accident with, 2402 c.<sub>2</sub>at a safe distance 2401 c<sub>1</sub>Please note that you are not at fault unless you are within the zone of Note that the end result is very similar to regular RSS (this is exactly the case for a car on a straight road trying to change lanes). Note that there may be cases where the route used by another agent is unknown. For example, in FIG. 26, car 2601 may not be able to determine whether car 2602 will take route "a" or route "b." In this case, RSS can be obtained by iteratively examining all possibilities.
[0561] Two-way traffic [0562] To address two-way traffic, amendments to the definition of fault have been accomplished to sharpen the dependence on the rear/front relationship, because in such cases This is because they have slightly different meanings. Driving in the opposite vertical direction on some straight two-lane road, i.e. v<sub>1,long</sub>.v<sub>2, long</sub>Two cars c for which <0 holds true<sub>1</sub>,c<sub>2</sub>Consider. The driving direction relative to the lane may be negative in a reasonable urban scenario, such as a car veering into an oncoming lane to pass a parked truck or a car backing into a parking lot. Therefore, it is necessary to extend the definition of a safe longitudinal distance introduced in the case where negative longitudinal velocities are assumed to be unrealistic. c<sub>f</sub>c before colliding with<sub>r</sub>c has enough reaction time to apply the brakes.<sub>f</sub>If the maximum brake given by c<sub>r</sub>and c<sub>f</sub>Recall that the distance between is safe. In our case, we revisit the "worst case" of an oncoming vehicle in a slightly different way, and understand that the "worst case" is that the oncoming vehicle accelerates toward itself. Without thinking, you actually apply the brakes to avoid a collision, but think of using only some moderate braking force. To capture differences in vehicle liability, we start by defining the "correct" driving direction when one of those vehicles is clearly driving in the opposite direction.
[0563] In the RSS definition of parallel lanes, the relevant lane is defined as the one whose center is closest to the interrupt position. We can then reduce ourselves to considering this lane (or, in the case of symmetry, address the two lanes separately as in Definition 22). In the following definitions, the term "heading" refers to the arctangent (in radians) of the transverse velocity divided by the longitudinal velocity.
[0564] Definition 23 (depending on the correct driving direction (μ<sub>1</sub>,μ<sub>2</sub>,μ<sub>3</sub>)-win) c<sub>1</sub>,c<sub>2</sub>is running in the opposite direction, i.e. v<sub>1,long</sub>.v<sub>2, long</sub>Assume that <0 holds true. x their lateral position and heading relative to the lane<sub>i</sub>,h<sub>i</sub>shall be. If all of the following conditions apply, c<sub>1</sub>is the correct driving direction (μ<sub>1</sub>,μ<sub>2</sub>,μ<sub>3</sub>)-You can say you win:.|h<sub>1</sub>|<μ<sub>1</sub>, .|h2-π|<μ<sub>2</sub>,.|x<sub>1</sub>|<μ<sub>3</sub>。
[0565] The indicator for this event is denoted by WCDD(i).
[0566] In words, c1 wins if it drives near the center of the lane in the correct direction while c2 takes the opposite direction. At most one car can win, and sometimes none of the cars win. Intuitively, assume that there is a conflict in the situation discussed. It is reasonable to assign more responsibility to the losing car c1 in the correct driving direction. If the car wins with the correct driving direction, this means that a<sub>max,brake</sub>This is done by redefining .
[0567] Definition 24 (reasonable braking force) α<sub>max,brake,wcdd</sub>>0 to a<sub>max,brake</sub>be a constant less than or equal to Assume that c1 and c2 are traveling in opposite directions. RBP<sub>i</sub>For each car, denoted by c<sub>i</sub>A reasonable braking force for c<sub>i</sub>is the correct driving direction (μ<sub>1</sub>,μ<sub>2</sub>,μ<sub>3</sub>) - a if you win<sub>max,brake,wcdd</sub>, otherwise a<sub>max,brake</sub>It is.
[0568] A for winning/not winning with the correct driving direction<sub>max,brake,wcdd</sub>,a<sub>max,brake</sub>The exact value of is a constant to be defined and may depend on the type of road and lane on which each vehicle is traveling. For example, on a narrow urban road, winning in the correct driving direction may not imply much smaller braking values, and in congested traffic, someone in their own lane may clearly be driving the vehicle regardless of whether or not they deviate. We expect that the brakes will be applied with similar force. However, consider the example of a local two-way road where high speeds are permitted. When veering into oncoming lanes, it is unpredictable that a car traveling in the right direction would apply very strong braking force to avoid a collision with the host vehicle, placing more responsibility on the host vehicle than on those cars. In the case where two cars are in the same lane with one backing into a parking lot, a different constant can be defined.
[0569] The safe distance between cars traveling in opposite directions and the immediate derivation of its exact value is defined next.
[0570] Definition 25 (Safe longitudinal distance - two-way traffic) Vehicles traveling in opposite directions and both within each other's front zonec<sub>1</sub>and another car c<sub>2</sub>The longitudinal distance between is c until the reaction time ρ<sub>1</sub>,c<sub>2</sub>Any acceleration command performed by a, |a|<a<sub>max, accel</sub>About, c<sub>1</sub>and c<sub>2</sub>If it applies its reasonable braking force from time ρ to a complete stop, it is safe for time ρ and the car will not crash.
[0571] Lemma 10 c<sub>1</sub>,c<sub>2</sub>Let be the same as in Definition 25. RBP reasonable brake and acceleration commands (for each i)<sub>i</sub>,a<sub>max, accel</sub>Let the reaction time of the car be ρ. The longitudinal speed of the car is v<sub>1</sub>, v<sub>2</sub>and their length is l<sub>1</sub>, l<sub>2</sub>shall be. v<sub>i,ρ,max</sub>=|v<sub>i</sub>|+ρ.a<sub>max, accel</sub>It is determined that L=(l<sub>r</sub>+l<sub>f</sub>)/2. Therefore, the minimum safe longitudinal distance is:
<math num="233"><img file="JP7367255B2_D0233.tif" /></math>
[0572] It can be seen that the term in this sum is the maximum distance that each car will travel until it comes to a complete stop when performing the operation of Definition 25. Therefore, for a complete stop to be at a distance greater than L, the initial distance must be greater than this sum plus an additional term in L.
[0573] To determine fault/accident liability for two-way traffic scenarios, we use the same definition of RSS's point of fault for critical vertical distance as defined in Definition 25.
[0574] Definition 26 (negligence in two-way traffic) Vehicles traveling in the opposite directionc<sub>1</sub>and c<sub>2</sub>Fault in two-way traffic for an accident during a period of time is a function of the conditions at the time of fault and is determined as follows: If the time of fault is also the time of the cut-in, fault is defined as in the normal RSS definition. It will be done.
-Otherwise, for every i, at some t that occurs after the point of error, c<sub>i</sub>is at least RBP<sub>i</sub>If the brakes are not applied with the force of c.<sub>i</sub>It is in.
[0575] For example, assume a safe interrupt that occurs prior to the point of failure. For example, c<sub>1</sub>veered into the oncoming lane, c<sub>2</sub>are cutting into the zone at a safe distance. c<sub>2</sub>wins with the correct driving direction, so this distance can be very long (c<sub>2</sub>do not expect the vehicle to apply strong braking force, only reasonable braking force). Therefore, both cars are responsible for not colliding with each other. But if the interrupt was not at a safe distance, c<sub>2</sub>If the driver drives in the center of his own lane without any lateral movement, then c<sub>2</sub>use the usual definition of no-fault. Negligence is c<sub>1</sub>Only in This allows cars to drive naturally in the center of their lane without worrying about traffic potentially straying dangerously into their zone. On the other hand, safe departure into oncoming lanes is permitted, a common maneuver required in congested urban traffic. Consider the example of a car initiating a reversing parking maneuver, and the car should begin reversing while ensuring that the distance to the car behind it is safe.
[0576] Traffic Lights [0577] In scenarios involving intersections with traffic lights, a simple rule for traffic lights is: If one car's route is a green light and the other car's route is a red light, then the red light route is Some people may think that the fault lies with the person at the top. However, this is not the correct rule, especially in all cases. For example, consider the scenario shown in Figure 27. Even if car 2701 is on a route with a green light, it is unexpected for car 2701 to ignore car 2703, which is already in the intersection. The correct rule is that green light routes have priority over red light routes. There is therefore a clear traffic light-derived reduction to the route priority concept described above.
[0578] Unstructured Roads [0579] Next, consider a road where no clear geometry of the route can be defined, and first consider a scenario without any lane structure (eg, a parking lot). A method to ensure that there are no accidents requires that while all cars drive in a straight line, if a change in heading occurs, such change must be made when there are no cars nearby. That could be the case. The rationale behind this method is that cars can predict what other cars will do and behave accordingly. If other cars deviate from this prediction (by changing heading), the maneuver will be done over a sufficiently long distance so there may be enough time to correct the prediction. If there is a lane structure, smarter predictions about what other cars will do may be possible. If there is no lane structure, the car will proceed according to its current heading. Strictly speaking, this is equivalent to assigning a virtual straight route to all cars according to their heading. Then (for example, the Arc de Paris in Paris) We consider a scenario at a large unstructured roundabout (around Triomphe). Here, a sensible prediction is to assume that the car follows the roundabout geometry while maintaining its offset. Strictly speaking, this is equivalent to assigning every car a virtual arc path according to its current offset from the center of the roundabout.
[0580] together with one or more of the accident liability rules described above, such as a driving policy system (e.g., an RL system), to provide a navigation system that takes into account potential accident liability when determining the particular navigation instructions to implement. Can be implemented. Such rules may be applied during the planning phase, for example within a set of programmed instructions or within a trained model, so that suggested navigation behaviors are developed by a system that already complies with the rules. For example, the driving policy module may be responsible for, or trained using, one or more navigation rules on which the RSS is based, for example. Additionally or alternatively, RSS safety constraints may be used as a filter layer to test all proposed navigation actions proposed by the planning stage against relevant accident liability regulations to ensure that the proposed navigation actions are compliant. Can be applied. A particular action may be performed if it complies with the RSS safety constraints. Otherwise, if the proposed navigation action does not comply with the RSS safety constraints (e.g., if the proposed action could give rise to accident liability for the host vehicle under one or more of the above rules), The action is not performed.
[0581] In fact, certain implementations may include a navigation system for the host vehicle. The host vehicle may include an image capture device (eg, one or more cameras, such as any of those described above) that captures images representative of the host vehicle's environment during operation. Using the image information, the driving policy can take in multiple inputs and output planned navigation actions to achieve the host vehicle's navigation goals. Driving policies are based on various inputs (e.g. images from one or more cameras showing the host vehicle's surroundings including target vehicles, roads, objects, pedestrians, etc., outputs from lidar or radar systems, speed sensors and suspensions). A set of programmed instructions or a trained network capable of accepting outputs from sensors, etc., one or more goals of the host vehicle, e.g. information representing a navigation plan to deliver a passenger to a particular location, etc. may be included. Based on the input, the processor can identify target vehicles within the host vehicle's environment by, for example, analyzing camera images, lidar output, radar output, and the like. In some embodiments, the processor identifies a target vehicle within the host vehicle's environment by analyzing one or more inputs, such as one or more camera images, lidar output, and/or radar output. I can do it. Further, in some embodiments, the processor detects a target vehicle (e.g., analyzes one or more camera images, lidar output, and/or radar output and identifies a target vehicle based on a majority match or combination of the inputs). A target vehicle within the host vehicle's environment can be identified based on a match of a majority or combination of sensor inputs (receiving results).
[0582] Based on the information available to the driving policy module, output may be provided in the form of one or more planned navigation actions to achieve the host vehicle's navigation goals. In some embodiments, RSS safety constraints may be applied as a filter for planned navigation operations. That is, the planned navigation operation, once developed, uses at least one accident liability rule (e.g., the accident liability rule discussed above) to determine the host vehicle's potential accident liability for the identified target vehicle. ) can be tested against. As previously mentioned, if testing of the planned navigation operation against at least one accident liability rule indicates that there could be potential accident liability for the host vehicle if the planned navigation operation is performed, the processor may cause the host vehicle not to perform the planned navigation operation. On the other hand, if a test of the planned navigation operation against at least one accident liability rule indicates that no accident liability will arise for the host vehicle if the planned navigation operation is performed, the processor The host vehicle can be caused to perform navigation operations based on the navigation information.
[0583] In some embodiments, the system can test multiple potential navigation actions against at least one accident liability rule. Based on the results of the test, the system can filter those potential navigation actions into a subset of multiple potential navigation actions. For example, in some embodiments, the subset may include a potential navigation action in which a test against at least one accident liability rule indicates that accident liability would not arise for the host vehicle if the potential navigation action was performed. May include only navigation operations. The system then scores and/or prioritizes potential navigation actions that are not responsible for the accident and selects one of the navigation actions to perform, e.g. based on the optimized score or highest priority. can do. This score and/or priority may be based on one or more factors, such as, for example, potential navigation actions deemed safest, most efficient, most comfortable, etc. for passengers.
[0584] In some examples, the decision whether to perform a particular planned navigation operation may also depend on whether default emergency procedures are available in the next state after the planned operation. If DEP is available, the RSS filter can approve its planned operation. On the other hand, if DEP is not available, the next condition may be considered dangerous and the planned navigation operation may be rejected. In some embodiments, a planned navigation operation may include at least one default emergency procedure.
[0585] One advantage of the described system is that only the behavior of the host vehicle relative to a particular target vehicle needs to be considered to ensure safe operation by the vehicle. Therefore, if there is more than one target vehicle, the host vehicle's planned Operation can be tested sequentially with respect to accident liability rules. In practice, the at least one processor detects a plurality of other target vehicles within the host vehicle's environment based on an analysis of at least one image representative of the host vehicle's environment (or based on lidar information, radar information, etc.). The host vehicle may be further programmed to repeat testing of the planned navigation operation against at least one accident liability rule to identify and determine potential accident liability of the host vehicle with respect to each of the plurality of other target vehicles. If repeated testing of the planned navigation operation against at least one accident liability rule indicates that there is potential accident liability for the host vehicle if the planned navigation operation is performed, the processor It is possible to prevent the host vehicle from performing the navigation operations that are performed by the host vehicle. If repeated testing of the planned navigation operation against at least one accident liability rule indicates that accident liability would not arise for the host vehicle if the planned navigation operation was performed, the processor The host vehicle can be caused to perform navigation operations based on the navigation information.
[0586] As mentioned earlier, any of the above rules can be used as the basis for RSS safety testing. In some embodiments, the at least one accident liability rule is a tracking rule that defines a distance behind the identified target vehicle within which the host vehicle cannot proceed without potential accident liability. Contains rules. In other cases, the at least one accident liability rule includes a preceding rule that defines a distance in front of the identified target vehicle that the host vehicle cannot proceed without potential accident liability. .
[0587] The above system applies an RSS safety test to a single planned navigation operation to test compliance with the regulation that the host vehicle should not perform any operation that would make it liable for a resulting accident. However, the test can also be applied to more than one planned navigation behavior. For example, in some embodiments, the at least one processor determines two or more planned navigation actions to achieve the host vehicle's navigation goals based on applying the at least one driving policy. There are cases. In these situations, the processor may test each of the two or more planned navigation operations against at least one accident liability rule to determine potential accident liability. For each of the two or more planned navigation operations, if the test indicates that there may be potential accident liability for the host vehicle if the particular one of the two or more planned navigation operations is performed; The processor may prevent the host vehicle from performing a particular one of the planned navigation operations. On the other hand, for each of the two or more planned navigation operations, the test shows that if a particular one of the two or more planned navigation operations is performed, no accident liability will arise for the host vehicle. If so, the processor may identify a particular one of the two or more planned navigation operations as a viable candidate for implementation. The processor then selects a navigation operation to perform from among viable candidates for implementation based on the at least one cost function and causes the host vehicle to perform the selected navigation operation. I can do it.
[0588] The implementation of RSS involves the determination of the relative potential liability for an accident between a host vehicle and one or more target vehicles, and therefore the system, along with testing planned navigation operations for safety compliance. can track the accident liability potential of vehicles encountered. For example, not only may the system potentially be able to avoid performing actions where the resulting accident would result in liability for the host vehicle, but the host vehicle system may also be able to track one or more target vehicles; It may also be possible to identify and track which accident liability rules are being violated by those target vehicles. In some embodiments, an accident liability tracking system for a host vehicle receives from an image capture device at least one image representative of the environment of the host vehicle, and analyzes the at least one image to determine the environment of the host vehicle. and at least one processing device programmed to identify a target vehicle within the vehicle. Based on the analysis of the at least one image, the processor may include programming to determine one or more characteristics of the identified target vehicle's navigation condition. Vehicle speed, proximity to the center of the lane, lateral speed, direction of travel, distance from the host vehicle, heading or others that may be used to determine potential accident liability based on any of the above rules. The navigation state may include various operating characteristics of the target vehicle, such as any parameters of the vehicle. The processor combines the determined one or more characteristics of the identified target vehicle's navigation state into at least one accident liability rule (e.g., winning lateral speed, directional right-of-way, center-of-lane Winning by distance, chasing distance or leading distance, cutting in, etc.) can be compared to any of the above rules). Based on the comparison of the condition with one or more rules, the processor can store at least one value indicative of potential accident liability on the part of the identified target vehicle. In the event of an accident, the processor may provide an output of the at least one stored value (eg, via any suitable data interface, wired or wireless). Such an output can be provided, for example, after an accident between a host vehicle and at least one target vehicle, and the output can be used to indicate accident responsibility, or to provide an accident responsibility indication. I can do it.
[0589] The at least one value indicative of potential accident liability may be stored at any suitable time and under any suitable conditions. In some embodiments, if it is determined that the host vehicle is unable to avoid a collision with the identified target vehicle, the at least one processing device assigns and stores a collision liability value for the identified target vehicle. I can do it.
[0590] Accident liability tracking functionality is not limited to a single target vehicle, but rather can be used to track potential accident liability of multiple target vehicles encountered. For example, the at least one processing device is configured to detect a plurality of target vehicles within the host vehicle's environment, determine navigation state characteristics for each of the plurality of target vehicles, and determine a respective navigation state for each of the target vehicles. and determining and storing a value indicative of potential accident liability of one side of each of a plurality of target vehicles based on a comparison of the characteristics with at least one accident liability rule. As mentioned above, accident liability rules that can be used as a basis for tracing liability may include any of the above rules or any other suitable rules. For example, the at least one accident liability rule may include a lateral speed rule, a lateral position rule, a driving direction right-of-way rule, a traffic light-based rule, a traffic sign-based rule, a route right-of-way rule, and the like. Accident liability tracking functionality may also be combined with safe navigation based on considering RSS (e.g., whether any action of the host vehicle incurs potential liability for a resulting accident).
[0591] In addition to navigation based on considering accident liability according to the RSS, navigation can also be based on the vehicle's navigation state and whether a particular future navigation state is considered safe (as detailed above, e.g. Consideration may also be given to determining whether a DEP exists so that an accident can be avoided or any resulting accident is not considered the responsibility of the host vehicle. The host vehicle can be controlled and navigated from safe to safe conditions. For example, in any particular state, a driving policy may be used to generate one or more planned navigation actions and determine whether the future predicted state corresponding to each planned action provides a DEP. By making a decision, you can test their behavior. If a DEP is provided, one or more planned navigation operations that provide a DEP may be considered safe and may be eligible for implementation.
[0592] In some embodiments, a navigation system for a host vehicle receives from an image capture device at least one image representing an environment of the host vehicle and a planned navigation to achieve a navigation goal of the host vehicle. determining a behavior based on at least one driving policy; analyzing the at least one image to identify a target vehicle in the host vehicle's environment; and determining a potential accident of the host vehicle with respect to the identified target vehicle. To determine liability, testing the planned navigation move against at least one accident liability rule and testing the planned navigation move against at least one accident liability rule determines whether the planned navigation move is performed. If this indicates that there could be potential accident liability for the host vehicle if the host vehicle at least one processing device programmed to: cause the host vehicle to perform the planned navigation action if the planned navigation action indicates that accident liability would not arise for the host vehicle if the planned navigation action is performed; may include.
[0593] In some embodiments, a navigation system for a host vehicle receives at least one image representing an environment of the host vehicle from an image capture device and at least a plurality of potential navigation operations for the host vehicle. determining based on a driving policy; analyzing at least one image to identify a target vehicle within the host vehicle's environment; and determining potential accident liability of the host vehicle with respect to the identified target vehicle. testing a plurality of potential navigation actions against at least one accident liability rule; and selecting one of the potential navigation actions, the one of the potential navigation actions For one, the test involves selecting one of the potential navigation actions and indicating that no accident liability would arise for the host vehicle if the selected potential navigation action was taken. and causing the host vehicle to perform potential navigation operations. In some examples, the selected potential navigation behavior is a subset of the multiple potential navigation behaviors, and for that subset of the multiple potential navigation behaviors, the test A selection may be made from a plurality of potential subsets of navigation operations indicating that if any of the subsets of operations were performed, no accident liability would arise for the host vehicle. Furthermore, in some examples, the selected potential navigation actions may be selected according to a scoring parameter.
[0594] In some embodiments, a system for navigating a host vehicle includes receiving at least one image representing an environment of the host vehicle from an image capture device and a system for navigating a host vehicle. and determining navigation operations based on the at least one driving policy. The processor is configured to analyze the at least one image to identify a target vehicle within the host vehicle's environment and to determine the distance between the host vehicle and the target vehicle that will occur if the planned navigation operation is performed. determining a distance of a next state; determining a current maximum braking capacity of the host vehicle and a current speed of the host vehicle; determining a current speed of the target vehicle; and at least one recognition of the target vehicle. Assuming a maximum braking capacity of the target vehicle based on the characteristics determined, and given the maximum braking capacity of the host vehicle and the current speed of the host vehicle, the current speed of the target vehicle and the assumed maximum of the target vehicle. performing a planned navigation operation if the host vehicle can be stopped within a stopping distance that is less than a travel distance of the target vehicle determined based on braking capability plus a determined next state distance; You can also do things. The stopping distance may further include the distance the host vehicle travels during the no-brake reaction time.
[0595] The recognized characteristics of the target vehicle based on which the target vehicle's maximum braking capability is determined may include any suitable characteristics. In some embodiments, the characteristics include vehicle type (e.g., motorcycle, car, bus, truck, each of which may be associated with a different braking profile), vehicle size, and expected or known vehicle weight. , the model of the vehicle (which may be used, for example, to determine known braking capabilities), etc.
[0596] In some cases, safe condition determinations may be made for more than one target vehicle. For example, in some cases, the determination of a safe condition (based on distance and braking ability) may be based on two or more identified target vehicles preceding the host vehicle. Such a determination may be particularly useful when information about what is in front of the lead target vehicle is not available. In this case, in order to determine the safe condition, safe distance and/or available DEP, it can be assumed that the detectable lead vehicle experiences an imminent collision with a stationary or nearly stationary obstacle. , so that a trailing target vehicle may reach a stop faster than its own braking profile allows (e.g., a second target vehicle may collide with a first leading vehicle, resulting in the expected maximum (can reach a stop faster than braking conditions). In that case, it may be important to base the determination of safe conditions, safe pursuit distance, and DEP on the position of the identified lead target vehicle relative to the host vehicle.
[0597] In some embodiments, such a safe state to safe state navigation system includes at least one processing device programmed to receive at least one image representative of the host vehicle's environment from an image capture device. may include. Again, as with other embodiments, the image information captured by an image capture device (eg, a camera) can be supplemented with information obtained from one or more other sensors, such as a lidar or radar system. In some embodiments, the image information used for navigation may even originate from a lidar or radar system rather than from an optical camera. The at least one processor can determine a planned navigation operation to achieve a navigation goal of the host vehicle based on the at least one driving policy. The processor analyzes at least one image (e.g., obtained from a camera, radar, lidar or any other device from which an image of the environment of the host vehicle can be obtained based on optics, range maps, etc.) and the host vehicle. A first target vehicle in front of the vehicle and a second target vehicle in front of the first target vehicle can be identified. The processor may then determine a next state distance between the host vehicle and the second target vehicle that would occur if the planned navigation operation was performed. The processor may then determine the host vehicle's current maximum braking capacity and the host vehicle's current speed. The processor controls the host vehicle within a stopping distance that is less than the determined next state distance between the host vehicle and the second target vehicle, given the host vehicle's maximum braking capacity and the host vehicle's current speed. If the navigation operation can be stopped, the planned navigation operation can be performed.
[0598] That is, if there is sufficient distance between the preceding visible target vehicle and the host vehicle to stop without collision or without a collision attributable to the host vehicle, the host vehicle If the vehicle's processor determines, and assuming that the preceding visible target vehicle suddenly comes to a complete stop at any instant, the host vehicle's processor can perform the planned navigation action. On the other hand, if there is not enough space to stop the host vehicle without a collision, the planned navigation maneuver may not take place.
[0599] Additionally, in some embodiments, the distance of the next state may be used as a benchmark, while in other instances another distance value may be used to determine whether to perform the planned navigation action. obtain. Similar to what was discussed above, in some cases the actual distance the host vehicle may have to stop to avoid a collision may be less than the predicted next state distance. . For example, if a leading visible target vehicle is followed by one or more other vehicles (the first target vehicle in the example above), the actual predicted required stopping distance will be It is calculated by subtracting the length of the target vehicle chasing the visible target vehicle in front from the distance. Assume that if the leading visible target vehicle suddenly stops, the trailing target vehicle will collide with the leading visible target vehicle and as a result those vehicles also have to be avoided by the host vehicle to avoid a collision. can do. Therefore, the host vehicle's processor evaluates the distance of the next state minus the total length of any intervening target vehicles between the host vehicle and the preceding visible/detected target vehicle to determine whether the collision will occur. It can be determined whether there is sufficient space to stop the host vehicle under maximum braking conditions without any brakes.
[0600] In other embodiments, a benchmark distance for evaluating a collision between a host vehicle and one or more preceding target vehicles may be greater than a predicted next state distance. For example, in some cases the leading visible/detected target vehicle may stop quickly, but not immediately, thereby causing the leading visible/detected target vehicle to stop a short distance after the anticipated collision. Moving. For example, if the vehicle collides with a parked car, the colliding vehicle may continue to travel some distance until it comes to a complete stop. The expected post-crash travel distance may be less than the expected or determined minimum stopping distance for the relevant target vehicle. Thus, in some cases, the host vehicle's processor may increase the distance of the next state when evaluating whether to perform a planned navigation operation. For example, when making this decision, the distance for the next state may be increased by 5%, 10%, 20%, etc. to accommodate a reasonable distance that the leading/visible target vehicle may travel after a hypothetical impending collision. or can be supplemented with a predetermined fixed distance (10m, 20m, 50m, etc.).
[0601] In addition to increasing the distance of the next state by the assumed distance value during the evaluation, to account for the distance traveled after a collision by a preceding visible/detected target vehicle; For any target vehicle following a visible/detected target vehicle (which may be assumed to have multiple collisions with a preceding visible/detected vehicle after it suddenly stops), both the length of The state distance can be modified.
[0602] A decision on whether to perform a planned navigation maneuver between the host vehicle and the preceding visible/detected target vehicle (post-collision movement of the preceding visible/detected target vehicle and/or the preceding visible/detected target vehicle) In addition to basing the distance on the next state (modified by taking into account the length of the vehicle following the visible/detected target vehicle), the host vehicle may It is possible to continue to take into account the braking capabilities of multiple preceding vehicles. For example, the host vehicle's processor may determine whether the host vehicle and a first target vehicle (e.g., a target following a leading visible/detected target vehicle) will occur if a planned navigation operation is performed. determining a current speed of the first target vehicle; and determining a current speed of the first target vehicle based on at least one recognized characteristic of the first target vehicle. Assuming a maximum braking capacity of the target vehicle and given the maximum braking capacity of the host vehicle and the current speed of the host vehicle, the current speed of the first target vehicle and the assumed maximum of the first target vehicle The host within a stopping distance that is less than the travel distance of the first target vehicle determined based on braking capability plus the determined next state distance between the host vehicle and the first target vehicle. If the vehicle cannot be stopped, the planned navigation operation may continue to not be performed. Again, similar to the example above, the recognized characteristics of the first target vehicle may include vehicle type, vehicle size, vehicle model, etc.
[0603] In some cases, the host vehicle (eg, due to the actions of other vehicles) may determine that a collision is imminent and unavoidable. In that case, the host vehicle's processor may be configured to select navigation operations (if any) in which the resulting collision does not create liability for the host vehicle. Additionally or alternatively, the host vehicle's processor may cause less potential damage to the host vehicle or cause less potential damage to the target than the current trajectory or compared to one or more other navigation operations. The navigation behavior may be configured to select which navigation behavior to provide. Additionally, in some cases, the host vehicle's processor may select navigation operations based on consideration of the type of object or objects that are expected to collide. For example, if a first navigation maneuver faces a collision with a parked car, or a second navigation maneuver faces a collision with a stationary object, a maneuver that causes less potential damage to the host vehicle (e.g. , actions that result in a collision with a parked car). Less potential damage to the host vehicle if the first navigation operation faces a collision with a car moving in a similar direction to the host vehicle, or the second navigation operation faces a collision with a parked car. (e.g., an action that results in a collision with a moving car). Provide an optional alternative to a collision with a pedestrian if a collision with a pedestrian is faced as a result of the first navigation action or a collision with any other object is faced in the second navigation action. You can choose the action.
[0604] In practice, a system for navigating a host vehicle includes receiving from an image capture device at least one image (e.g., a visible image, a lidar image, a radar image, etc.) representative of the host vehicle's environment; between the host vehicle and the one or more objects based on the analysis of the at least one image and the indicator of the current navigation state of the host vehicle; at least one processing device programmed to: determine that a collision is unavoidable; The processor may evaluate available alternatives. For example, the processor determines a first planned navigation operation for the host vehicle that involves an expected collision with a first object based on the at least one driving policy and an expected collision with a second object. and a second planned navigation operation for the host vehicle. The first planned navigation motion and the second planned navigation motion may be tested against at least one accident liability rule to determine potential accident liability. If testing the first planned navigation operation against at least one accident liability rule indicates that there could be potential accident liability for the host vehicle if the first planned navigation operation is performed, the processor , the host vehicle may not perform the first planned navigation operation. If the test of the second planned navigation operation against the at least one accident liability rule indicates that accident liability would not arise for the host vehicle if the second planned navigation operation is performed, the processor , may cause the host vehicle to perform a second planned navigation operation. Objects may include other vehicles or non-vehicle objects (eg, road debris, trees, poles, signs, pedestrians, etc.).
[0605] The drawings and discussion below provide examples of various scenarios that may occur when navigating and implementing the disclosed systems and methods. In these examples, the host vehicle may avoid performing certain actions that would result in the host vehicle being attributable to the resulting accident if performed.
[0606] Figures 28A and 28B illustrate the following example scenario and rules. As shown in FIG. 28A, the area surrounding vehicle 2804 (e.g., a target vehicle) is a minimum safe distance zone for vehicle 2802 (e.g., a host vehicle) traveling in a lane at a distance behind vehicle 2804. represents. According to one rule consistent with the disclosed embodiments, to avoid an accident attributable to vehicle 2802, vehicle 2802 must maintain a minimum safe distance by remaining within the area surrounding vehicle 2802. . In contrast, if vehicle 2804 applies the brakes as shown in FIG. 28B, vehicle 2802 will be at fault if an accident occurs.
[0607] FIGS. 29A and 29B illustrate an example of a fault in an interrupt scenario. In these scenarios, the safety zone around vehicle 2902 determines the attribution of the interrupt operation. As shown in FIG. 29A, vehicle 2902 has cut in front of vehicle 2904, violating the safe distance (as indicated by the area surrounding vehicle 2904), and therefore has an attribution. As shown in FIG. 29B, vehicle 2902 cuts in front of vehicle 2904, but maintains a safe distance in front of vehicle 2904.
[0608] FIGS. 30A and 30B illustrate an example of a fault in an interrupt scenario. In these scenarios, the safety zone around vehicle 3004 determines whether vehicle 3002 is at fault. In FIG. 30A, vehicle 3002 is driving behind vehicle 3006 and changes lanes to the lane in which target vehicle 3004 is moving. In this scenario, vehicle 3002 violates the safe distance and is therefore attributable if an accident occurs. In Figure 30B, vehicle 3002 cuts in behind vehicle 3004 and maintains a safe distance.
[0609] FIGS. 31A to 31D show examples of mistakes in a drift scenario. In FIG. 31A, this scenario begins with a slight lateral maneuver by vehicle 3104 cutting into a wide zone of vehicle 3102. In FIG. 31B, vehicle 3104 continues to cut into the normal zone of vehicle 3102, violating the safe distance zone. When an accident occurs, vehicle 3104 is at fault. In FIG. 31C, vehicle 3104 maintains its initial position while "forcing" vehicle 3102 to move laterally and violate the normal safety distance zone. When an accident occurs, vehicle 3102 is at fault. In FIG. 31B, vehicle 3102 and vehicle 3104 move laterally toward each other. If an accident occurs, the fault is shared by both vehicles.
[0610] FIGS. 32A and 32B illustrate an example of a fault in a two-way traffic scenario. In FIG. 32A, vehicle 3202 has passed vehicle 3206 and vehicle 3202 is performing an interrupt maneuver to maintain a safe distance from vehicle 3204. When an accident occurs, vehicle 3204 is at fault for not applying the brakes with reasonable force. In FIG. 32B, vehicle 3202 cuts in without maintaining a safe vertical distance from vehicle 3204. When an accident occurs, vehicle 3202 is at fault.
[0611] FIGS. 33A and 33B illustrate an example of a fault in a two-way traffic scenario. In Figure 33A, vehicle 3302 has drifted into the path of oncoming vehicle 3204, maintaining a safe distance. When an accident occurs, vehicle 3204 is at fault for not applying the brakes with reasonable force. In FIG. 33B, vehicle 3202 has drifted into the path of oncoming vehicle 3204, violating the safe longitudinal distance. When an accident occurs, vehicle 3202 is at fault.
[0612] FIGS. 34A and 34B illustrate an example of a fault in a route priority scenario. In Figure 34A, vehicle 3402 ignores a stop sign. Fault is attributed to vehicle 3402 for not respecting the right of way given to vehicle 3404 by the traffic light. In Figure 34B, vehicle 3402 does not have the right of way, but was already entering the intersection when vehicle 3404's light turned green. If vehicle 3404 collides with vehicle 3402, vehicle 3404 is at fault.
[0613] FIGS. 35A and 35B illustrate an example of a fault in a route priority scenario. In FIG. 35A, vehicle 3502 is backing into the path of approaching vehicle 3504. Vehicle 3502 is performing an interrupt operation to maintain a safe distance. When an accident occurs, vehicle 3504 is at fault for not applying the brakes with reasonable force. In Figure 35B, vehicle 3502 cuts in without maintaining a safe vertical distance. When an accident occurs, vehicle 3502 is at fault.
[0614] FIGS. 36A and 36B illustrate an example of a fault in a route priority scenario. In FIG. 36A, vehicle 3602 and vehicle 3604 are traveling in the same direction, while vehicle 3602 crosses the path of vehicle 3604 and turns left. Vehicle 3602 is performing an interrupt operation to maintain a safe distance. When an accident occurs, vehicle 3604 is at fault for not applying the brakes with reasonable force. In Figure 36B, vehicle 3602 cuts in without maintaining a safe vertical distance. When an accident occurs, vehicle 3602 is at fault.
[0615] FIGS. 37A and 37B illustrate an example of a fault in a route priority scenario. In FIG. 37A, vehicle 3702 wants to make a left turn, but must yield to an oncoming vehicle 3704. Vehicle 3702 violates a safe distance relative to vehicle 3704 by turning left. Vehicle 3702 is at fault. In FIG. 37B, vehicle 3702 is turning left and maintaining a safe distance from vehicle 3704. When an accident occurs, vehicle 3704 is at fault for not applying the brakes with reasonable force.
[0616] FIGS. 38A and 38B illustrate an example of a fault in a route priority scenario. In FIG. 38A, vehicle 3802 and vehicle 3804 are traveling straight, and vehicle 3802 has a stop sign. Vehicle 3802 enters the intersection and violates the safe distance relative to vehicle 3804. Vehicle 3802 is at fault. In FIG. 38B, vehicle 3802 is entering an intersection while maintaining a safe distance from vehicle 3804. When an accident occurs, vehicle 3804 is at fault for not applying the brakes with reasonable force.
[0617] FIGS. 39A and 39B illustrate an example of a fault in a route priority scenario. In Figure 39A, vehicle 3902 wants to turn left but must yield to vehicle 3904 coming from the right. Vehicle 3902 enters the intersection and violates the right of way and safe distance from vehicle 3904. Vehicle 3902 is at fault. In FIG. 39B, vehicle 3902 is entering an intersection while maintaining the right of way and a safe distance from vehicle 3904. When an accident occurs, vehicle 3904 is at fault for not applying the brakes with reasonable force.
[0618] FIGS. 40A and 40B illustrate an example of a fault in a traffic light scenario. In FIG. 40A, vehicle 4002 ignores a red light. Fault is attributed to vehicle 4002 for not respecting the right of way given to vehicle 4004 by the traffic light. In FIG. 40B, vehicle 4002 does not have the right of way, but vehicle 4002 was already entering the intersection when vehicle 4004's light turned green. If vehicle 4004 collides with vehicle 4002, vehicle 4004 is at fault.
[0619] FIGS. 41A and 41B illustrate an example of a fault in a traffic light scenario. Vehicle 4102 is turning left across the path of oncoming vehicle 4104. Vehicle 4104 has priority. In FIG. 41, vehicle 4102 has turned left, violating a safe distance relative to vehicle 4104. Fault is attributed to vehicle 4102. In FIG. 41B, vehicle 4102 is turning left and maintaining a safe distance from vehicle 4104. When an accident occurs, vehicle 4104 is at fault for not applying the brakes with reasonable force.
[0620] FIGS. 42A and 42B illustrate an example of a fault in a traffic light scenario. In FIG. 42A, vehicle 4202 is turning right and cutting into the path of vehicle 4204, which is traveling straight. Turning right at a red light is assumed to be a legal operation, but because vehicle 4202 violates the safe distance from vehicle 4204, vehicle 4204 has the right of way. Fault is attributed to vehicle 4202. In FIG. 42B, vehicle 4202 is turning right and maintaining a safe distance from vehicle 4204. When an accident occurs, vehicle 4204 is at fault for not applying the brakes with reasonable force.
[0621] FIGS. 43A-43C illustrate examples of vulnerable road user (VRU) scenarios. Accidents with animals or VRUs where the vehicle performs the maneuver are treated as a variant of interruption and, with some exceptions, the default fault lies with the vehicle. In FIG. 43A, a vehicle 4302 has cut into the path of the animal (or VRU) while ensuring a safe distance and avoiding an accident. In FIG. 43B, vehicle 4302 has cut into the path of the animal (or VRU), violating the safe distance. Fault is attributed to vehicle 4302. In Figure 43C, vehicle 4302 recognizes the animal and stops, giving the animal sufficient time to stop. If an animal collides with a car, the animal is at fault.
[0622] FIGS. 44A-44C illustrate examples of vulnerable road user (VRU) scenarios. In FIG. 44A, vehicle 4402 turns left at an intersection with a traffic light and encounters a pedestrian in a crosswalk. Vehicle 4402 has a red light, and VRU has a green light. There is a cause attributable to vehicle 4402. In Figure 44B, vehicle 4402 has a green light and the VRU has a red light. If the VRU enters a crosswalk, the VRU is at fault. In Figure 44C, vehicle 4402 has a green light and the VRU has a red light. If the VRU was already in the crosswalk, vehicle 4402 is at fault.
[0623] FIGS. 45A-45C illustrate examples of vulnerable road user (VRU) scenarios. In Figure 45A, vehicle 4502 turns right and encounters a bicycle. The bicycle light is green. There is a cause attributable to vehicle 4502. In Figure 45B, the bicycle is at a red light. If a bicycle enters an intersection, the bicycle is at fault. In Figure 45C, the bicycle is at a red light, but has already entered the intersection. There is a cause attributable to vehicle 4502.
[0624] FIGS. 46A-46D illustrate examples of vulnerable road user (VRU) scenarios. With some exceptions, the car is by default at fault for an accident with a VRU if the vehicle is not operated. In Figure 46A, one must always ensure that the vehicle 4602 maintains a safe distance to ensure that an accident with the VRU can be avoided. In Figure 46B, vehicle 4602 is at fault if it does not maintain a safe distance. In FIG. 46C, vehicle 4602 is at fault if it does not maintain a speed slow enough to avoid a collision with a VRU, which may be obscured by vehicle 5604, or if it travels above the legal speed limit. In FIG. 46D, another scenario in which the VRU is potentially obscured by vehicle 4604, if vehicle 4602 maintains a sufficiently slow speed but the VRU's speed exceeds a reasonable threshold, then the VRU is at fault.
[0625] As disclosed herein, RSS defines a framework for multi-agent scenarios. The host is at fault for accidents with static objects, departure from the road, loss of control or vehicle failure. RSS prescribes prudent operation to avoid accidents with other objects unless they come dangerously into the path of the host (in which case the fault lies with that object). In case of a definite collision where the target is at fault, the host applies its own brakes. The system may only consider evasive maneuvers if the maneuver is "prudent" (certain that it will not cause another accident).
[0626] Non-collision incidents include accidents caused by vehicle fires, potholes, falling objects, and the like. In these cases, a host of potholes and falling objects can be classified as a "static object" scenario, assuming that the "static object" becomes visible at a safe distance or that careful avoidance maneuvers exist. Except in avoidable scenarios, the fault can be on the host by default. In a multi-agent scenario where the host vehicle is stationary, the host is not at fault. In this case, the target is essentially performing a dangerous interruption. For example, if a bicycle hits a stationary car, the host is not at fault.
[0627] The RSS also includes guidelines for assigning fault where roads are not clearly structured, such as parking lots with no lane markings or wide roundabouts. In these unstructured road scenarios, each vehicle's deviation from its own path is examined to determine whether each vehicle has given enough distance to allow other objects in the area to make adjustments. Fault is assigned by determining.
[0628] Further details on Responsibility Sensitive Safety (RSS) [0629] As discussed above, RSS is used to ensure that vehicles implementing RSS are not involved in accidents of their own making. Provide a set of possible formulas. As a result, in some implementations, the RSS may set an envelope of extreme values (eg, minimum safe longitudinal distances) that ensure that a vehicle implementing the RSS will not be involved in an accident for which it is at fault. In some embodiments, the system may implement a modified RSS, and such modified RSS may include a predetermined operating envelope that may be larger than the RSS protocols described above. Such a modified RSS may, in at least some situations, provide an increased level of safety and reduce the probability that a vehicle employing the RSS may be attributable. Such a modified RSS system is described in more detail below, for example with respect to the described Comfort RSS system.
[0630] As discussed above, RSS may assume that the host vehicle brakes at its maximum braking capacity to avoid collision with the target vehicle, VRU, or another object. RSS determines the safe longitudinal distance between the host vehicle, the speed of the host vehicle and the target vehicle as described above, the maximum braking capacity of the host vehicle and the target vehicle, and the maximum of the host vehicle over the reaction time of the host vehicle. A buffer zone can be determined whose size is based on acceleration capability. As shown in the examples in Figures 28A to 46D, if the host vehicle comes within a distance less than this safe distance (e.g., the RSS distance), the host vehicle will brake ( (at least in some cases) may not be able to stop without contacting the target vehicle. Applying maximum brakes may be considered an extreme response, especially if such application is sudden, and such a response should be reserved for cases where it is unavoidable. Under certain circumstances, applying maximum braking may not be comfortable for passengers, may imply tailing vehicles, and may cause excessive wear on vehicle hardware (tires, brake pads, etc.) etc. are possible. Therefore, to avoid maximum braking scenarios and potential contact with the target vehicle, RSS safe distance-based driving policies may be employed at least under certain circumstances or in some implementations to ensure a safe vertical distance to the target vehicle. Directional distance can be maintained.
[0631] As stated above, the RSS distance is the maximum possible range that the host vehicle may experience during the period between a sensing event and the host vehicle's reaction to the sensing event (i.e., the reaction time associated with the host vehicle). It may contain components that depend on acceleration capabilities. For example, RSS may take into account the delay between the period during which a target vehicle, VRU, or other object is detected by the host vehicle and the time when the host vehicle begins to brake or perform another navigation maneuver. can. For example, there may be a few milliseconds (or approximately a few milliseconds) between the time the host vehicle detects that the target vehicle is braking and the time the host vehicle begins to brake or initiate another avoidance maneuver. ) is possible. Further details regarding RSS and RSS safety distances are explained above.
[0632] Definition 27 (RSS safety distance) car c<sub>r</sub>and c<sub>r</sub>Another car within the forward zone of c<sub>f</sub>The vertical distance between c<sub>f</sub>The maximum α<sub>max,brake</sub>For any brake of c<sub>r</sub>is the maximum α during the response time p<sub>max, accel</sub>then accelerates at its maximum brake α from point p until a complete stop<sub>max,brake</sub>If you apply the brakes at and do not collide with cf, it is safe in terms of response time p.
[0633] Lemma 11 below calculates an example RSS safety distance according to this embodiment for a host vehicle following a target vehicle. The RSS safety distance may include the assumed maximum braking force of the target vehicle and the determined current speed.
[0634] Lemma 11 The host vehicle behind the target vehicle cf on the vertical axis is c<sub>r</sub>shall be. The maximum braking force of the vehicle is α<sub>max,brake</sub>and the maximum acceleration is α<sub>max, accel</sub>shall be. Let v be the current speed of the vehicle. host vehicle c<sub>r</sub>Let p be the response time of Therefore, host vehicle c<sub>r</sub>The RSS safety distance for is given by the following formula.
<math num="234"><img file="JP7367255B2_D0234.tif" /></math>
[0635] Proof RSS safety distance is maximum braking force α<sub>max,brake</sub>may include calculating the minimum distance required for the vehicle to stop if the brakes are applied at This minimum distance can be calculated in any manner described above for RSS, and the formula used in Lemma 11 is for illustration only. For example, the minimum distance required to stop can be calculated using Lemma 2. Lemma 11 provides an example of the modifications needed to account for accelerations that may occur during the response period p. host vehicle c<sub>r</sub>The current speed of v<sub>1</sub>and target vehicle c<sub>f</sub>Let v2 be the current speed of . Therefore, for host vehicle cr, RSS<sub>Distance</sub>is the maximum acceleration force α over the response period p<sub>max, accel</sub>The host vehicle c after accelerating with<sub>r</sub>and multiplying that speed by the response period p to determine the maximum distance traveled during the response period p. The maximum distance traveled during the response period p is as follows.
d<sub>max,p</sub>=(v<sub>1</sub>+(v<sub>1</sub>+pα<sub>max, accel</sub>))p
[0636] The speed of the host vehicle used to determine the safe distance may correspond to the maximum possible speed of the host vehicle after accelerating over its response period p. For example, let the current speed of the host vehicle be v1, let the response period be p, and let the maximum acceleration of the host vehicle be α<sub>max, accel</sub>Then, the maximum possible speed v of the host vehicle after the response period p<sub>max,p</sub>is as follows.
v<sub>max,p</sub>=v<sub>1</sub>+pα<sub>max, accel</sub>
[0637] In some embodiments, the RSS safety distance is reliable even when both the host vehicle and the target vehicle come to a stop from their current speed at maximum braking capacity (and after the host vehicle's maximum acceleration over its own reaction time). It may also contain ingredients for making it. Such components may include a minimum approach distance component. The minimum approach distance may correspond to the minimum distance component in the RSS safety distance calculation, so that even if the host vehicle and target vehicle apply maximum braking while stationary (and after reaction time of the host vehicle at maximum acceleration) , the host vehicle stops at least a minimum distance from the target vehicle, VRU, or other object being detected. For example, if a host vehicle is driving behind a target vehicle and the target vehicle begins to brake with its maximum braking force, the host vehicle with RSS will respond by braking with its maximum braking force. be able to. If the RSS safety distance includes a minimum approach distance component, the host vehicle may stop at least a minimum distance from the target vehicle. The minimum distance may be predetermined (e.g., a minimum distance of 0.5m, 1m, 2m, etc.) or any factor consistent with this disclosure (e.g., the speed relative to the host vehicle and/or target vehicle, the detected weather conditions, user preferences, etc.). The following formula for RSS safe distance is the minimum distance d<sub>min</sub>including.
<math num="235"><img file="JP7367255B2_D0235.tif" /></math>
[0638] FIGS. 47A and 47B further illustrate the concept of RSS safety distance when the host vehicle is driving behind the target vehicle. For example, FIG. 47A shows an RSS safety distance 4702 between a host vehicle 4704 and a target vehicle 4706. Even in the worst case scenario, where the target vehicle 4706 brakes with its maximum braking capacity, at the RSS safety distance 4702, the host vehicle 4704 brakes with its maximum braking capacity, even if during that time the host vehicle brakes with its maximum acceleration. Even if maximum braking is initiated after the response time of the host vehicle that accelerates at its capacity, the host vehicle can be stopped without colliding with the target vehicle 4706. In most instances, target vehicle 4706 may not need to brake until it comes to a complete stop at its maximum braking capacity. In such an example, host vehicle 4704 may brake at its maximum braking capacity until it is at least RSS safe distance 4702 away from target vehicle 4706.
[0639] FIG. 47B shows an RSS safety distance 4722 between a host vehicle 4724 and a target vehicle 4726. In this embodiment, RSS safety distance 4722 includes RSS safety distance 4702 and minimum approach distance 4710. As discussed above, minimum approach distance 4710 means that both host vehicle 4724 and target vehicle 4726 brake at their maximum braking capability (even after the host vehicle has accelerated at its maximum acceleration capability over its reaction time). may be the minimum separation distance that would exist between the host vehicle 4724 and the target vehicle 4726 if the host vehicle 4724 and the target vehicle 4726 were to stop.
[0640] Consistent with this disclosure, a system for navigating a host vehicle is disclosed. The system may include at least one processing device programmed to perform one or more methods, processes, functions or operations consistent with this disclosure. In some embodiments, the system may be an ADAS system or another navigation system disclosed herein. Similarly, the processing device may be processing device 110 or another processor or processing device within or in communication with the host vehicle.
[0641] FIG. 48 is a flowchart illustrating an example process 4800 that may be performed by at least one processing device. Process 4800 is exemplary only. Those skilled in the art with the benefit of this disclosure will understand that process 4800 may include additional steps, exclude certain steps, or be otherwise modified in a manner consistent with this disclosure. can do.
[0642] The process 4800 may include a step 4802 of receiving at least one image representative of the host vehicle's environment. Consistent with this disclosure, at least one processing device may be programmed to receive at least one image representative of the host vehicle's environment. As discussed above, at least one image may be received from an image capture device. The image capture device can be anything consistent with this disclosure, including image capture device 122. In some embodiments, the at least one image is an image obtained from either a camera, radar, lidar, or any other device (whether optical or otherwise) from which an image can be obtained. It can be. There may be some delay period between the time the image is captured and the time the processing device receives the image. Additionally, there may be a delay period between the time an event occurs and the time an imaging device captures an image of the event. For example, if a target vehicle enters a lane in front of a host vehicle, there will be a short period of time (e.g., 1 ms, 2 ms, 5 ms, etc.).
[0643] The process 4800 may include a step 4804 for determining a planned navigation operation of the host vehicle. Consistent with this disclosure, at least one processing device may be programmed to determine a planned navigation operation to achieve a host vehicle navigation goal. Navigation behavior may be determined based on at least one driving policy. The planned navigation maneuver and/or at least one driving policy may be any consistent with this disclosure, including those discussed above. For example, the planned navigation operation may include at least one of a lane change operation, a merging operation, an overtaking operation, a pursuit distance reduction operation, or a throttle maintenance operation. The processing device may be programmed to analyze the at least one image to identify a target vehicle within the environment of the host vehicle. The at least one image may be an image received from an image capture device, such as image capture device 122. The at least one image may be one or more images received as part of step 4802 of process 4800.
[0644] The process 4800 may include step 4806 to determine the distance of the next state associated with the planned navigation operation. Consistent with this disclosure, the processing device may be programmed to determine the next state distance between the host vehicle and the target vehicle that would occur if the planned navigation operation was performed. The next state distance can be calculated by any means disclosed herein, including the RSS safe distance formula above. For example, if the planned navigation action is to accelerate the host vehicle, the next state distance may be the distance between the host vehicle and the vehicle in front of the host vehicle. In some examples, multiple next state distances can be determined. For example, if the planned navigation operation is to merge into an adjacent lane, a first next state distance may be determined with respect to the host vehicle and a first target vehicle in front of the host vehicle after the merge; A second next state distance may be determined for a second target vehicle behind the host vehicle after the merge.
[0645] Process 4800 may include step 4808 to determine maximum braking capacity, maximum acceleration capacity, and current speed of the host vehicle. Consistent with this disclosure, the processing device may be programmed to determine a maximum braking capability of the host vehicle, a maximum acceleration capability of the host vehicle, and/or a speed of the host vehicle. Each of those factors (ie, maximum braking capacity, maximum acceleration capacity, and speed of the host vehicle) may be determined by any means consistent with this disclosure. In some embodiments, the host vehicle's maximum braking capacity and/or maximum acceleration capacity may depend, for example, on the vehicle's current speed, road characteristics (e.g., road slope, road material, etc.), weather conditions (e.g., snow raining, high humidity, etc.), the condition of the vehicle (for example, tire pressure, brake pad condition, current load of the vehicle, etc.). In some embodiments, one or more of those factors may be determined based on the output of one or more sensors. For example, the host vehicle can include an accelerometer that can provide output to the processing device, which output can include the host vehicle's current speed and/or the host vehicle's acceleration force. In some embodiments, the processing device may determine a current speed of the host vehicle and use the current speed to determine a current maximum braking capability and/or a current acceleration capability. For example, the maximum braking capacity of a vehicle traveling at a first speed (e.g. 15 km/h) is significantly different from the maximum braking capacity of the same vehicle traveling at a second speed (e.g. 30 km/h). obtain. In some embodiments, predetermined values may be used to assume one or more of the host vehicle's maximum braking capacity, maximum acceleration capacity, and/or speed. For example, the processing device may be configured to assume that the host vehicle has a maximum braking capacity that corresponds to a predetermined value that corresponds to an average (or lowest) maximum braking capacity for the type of vehicle associated with the host vehicle. . In some embodiments, each factor may be determined based on external conditions of the road or temporal characteristics of the vehicle. For example, the host vehicle's maximum braking capacity may be determined based on sensed road surface conditions. In this example, the detected road condition may include the roughness of the road, the slope or slope of the road, the presence or absence of materials or objects on the road, whether the road is asphalt, cement, gravel, or another material, or Any other terms consistent with the disclosure may be included. As another example, the host vehicle's maximum braking capacity may be determined based on sensed weather conditions. In this example, weather conditions include any precipitation (e.g., rain, sleet, snow, ice, etc.), weather conditions that affect visibility (e.g., fog, haze, smoke, etc.), and weather conditions that affect vehicle maneuverability. This may include weather conditions that may be present (eg, strong winds, high temperatures, etc.) or any other weather conditions that may affect the host vehicle's navigation response. In another example, the processing device determines the maximum braking capacity based on whether the host vehicle includes, for example, one passenger, multiple passengers, a significantly heavy load, a trailer, etc. I can do it.
[0646] In some embodiments, maximum braking capability and/or maximum acceleration capability may be determined based on one or more predetermined factors. For example, a government or industry entity may provide one or more regulations that define the maximum braking capacity and/or maximum acceleration capacity that a vehicle or class of vehicles may have, and at least one processor may It may be assumed that the host vehicle has braking and/or maximum acceleration capabilities.
[0647] Process 4800 may include step 4810 to determine the host vehicle's current stopping distance. Consistent with this disclosure, the processing device is configured to determine a current stopping distance of the host vehicle based on a current maximum braking capability of the host vehicle, a current maximum acceleration capability of the host vehicle, and/or a current speed of the host vehicle. may be configured. The host vehicle's current stopping distance may be determined by any means consistent with this disclosure. For example, the processing device may use one or more of the formulas discussed above for RSS.
[0648] In some embodiments, the current stopping distance is defined as the current speed of the host vehicle if the host vehicle accelerates at its maximum acceleration capability for a period of time and then brakes at its maximum braking capability. may be the distance required for the host vehicle to stop. For example, the host vehicle's current stopping distance is an acceleration distance that corresponds to the distance that the host vehicle could travel over a given period of time at the host vehicle's current maximum acceleration capability starting from the host vehicle's determined current speed. may include. The predetermined time period may be a time period assumed by the processing device. For example, a constraint may specify that the processing device assumes that the host vehicle accelerates over a period of time (eg, 0.5 milliseconds, 1 millisecond, 2 milliseconds, 3 milliseconds, 10 milliseconds, etc.). The predetermined period of time may be a reaction time associated with the host vehicle. For example, the predetermined time period may be defined as the point in time when a host vehicle's sensor (e.g., an imaging device) detects a condition that requires a brake response of the host vehicle (e.g., a target vehicle entering the host vehicle's path); There may be a delay between when the vehicle begins to brake at its maximum braking capacity. In this example, it can be assumed that in the worst case scenario, the host vehicle accelerates at its maximum braking capacity for the entire delay period. In some embodiments, the period associated with reaction time may be determined or estimated by the processing device. For example, the processing device may monitor the time between receiving the image and determining that a brake response is required. As another example, the processing device may determine an average, median, or mode reaction time associated with the plurality of navigation responses and use the average, median, or mode reaction time when calculating the current stopping distance. I can do it. For example, after tens, hundreds, or thousands of navigation responses, the processing device determines that the average time between a sensing event and a navigation response is a certain value (e.g., any real number of milliseconds). , that value can be used as the reaction time associated with the host vehicle.
[0649] Consistent with this disclosure, the determined current stopping distance may include the minimum distance between the host vehicle and another object (eg, a target vehicle or VRU) after coming to a complete stop. The determined or predetermined minimum distance may correspond to a predetermined separation distance maintained between the host vehicle and other vehicles. For example, the processing device may be programmed to include the determined or predetermined minimum distance in the stopping distance calculation. For example, the predetermined separation distance can be at least 1 meter. In this example, the determined or predetermined minimum distance may be dmin as discussed above.
[0650] Process 4800 may include step 4812 to determine the target vehicle's current speed and assume the target vehicle's maximum braking capability. Consistent with this disclosure, at least one processing device may be programmed to determine a current speed of the target vehicle. In some embodiments, the speed of the target vehicle can be determined based on the speed of the host vehicle. For example, one or more sensors on the host vehicle may provide information related to the speed of the host vehicle and an indication of a change in acceleration or direction of the target vehicle relative to the host vehicle. In some embodiments, the speed of the target vehicle can be determined based on analysis of at least one image. The at least one image can be, for example, the same image received by the processing device in step 4802 that is used to identify the target vehicle in step 4806. In some embodiments, multiple images may be analyzed to determine the speed of the target vehicle. For example, the first image may show the target vehicle at a first distance from the host vehicle, the second image may show the target vehicle at a second distance from the host vehicle, and the processing device can determine the speed associated with the target vehicle using the change in distance and time between the first image and the second image. In some embodiments, the speed of the target vehicle may be determined based on an analysis of output from at least one lidar or radar system associated with the host vehicle. For example, the processing device can determine the speed of the host vehicle using the speed of the host vehicle and the distance between the host vehicle and the target vehicle (eg, as sensed by the lidar system).
[0651] In some embodiments, to address the worst-case scenario, the processing device may be programmed to assume that the target vehicle's current speed is below or above the target vehicle's sensed speed. For example, if the target vehicle is traveling in front of the host vehicle, the processing device may be programmed to reduce the detected speed using a predetermined value, percentage, formula, etc. For example, if the processing device determines that the target vehicle is traveling at 100 km/h based on information from one or more sensors, the processing device adjusts the sensed speed by 10% so that the vehicle It can be assumed that the vehicle is traveling at 90 km/h. The speed of the target vehicle may be correlated with the braking force of the target vehicle, such that assuming a reduced speed of the target vehicle will reduce the speed of the target vehicle more than if the target vehicle did not use the reduced speed. such as assuming that the target vehicle can stop quickly (e.g., the target vehicle can stop in a shorter distance if traveling at 90 km/h than at 100 km/h).
[0652] Consistent with this disclosure, at least one processing device may be programmed to assume a maximum braking capability of the target vehicle based on at least one recognized characteristic of the target vehicle. Maximum braking capacity may be assumed as part of step 4812 of process 4800. The processing device may be programmed to recognize at least one characteristic from information from one or more sensors (eg, lidar, radar, imaging device, etc.). In some embodiments, perceived characteristics of the target vehicle can be determined based on analysis of the at least one image. For example, an image of the target vehicle may include text external to the target vehicle that may be used to determine the type of vehicle, model of the vehicle, size of the vehicle, or other characteristics. As another example, the image of the target vehicle may include other objects that may be used to approximate the size or other characteristics of the vehicle based on comparison thereto. As a specific example, a large vehicle may appear taller than a speed limit sign, whereas a small vehicle may appear lower than a speed limit sign. In some embodiments, at least one characteristic of the target vehicle can be determined based on at least one of the lidar output or the radar output. For example, the lidar output may include a first distance related to a distance to a rear bumper of the target vehicle and a second distance related to a distance to a front bumper (or other area) of the target vehicle; The difference between them can be used to estimate the size of the target vehicle.
[0653] The at least one characteristic may be any characteristic known to be or capable of being correlated with braking ability. In some embodiments, the recognized characteristics of the target vehicle may include vehicle type. The type of vehicle can be, for example, a general category to which the target vehicle belongs (eg, large sedan, compact car, SUV, crossover SUV, motorcycle, etc.) or a more specific category or subcategory related to the target vehicle. The processing device may be programmed to assume that the target vehicle has a maximum braking capacity that matches a maximum braking capacity of the known braking capacities for that vehicle type or class of vehicles. For example, if the target vehicle is determined to be a large sedan, the processing device may select a large sedan that has the best maximum braking capability (e.g., a large sedan that can stop in the shortest distance compared to other large sedans). It may be assumed that the target vehicle has a maximum braking capacity that matches the maximum braking capacity of ). In some embodiments, the target vehicle's recognized characteristics may include the target vehicle's vehicle model, brand name, or other classifier (e.g., Toyota Prius®, BMW X5 (registered trademark), etc.). The processing device may use the vehicle type to access a database containing, for example, the known braking capabilities of the respective vehicle type. For example, if the detected vehicle type is a BMW X5®, the processing device may search the database for the reported braking ability of the BMW X5®. If more than one maximum braking capacity is reported for a vehicle model (e.g. a BMW X5® has an xDrive40i trim level or an xDrive50i trim level) the processing device assumes that the target vehicle has the best maximum braking capacity reported for that model of vehicle. can be programmed to do so. In some embodiments, the recognized characteristics of the target vehicle may include the size of the vehicle. The size of the vehicle may be a relative size, a predetermined size, etc. The size of the vehicle may correspond to the physical size of the vehicle and/or the estimated weight of the vehicle. For example, the processing device may determine that the target vehicle is larger or smaller than the host vehicle. In another example, the processing device may be programmed to classify the target vehicle into one or more predetermined vehicle sizes, and each such size may include a range of sizes (e.g., category 1=1,000 (less than kg, category 2 = 1,000~1,500kg, category 3 = 1,500~1,800kg, etc.). Some vehicles may include an indication of their size, which may be used to reveal the target vehicle's vehicle size. For example, the target vehicle is GMC Sierra® and may include instructions on the exterior of the vehicle relating to its size, such as 1500, 2500, 2500HD, etc.
[0654] Using the target vehicle's current speed and the target vehicle's assumed maximum braking capacity, determine the amount necessary for the target vehicle to come to a complete stop from its current speed by braking at its maximum braking capacity. The distance traveled by the target vehicle, including the distance, can be determined. This distance can be determined by any means consistent with this disclosure.
[0655] Process 4800 may include step 4814 for implementing the planned navigational action if the planned navigational action is safe. In step 4814, one example of when a planned navigation operation can be considered safe is when the host vehicle's determined current stopping distance is equal to the target vehicle's travel distance, and the determined next state distance. This is the case when it is less than the sum of . In some embodiments, the at least one processing device determines if the host vehicle's determined current stopping distance is less than the target vehicle's traveled distance plus the determined next state distance; It may be programmed to carry out planned navigation operations. The distance traveled by the target vehicle may be determined by any means consistent with this disclosure. For example, the distance traveled by the target vehicle may be based on the target vehicle's current speed and the target vehicle's assumed maximum braking capability. The distance traveled by the target vehicle may correspond to the total distance traveled by the target vehicle before coming to a complete stop. In this situation, a navigation operation can be performed that determines the host vehicle's current stopping distance (maximum acceleration during the reaction period) before the target vehicle brakes with its maximum braking capacity and still does not collide with the target vehicle. This is because there is no collision between the host vehicle and the target vehicle, since the assumed potential travel distance (including the assumed potential travel distance) allows the host vehicle to come to a complete stop. The processing device performs the navigation operation because the host vehicle is at least an RSS safe distance from the target vehicle after performing the navigation operation. In other words, the distance of the next state is greater than or equal to the RSS safe distance. Conversely, the next state distance is less than the RSS safe distance (e.g., the host vehicle's current stopping distance is not less than the target vehicle's traveled distance plus the determined next state distance). In this case, the processing device may be programmed to reject or abort the navigation operation, or to perform a different navigation operation.
[0656] In step 4814, one example of when a planned navigation operation can be considered safe is when the host vehicle's determined current stopping distance is equal to or less than the target vehicle's travel distance. The distance traveled by the target vehicle is determined based on the current speed of the target vehicle and the assumed maximum braking capacity of the target vehicle, where the distance traveled by the target vehicle is at least a predetermined minimum distance less than the sum of the state distances. In some embodiments, the at least one processing device determines that the determined current stopping distance of the host vehicle is at least a predetermined distance greater than the distance traveled of the target vehicle plus the determined next state distance. If the minimum distance is less, the planned navigation operation may be configured to be performed, and the distance traveled by the target vehicle is determined based on the target vehicle's current speed and the assumed maximum braking capability of the target vehicle. In this embodiment, the predetermined minimum distance is d<sub>min</sub>may be predetermined or determined by any means consistent with this disclosure, and may have any of the attributes discussed above. In some embodiments, the predetermined minimum distance may correspond to a predetermined separation distance maintained between the host vehicle and other vehicles. For example, an autonomous navigation system constraint may stipulate that an autonomous vehicle never comes within (less than) a specified distance of another vehicle. The predetermined separation distance can be a distance of any size. In some embodiments, the predetermined separation distance is at least 1 m, 2 m, 5 m, etc. In some embodiments, the predetermined distance may vary depending on, for example, the speed at which the host vehicle is traveling, the location of the host vehicle (eg, city road, local highway, etc.).
[0657] Consistent with this disclosure, RSS models may be used to navigate fully autonomous vehicles, partially autonomous vehicles (e.g., having a human-operable mode and a selective autonomous mode). vehicles) or as a basic safety feature in human-operable vehicles (e.g., human safety features in human-operable vehicles that can be dangerous under the RSS model). inputs may be prevented or prohibited by safety systems).
[0658] Comfort Responsibility Sensitive Safety (CRSS) [0659] RSS is effective for determining a safe distance between a host vehicle and another object (e.g., one or more target vehicles or VRUs). It is. Some of the basic calculations in RSS may require the host vehicle to assume a worst-case scenario and respond to its maximum capacity. For example, some embodiments of the RSS can define a safe area in which both the target vehicle and the host vehicle brake at their maximum braking capacity, while the host vehicle accelerates at its maximum capacity. If the host vehicle begins to brake after a period of time, no collision will occur within such a safe area.
[0660] Although RSS may prevent or avoid accidents, some users of autonomous vehicles realize that the responses required by RSS may not result in the most comfortable or relaxing ride in certain situations. Recognizable. For example, if a worst-case scenario occurs, a vehicle operating in RSS mode can brake at its maximum braking capacity in at least some scenarios. Additionally, RSS can define a safety distance between the host vehicle and another object. If the target vehicle, VRU, or other object is further away from the vehicle than a safe distance, the vehicle does not need to make any navigation response. However, when an object comes within a distance less than or equal to the RSS safety distance, the vehicle may respond by braking at maximum braking capacity until it stops or until an RSS distance is established with the object or target vehicle. Such maneuvers can subject the host vehicle's passengers to uncomfortable acceleration rates (eg, during periods when the host vehicle's brakes are applied at maximum levels). Therefore, there may be a need for a safety model for autonomous vehicles that allows for a more comfortable passenger experience.
[0661] To provide a more comfortable human-like driving experience, an autonomous vehicle may be equipped with a navigation system configured to operate using a submaximal navigation response. For example, if an autonomous vehicle needs to slow down or stop to avoid a collision, the navigation system may allow the brakes to be applied with less than the vehicle's maximum braking capacity during at least a portion of the braking period. Similarly, if the vehicle needs to turn, swerve, or veer off course to avoid a collision, the navigation system may allow the vehicle to turn less than its maximum turning capability. The same may be true for acceleration responses that may be performed at less than the vehicle's maximum acceleration capability during at least part of the acceleration period. By limiting the amount of braking or acceleration, the braking response by the host vehicle can be less abrupt, smoother, and feel more like scheduled braking rather than emergency braking or delayed braking. For example, scheduled braking may include gradual decelerations that do not cause seat belt securing. In another example, scheduled braking may include gradually increasing the braking force, possibly until the vehicle's maximum braking force is reached, at which point braking may be resumed at the maximum braking force.
[0662] Consistent with the disclosed embodiments, a Comfort Responsibility Sensitive Safety (CRSS) safety distance may be determined. The CRSS safety distance may be the distance between the host vehicle and another object (eg, a target vehicle or VRU) that the host vehicle can comfortably stop without colliding with the object. Consistent with this disclosure, the CRSS may assume that the brakes are applied gradually from an initial braking force (less than the maximum braking capacity of the vehicle) to the maximum braking capacity of the vehicle, rather than immediately applying the maximum braking capacity. At least in the context of CRSS, the term minimum braking capacity is sometimes used interchangeably with the term initial braking force. The initial braking force may be a minimum braking force of the vehicle or a small fraction of the maximum braking capacity of the vehicle. The CRSS safety distance may be determined based on the instantaneous calculated speed of the host vehicle, the instantaneous speed of the target vehicle, comfortable or submaximal brake acceleration and response time.
[0663] Definition 28 (CRSS safety distance) car c<sub>r</sub>and c<sub>r</sub>Another car within the forward zone of c<sub>f</sub>The vertical distance between c<sub>f</sub>The maximum α<sub>max,brake</sub>For any brake of c<sub>r</sub>is jerk bound brake and acceleration plus c<sub>f</sub>is safe in terms of response time p if it does not collide with .
[0664] In some embodiments, a comfortable navigation response may be achieved by reducing the speed or velocity of the host vehicle with a restraining brake profile that may significantly reduce jerk to the vehicle and/or vehicle passengers. . Lemma 12 below calculates the CRSS safety distance according to an embodiment in which the vehicle's brakes are subject to a graded braking profile, rather than immediately applying maximum braking capacity.
[0665] Lemma 12 Target vehicle c on the vertical axis<sub>f</sub>c the host vehicle behind<sub>r</sub>shall be. The maximum braking force is α<sub>max,brake</sub>shall be. Let the initial speed of the host vehicle be v<sub>0</sub>Let the original speed of the target vehicle be v<sub>f</sub>shall be. j the slope of the linear application of the host vehicle's brakes<sub>max</sub>Let T be the time it takes for the host vehicle to reach full braking. Therefore, the CRSS safety distance of the host vehicle is as follows.
<math num="236"><img file="JP7367255B2_D0236.tif" /></math>
[0666] Proof CRSS is the maximum braking capacity α<sub>max,brake</sub>The acceleration of the host vehicle until it reaches (slope j<sub>max</sub>) and then decrease linearly until it stops (α<sub>max,brake</sub>Assume a jerk-bound braking capability of the host vehicle associated with continuing to brake at a constant deceleration (correlated with Initial acceleration α<sub>0</sub>0m/s, initial velocity v<sub>0</sub>0m/s<sup>2</sup>and initial position x<sub>0</sub>j with 0m<sub>max</sub>A vehicle that brakes with a jerk of has the following dynamics:
<math num="237"><img file="JP7367255B2_D0237.tif" /></math>
[0667] The brake distance can be determined based on these formulas. The minimum braking capacity of the host vehicle is α<sub>min,brake</sub>and α(T)=-α<sub>min,brake</sub>Or the first point at which v(T)=0 holds, i.e.<math num="238"><img file="JP7367255B2_D0238.tif" /></math>shall be.
[0668] j<sub>max</sub>The time to reach maximum braking capacity by linearly applying the brakes using:
<math num="239"><img file="JP7367255B2_D0239.tif" /></math>
[0669] Let the speed (velocity) of the host vehicle at any given time be j<sub>max</sub>It can also be determined accordingly.
<math num="240"><img file="JP7367255B2_D0240.tif" /></math>
[0670] Formula and j for time to reach maximum braking force<sub>max</sub>(until maximum braking capacity is reached)<sub>max</sub>) Maximum brake α<sub>max,brake</sub>The speed of the host vehicle during the period in which the host vehicle is being applied allows the host vehicle's braking distance to be determined.
<math num="241"><img file="JP7367255B2_D0241.tif" /></math>
[0671] Consistent with this disclosure, submaximal braking rate j<sub>max</sub>can be predetermined or determined by a processing device. In some embodiments, j<sub>max</sub>may be an ideal or statistically acceptable braking rate. For example, most people would not be able to access data related to braking rate at a stop.<sub>max</sub>It can be used for. In some embodiments, j<sub>max</sub>can be set according to user preferences. For example, the first passenger chooses the first j that is comfortable for him.<sub>max</sub>The second passenger can provide himself with a comfortable second j<sub>max</sub>etc. can be provided. In some embodiments, the processing device determines, for example, one or more road characteristics, traffic conditions (congestion level, average distance between the host vehicle and surrounding vehicles, cut-in rate in front of the host vehicle, etc.). , one or more vehicle characteristics, one or more weather characteristics, etc.<sub>max</sub>can be determined. For example, the processing device provides maximum safety and comfort on snowy roads.<sub>max</sub>, second j giving maximum safety and comfort on dry highways<sub>max</sub>etc. can be determined. j<sub>max</sub>Implementing a j<sub>max</sub>It will be appreciated that it may also have a less desirable effect if is small, so that the brakes are "softened" or less strong. For example, small j<sub>max</sub>The use of can have a strong negative impact on efficiency by forcing the vehicle to always maintain an excessive distance from the vehicle in front. Such settings may cause annoyance to other road users and prevent the host vehicle from efficiently negotiating complex scenarios such as merging and driving in congested areas. Therefore, in some instances, balancing performance or efficiency with comfort (while maintaining RSS security)<sub>max</sub>can be set, selected or determined.
[0672] Consistent with this disclosure, submaximal braking rate j<sub>max</sub>can be a constant. For example, the submaximal braking rate is 2m/s<sup>2</sup>,10m/s<sup>2</sup>or any other constant braking rate. In some embodiments, the submaximal braking rate may be a constant that is proportional to the host vehicle's maximum braking rate capability. For example, the submaximal braking rate may be 20%, 33%, 50%, 70% or any other percentage of the host vehicle's maximum braking capacity. In some embodiments, the submaximal braking rate may be a linear function of the host vehicle's current speed, such that the brakes range from the host vehicle's minimum braking capacity to at most the host vehicle's maximum braking capacity. The ability can be gradually increased. For example, the host vehicle may determine from its current speed a submaximal braking rate that allows it to stop without reaching the vehicle's maximum braking capacity. As another example, the maximum braking capacity of the host vehicle may be such that the host vehicle brakes at less than its maximum braking capacity for a first part of the stopping maneuver and at its maximum braking capacity for a second part of the stopping maneuver. The maximum sub-brake rate that will ultimately be reached can be determined. In some embodiments, the submaximal braking rate may be a different function of the vehicle's current speed. For example, a submaximal braking rate may be a braking rate that is applied exponentially or logarithmically until the vehicle is stopped.
[0673] Consistent with this disclosure, the CRSS safety distance can be used in any embodiment discussed above with respect to RSS. In practice, CRSS requires autonomous vehicles to begin braking earlier than vehicles operating in RSS mode. In CRSS, the vehicle is at most j<sub>max</sub>In RSS, the vehicle can brake at its maximum braking capacity, whereas in RSS the vehicle can brake at its maximum braking capacity. Therefore, the CRSS safety distance may be longer than the RSS safety distance.
[0674] Consistent with this disclosure, the CRSS model allows the host vehicle to accelerate at its maximum acceleration capability for an initial response period and then<sub>max</sub>may include the RSS assumption that the brakes can be applied at a rate related to . As another example, CRSS uses j until it stops<sub>max</sub>The RSS assumption may include that the host vehicle ceases to leave a minimum approach distance from the target vehicle after braking at a rate related to .
[0675] In some embodiments, the vehicle may brake at a submaximal braking period during a first period and at its maximum braking capacity during a second period. CRSS can be fixed. For example, a host vehicle that is traveling behind a target vehicle may detect that the target vehicle is stationary, and then the host vehicle travels for a first duration (or a first distance). Apply the brakes at submaximum braking capacity (for example, until j<sub>max</sub>(applying the brakes gradually) and applying the brakes at maximum braking capacity for a second duration (or until a second distance is traveled). In this example, the CRSS safety distance may be shorter than if the vehicle were programmed to brake only at submaximal braking capacity for the entire period. In some embodiments, the host vehicle may brake at a first submaximal braking capacity for a first period of time and may brake at a second submaximal braking capacity for a second period of time. CRSS can be fixed. For example, the host vehicle may operate at a first rate (e.g., a first j<sub>max</sub>) and brake at a second rate (e.g., a second j<sub>max</sub>) can be programmed to apply the brakes. In this example, the first rate may be less than the second rate. Such a mode of operation may allow the host vehicle to re-establish CRSS distance without braking at a higher rate than the level associated with jmax, for example.
[0676] In some embodiments, a passenger or user of the host vehicle may select one or more navigation modes that determine a submaximal braking rate. For example, the passenger may determine whether the host vehicle brakes using a brake profile associated with a constant submaximal braking rate, or with a brake profile associated with a gradually increasing submaximal braking rate; It may be possible to choose to brake using a brake profile associated with an exponentially increasing submaximal braking rate. Any other braking rate profile may be employed. For example, the selectable mode may instruct the host vehicle to always apply a predetermined submaximal braking rate until the host vehicle is stopped or it is determined that a braking condition no longer exists. As another example, a selectable mode may always apply a predetermined submaximal braking rate for at least a portion of the period during which a braking condition is determined to exist, and then apply the maximum braking rate of the host vehicle at the host vehicle's maximum braking rate capability. The host vehicle can be instructed to apply the brakes. As another example, the selectable mode may instruct the host vehicle to apply the brakes starting at a predetermined submaximal braking rate, and to progressively increase the braking rate up to the host vehicle's maximum braking rate capability. can. Any other combination of braking rates may be selected.
[0677] Consistent with this disclosure, CRSS may be used in conjunction with an RSS model or in place of RSS mode. In some embodiments, CRSS may be selectively used in some situations, while RSS may be used in other situations. For example, CRSS can be used on a highway (e.g., where the host vehicle is traveling at a high speed rate and braking at maximum braking capacity can be very unpleasant), and RSS is For example, it can be used in an urban setting (where the host vehicle is traveling at low speeds and frequently stops at traffic lights, stop signs, etc.). In some embodiments, the host vehicle may be able to automatically switch between CRSS mode and RSS mode. For example, the system may switch to CRSS when it detects that the host vehicle enters a highway environment, may switch to RSS upon detecting an urban environment, and/or the system may switch to RSS upon detecting certain weather conditions ( For example, it is possible to switch to CRSS in the event of snow, rain, etc.).
[0678] In some embodiments, CRSS may be used to ensure that the host vehicle remains at least a CRSS safe distance from the target vehicle (or VRU or other object) at all times. For example, if the target vehicle changes lanes and is in front of the host vehicle at a distance less than the CRSS distance after the lane change, the host vehicle will maintain a submaximal braking rate (e.g., j<sub>max</sub>braking rate). In this example, if the target vehicle comes to a complete stop, the host vehicle can brake at a submaximal braking rate until it comes to a complete stop, or start at a submaximal rate and increase to the maximum braking rate. Brake profiles can be applied. Similarly, if the target vehicle brakes for a period of time but continues to drive in front of the host vehicle, the host vehicle shall brake at a submaximal braking rate until the target vehicle is at least a CRSS safe distance from the target vehicle. The host vehicle can then continue traveling at its current speed or at a new speed that allows the host vehicle to maintain the CRSS distance. As mentioned above, if the CRSS distance cannot be maintained in this mode, the host vehicle will not perform active navigation operations toward the target vehicle. In some embodiments, the processing device determines whether the host vehicle is approaching a CRSS safe distance (e.g., the host vehicle is more than the CRSS safe distance from the target vehicle, but the processing device does not detect its own current acceleration or current speed). If you continue at , you will be at the CRSS safe distance), and you can cancel the acceleration without applying the brakes to maintain a distance above the CRSS safe distance. For example, if the host vehicle did not coast otherwise the host vehicle would be within the CRSS safe distance, the host vehicle would release the throttle and coast the host vehicle away from the target vehicle in front of the host vehicle. can be released.
[0679] In some embodiments, RSS may be implemented as a contingency measure when CRSS is used. For example, if a target vehicle cuts in front of a host vehicle and the distance to the target vehicle in front of the host vehicle (not attributable to the host vehicle) is less than the CRSS safe distance, the host vehicle will It is possible to apply the brakes (eg, apply the brakes immediately with its maximum braking capacity). In particular, RSS braking may be used if the interrupt creates a distance less than or equal to the RSS safety distance (less than the CRSS safety distance). If the distance is greater than the RSS safe distance but less than the CRSS safe distance, in one example, the system safely stops the host vehicle according to the current distance (which is somewhere between the CRSS safe distance and the RSS safe distance). j available for<sub>max</sub>and during at least part of the braking period, the determined j<sub>max</sub>can be used to adjust the CRSS accordingly so that the vehicle comes to a stop.
[0680] FIGS. 49A to 49D show the concept of safe distance under the CRSS model when the host vehicle is running behind the target vehicle. FIG. 48A shows a CRSS safety distance 4902 between a host vehicle 4904 and a target vehicle 4906. As discussed above, the CRSS safety distance 4902 is a comfortable braking rate for the host vehicle 4904 below its own maximum braking capacity.<sub>max</sub>This is the distance required to stop without colliding with target vehicle 4906 when applying the brakes. For comparison, RSS safety distance 4702 (shown in FIGS. 47A to 47D) is included. RSS safety distance 4702 is the distance required for host vehicle 4904 to stop when braking at its maximum braking capacity.
[0681] FIG. 49B shows a CRSS safety distance 4912 between a host vehicle 4914 and a target vehicle 4916. The difference between CRSS safety distance 4902 and CRSS safety distance 4912 may be that CRSS safety distance 4912 is associated with a lower braking rate than CRSS safety distance 4902. i.e. j associated with host vehicle 4814<sub>max</sub>is associated with host vehicle 4904<sub>max</sub>Therefore, host vehicle 4914 requires a longer distance to stop.
[0682] FIG. 49C shows a CRSS safety distance 4922 between a host vehicle 4924 and a target vehicle 4926. In this example, CRSS safe distance 4922 includes a minimum distance 4910, which (as indicated by CRSS safe distance 4902)<sub>max</sub>The minimum separation distance between host vehicle 4924 and target vehicle 4926 after stopping by applying the brakes at a rate of . As discussed above, the minimum distance 4910 is d<sub>min</sub>It can be done.
[0683] FIG. 49D shows a CRSS safety distance 4932 between a host vehicle 4934 and a target vehicle 4936. In this example, CRSS safety distance 4932 includes both minimum distance 4910 and acceleration distance 4908. Acceleration distance 4908 may be the distance that host vehicle 4934 would travel if it were accelerated to maximum acceleration capability during the initial reaction period.
[0684] The CRSS safety distance concept can also be extended laterally. A CRSS safety distance can include a lateral distance between the host vehicle and another object (e.g., a target vehicle or VRU) in which the host vehicle can avoid the object while avoiding uncomfortable steering adjustments. Those skilled in the art who have the benefit of this disclosure will understand that the host vehicle's steering may be adjusted to avoid colliding with the host vehicle.
[0685] Consistent with this disclosure, a system for applying the brakes of a host vehicle is disclosed. This system may be, for example, a brake system 230, or may be included within a navigation system, such as an ADAS system. The brake system may include at least one processing device. A processing device may be programmed to perform one or more methods, processes, operations or functions consistent with this disclosure.
[0686] FIG. 50 is a flowchart illustrating an example process 5000 that may be performed by at least one processing device of a brake system. Process 5000 is used for illustrative purposes and is not intended to be limiting. Those skilled in the art with the benefit of this disclosure will understand that process 5000 may include additional steps, exclude certain steps, or be otherwise modified in a manner consistent with this disclosure. can do.
[0687] Process 5000 may include step 5002 of receiving output representative of the host vehicle's environment. Consistent with this disclosure, at least one processing device may be programmed to receive output representative of the host vehicle's environment. The output may be received from at least one sensor included within or in communication with the brake system, for example. The at least one sensor can be any sensor disclosed herein, including, for example, an image capture device, a lidar system, a radar system, an accelerometer, and the like. In some embodiments, a processing device may be programmed to receive multiple outputs from one or more sensors. For example, the processing device can receive a first output from an image capture device, a second output from a lidar system, and so on. As another example, the processing device may be programmed to receive a first output from the image capture device, receive a second output from the image capture device, and so on. In some embodiments, one or more sensors may be configured to capture information related to different areas of the host vehicle's environment. For example, the first image capture device may be configured to capture an image of the environment in front of the host vehicle, and the second image capture device may be configured to capture an image of the environment behind the host vehicle. It is possible to configure, etc. As another example, the image capture device may be configured to capture images of the environment in front of the host vehicle, and the one or more lidar systems may be configured to capture images of the environment on either side of the host vehicle. It may be configured to capture relevant information.
[0688] Process 5000 may include step 5004 for detecting a target vehicle within the host vehicle's environment. In some embodiments, a target vehicle may be detected based on the power received in step 5002. Consistent with this disclosure, at least one processing device of the brake system may be programmed to detect a target vehicle within the host vehicle's environment based on the output. The target vehicle may be detected by any means consistent with this disclosure. For example, if the output is one or more images, the processing device analyzes the images (e.g., by comparing those images with images known to include the vehicle) to detect the target vehicle. be able to. As another example, one or more outputs from one or more lidar or radar systems can be used to detect the contour and location of a target vehicle. In some embodiments, the processing device may detect multiple target vehicles. For example, the processing device may detect a target vehicle in front of the host vehicle, a target vehicle behind the host vehicle, and/or a target vehicle next to the target vehicle, or a combination thereof. Although these examples disclose detecting a target vehicle, it will be appreciated that any other object (eg, VRU, road feature, etc.) may also be detected by the same means.
[0689] Process 5000 may include step 5006 to determine the speed of the host vehicle and the distance between the host vehicle and the target vehicle. Consistent with this disclosure, at least one processing device of the braking system may be programmed to determine the current speed of the host vehicle and the current distance between the host vehicle and the target vehicle. Current speed and/or current distance may be determined based on one or more outputs from one or more sensors. In some embodiments, the current speed and/or current distance may be determined based on the power used to detect the target vehicle in step 5004. For example, the output from the lidar system may output one or more distances between various points on the target vehicle and the host vehicle, where the distance is the current distance between the host vehicle and the target vehicle. can be used to determine. In some embodiments, the current speed of the host vehicle may be provided by an accelerometer and/or speedometer associated with the host vehicle.
[0690] Process 5000 may include step 5008 to determine whether a braking condition exists. Consistent with this disclosure, at least one processing device of the braking system can determine whether a braking condition exists. A braking condition may be any condition under which the host vehicle needs to brake to avoid a dangerous situation, avoid a collision, or come to a stop. For example, a braking condition may be a determination that the host vehicle is less than a CRSS safe distance from a target vehicle or other object. Determining whether a braking condition exists may be based at least on a current speed of the host vehicle and a current distance between the host vehicle and the target vehicle. For example, the processing device may be configured based on the current speeds of the host vehicle and target vehicle, the braking capability of the host vehicle, the assumed braking capability of the target vehicle, and optionally the maximum acceleration rate of the host vehicle as discussed above. A CRSS safety distance (or RSS safety distance) between the target vehicle and the host vehicle can be determined. The processing device may compare the CRSS safety distance to the current distance between the host vehicle and the target vehicle. Based on that comparison, if the current distance between the host vehicle and the target vehicle is less than the CRSS safe distance, the processing device may determine that a braking condition exists. Any other hazardous condition disclosed herein to which the host vehicle responds by applying the brakes may be a braking condition. A braking condition also exists if the host vehicle detects that the host vehicle approaches the target vehicle closer than the CRSS distance at the current speed and/or acceleration. In that case, the host vehicle takes action by applying the brakes or by initiating another maneuver (such as a lane change) so that at least the CRSS distance is maintained between the host vehicle and the target vehicle.
[0691] Process 5000 may include step 5010 for applying a braking device according to a predetermined braking profile when a braking condition exists. The brake condition may be any brake condition determined in step 5008. Consistent with this disclosure, at least one processing device of the braking system may be programmed to apply braking devices associated with the host vehicle according to a predetermined braking profile when a braking condition is determined to exist. For example, if the processing device determines that the host vehicle is less than or approaching the CRSS safe distance relative to the target vehicle, the braking system applies the brakes of the host vehicle at a predetermined submaximal braking rate. You can start playing. The predetermined braking profile may include a segment that begins at a submaximal braking rate of the host vehicle and may progressively increase to a maximum braking rate of the host vehicle. The submaximal braking rate may be progressively increased according to any of the embodiments disclosed herein. For example, a gradual increase can be a non-linear increase. In another example, the gradual increase may be a linear increase.
[0692] For example, the braking system may only apply the brakes during the duration of the braking condition. In some embodiments, once the maximum braking rate of the host vehicle is achieved, the at least one processor determines that the braking condition no longer exists (e.g., the host vehicle has stopped or the brake condition between the host vehicle and the target vehicle has stopped). The host vehicle's braking device may be configured to continue applying the host vehicle's braking device at the host vehicle's maximum braking rate until the CRSS distance is re-established at the host vehicle's maximum braking rate. In some cases, even when applying the braking mechanism at its maximum braking rate, the braking condition may disappear before the host vehicle comes to a complete stop, and the processing device may terminate the braking in response to the end of the braking condition. I can do it. In some embodiments, the maximum braking rate of the host vehicle may not be achieved. For example, if the braking condition ceases to exist before the braking system reaches the maximum braking rate, the at least one processor may be programmed to terminate the braking without ever reaching the host vehicle's maximum braking rate. .
[0693] In some embodiments, the maximum braking capacity used to calculate braking distance is a constant number (or other can be any mathematical expression). In some embodiments, the maximum braking force may vary with respect to that used in the formula, and may vary with respect to that which the vehicle can provide at each moment and depends on, for example, road surface characteristics, weather conditions, and the vehicle and its various systems and It can be the actual braking force, which can be caused by dynamic factors including component conditions.
[0694] Consistent with this disclosure, a system for navigating a host vehicle is disclosed. In some embodiments, this system may be an ADAS system or other navigation system disclosed herein. A navigation system may include at least one processor or processing device programmed to perform one or more methods, processes, functions or operations consistent with this disclosure. The processor or processing device may be processing device 110 or another processor or processing device within or in communication with the host vehicle (eg, at least one processing device of a brake system).
[0695] FIG. 51 is a flowchart illustrating an example process 5100 that may be performed by at least one processing device of a navigation system. Process 5100 is for illustrative purposes and is not intended to be limiting. Those skilled in the art with the benefit of this disclosure will appreciate that process 5100 may include additional steps, exclude certain steps, or be otherwise modified in a manner consistent with this disclosure. can do.
[0696] The process 5100 may include a step 5102 of receiving at least one image representative of the host vehicle's environment. Consistent with this disclosure, at least one processing device may be programmed to receive at least one image representative of the host vehicle's environment. At least one image may be received from an image capture device. The image capture device can be anything consistent with this disclosure, including image capture device 122. In some embodiments, the at least one image is an image obtained from either a camera, radar, lidar, or any other device (whether optical or otherwise) from which an image can be obtained. It can be.
[0697] Process 5100 may include step 5104 for determining a planned navigational movement. Consistent with this disclosure, at least one processing device may be programmed to determine a planned navigation operation to achieve a host vehicle navigation goal. Navigation behavior may be determined based on at least one driving policy. The planned navigation maneuver and/or at least one driving policy may be any consistent with this disclosure, including those discussed above. For example, the planned navigation operation may include at least one of a lane change operation, a merging operation, an overtaking operation, a pursuit distance reduction operation, or a throttle maintenance operation. Navigation motion may be determined in substantially the same manner as discussed with respect to step 4804 of process 4800. The processing device may be programmed to analyze the at least one image to identify a target vehicle within the environment of the host vehicle. The at least one image may be an image received from an image capture device, such as image capture device 122. The target vehicle may be a vehicle anywhere in the host vehicle's current neighborhood or in the host vehicle's future neighborhood. For example, the target vehicle may be a vehicle in front of, next to, or behind the host vehicle and/or in front of or next to the host vehicle if the host vehicle performs a planned navigation maneuver. Or it can be a vehicle behind you.
[0698] Process 5100 may include step 5106 to determine the distance of the next state associated with the planned navigational operation. Consistent with this disclosure, the at least one processing device may be programmed to determine a next state distance between the host vehicle and the target vehicle that would occur if the planned navigation operation was performed. . The next state distance can be calculated by any means disclosed herein, including the CRSS safety distance formula above. For example, if the planned navigation action is to accelerate the host vehicle (or change lanes, or even continue at the current speed without braking, etc.), the next state distance is It can be the distance between the vehicle and a target vehicle in front of the host vehicle. In some examples, multiple next state distances can be determined. For example, if the planned navigation operation is to merge into an adjacent lane, a first next state distance may be determined with respect to the host vehicle and a first target vehicle in front of the host vehicle after the merge; A second next state distance may be determined for a second target vehicle behind the host vehicle after the merge.
[0699] Process 5100 may include step 5108 to determine the current speed of the host vehicle. Consistent with this disclosure, at least one processing device may be programmed to determine a current speed of the host vehicle. The current speed may be determined by any means consistent with this disclosure, including those discussed with respect to step 4808 of process 4800. In some embodiments, the current speed of the host vehicle may be determined based on the output of one or more sensors. For example, current speed may be determined from the output of an accelerometer, lidar system, radar system, GPS unit, etc. As another example, the host vehicle's current speed may be determined by analyzing one or more images (e.g., based on changes in the scaling rate of a fixed object detected within two or more images). can be determined.
[0700] Process 5100 may include step 5110 to determine the current speed of the target vehicle and assume maximum braking capability of the target vehicle. Consistent with this disclosure, at least one processing device may be programmed to determine a current speed of the target vehicle and assume a maximum braking rate capability of the target vehicle. A maximum braking rate capability may be assumed based on at least one characteristic of the target vehicle. The target vehicle's maximum braking capacity may be determined (eg, as part of step 4812 of process 4800) by any means consistent with this disclosure, including those discussed with respect to RSS.
[0701] In some embodiments, the current speed of the target vehicle may be determined based on the current speed of the host vehicle. For example, one or more sensors of the host vehicle may provide information related to the host vehicle's speed, and one or more sensors may provide information related to the host vehicle's speed and/or the target vehicle's location relative to the host vehicle. , which can be used to determine the current speed of the target vehicle. In some embodiments, the speed of the target vehicle can be determined based on analysis of at least one image. The at least one image may be the same image in which the target vehicle was first detected or a different image. In some embodiments, multiple images may be analyzed to determine the target vehicle's current speed. For example, an image captured at a first time point may show the target vehicle at a first distance from the host vehicle, and an image captured at a second time point may show the target vehicle at a second distance from the host vehicle. The target vehicle can be indicated and the processing device can determine the current speed based on the change in distance between the two images. In some embodiments, the speed of the target vehicle may be determined based on an analysis of output from at least one lidar or radar system associated with the host vehicle. For example, the processing device can determine the speed of the host vehicle using the speed of the host vehicle and the distance between the host vehicle and the target vehicle (eg, as sensed by the lidar system).
[0702] At least one characteristic of the target vehicle may be determined by any means consistent with this disclosure. In some embodiments, the processing device can determine at least one characteristic based on one or more outputs from one or more sensors associated with the host vehicle. For example, at least one characteristic may be determined based on at least one of lidar output or radar output. For example, the lidar output can include one or more distances related to, for example, the distance between the host vehicle and the top of the target vehicle and the bottom of the target vehicle, and the difference is used to size the target vehicle. can be determined. In some embodiments, at least one characteristic can be determined based on analysis of at least one image. The at least one image may be the image in which the target vehicle was first detected or a different image. For example, the image of the target vehicle may include text, logos, or other information that may be used to determine the type, model, trim level, or other characteristics of the vehicle. As another example, the image of the target vehicle may include other vehicles or objects that may be used to determine the size of the target vehicle in comparison to the target vehicle.
[0703] The recognized characteristics of the target vehicle may be any that may be useful in determining the maximum braking capability of the target vehicle. In some embodiments, the recognized characteristics of the target vehicle may include vehicle type. The type of vehicle can be, for example, a general category to which the target vehicle belongs (eg, large sedan, compact car, SUV, crossover SUV, motorcycle, etc.) or a more specific category or subcategory related to the target vehicle. The processing device may be programmed to assume that the target vehicle has a maximum braking capacity that matches a maximum braking capacity of the known braking capacities for that vehicle type or class of vehicles. For example, if the target vehicle is determined to be a motorcycle, the processing device determines the maximum braking capability of the motorcycle that has the best maximum braking capability (e.g., the motorcycle that can stop most quickly or with the shortest difference). It may be assumed that the target vehicle has a matching maximum braking capacity. In some embodiments, the recognized characteristics of the target vehicle may include a model of the vehicle. The vehicle model may include, for example, the vehicle brand name and/or model name. The vehicle model of the target vehicle may be revealed by any means consistent with this disclosure, including, for example, detecting text, logos, or other identifiers of the vehicle model or detecting characteristics of features of the vehicle model. . For example, the processing device may analyze an image of the target vehicle and recognize text associated with the model of the vehicle (eg, the text "Honda Accord" may be detected on the rear of the target vehicle). As another example, the processing device may analyze the image and recognize one or more differential characteristics of the target vehicle (e.g., tail light shape, presence or absence of a spoiler, general shape of the target vehicle, etc.). The type of vehicle can be determined, for example, by comparing the recognized features with one or more images of known vehicle types. The processing device may use the vehicle type to determine the target vehicle's braking capabilities. For example, if the vehicle type is a Ford Escape®, the processing device may access a database or search the Internet to find the reported braking capabilities of the Ford Escape®. Some vehicle types may have multiple reported braking capabilities depending on the features offered in the vehicle's trim level, for example. The processing device may be programmed to assume that the target vehicle has the best maximum braking capacity for that type of vehicle. In some embodiments, the recognized characteristics of the target vehicle may include the size of the vehicle. The size of a vehicle can be the physical size of the vehicle, the weight associated with the vehicle, a combination thereof, and the like. The size of the vehicle may be a relative size, one of a plurality of predetermined sizes, etc. For example, the processing device may be programmed to classify the target vehicle into one of a plurality of predetermined size ranges (eg, compact = less than 1,000 kg, compact = 1,000-1,500 kg, etc.). In some embodiments, the size of the target vehicle is based on a comparison of the size of the target vehicle and the size of the host vehicle, based on a comparison with other vehicles on the road, and compared to objects near the target vehicle. It can be estimated based on the following. For example, the processing device may analyze an image of the target vehicle and determine that the target vehicle is larger than the host vehicle, but smaller than the target vehicle's neighbors. The size of the vehicle may be used to assume the maximum braking capacity of the target vehicle. For example, the processing device may be programmed to assume that larger, heavier vehicles take longer to stop.
[0704] The target vehicle's maximum braking capacity may be determined based on additional factors. For example, the target vehicle's current maximum braking capacity may be determined based on sensed road surface conditions. In this example, the detected road condition may include the roughness of the road, the slope or slope of the road, the presence or absence of materials or objects on the road, whether the road is asphalt, cement, gravel, or another material, or Any other terms consistent with the disclosure may be included. As another example, the target vehicle's current maximum braking capacity may be determined based on sensed weather conditions. In this example, weather conditions include any precipitation (e.g., rain, sleet, snow, ice, etc.), weather conditions that affect visibility (e.g., fog, haze, smoke, etc.), and weather conditions that affect vehicle maneuverability. This may include weather conditions that may be present (eg, strong winds, high temperatures, etc.) or any other weather conditions that may affect the host vehicle's navigation response. In another example, the processing device determines whether the host vehicle includes, for example, new tires, old tires, one passenger, multiple passengers, a significantly heavy load, or a trailer. The maximum braking capacity can be determined based on, etc. In some embodiments, the maximum braking capacity of the target vehicle may be determined based on predetermined factors, such as rules providing maximum braking capacity allowed for the vehicle or type of vehicle.
[0705] The assumed maximum braking capacity is used to determine the distance the target vehicle could travel to stop from its current speed if the target vehicle were to brake at its maximum braking capacity. obtain. This distance can be calculated by any means consistent with this disclosure, including those discussed with respect to RSS and/or CRSS.
[0706] Process 5100 may include step 5112 for implementing the planned navigation operation if it is safe. For purposes of step 5112, one example of when a planned navigation maneuver can be considered safe is to use a target vehicle that is determined based on the target vehicle's current speed and the target vehicle's assumed maximum braking rate capability. This is the case when the host vehicle can be stopped using a submaximal braking rate within a stopping distance of the host vehicle that is less than the moving stopping distance of the vehicle plus the determined next state distance. In some embodiments, the at least one processing device calculates the determined next state distance to the distance traveled of the target vehicle for the determined current speed of the host vehicle and at a predetermined submaximal braking rate. If the host vehicle can be stopped within a stopping distance of the host vehicle that is less than the sum, the host vehicle may be programmed to perform the planned navigation operation. The predetermined submaximal braking rate may be less than the maximum braking rate capability of the host vehicle. The distance traveled by the target vehicle may be a distance determined based on the target vehicle's current speed and the target vehicle's assumed maximum braking rate capability. As explained above, the submaximal braking rate is j<sub>max</sub>The jerk-bound braking rate can be expressed as . In some embodiments, the submaximal braking rate may be any deceleration rate that is related to, but below, the maximum braking capability of the host vehicle. For example, the predetermined submaximal braking rate may relate to a deceleration rate of up to 50% of the deceleration rate associated with the host vehicle's maximum braking rate capability. In another example, the predetermined submaximal braking rate may be associated with a deceleration rate of up to 20% of the deceleration rate associated with the maximum braking rate capability of the host vehicle. In some embodiments, a maximum braking rate capability of the host vehicle can be determined such that a submaximal braking rate can be determined. The maximum braking rate capability may be determined based on any means consistent with this disclosure, including those discussed with respect to RSS. For example, the host vehicle's maximum braking rate capability may be determined based on sensed road conditions. As another example, the host vehicle's maximum braking rate capability may be determined based on sensed weather conditions.
[0707] In some embodiments, as discussed above, the predetermined submaximal braking rate may be a submaximal braking rate selected by the user. For example, a predetermined submaximal braking rate is a mode in which the host vehicle's brakes are always applied at a predetermined submaximal braking rate until the host vehicle is stopped or until it is determined that the braking condition no longer exists. a mode in which the host vehicle's brakes are always applied at a predetermined submaximal braking rate for at least a portion of the period during which the condition is determined to exist, and then the host vehicle's brakes are applied at the host vehicle's maximum braking rate; and/or or may be determined based on user-selectable braking modes, including modes in which the host vehicle's brakes are applied starting at a predetermined submaximal braking rate and increasing progressively to a maximum braking rate for the host vehicle.
[0708] Consistent with this disclosure, the host vehicle's stopping distance may be determined using one or more of the CRSS equations discussed above. For example, the host vehicle's stopping distance may be determined based on the host vehicle's current speed and the host vehicle's predetermined submaximal braking rate. As another example, the host vehicle's stopping distance may be determined based on the host vehicle's current speed, the maximum acceleration that the host vehicle can achieve during the response period, and a predetermined submaximal braking rate of the host vehicle. In some embodiments, the stopping distance of the host vehicle is an acceleration distance that corresponds to a distance that the host vehicle can travel at the host vehicle's maximum acceleration capability over a predetermined period of time and a host vehicle at the host vehicle's maximum braking rate capability. The corresponding maximum braking rate distance can be greater than the distance that the host vehicle can travel while decelerating from its current speed to zero speed. In some embodiments, the stopping distance of the host vehicle is an acceleration distance that corresponds to the distance that the host vehicle can travel at the host vehicle's maximum acceleration capability over a predetermined period of time and a predetermined submaximal braking rate of the host vehicle. The maximum braking rate can be greater than the distance corresponding to the distance that the host vehicle can travel while decelerating from the host vehicle's current speed to zero speed. In any embodiment, the predetermined period of time may be a reaction time associated with the vehicle. As discussed above with respect to RSS, for example, the predetermined period of time may be the period between when the processing device first detects the target vehicle and when the host vehicle begins to brake.
[0709] In some embodiments, the stopping distance of the host vehicle is the first distance at which the host vehicle is braked at a predetermined submaximal braking rate and the distance at which the host vehicle is able to brake at a maximum braking rate of the host vehicle. and a second distance over which the brakes are applied. For example, as discussed above, the host vehicle may brake at a submaximal braking rate with CRSS for a first distance and then brake at a maximum braking rate for a second distance. In another example, the host vehicle may brake at a submaximal braking rate, gradually increase the submaximal braking rate until the maximum braking rate is reached, and then continue to brake at the maximum braking rate. The processing device causes the host vehicle to brake at a predetermined submaximal braking rate for a first distance before braking the host vehicle at a maximum braking rate capability of the host vehicle for a second duration. can be configured. For example, the host vehicle may brake at a submaximal braking rate with CRSS for a first duration and then brake at a maximum braking rate for a second duration.
[0710] Another example of when the navigation maneuver planned in step 5112 may be considered safe is the ability of the target vehicle to be determined based on the target vehicle's current speed and the target vehicle's assumed maximum braking rate capability. If the host vehicle can be stopped using a predetermined submaximal braking rate within a stopping distance of the host vehicle that is less than the distance traveled plus the determined next state distance, and the predetermined braking rate The profile increases progressively from a submaximal braking rate to a maximum braking rate of the host vehicle. In some embodiments, the at least one processing device determines the current speed of the target vehicle and the assumed maximum braking rate capability of the target vehicle for the determined current speed of the host vehicle and for the predetermined braking rate profile. performing the planned navigation operation if the host vehicle can be stopped within a stopping distance of the host vehicle that is less than a travel distance of the target vehicle determined based on the determined next state distance; The predetermined braking rate profile can be programmed to progressively increase from a submaximal braking rate to a maximum braking rate of the host vehicle. The stopping distance of the target vehicle may be determined as discussed with respect to the embodiments above. The difference between this embodiment and the previous embodiment is that the braking profile of this embodiment can be progressively increased up to the maximum braking rate of the host vehicle. The CRSS safety distance is defined when the host vehicle brakes at its submaximal rate for at least part of the braking period and the target vehicle brakes at its maximum rate (at the current speeds of the host and target vehicles). ) may include the distance within which the host vehicle can stop and not collide with the target vehicle.
[0711] In this embodiment, the braking rate profile may be increased by any means consistent with this disclosure. For example, the predetermined braking rate profile may increase linearly from a submaximal braking rate to a maximum braking rate of the host vehicle. As another example, the predetermined braking rate profile may increase non-linearly from a submaximal braking rate to a maximum braking rate of the host vehicle. In this example, the predetermined braking rate profile may increase exponentially, logarithmically or according to any other function, or may increase sporadically. In either example, it is still possible that the brake profile never reaches its maximum braking capacity. For example, before the vehicle reaches its maximum braking rate capability, braking conditions may no longer exist that would cause the vehicle to begin braking with a predetermined braking rate profile, and the vehicle may The brakes may be applied accordingly.
[0712] Consistent with this disclosure, the at least one processing device determines that the determined stopping distance of the host vehicle is determined based on the current speed of the target vehicle and an assumed maximum braking rate capability of the target vehicle. The planned navigation operation may be configured to perform the planned navigation operation if the target vehicle's travel distance is less than the determined next state distance by at least a predetermined minimum distance. In some embodiments, as discussed above with respect to CRSS and RSS, the minimum distance is d<sub>min</sub>may be predetermined or determined by any means consistent with this disclosure. In some embodiments, the predetermined minimum distance may correspond to a predetermined separation distance maintained between the host vehicle and other vehicles. For example, if the host vehicle and target vehicle are traveling in the same direction and both come to a stop, the predetermined separation distance may be the minimum distance between the host vehicle and the target vehicle after both come to a stop. The predetermined separation distance can be any length of distance. For example, the predetermined separation distance can be at least 1 meter or other suitable minimum approach distance. In some embodiments, the predetermined distance may vary depending on, for example, the speed at which the host vehicle is traveling, the location of the host vehicle (eg, city road, local highway, etc.). For example, d<sub>min</sub>can be increased by increasing the speed of the host vehicle.
[0713] The embodiments discussed with respect to CRSS are exemplary only. Those skilled in the art having the benefit of this disclosure can understand that CRSS may be used in place of or in conjunction with RSS in any of the embodiments discussed in this disclosure. In some embodiments, the host vehicle may be navigable under CRSS mode or RSS mode. For example, a fully autonomous vehicle may be configured to navigate under the rules of the CRSS model, but a passenger within the vehicle may disengage the CRSS model and choose to navigate under the RSS mode. For example, selecting such a mode may allow the host vehicle to move closer to the target vehicle while still maintaining a safe distance. However, the trade-off is that RSS mode may be associated with higher deceleration rates than CRSS mode and may be less comfortable for the host vehicle's passengers. In some embodiments, the autonomous vehicle may be able to navigate under each of CRSS and RSS modes. For example, when a vehicle is traveling under a first condition (e.g., at high speed on a rural expressway), it adopts the CRSS model, and under a second condition (e.g., on an urban expressway), the vehicle adopts the CRSS model. The vehicle may be configured to employ RSS mode when traveling at low speeds).
[0714] Consistent with this disclosure, CRSS modes may be used to navigate fully autonomous vehicles, partially autonomous vehicles (e.g., having a human-operable mode and a selective autonomous mode). human-operable vehicles) or as a basic safety feature in human-operable vehicles (e.g., human-operable vehicles are required to navigate potentially dangerous situations under the CRSS model) or as a basic safety feature in human-operable vehicles (e.g., (can be prevented or avoided).
[0715] Vision Zero Safety System for Driver Operated Vehicles [0716] In some embodiments, a safety system (e.g., Vision Zero) may be employed in a host vehicle that has autonomous or partially autonomous capabilities, but where a human driver is allowed to operate in a driver-controlled mode. . In that case, the safety system can operate in the background. The human driver may be allowed to perform any navigation actions he or she desires, as long as the navigation actions do not result in less than a CRSS or RSS distance to the detected target vehicle. A human driver may not be aware of the operation of the safety system if the driver does not perform any navigation actions that result in an approach of less than the CRSS or RSS distance (e.g., depending on the selected operating mode) to the target vehicle. On the other hand, if the driver initiates a navigation action that brings the target vehicle closer than the CRSS or RSS distance, the safety system takes one or more actions to prevent completion of the initiated navigation action. In other words, the vehicle can take control of the vehicle away from the driver and avoid navigating into dangerous situations involving approaches less than CRSS or RSS. The driver regains control when the driver aborts the attempt to navigate into conditions that the vehicle detects as dangerous, or when the driver initiates a different navigation action that is deemed safe by the host vehicle.
[0717] The disclosed safety system is capable of controlling a vehicle to prevent navigation operations by the driver associated with potentially dangerous navigation operations (e.g., operations detected as having the result of violating a CRSS envelope or an RSS envelope). inputs, inhibit certain navigation inputs, alert the driver, and/or a combination thereof. For example, the safety system may require that the vehicle's Lateral movement can be prevented. An example of a dangerous interrupt that can be prevented by a safety system is shown in FIG. 30A. As another example, the safety system may prevent navigation operations that would result in longitudinal movement that would therefore result in less than the RSS or CRSS distance between the host vehicle and the target vehicle (eg, a leading target vehicle). An example of dangerous vertical movement that can be prevented by a safety system is shown in Figure 28B. As another example, the safety system may prevent lateral or vertical movement of the host vehicle (in response to driver input) that would result in less than a predetermined distance or less than a safe distance between the host vehicle and a pedestrian. . Examples of hazardous movement relationships for pedestrians that can be prevented by safety systems are shown in FIGS. 44A and 44C.
[0718] Consistent with the present disclosure, the safety system can prevent a navigation maneuver attempted by a human driver if the maneuver is dangerous. As used herein, "preventing" may include any means for preventing a human driver from performing navigation operations. For example, if the dangerous navigation maneuver is a merge, the safety system can prevent equal but opposite steering inputs by locking the steering wheel or other steering mechanism ( e.g., so that the steering wheel cannot be turned in the direction of the merge ). by canceling the driver's steering inputs, by blocking or interfering with the electrical signals associated with the driver's steering inputs, or by eliminating the navigation response of the driver's steering inputs or by reducing the navigation response to less than the intended navigation response. Merging can be stopped by having a navigation response. As another example, if the dangerous navigation maneuver is acceleration, the safety system may accelerate the acceleration by eliminating the effect of the acceleration input (e.g., blocking the electrical signal, locking the gas pedal, applying the brakes, etc.). can be stopped. Such prevention is sufficient to avoid or stop navigation maneuvers that may otherwise be dangerous (e.g. maneuvers determined to have a resulting CRSS or approach to a target vehicle below RSS, etc.). It can be.
[0719] Consistent with this disclosure, the safety system may take over the vehicle to avoid dangerous navigation maneuvers attempted by the human driver. During the takeover period, the safety system may operate in an autonomous mode, e.g., taking full control of the host vehicle until it is determined that driver inputs no longer result in dangerous navigation maneuvers. As used herein, "takeover" refers to one or more autonomously controlled systems in which the safety system can operate or control one or more of the steering, braking, acceleration or other vehicle control systems. may include any manipulation of the actuator. For example, if a driver input is determined to be associated with a dangerous merge, the safety system may prevent the driver from providing additional steering input to cause the merge or prevent the driver input from having any effect. be able to temporarily control the vehicle's steering mechanisms to steer the vehicle in a direction to avoid dangerous maneuvers (e.g., perform anti-phase steering in response to driver inputs initiated on the steering wheel) )be able to. As another example, if the dangerous operation is acceleration, the safety system may control the acceleration mechanism such that, for example, the driver cannot provide additional input to the throttle, or the driver input has no effect. The vehicle can be slowed down, maintained at its current speed, or coasted to avoid a dangerous situation. Safety system takeover is designed to avoid navigation maneuvers that are determined to be unsafe by the system and, among other things, to avoid collisions that may be attributable to the host vehicle.
[0720] As used herein, the term "replace" is used to refer to a safety system replacing a human driver, inhibiting, preventing control by the human driver. , or taking control away from the human driver.
[0721] As previously mentioned, the autonomous safety system may operate in the background until a driver input is detected that is determined to be associated with a dangerous condition or dangerous maneuver. A safety system may replace (at least temporarily) human driver control if driver input is detected that is determined to be associated with an unsafe condition or unsafe operation. A dangerous condition may include, for example, a determination that the driver's input is associated with a navigation maneuver that would place the host vehicle at a distance less than a CRSS safe distance or an RSS safe distance from the target vehicle. As an example, the processing device of the safety system uses the longitudinal and/or lateral safety distances developed under the RSS model and/or the CRSS model to determine the proximity to the host vehicle (or detected target vehicle). Buffers can be built. In this example, a safe condition may include any condition in which there are no objects (eg, a target vehicle or VRU) within the proximity buffer. Conversely, a hazardous condition may include any condition in which there is or will be an object within the proximal buffer. The proximity buffer may include a two-dimensional area or a three-dimensional area around the host vehicle. The dimensions of the proximity buffer can be determined using the RSS and/or CRSS safety distance calculations described above. For example, the proximity buffer may include, for example, a distance in front of the host vehicle corresponding to a CRSS safe longitudinal distance or an RSS distance, a distance on either side of the host vehicle corresponding to a predetermined safe lateral distance, and /or may include a distance behind the host vehicle corresponding to the RSS safety distance, for example.
[0722] 52A-52D are visual representations of example proximity buffers consistent with this disclosure. FIG. 52A shows a proximity buffer 5202 around a host vehicle 5201. In this example, proximity buffer 5202 is derived from a determined safety distance 5204 in front of host vehicle 5201 and from a determined safety distance 5206 behind host vehicle 5201. The determined safe distance 5204, which is the shortest safe distance in the examples of FIGS. 52A-52D, may be the RSS longitudinal safe distance. Accordingly, the determined safe distance 5204 may be determined as described above. Similarly, the determined safety distance 5206 may be the RSS longitudinal safety distance. Although the longitudinal safety distances including the sides of the proximal buffer area 5202 are not labeled, it will be appreciated that they may also be determined by any means consistent with this disclosure. Such a lateral safety distance may be predetermined to maintain a minimum lateral separation between the host vehicle and other vehicles or objects. Such a lateral safety distance may be, for example, 0.25 m, 0.5 m, 1.0 m or more. In some embodiments, the sides of the proximity buffer may correspond to predetermined lateral distance thresholds that may define a minimum safe lateral distance between the host vehicle and the object.
[0723] FIG. 52B shows a proximity buffer 5212 around a host vehicle 5211. In this example, the determined safe distance 5214 in front of host vehicle 5211 is longer than the determined safe distance 5204 in front of host vehicle 5201. In some embodiments, the determined safety distance 5214 may be longer than 5204, for example, because the determined safety distance 5214 may correspond to a CRSS safety distance. As explained above, the determined safe distance 5214 may correspond to a minimum approach distance as well as an RSS safe distance. As mentioned above, the determined safe distance 5214 may correspond to a CRSS safe distance that includes a braking distance associated with a submaximal braking rate. In some embodiments, the determined safe distance 5214 may be longer than the determined safe distance 5204 because the host vehicle 5211 may have different braking characteristics or features, etc. compared to the host vehicle 5201. be. For example, both determined safe distance 5204 and determined safe distance 5214 may be RSS safe distances, with a higher current speed and/or lower maximum braking of host vehicle 5211 compared to host vehicle 5211. The determined safety distance 5214 may be longer as it may be related to capacity.
[0724] FIG. 52C shows a proximity buffer 5222 around a host vehicle 5221. In this example, determined safe distance 5224 is longer than determined safe distance 5204 and shorter than determined safe distance 5214. In some embodiments, the determined safe distance 5224 may be shorter than the determined safe distance 5214 and longer than the determined safe distance 5204 for any of the reasons described with respect to FIG. 52B. For example, as described with respect to the RSS model, the determined safe distance 5224 may be the RSS safe distance that includes the minimum approach distance, but the maximum acceleration capability of the host vehicle 5221 is less than the maximum acceleration capability of the host vehicle 5211. The possible and/or predetermined approach distance may be shorter for host vehicle 5221 than for host vehicle 5211. As another example, the passenger or driver of host vehicle 5221 may have selected a first submaximal braking rate, and the passenger or driver of host vehicle 5211 may have selected less than the first submaximal braking rate. may have selected a second submaximal braking rate, and therefore, host vehicle 5211 may require a longer distance (eg, determined safe distance 5214) to stop safely. In this example, host vehicle 5221 has a longer determined safe distance 5226 behind the vehicle. The model used to determine the distance (e.g., RSS and/or CRSS), the state or characteristics of host vehicle 5221 (e.g., maximum acceleration capability, Maximum braking capacity, current speed, current braking capacity, submaximal braking rate, etc.) and/or road conditions (e.g. weather conditions, road material, etc.), the length of the determined safety distance 5226 is: It may be longer than the determined safe distance 5216.
[0725] FIG. 52D shows a proximity buffer 5232 around a vehicle 5231. Proximity buffer 5232 is larger than any of proximal buffers 5205, 5212, or 5222. In some embodiments, a proximity buffer may be used, e.g., because the CRSS model was used for both the determined safety distance 5234 in front of the host vehicle 5231 and the determined safety distance 5236 behind the host vehicle 5231. 5232 may be larger than other buffers. In some embodiments, the size of the proximal buffer zone 5232 may be due to any of the factors discussed with respect to FIGS. 52A-52C.
[0726] Consistent with this disclosure, a host vehicle may have multiple proximity buffers. For example, a host vehicle may have a first proximity buffer (eg, proximity buffer 5202) associated with an RSS model and a second proximity buffer (eg, proximity buffer 5232) associated with a CRSS model. In some embodiments, the safety system can replace the human driver if the first proximity buffer is violated. For example, if the host vehicle approaches the target vehicle at a distance less than the RSS safety distance, the safety system may displace the driver to prevent a collision. In the same embodiment, the safety system alerts the human driver if the second proximity buffer is violated, but can do without driver replacement. For example, if the host vehicle approaches the target vehicle at a distance less than the CRSS safety distance, the safety system may e.g. issue an audible warning over the speaker system, vibrate the seats, steering mechanism or other components in the passenger compartment. The host vehicle may alert the driver that the host vehicle is too close to the target vehicle by causing the host vehicle to approach the target vehicle, by displaying the alert on a heads-up display or augmented reality display, or by any other means. In some embodiments, the safety system can replace the human driver if the second proximity buffer is violated. For example, if the host vehicle approaches the target vehicle at a distance less than the CRSS safety distance, the safety system may decelerate the vehicle at a rate corresponding to the submaximal braking rate of the CRSS model. If the distance between the host vehicle and the target vehicle violates the first proximity buffer by becoming closer than the RSS safe distance, the safety system brakes the vehicle at a rate corresponding to the maximum braking rate of the host vehicle according to the RSS model. can be slowed down. As explained above, the safety system may also replace the human driver whenever input from the driver is determined to result in an approach of less than the RSS or CRSS distance to another vehicle or object. .
[0727] As explained above, each of the RSS and CRSS models may calculate a safe distance for a detected object. For example, each model can determine the braking capability of the target vehicle and use that braking capability to determine a safety distance. As such, the dimensions of the proximity buffer may vary depending on the objects detected within the environment of the host vehicle. As an example, if the second target vehicle has a higher maximum braking capacity than the first target vehicle (i.e., the second target vehicle can stop in a shorter distance than the first target vehicle), then the host vehicle and the first The RSS safe distance between the target vehicle and the host vehicle may be longer than the RSS safe distance between the host vehicle and the second target vehicle. As another example, the minimum separation distance between the host vehicle and the pedestrian may be greater than the minimum separation distance between the host vehicle and the target vehicle, so the RSS between the host vehicle and the pedestrian The safety distance or CRSS safety distance may be significantly longer than the RSS safety distance or CRSS safety distance between the host vehicle and any target vehicle.
[0728] The safety system must be able to operate for a duration corresponding to a hazardous condition, such as a violation of the proximity buffer, or hazardous driver input (e.g., resulting in an approach less than the RSS distance or CRSS distance to another vehicle or object). The human driver may be replaced during the period during which at least one driver input determined to be detected is detected. For example, the safety system may control the vehicle until the hazardous condition no longer exists (eg, the vehicle moves away from the target vehicle by more than a CRSS or RSS safety distance).
[0729] FIGS. 53A and 53B show examples of safe and unsafe conditions between two vehicles. FIG. 53A shows a host vehicle 5301 and a target vehicle 5303 traveling in a common direction along a road 5302. In this example, host vehicle 5301 may include a safety system and proximity buffer 5304 consistent with this disclosure. Similarly, target vehicle 5303 can include a safety system consistent with this disclosure and has a proximity buffer 5306. In this example, if host vehicle 5301 is being driven by a human driver, the human driver can continue to navigate host vehicle 5301 because there are no objects within proximity buffer 5304 (e.g., human driver are not replaced by the safety system). The same may be true for the human driver navigating target vehicle 5303.
[0730] FIG. 53B shows a host vehicle 5311 and a target vehicle 5313 traveling in a common direction along a road 5312. In this example, host vehicle 5301 may include a safety system and proximity buffer 5314 consistent with this disclosure. Similarly, target vehicle 5313 can include a safety system consistent with this disclosure and has a proximity buffer 5316. In this example, host vehicle 5311 is being driven by a human driver, and the safety system of target vehicle 5313 receives driver inputs that are determined to be one (or more) that cause the interrupt shown in FIG. 53B. If detected, the safety system can take control and replace the human driver to prevent the operation. For example, the safety systems associated with vehicle 5313 may be affected by situations where the lateral safety distance may be violated after an operation, insufficient longitudinal clearance to the rear of vehicle 5313 (e.g. the RSS distance is It can be determined that the driver's input will cause the driver's input to occur, such as a situation (that does not exist). In that case, the safety system responds by preventing the driver's input from causing a dangerous interrupt operation. In the same scenario of FIG. 53B, if vehicle 5313 performs an interrupt operation, e.g., if the driver does not apply the brakes to slow the vehicle to establish the CRSS distance or RSS distance associated with buffer zone 5314, vehicle 5311 Safety systems can replace driver input. In such situations, the driver's dangerous inputs may correspond to a lack of change in the inputs, in other words maintaining the current inputs/controls in a dangerous state.
[0731] FIGS. 54A and 54B illustrate additional examples in which a safety system consistent with this disclosure may replace a human driver. In FIGS. 54A and 54B, vehicle 5402, vehicle 5404, and vehicle 5406 are each traveling in a common direction on road 5400. In this example, road 5400 may be a highway. Vehicle 5402 may include a safety system consistent with this disclosure and may have a proximity buffer 5403. As discussed above, proximity buffer 5403 may correspond to either an RSS model and/or a CRSS model. In this discussion, the proximity buffer 5403 is assumed to have a determined safety distance related to the CRSS model in front of the vehicle 5402.
[0732] In FIG. 54A, a human driver within vehicle 5402 may approach vehicle 5402 to the rear of vehicle 5406 as shown. If a driver attempts to cause vehicle 5402 to approach vehicle 5406 at a distance less than the CRSS safe distance (as indicated by vehicle 5406 being within the proximity buffer 5403), vehicle 5402's safety systems displace the driver. to avoid dangerous situations and maintain CRSS distance. For example, the safety system can take control and prevent or ignore driver acceleration inputs, or apply the brakes of the vehicle 5402. For example, the safety system can decelerate vehicle 5402 at a rate corresponding to a submaximal braking rate of the CRSS model until vehicle 5402 moves away from vehicle 5406 at a distance greater than proximity buffer 5402. As another example, the safety system determines the next state location of vehicle 5402 (e.g., the location where vehicle 5402 would be if the driver input was performed) before proximity buffer 5403 is violated. and the driver can be replaced before vehicle 5402 approaches vehicle 5406 within a distance less than the CRSS safe distance.
[0733] In FIG. 54B, a human driver within vehicle 5402 may attempt to merge vehicle 5402 into a lane occupied by vehicle 5404. If the driver attempts a navigation maneuver that would cause vehicle 5402 to approach vehicle 5404 at a distance less than the CRSS safe distance (as indicated by the overlap between proximity boundary 5403 and vehicle 5404), the safety system replaces the human driver. navigation operations can be prevented. For example, if a human driver attempts to veer to the right by turning the steering wheel to the right, the safety system may determine that performing that veer change would result in the dangerous condition shown in Figure 54B. . Safety systems can respond by taking control and replacing the human driver. For example, the safety system can prevent turning the steering wheel to the right by applying an equal force to the left on the steering wheel. As another example, a safety system may prevent a right turn by mechanically and/or electrically interfering with a right turn input to eliminate the effect of a right turn of the steering wheel. As another example, the safety system may take over the vehicle 5402 and steer to the left if necessary to avoid the hazardous condition that would result if the vehicle 5402 were to veer to the right. For any of these examples, until vehicle 5402 is returned to a safe state (e.g., to a position where neither vehicles 5404 nor 5406 are within proximity boundary 5403), and/or the next state condition associated with vehicle 5402 is Substitutions may continue until determined to be safe.
[0734] The examples shown in FIGS. 53A to 53B and FIGS. 54A to 54B are merely illustrative. It should be appreciated by those skilled in the art who have the benefit of this disclosure that a safety system or similar navigation system can replace a human driver in any dangerous condition or in response to any input that could cause a dangerous condition. can be understood.
[0735] In some embodiments, the safety system may store or transmit data related to any instance in which the safety system replaces a human driver. For example, the safety system can identify the hazardous condition that caused the safety system to displace the driver, the type of displacement that occurred (e.g., interdiction, prevention, or takeover), the duration of the displacement, and the consequences of the displacement (e.g., the vehicle is now in a safe state. returning, the vehicle stopping, the vehicle being involved in a collision) or any other information relating to the replacement or events occurring before and after the replacement. Consistent with this disclosure, this information can be used to determine whether a safety system, RSS model, CRSS model, or any other autonomous vehicle characteristic is functioning as expected. Such information may be transmitted from the host vehicle to a remote server over one or more networks. For example, the information may be transmitted for analysis to an automobile manufacturer, safety device manufacturer, or other party responsible for implementing the safety system. In some embodiments, information may be displayed to the driver of the vehicle. For example, reports may be generated after each displacement event, after each trip, after receiving a request from a driver, or at scheduled intervals (eg, daily, weekly, biweekly, monthly, etc.). It is believed that providing the driver with information related to the safety system and the unsafe conditions avoided can increase the confidence the driver has in the safety system (or in autonomous vehicles in general).
[0736] In some embodiments, the vehicle may include a means (eg, a switch, button, voice-activated control, or other type of control) for deactivating or disabling the safety system. For example, the driver may deactivate the safety system to prevent the safety system from disengaging the driver, or disengage the safety system during a displacement event to regain control of the host vehicle. The controls for deactivating or disabling the safety system are considered independent of any of the vehicle's control mechanisms associated with navigating the vehicle. For example, a vehicle may include buttons that are not associated with the throttle, brakes, steering wheel, or any other control mechanism. In this example, the driver can only deactivate or disable the safety system by pressing a button (or other control of the safety system). Unlike traditional ADAS systems (i.e., modern driver assistance systems such as lane keeping systems, automatic braking systems, etc.), the disclosed safety system is deactivated or deactivated by providing input to the steering control, brake control, or throttle control. I can't. For example, if the safety system takes over the vehicle's steering mechanism to make a right turn, the driver may, for example, turn the steering wheel to the right (e.g., with increasing force in an attempt to overcome or disable the safety system). The safety system cannot be deactivated by continuing. Rather, the driver must first activate a designated safety system override control to disable the safety system. Similarly, if the disclosed safety system provides controls to prevent navigation operations that result in longitudinal distances less than CRSS or RSS, the driver cannot override the safety system with inputs to the vehicle's brakes or throttle. Rather, designated safety system controls should be used.
[0737] It will be appreciated that the safety system may be released or deactivated by any suitable control input, such as a lever, knob, virtual button, voice command, hand gesture, etc. In some embodiments, the navigation system may include an image capture device within the vehicle that is configured to capture one or more images or videos of the human driver. The at least one processing device is capable of analyzing the captured image and/or video to recognize a hand gesture or other non-verbal command, and deactivates the safety system in response to the hand gesture or other non-verbal command. or can be programmed to release. In some embodiments, the navigation system may include one or more microphones configured to capture audio data. The at least one processing device analyzes the voice data using a voice recognition process (e.g., using a neural network, a trained system, or an untrained system, etc.) to generate voice data including recognized commands. The safety system can be deactivated or deactivated accordingly.
[0738] Consistent with this disclosure, the safety system may collect information related to any instance in which the safety system is deactivated or deactivated. For example, the safety system may record navigation conditions (eg, host vehicle speed, acceleration, pursuit distance, etc.) and/or the time and location of deactivation immediately before deactivating the safety system. In some embodiments, the safety system may continue to monitor the driver's driving conditions and store or transmit that data as described above even after the safety system is deactivated or disarmed.
[0739] Consistent with this disclosure, a navigation system for selectively replacing human driver control is disclosed. The navigation system can partially replace the driver, e.g. by preventing or inhibiting the driver's movements, or completely replace the driver, e.g. by taking over the vehicle and performing navigation operations independently of the driver. can do. The navigation system may be a safety system or any other system disclosed herein. In some embodiments, the system may be completely housed within the host vehicle. In other embodiments, one or more components of the system may be located remote from the host vehicle, such as within a server or other device. The system may include at least one processing device. At least one processing device may be programmed to perform one or more methods, processes, operations or functions consistent with this disclosure. The at least one processing device may be, for example, processing device 110.
[0740] FIG. 55 is a flowchart illustrating an example process 5500 for selectively replacing a human driver. Consistent with this disclosure, at least one processing device within the navigation system may be programmed to perform all or a portion of process 5500. It is understood that process 5500 is exemplary only and that process 5500 may include additional steps, exclude one or more steps, or be otherwise modified in a manner consistent with this disclosure. This can be understood by one of ordinary skill in the art who has the benefit of this disclosure.
[0741] Process 5500 may include step 5502 of receiving one or more images. The at least one image may, for example, represent the environment of the host vehicle. In some embodiments, the processing device may be programmed to receive at least one image representative of the host vehicle's environment. At least one image may be received from at least one image capture device. For example, the navigation system may include one or more image capture devices, such as image capture device 122, that may capture images of the host vehicle's environment. The image capture device may have a field of view that corresponds to a human driver's field of view. In some embodiments, a navigation system may include multiple image capture devices, each having a different view of the host vehicle's environment. For example, at least one of the plurality of image capture devices can be configured to capture an image representative of the host vehicle's environment behind the host vehicle, and at least one of the plurality of image capture devices can be configured to capture an image representative of the host vehicle's environment behind the host vehicle; The at least one of the plurality of image capture devices may be configured to capture an image representative of an environment of the host vehicle to the side, and/or at least one of the plurality of image capture devices capture an image representative of the environment of the host vehicle in front of the host vehicle. It can be configured as follows. For example, a first image capture device may have a field of view that corresponds to an area in front of the host vehicle, a second image capture device may have a field of view that corresponds to the right side of the host vehicle, and so on. In disclosed embodiments, a processing device may be programmed to receive multiple images. For example, in embodiments where the navigation system includes only one image capture device, the processing device may receive multiple images from the imaging device. As another example, the processing device processes one or more images from a first image capture device and/or one or more images from a second image capture device and one image from a third image capture device. One or more images may be received, etc.
[0742] Process 5500 may include step 5504 of detecting at least one obstacle in the host vehicle's environment. At least one obstacle can be detected by analyzing at least one image. The at least one image may be one or more images received in step 5502. For example, the processing device may be programmed to detect at least one obstacle in the host vehicle's environment based on analysis of the at least one image. For example, the processing device may compare the received image to one or more images of known obstacles, or identify shapes, characters or other objects in the image that correspond to the one or more obstacles. can be detected. The one or more obstacles may be any object within the host vehicle's environment. For example, the one or more obstacles may be a target vehicle or VRU on or near a roadway on which the host vehicle is traveling. As another example, the one or more obstacles may be a pothole, a handrail, a road sign, a traffic light, a cone, a handrail or barrier or any other object on or along the road. In some embodiments, multiple obstacles may be detected within the host vehicle's environment. For example, the processing device may detect a first host vehicle, a second host vehicle, a traffic light, a pothole, and any number of additional obstacles.
[0743] In embodiments in which the navigation system includes a plurality of image capture devices, each of which may have a different field of view with respect to the environment of the host vehicle, the at least one processing device receives one or more images from each of the plurality of image capture devices. and detect at least one obstacle in the environment of the host vehicle based on analysis of the one or more images received from the plurality of image capture devices. For example, at least one of the plurality of image capture devices may be configured to capture an image representative of the host vehicle's environment on the side of the host vehicle, and may be configured to capture one or more images received from the image capture device. It can be used to detect one or more obstacles located on the side of the host vehicle. Similarly, at least one of the plurality of image capture devices may be configured to capture images representative of the host vehicle's environment behind the host vehicle, and may be configured to capture one or more images received from the image capture device. can be used to detect one or more obstacles located behind the host vehicle.
[0744] In some embodiments, the at least one obstacle may be a target vehicle within the host vehicle's environment. For example, the at least one obstacle may be a target vehicle that is determined to be in front of the host vehicle. As another example, the at least one obstacle may be a target vehicle that is determined to be in a different lane than the host vehicle. In some embodiments, the at least one obstacle may include a pedestrian or object in the roadway.
[0745] In some embodiments, the processing device may detect one or more obstacles in the environment using output from one or more sensors associated with the host vehicle. The sensor can be any sensor disclosed herein. For example, the processing device can receive output from the lidar system and/or radar system and use that information to detect one or more obstacles in the vicinity of the host vehicle.
[0746] Process 5500 may include step 5506 for monitoring driver input into the throttle mechanism, brake mechanism, and/or steering mechanism. Consistent with this disclosure, at least one processing device may be programmed to monitor driver input to at least one or more throttle control, brake control, or steering control associated with the host vehicle. Input may be monitored by any technique consistent with this disclosure. For example, one or more sensors may be configured to detect electrical signals related to either throttle, brake or steering control. In another example, one or more sensors may be configured to detect mechanical input to either throttle, brake or steering controls. The throttle control, brake control and/or steering control may be of any type now known or later developed. For example, the throttle control may include an accelerator pedal, the brake control may include a brake pedal, and the steering control may include a steering wheel. In another example, the one or more control systems of the host vehicle include at least one steering actuator for controlling heading of the host vehicle, a brake actuator for applying a braking device of the host vehicle, or a throttle of the host vehicle. may include an accelerator actuator for applying. For example, the throttle control system may include an accelerator actuator for applying the host vehicle's throttle, the brake control system may include a brake actuator for applying the host vehicle's brake pedal, and/or The steering control system can include at least one steering actuator for controlling heading of the host vehicle. The driver input may be any active operation or other activation or lack of operation or other activation of one or more of the controls. Continuing with the example above, the driver inputs may include at least one of depressing the gas pedal, depressing the brake pedal, no depressing the brake pedal, turning the steering wheel, or not turning the steering wheel. .
[0747] Process 5500 may include step 5508 for determining whether the driver's input results in a safe navigation operation. As previously explained, navigation changes that occur when driver input is implemented at a distance less than a proximal buffer around the host vehicle (e.g., an RSS distance, a CRSS distance, or a predetermined minimum lateral buffer distance, among others) Driver inputs may result in a safe condition if the host vehicle does not approach the object (within the vehicle). Consistent with this disclosure, at least one processing device may be programmed to determine whether the driver input causes the host vehicle to navigate within a proximity buffer to the at least one obstacle. The at least one processing device may determine whether the proximity buffer may be violated by any technique consistent with this disclosure. For example, the processing device determines a current distance between the host vehicle and one or more objects (e.g., a target vehicle or VRU), and determines a current distance between the host vehicle and the one or more objects (e.g., a target vehicle or VRU) and The next state distance between multiple objects can be determined. This distance to the object may be determined based on, for example, image analysis (eg, scaling observations), lidar output, radar output, etc. As an example, referring to FIGS. 53A and 53B, if a human driver provides input that causes vehicle 5303 to veer to the left (e.g., by turning the steering wheel of vehicle 5303), the processing device 5301 may be determined to result in a dangerous state as shown in FIG. 53B. As another example, referring to FIG. 54A, if the human driver of vehicle 5402 provides an accelerator input (e.g., by depressing the accelerator pedal), the processing device determines that the acceleration of vehicle 5402 is less than the proximity buffer 5403. vehicle 5402 may approach vehicle 5406 at a distance of , and therefore may be determined to be dangerous. In this case, the safety system can control the host vehicle and prevent dangerous operations related to the detected driver inputs.
[0748] Process 5500 is a function that may define how the navigation system handles instances where driver input causes the host vehicle to violate the proximity buffer and instances where driver input does not cause the host vehicle to violate the proximity buffer. 5510 may be included. In some embodiments, function 5510 may be included in step 5508 of process 5500. The proximity buffer may be any proximity buffer consistent with this disclosure, including those determined using an RSS model and/or a CRSS model. The proximity buffer may depend on detected obstacles or objects in the host vehicle's environment on which the RSS and CRSS models depend. In some embodiments, the proximity buffer may include one or more predetermined distances. For example, the proximity buffer may correspond to a predetermined lateral distance threshold. The lateral distance threshold may include a predetermined minimum distance between the host vehicle and an object located on either side of the host vehicle. As another example, a proximity buffer may correspond to a predetermined longitudinal distance. The longitudinal distance is, for example, the minimum distance d maintained between the host vehicle and the object as described for the RSS model.<sub>min</sub>It can be.
[0749] Consistent with this disclosure, the at least one obstacle may include a target vehicle, and the proximity buffer to the target vehicle may include the host vehicle's detected current speed, the host vehicle's maximum braking rate capability, the target The determination may be based on the determined current speed of the vehicle, the assumed maximum braking rate capability of the target vehicle, and the determined maximum acceleration capability of the host vehicle. In that case, the proximity buffer is at least the acceleration distance of the host vehicle determined as the distance traveled by the host vehicle if accelerated at the host vehicle's maximum acceleration capability over the reaction time associated with the host vehicle, the host vehicle's maximum The host vehicle's stopping distance is determined as the distance required to reduce the host vehicle's current speed to zero at the braking rate capability and the target vehicle's current speed at the target vehicle's assumed maximum braking rate capability. It may include the sum of the target vehicle's stopping distance determined as the distance required to reduce to zero. For example, the proximity buffer can be determined using an RSS safety distance that includes the host vehicle's acceleration and stopping distances and the target vehicle's stopping distance. In some embodiments, the proximity buffer for the target vehicle may be further determined based on a predetermined minimum distance maintained between the host vehicle and the target vehicle. For example, as discussed above, a proximity buffer is defined by a minimum distance d<sub>min</sub>may include.
[0750] Consistent with this disclosure, the at least one obstacle may include a target vehicle, and the proximity buffer to the target vehicle may include the host vehicle's detected current speed, the host vehicle's maximum braking rate capability, the target The determination may be based on the determined current speed of the vehicle, the assumed maximum braking rate capability of the target vehicle, and the maximum acceleration capability of the host vehicle. In some embodiments, the proximity buffer for the target vehicle may be further determined based on a predetermined submaximal braking rate that is less than the maximum braking rate capability of the host vehicle, such that the proximity buffer is , at least at a stopping distance of the host vehicle determined as the distance required to reduce the current speed of the host vehicle to zero at a predetermined submaximal braking capacity of the host vehicle and an assumed maximum braking capacity of the target vehicle. Includes the sum of the target vehicle's stopping distances determined as the distance required to reduce the target vehicle's current speed to zero. For example, the proximity buffer is the stopping distance of the host vehicle when the host vehicle decelerates at a rate corresponding to a predetermined submaximal braking rate and when the target vehicle decelerates at a rate corresponding to its own maximum braking rate capability. The target vehicle's stopping distance can be determined using the CRSS safety distance. In some embodiments, the proximity buffer for the target vehicle may be further determined based on a predetermined minimum distance maintained between the host vehicle and the target vehicle. For example, as discussed above, a proximity buffer is defined by a minimum distance d<sub>min</sub>may include.
[0751] In any of the above embodiments, the target vehicle may be in front of, next to, or behind the host vehicle. For example, the target vehicle may be in front of the host vehicle, and the at least one processing device is configured to determine that the driver input will result in a change in longitudinal distance between the target vehicle and the host vehicle. can be done. For example, the host vehicle may be away from the target vehicle at a distance greater than the proximity buffer distance in front of the host vehicle, and driver input causes the host vehicle to approach the target vehicle at a distance less than the proximity buffer distance. obtain. In some embodiments, it may be determined that the target vehicle is in a different lane than the host vehicle, and the at least one processing device determines that the driver's input will cause a lateral movement of the host vehicle, thereby causing a lateral movement of the host vehicle. The target vehicle may be configured to determine that the target vehicle will be in front of the host vehicle after moving. For example, driver input can cause the host vehicle to merge behind the target vehicle, such merging causing the host vehicle to increase the distance from the target vehicle in front of the host vehicle by exceeding the proximity buffer distance. It can be safe if maintained, and dangerous if the host vehicle comes within a distance of the target vehicle that is less than the distance of the proximity buffer in front of the host vehicle. In some embodiments, the target vehicle is determined to be in a different lane than the host vehicle, and the at least one processing device causes the driver input to cause a lateral movement of the host vehicle, thereby causing the lateral movement The host vehicle is configured to later determine that the host vehicle will be in front of the target vehicle. For example, driver input may cause the host vehicle to cut in front of the target vehicle, and such cutting may be safe if the host vehicle maintains a distance greater than a proximity buffer at the rear of the vehicle; It can be dangerous if the host vehicle comes within a distance of the target vehicle that is less than the proximity buffer at the rear of the host vehicle.
[0752] Consistent with this disclosure, the at least one obstacle can include a pedestrian or an object in the roadway, and a proximity buffer to the at least one obstacle is maintained between the host vehicle and the at least one obstacle. may include at least the minimum distance to be determined. As discussed above, the minimum distance may vary depending on the classification of the obstacle. For example, if the obstacle is a pedestrian, the minimum distance may be a distance within which there is certainty within a predetermined threshold that the host vehicle can avoid a collision with the pedestrian. As such, the minimum distance for a pedestrian may be the longest minimum approach distance consistent with this disclosure. In some embodiments, a proximity buffer for a pedestrian may be determined based on the current speed of the host vehicle, and the proximity buffer may increase as the speed of the host vehicle increases. Even in the example where the proximity buffer for pedestrians is based on the host vehicle's current speed, the proximity buffer for pedestrians is considered to be longer than the stopping distance of the host vehicle (and generally significantly longer than the stopping distance of the host vehicle). It will be done. For example, the proximity buffer for pedestrians may be the host vehicle's stopping distance (calculated under the RSS or CRSS model) with respect to the host vehicle's current speed plus an additional minimum distance. As another example, the proximity buffer for pedestrians may be 120% of the host vehicle's stopping distance with respect to the host vehicle's current speed or a different modification to the stopping distance (e.g., 110%, 150%, 200%, etc.) .
[0753] Process 5500 may include step 5514 for enabling driver input if it does not bring the host vehicle within a proximity buffer for the object. Consistent with this disclosure, if the at least one processing device determines that the driver input will not cause the host vehicle to navigate within a proximity buffer with respect to the at least one obstacle, the at least one processing device: It may be programmed to allow driver input to cause corresponding changes in one or more motion control systems of the host vehicle. For example, if the processing device determines that driver input does not place the host vehicle within a proximity buffer for any obstacle (e.g., a target vehicle or VRU), the processing device may be able to do without human driver replacement. can. Several figures disclosed herein, including FIG. 53A, illustrate conditions in which the safety system does not replace the driver.
[0754] The one or more motion control systems of the host vehicle may be any systems consistent with this disclosure. For example, one or more motion control systems may include a throttle system 220, a brake system 230, and/or a steering system 240. In some embodiments, the one or more control systems of the host vehicle include at least one steering actuator for controlling the heading of the host vehicle, a brake actuator for applying a braking device of the host vehicle, or the host vehicle. may include an accelerator actuator for applying the throttle. For example, throttle system 220 may include one or more accelerator actuators, brake system 230 may include one or more brake actuators, and steering system 240 may include one or more steering actuators. can include.
[0755] The process 5500 may include a step 5512 for replacing the human if the driver's input puts the host vehicle within a proximity buffer for the object. Consistent with this disclosure, if the at least one processing device determines that the driver input will not cause the host vehicle to navigate within a proximity buffer with respect to the at least one obstacle, the at least one processing device: It may be programmed to prevent driver input from causing a corresponding change in one or more motion control systems of the host vehicle. Any of the multiple examples of hazardous conditions disclosed herein can be the basis for a safety system to replace a human driver. For example, if the driver's input brings the host vehicle within distance of an object that is less than the RSS safe distance (or CRSS safe distance, depending on which model is used), the safety system indicates that the driver's input is dangerous. The driver can be replaced and control can be taken over.
[0756] Consistent with this disclosure, preventing the driver input from causing a corresponding change in the host vehicle may include preventing the driver input. In some embodiments, the at least one processing device changes the throttle control, brake control, or steering control to prevent driver input from causing a corresponding change in one or more motion control systems of the host vehicle. It may be configured to prevent at least one action under certain conditions or to prevent further action in response to detected driver input. For example, if the driver's input is an acceleration caused by pressing down on the gas pedal, the safety system may prevent the acceleration by locking the gas pedal so that it cannot be pressed down. Similarly, if the driver's input is to turn the steering wheel clockwise, the safety system locks the steering wheel so that it cannot be turned or can only be turned in a direction that corresponds to a safe condition. (e.g., turning the handle clockwise locks it, but turning it counterclockwise allows it). In some embodiments, the at least one processing device controls at least one of the throttle control, brake control, or steering control to prevent the driver from causing a corresponding change in the one or more motion control systems of the host vehicle. may be configured to disable one operability. For example, if the driver's input is to press down on the brake pedal, the safety system can remove the brake pedal from the rest of the braking system (e.g., electronically or mechanically) so that pressing down on the brake pedal has no effect. Input can be prevented by separating the Similarly, if the driver's input is to turn the steering wheel, the safety system prevents the input by disconnecting the steering wheel so that turning the steering wheel has no effect (at least in a direction that would create a dangerous condition). be able to. In some embodiments, preventing the driver input from causing a corresponding change in one or more motion control systems of the host vehicle includes preventing the driver input to the steering wheel from causing a corresponding response by the at least one steering actuator. preventing a driver's input on a brake pedal from causing a corresponding response by a brake actuator; or preventing a driver's input on an accelerator pedal from causing a corresponding response by an accelerator actuator. may include at least one of the following. For example, if the driver's input is to press down on the accelerator pedal, instead of locking the pedal or disconnecting the pedal, the safety system will allow the driver to give the input, but e.g. an electrical signal from the accelerator pedal. By blocking the electrical signal before it reaches the accelerator actuator, the input can be prevented from having an effect. In any embodiment, preventing driver input may continue for a period of time corresponding to the hazardous condition. For example, if the processing device determines that accelerating the host vehicle will not bring the host vehicle within a distance of an object less than the distance of the proximity buffer, the safety system may unlock the accelerator pedal.
[0757] Consistent with this disclosure, preventing the driver input from causing a corresponding change in the host vehicle may include inhibiting the driver input. In some embodiments, the at least one processing device controls the throttle control in response to the driver input to prevent the driver input from causing a corresponding change in one or more motion control systems of the host vehicle. It may be configured to prevent operation of at least one of the brake control or steering control and to apply an impulse force to at least one of the throttle control, brake control or steering control. For example, if the driver's input is to turn the steering wheel clockwise, the safety system will lock the steering wheel so that it cannot be turned in the clockwise direction, and by locking the steering wheel so that it cannot be turned in the clockwise direction. Rotation can be prohibited by applying a rotation force. In the same example, the safety system allows the steering wheel to be turned in a clockwise direction, but prohibits turning by partially locking the steering wheel so that the host vehicle does not fully respond to turning the steering wheel clockwise. The safety system can apply force to the handle in a counterclockwise direction. In some embodiments, the at least one processing device controls the throttle control, brake control until a driver input is received that would not cause the host vehicle to navigate within a proximity buffer to the at least one obstacle. Alternatively, it may be configured to prevent operation of at least one of the steering controls and continue to apply an impulse force to at least one of the throttle control, the brake control, or the steering control. For example, continuing with the example above, the safety system may be configured to operate in an amount sufficient to cause the host vehicle to avoid the hazardous condition or to return the host vehicle to a safe condition (e.g., no obstruction within the proximity buffer). , can continue to prohibit turning the steering wheel in the clockwise direction until the human driver turns the steering wheel in the counterclockwise direction. As another example, if the driver's input is to press down on the gas pedal, the safety system can inhibit the input by applying a force on the gas pedal that is opposite to the press down, preventing the driver from pressing down on the gas pedal. You can keep applying force until you stop or start pressing down on the brake pedal. In any embodiment, this prohibition may continue until the host vehicle is in a safe location with no obstacles within the proximity buffer and there are no driver inputs that would cause the host vehicle to approach the obstacle at a distance less than the proximity buffer. .
[0758] Consistent with this disclosure, preventing driver input from causing a corresponding change in the host vehicle may include or be associated with taking control from a human driver. In some embodiments, the at least one processing device controls the host vehicle during an interval in which the at least one processing device prevents the driver input from causing a corresponding change in one or more motion control systems of the host vehicle. may be configured to autonomously control navigation of. For example, if the driver's input is to turn the steering wheel clockwise, the safety system will prevent or inhibit the input as described above, and the driver's input will have no effect and the processing device will steer the host vehicle. Steering system 240 can be controlled to control. In some embodiments, the processing device has full autonomous control of the host vehicle as well as control of the system responsive to the driver's hazardous input if any input of the driver creates a hazardous condition. Safety systems can completely replace human drivers. For example, if the driver's dangerous input is to press down on the accelerator pedal, a safety DAS system can control the accelerator system, brake system, and steering system so that the processing device can fully navigate the host vehicle. The safety system and processing devices therein can autonomously navigate the host vehicle according to any embodiment of the present disclosure. In some embodiments, the at least one processing device controls navigation of the host vehicle after a driver input is received that would not cause the host vehicle to navigate within a proximity buffer with respect to the at least one obstacle. May be configured to return control to the driver. For example, if the safety system is autonomously navigating the host vehicle and the driver attempts to provide input that does not create a dangerous condition, the safety system will terminate autonomous navigation and the human driver will It may be possible to take over control of a host vehicle.
[0759] Consistent with this disclosure, at least one processing device may be programmed to alert a human driver. In some embodiments, the at least one processing device can alert the driver if the processing device determines that the driver's input creates an unsafe condition. The alert may be any alert consistent with this disclosure. For example, a vehicle may include a speaker system, and the safety system may provide an audible alert on the speaker system. As another example, the vehicle may include one or more displays (e.g., a screen in the dashboard or entertainment system), and the safety system may provide visual alerts on the one or more displays. I can do it. As another example, a vehicle may have a heads-up display or an augmented reality display (e.g., on the vehicle's windshield), and the safety system may provide alerts on the head-up display or augmented reality display. . The display may include, for example, an indication of which driver inputs would result in an unsafe condition, an indication of which inputs would result in a safe condition, an indication that the safety system is replacing the driver, etc. .
[0760] Consistent with this disclosure, at least one processing device may be programmed to collect and transmit data related to instances in which a safety system replaces a human driver. The data may include, for example, information relating to driver inputs determined to be dangerous, the time, location and duration of the displacement, the type of displacement (e.g. interdiction, prevention or takeover), the outcome of the takeover (e.g. if a collision avoided) and any other information related to the replacement. Data may be transmitted to one or more systems or devices. For example, the data may be transmitted to a device associated with the driver for use in informing the driver of avoided accidents, driver driving ability, etc. As another example, information can be transmitted to one or more systems for use in research related to autonomous vehicles, road safety, human driving habits, and the like.
[0761] Consistent with this disclosure, an autonomous system for selectively replacing control of a host vehicle by a human driver includes a system override control for disabling an autonomous system for selectively replacing control by a human driver. may include. System override control may be different from throttle control, brake control and steering control. For example, making the system override control different from brake control, throttle control, and steering control requires the driver to provide an input to the override control that is different from the navigation input. In some embodiments, the system override control may be a steering wheel, button, lever or other physical device that can obtain input from the driver. For example, the override control may be a lever and the input that overrides the autonomous system may be a pull of the lever or other displacement of the lever. In some embodiments, the system override control may be an audible phrase, visual gesture, or other type of input that may be detected by one or more microphone and/or image capture device systems. For example, system override controls may include hand gestures. The system override control may be any other means for deactivating or deactivating the system as described above.
[0762] Consistent with this disclosure, at least one processing device may be configured to track when an autonomous system for selectively replacing control by a human driver is overridden by operation of a system override control. For example, the processing device may include a date related to the time and location at which the human driver activated the system override control, as well as visual information related to the deactivation displayed, e.g., on a display or instrument panel within the vehicle. information related to the duration the system was disabled, including images or video clips). In some embodiments, if a vehicle is involved in a crash, the processing device may be programmed to report whether the safety system was disabled prior to the crash. In some embodiments, the processing device may be programmed to monitor driver input even if the system is disabled. For example, if disabled, the processing device may be configured to perform process 5500 except for step 5512. In this example, the processing device collects and stores data related to whether the human driver caused a hazardous condition in which the host vehicle was within a proximity buffer to the object and other information related to navigation of the vehicle. I can do it. That information can be used to determine liability in the event of a conflict. For example, if the processing device determines that the AD safety AS system was ineffective when a collision occurred between the host vehicle and the target vehicle, the human driver of the host vehicle or the driver of the target vehicle may be at fault. Similarly, if the safety system has been disabled and the processing device determines that the human driver provided an input that would have been replaced if the safety system had not been disabled, the human driver of the host vehicle is responsible. obtain.
[0763] The examples and embodiments discussed above are illustrative only and are not intended to limit the scope of autonomous systems for selectively replacing human driver control. Those skilled in the art having the benefit of this disclosure will be able to understand how to modify this autonomous system in any manner consistent with this disclosure.
[0764] Route prediction to compensate for control delays [0765] The autonomous navigation system disclosed herein can determine one or more navigation operations of a host vehicle and can perform the navigation operations. As explained above, navigation operations may be based on one or more navigation policies, navigation goals, etc. A navigation system may include one or more sensors that provide output that is used to determine the host vehicle's navigation response. For example, the system may include a plurality of image capture devices configured to capture images of the host vehicle's environment and at least one processor configured to analyze the images to determine a navigation response. However, there may be a delay period between the time the image (or other output from the sensor) is captured and the time the navigation response is implemented to cause host vehicle operation. In some cases, there may be an additional delay between the time the navigation response is implemented and the desired effect occurs (eg, a steering wheel is turned and the wheels change orientation accordingly). The delay period can be very short, but in most cases the vehicle continues to move during the delay period. As a result, certain navigation decisions may be out of date sensor outputs because those sensor outputs are closer to the point in time when the vehicle's actuators cause a change in vehicle state in response to the implementation of the planned navigation action. or may be based on past operating conditions rather than actual operating conditions of the host vehicle at the same point in time. As a result, the host vehicle may exhibit a "sinusoidal" drive, in which the host vehicle may exhibit oscillations by making navigation decisions based on sensor information that does not represent the actual conditions at which the actuator responds. The vehicle may make corrections (e.g., brake, accelerator, brake, accelerator, etc.). Therefore, a system is disclosed for predicting vehicle behavior after vehicle sensor measurements are taken and before or simultaneously with the activation time. The disclosed system can avoid non-smooth shaky corrections that can occur if corrections are not made to account for changing vehicle behavior during the performance of sensor measurements, driving decisions and navigation actions, among other things. .
[0766] The problem of controlling the speed of a vehicle can be formulated as follows. speed command signal v<sub>cmd</sub>(t) may be the target speed of the host vehicle. The navigation system determines the actual speed of the vehicle, denoted by v(t), as much as possible<sub>cmd</sub>Throttle and brake control can be adjusted to approach (t). Let the pedal position be p(t)[-1,1], where -1 corresponds to full brake, 1 corresponds to full throttle, and 0 means no brake or throttle. Moving the pedal can change the speed of the vehicle, and the controller's goal is to generate the error signal e(t)=v<sub>cmd</sub>This can be done by moving the pedal so that (t)-v(t) becomes smaller in absolute value. In standard notation, v<sub>cmd</sub>(t) is the desired set point, v(t) is the process variable, and p(t) is the control variable.
[0767] The controller can operate in discrete time, with the pedal position being moved every Δ seconds. The controller can be based on the following simple formula: For all t=k, where k is a natural number,<math num="242"><img file="JP7367255B2_D0242.tif" /></math>It is.
[0768] However, pe is an error correction function regarding the pedal position, and p<sub>g</sub>(v,α) is a "guess" function that is a constant of the position of the pedal depending on the current speed and accelerator command, as the case may be (we discuss the guess function in more detail below; the dependence of v or α on the feedback is not, so p<sub>g</sub>Note that works in a completely "open-loop" fashion),<math num="243"><img file="JP7367255B2_D0243.tif" /></math>gives the discounted average of past errors (and<math num="244"><img file="JP7367255B2_D0244.tif" /></math>) signal. Please pay attention to the following equation.
<math num="245"><img file="JP7367255B2_D0245.tif" /></math>
[0769] For simplicity, the above representation does not include aspects related to saturation (error, pedal position, clipping of pedal changes). Those skilled in the art having the benefit of this disclosure will understand that the above equation may include saturation.
[0770] Next, the guess function p<sub>g</sub>(v,α) can be considered. For example, a vehicle may be driven while recording speed and pedal position as a function of time. The derivative of velocity can be interpreted as the actual acceleration (albeit potentially noisy) and one can attempt to fit a function that gives the acceleration depending on p(t) and v(t). By inverting this function, we get the guess function p<sub>g</sub>(v,α) is obtained.
[0771] The guess function may benefit from having no delay. If most of the information about the correct position of the pedal is within the desired acceleration and current velocity, the guessing function can make an estimate of the correct pedal position. The role of the controller is p<sub>e</sub>The purpose of this is to eliminate errors due to additional variables (e.g. tire pressure, engine heat, tailwind, etc.).
[0772] The main forces acting on the vehicle are friction, which depends linearly on v, lag (air resistance), which depends quadratically on v, gravity, which changes with altitude in the form 9.8sin(θ), and the force of the brake pedal. Most of the information depends on v and α since it is position dependent brake friction and throttle pedal position dependent engine.
[0773] A guess function can have several advantages. First, the initial guess (in most cases) will be very close to the optimal position of the pedal, and therefore there will be a negligible delay for accelerations that are very close to the desired acceleration. Second, the problem of learning the initial guess function is an offline problem (collecting examples from human/autonomous drivers and fitting the function to them). It can be easily adapted to another vehicle and even be fitted on the fly to an existing car (in our fitting selection using low-order polynomials) the number of parameters to be fitted can be collected. the number of examples, so there is no need to worry about overfitting). Third, based on empirical facts, a discounted integrator can yield the "no steady-state error" property of the integrator while having a much smaller delay due to discounting that quickly forgets past errors. Further, in some embodiments, even small delays that are present may only be on the controller side to reduce steady-state errors.
[0774] In some embodiments, the inference function may not receive feedback, but rather relies on comments. In some embodiments, this is the only controller in the navigation system that uses the inference function.<sub>e</sub>can do the term. p<sub>e</sub>Analyze the terms separately.
<math num="246"><img file="JP7367255B2_D0246.tif" /></math>
[0775] In some embodiments, a controller that operates according to the equations herein may rely on an integrator that has a coefficient that decays over time. It should be noted that the undamped term in the coefficients can be relatively small, such as Km>>1. As discussed above, this type of integrator may have a "no steady state error" property while incurring less delay due to past errors due to the discounting effect. This can also be interpreted as something between the P and I terms in a regular PI controller, i.e. no discounting at all gives a classical I term, while severe discounting gives , giving the classical P term. As such, the controller may offer unique advantages over existing PI controllers.
[0776] FIG. 56 shows an overview of a process 5600 that may be performed by a navigation system. The discussion below shows how the controller deals with processing delays. Process 5600 may include a sensing and output stage 5602 in which the navigation system may collect data from one or more sensors and analyze the data to generate one or more outputs. For example, in the sensing and output stage 5602, the image capture device 122 captures one or more images and at least one process is programmed to analyze the images to detect obstacles in the host vehicle's environment. You can provide those images to your device. The sensing and output stage 5602 may conclude with an output related to one or more sensors, such as a determination that the target vehicle is in front of the host vehicle. The sensing and output stage 5602 may also include measurements by any sensors associated with the host vehicle (eg, GPS, accelerometer, speedometer, tire pressure indicator, radar, lidar, etc.). The process 5600 includes a navigation command determination stage in which the navigation system may include at least one processing device programmed to determine one or more navigation commands for the host vehicle using the output of the sensing and output stage 5602. 5604. For example, the at least one processing device can determine navigation commands using one or more navigation policies, eg, in conjunction with outputs of sensor outputs. The navigation command determination step 5604 may conclude with the navigation command being transmitted to one or more motion control systems (eg, throttle control system, brake control system, steering control system, etc.). Process 5600 may include a navigation response stage 5606 for causing the vehicle to perform navigation operations consistent with the navigation commands of navigation command stage 5604. For example, the navigation response stage 5606 may include activating a throttle actuator in response to the navigation command of the navigation command determination stage 5604.
[0777] In this example, t(0) may represent the instant in time at which a sensor measurement is taken (e.g., image capture device 122 captures a first image), and t(1) may represent one or more processing steps. t(2) may represent the instant in time at which a sensor output is transmitted to a device, and t(2) is the moment at which the sensor output is received by a processor that is programmed to determine navigation commands based on the received sensor output or sensor outputs. t(3) may represent the instant at which the processor transmits the determined navigation command to one or more motion control systems, and t(4) may represent the instant at which the determined navigation command is transmitted. t(5) may represent the instant at which a navigation command is received by one or more motion control systems, and t(5) represents the instant at which the navigation command is implemented (e.g., an actuator causes a throttle, brake, or steering response). can be expressed. Consistent with this disclosure, there may be delays between any of these moments. For example, the total delay of process 5600 can be the difference between t(5) and t(0), which can range from 1 millisecond to hundreds of milliseconds, for example. For example, in some configurations of process 5600, the delay may be 150 milliseconds. The cumulative delay includes the processing time for collecting, packaging, and transmitting the sensor outputs in step 5602, and the processing time for collecting, packaging, and transmitting the sensor outputs in step 5604, and applying driving policy to one or more received sensor outputs to generate the planned navigation motion commands in step 5604. etc., and processing time for transmitting or implementing the navigation commands received in step 5606.
[0778] During any portion of the cumulative or total delay, the host vehicle may continue to navigate at the same or different speed, acceleration, and/or path than it was traveling at data acquisition time t(0). For example, if the brakes are applied at the time of sensor measurement, the speed of the vehicle may be slower by the time the navigation command is executed than at the time of sensor measurement. Similarly, if the accelerator is applied at the time of the sensor measurement, the vehicle may be faster by the time the navigation command is executed than at the time of the sensor measurement. If the navigation command is implemented at actuation time t(5), the vehicle may have traveled a significant distance and/or may have a different speed, acceleration, or path than the basis of the navigation command. As such, the navigation command may produce a navigation response at actuation time t(5) that is not based on the actual operating conditions of the vehicle at the time the navigation command is implemented by the motion control system.
[0779] FIGS. 57A-57C illustrate conditions in which prediction of vehicle path during the time between sensor measurements and actuation may be useful. 57A to 57C respectively show a host vehicle 5702 and a target vehicle 5706 on a road 5700 at data acquisition time t(0). Data acquisition time t(0) may be the moment at which one or more sensors on host vehicle 5702 capture information related to target vehicle 5706. Figures 57a and 57C also show host vehicle 5702' and target vehicle 5706' at delay time t(5). As discussed with respect to FIG. 56, the delay time t(5) may be the moment at which the host vehicle performs a navigation operation or otherwise implements a navigation command.
[0780] FIG. 57A shows a host vehicle 5702 at an initial time t(0) and the same host vehicle 5702' at a delayed time t(5). As shown, there is a slight positional difference between host vehicle 5702 at initial time t(0) and host vehicle 5702' at delay time t(5). This difference in position may be due to the distance traveled by the host vehicle 5702 during the response period Δt (not shown), which is the difference between the delay time t(5) and the initial time t(0). Similarly, host vehicle 5706 may also have traveled a predetermined distance during the response period Δt. In this example, the positions of the host vehicle 5702' and the target vehicle 5706' are different between the initial time t(0) and the delayed time t(5), so the positions of the host vehicle 5702 and the target vehicle 5706 at the initial time t(0) are different. The navigation command determined based on may not produce an ideal navigation response at delay time t(5).
[0781] FIG. 57B shows an example path 5704 that may correspond to a desired trajectory of host vehicle 5702. For example, at least one processor within the navigation system of host vehicle 5702 may determine that host vehicle 5702 should overtake target vehicle 5706 and may generate a navigation command consistent with route 5704. In this example, route 5704 is generated based on the conditions at initial time t(0). For example, path 5704 may include the velocity, acceleration, maximum braking capacity, and position of target vehicle 5706 at initial time t(0) and one or more of the velocity, acceleration, position, and location associated with host vehicle 5702 at initial time t(0). navigation policy.
[0782] FIG. 57C shows different routes 5704, 5704' and 5710 that host vehicles 5702, 5702' may travel. Route 5704 is a desired route determined under the conditions of initial time t(0) as shown in FIG. 57B. In this example, at delay time t(5), host vehicle 5702' is in a position where path 5704 cannot be followed because, at delay time t(5), the starting point of path 5704 is lower than the front of vehicle 5702'. Because it's also behind you. Therefore, the host vehicle 5702' at time t(5) cannot follow the route 5704 generated at time t(0). Route 5704' may be a trajectory that vehicle 5702' travels when it performs a navigation maneuver consistent with route 5704. In other words, route 5704' is the trajectory that would result if host vehicle 5702' executed at delay time t(5) the navigation command generated for host vehicle 5702 at initial time t(0). In this example, route 5704' is much closer to target vehicle 5706' at delay time t(5) than route 5704 is closer to target vehicle 5706 at initial time t(0). In some embodiments, the distance between path 5704' and target vehicle 5706' may be considered a critical distance (e.g., this distance may violate the proximity buffer of host vehicle 5702'). ). Route 5710 may be a route for host vehicle 5702' generated at initial time t(0) based on the predicted routes of host vehicle 5702 and target vehicle 5706. For example, at an initial time t(0), the navigation system of host vehicle 5702 can predict the position of target vehicle 5706 (which may be the position indicated by target vehicle 5706') at delay time t(5), and The location of host vehicle 5702 at time t(5) (which may be the location indicated by host vehicle 5702') can be predicted and generates a navigation command to be executed at delayed time t(5) consistent with route 5710. can do. As illustrated by this example, route 5710 based on predicted conditions may provide unique safety advantages over route 5704'.
[0783] The above examples are merely illustrative and do not limit the scope of the embodiments. For example, although FIGS. 57A-57C illustrate a predicted path for a host vehicle to overtake a target vehicle, the predicted path may be related to any navigation maneuvers of the host vehicle and any obstacles (e.g., VRU, multiple host vehicles, road characteristics, etc.). As an example, a predicted route may be generated for a host vehicle entering a turn in a road. Any other navigation operation may benefit from the predictive path analysis and/or inference functions described above.
[0784] In some embodiments, delayed responses may result in excessive corrections or unnecessary navigation responses by the host vehicle. For example, if the target vehicle in front of the host vehicle is traveling at low speed but high acceleration, the image captured at time t(0) is used to help the host vehicle avoid a collision with the target vehicle. It may be determined that it may be necessary to stop or turn around suddenly. However, at time t(5), the target vehicle may have traveled a significant distance due to its high acceleration, and the host vehicle may be, for example, coasting or braking with a submaximal braking rate. This makes it possible to avoid a collision with the target vehicle. In this example, if the host vehicle is equipped with a system that is programmed to determine a predicted route based on predicted conditions for time t(5) rather than based on current conditions at time t(0), The host vehicle can respond more accurately and comfortably to the detected target vehicle. In some embodiments, a delayed response may cause the host vehicle to perform an insufficient navigation response that may place the host vehicle in a dangerous condition. For example, if the target vehicle is traveling at a safe distance but decelerating rapidly, and the host vehicle is accelerating toward the target vehicle, the determination is made based on the conditions detected at time t(0). The distance determined based on the conditions at time t(0) may be significantly longer than the actual distance at time t(5) when the host vehicle implements the navigation command determined based on the conditions at time t(0). As a result, the host vehicle may not brake as programmed, turn suddenly, or otherwise avoid dangerous conditions.
[0785] Consistent with this disclosure, a navigation system for navigating an autonomous host vehicle according to at least one navigation goal of the host vehicle is disclosed. The navigation system can be any navigation consistent with this disclosure, including, but not limited to, the disclosed Vision Zero safety system. This system may reside within the host vehicle or may communicate with the host vehicle. The system may include one or more sensors configured to collect data related to the host vehicle's environment. For example, a navigation system may include one or more image capture devices, lidar systems, radar systems, accelerometers, and the like. The system may include at least one processor programmed to perform one or more methods, processes, operations or functions consistent with this disclosure. The at least one processor may be, for example, processing device 110 or any other processing device consistent with this disclosure.
[0786] FIG. 58 is a flowchart illustrating an example process 5800 for navigating an autonomous host vehicle according to at least one navigation goal of the host vehicle. Consistent with this disclosure, at least one processor may be programmed to perform all or a portion of process 5800. It is understood that process 5800 is exemplary only and that process 5800 may include additional steps, exclude one or more steps, or be otherwise modified in a manner consistent with this disclosure. This can be understood by one of ordinary skill in the art who has the benefit of this disclosure.
[0787] Process 5800 may include a step 5802 for receiving sensor output indicative of host vehicle operation. Consistent with this disclosure, at least one processor may be programmed to receive sensor output indicative of at least one aspect of host vehicle operation relative to the host vehicle's environment. Output may be received from one or more sensors. The one or more sensors may include any sensor disclosed herein, including, for example, an image capture device, a sensing system, an accelerometer, a GPS unit, and the like. For example, the one or more sensors may include a speed sensor, an accelerometer, a camera, a lidar system, or a radar system. The output may include or convey any information related to the host vehicle, the host vehicle's environment, obstacles within the host vehicle's environment, and the like. For example, the output may include the host vehicle's current speed and/or acceleration, the host vehicle's heading, the host vehicle's position, and the like. As another example, the output may include, for example, an identifier of a detected target vehicle, a position of the target vehicle, a speed of the target vehicle, and a hypothesis of the target vehicle detected based on analysis of one or more images, lidar or radar. This may include the braking capacity. The output may be determined according to any method, process, function or operation consistent with this disclosure.
[0788] In some embodiments, the sensor output is at a point in time that is later than the data acquisition point in time at which the measurement or data acquisition on which the sensor output is based is obtained, and after a second point in time at which the sensor output is received by the at least one processor. may also be generated at a previous first point in time. In some embodiments, a data acquisition point in time can be a moment at which one or more sensors make a measurement or data acquisition, the first point in time being an output based on the measurement or data. This can be the moment you are ready. For example, referring to FIG. 56, the output may be generated at a first time point t(1) and based on data acquired at a data acquisition time point t(0). In some embodiments, the time between the data acquisition point and the first point in time may be several milliseconds. For example, the time between the first time point t(1) and the data acquisition time point t(0) is greater than 2 ms, greater than 10 ms, or in the range of 5 ms to 50 ms. or less than 100 milliseconds, or any other period of time. In some embodiments, the second point in time may be the moment the controller receives the sensor output. For example, the second time point may be time point t(2). The controller may include, for example, the controller or at least one processor programmed to perform the guessing function as described above with respect to the guessing function.
[0789] Process 5800 may include step 5804 of generating a prediction of at least one aspect of host vehicle operation. Consistent with this disclosure, the at least one processor may be programmed to generate a prediction of at least one aspect of the host vehicle's operation for a predicted point in time. At least one processor is capable of determining or projecting a future point in time that corresponds to the predicted operation point in time. The motion prediction point in time may be a future point in time at which the host vehicle may perform a navigation response. In some embodiments, the predicted operation time point may approximately correspond to the actuation time point. For example, the motion prediction time point may be a prediction of when an actuation time point t(5) will occur at which one or more actuators may implement a navigation command. However, the motion prediction time point may correspond to any time point after t(0) and before t(5). For example, host vehicle operation may be predicted for any other time before t(1), t(2), t(3), t(4), or t(5).
[0790] In some embodiments, the predicted operation time point may approximately correspond to the time point at which the controller receives the sensor output (eg, time point t(2)). In other cases, the predicted time point may take into account the time it takes for the processor to act on the sensor output. For example, if the processor requires approximately 100 milliseconds to perform a function such as detecting a target vehicle, determining an appropriate navigation action in response to one or more sensing conditions, at least one processor can assume that the motion prediction time is 100 milliseconds after the second time t(2). In some embodiments, the predicted motion time point may approximately correspond to a third time point at which the actuation system receives a navigation command determined based on the sensor output (eg, at time point t(4)). The predicted motion time point may also correspond to a time point that is a predetermined or determined amount later than the third time point (eg, time point t(4)). For example, if brake system 230 receives a navigation command at time t(4), the predicted operation time may be a time after time t(4) that corresponds to the average response time of the actuators of brake system 230.
[0791] The motion prediction time point can be any period after the data acquisition time point. In some embodiments, the motion prediction time can be after the data acquisition time and before or equal to the actuation time. For example, the motion prediction time point can be at least 100 milliseconds after the data acquisition time point. In another example, the motion prediction time point can be at least 200 milliseconds after the data acquisition time point. In another example, the predicted motion time points are 50-300 ms after the data acquisition time, 75-275 ms after the data acquisition time, 100-250 ms after the data acquisition time, and 150-200 ms after the data acquisition time. It could be after milliseconds, etc.
[0792] In some embodiments, predicting at least one aspect of host vehicle operation includes predicting vehicle operation based on received sensor outputs and data acquisition points and operation prediction points (i.e., predicting vehicle operation based on previously acquired sensor outputs). may be based, at least in part, on an estimate of how at least one aspect of the operation of the host vehicle changes over a time interval between (a time point in time). For example, if at least one aspect of the host vehicle's motion is host vehicle acceleration, the prediction is that the host vehicle will move at the detected acceleration if the host vehicle moves at the detected acceleration during the time between the motion prediction time and the host vehicle's acceleration. may include the distance between Predictions may be based on received sensor outputs and any other information consistent with this disclosure. For example, the prediction may be based on output from an image sensor (such as target vehicle detection) and previously revealed road characteristics, weather conditions, or other factors that may influence future conditions for the host vehicle. In some embodiments, the prediction may include estimating at least one aspect of the host vehicle's behavior at a later time (eg, time t(5)). As a result, navigation decisions may be based on estimates of vehicle motion that may be closer to actual operating values that match the acquired sensor outputs sensed at an initial point in time (eg, time t(0)).
[0793] In some embodiments, predicting at least one aspect of the host vehicle's motion may include at least one predicting the host vehicle's velocity or acceleration at the time of the motion prediction. The at least one aspect may include, for example, speed, acceleration, position, yaw, or other aspects of host vehicle motion. For example, if the acceleration and/or velocity of the host vehicle is determined at time t(1) based on the sensor output obtained at data acquisition time t(0), the at least one processor determines at time t(5) or A predicted future velocity and future acceleration of the host vehicle at another predicted point in time can be determined. For example, the at least one processor may assume that the host vehicle has accelerated from time t(0) to time t(5) or another motion prediction time at the same acceleration rate determined at time t(0). , and thus may determine that the host vehicle has a speed at time t(5) that is greater than the speed at time t(0). Accordingly, the at least one processor is capable of predicting the velocity and/or acceleration at the predicted motion time in order to determine a navigation motion appropriate for the estimated velocity at time t(5) or other predicted motion time.
[0794] In some embodiments, predicting at least one aspect of the host vehicle's motion may include predicting the host vehicle's path at the time of the motion prediction. For example, if the host vehicle is traveling a path at time t(0), the at least one processor uses information from the one or more sensors to determine whether the host vehicle is traveling at time t(5). It is possible to predict the route (for example, change in direction of travel, etc.). As an example, if the sensor output indicates that the host vehicle is traveling at a constant speed on a straight path, then the at least one processor determines that the host vehicle is on the same straight path at time t(5), but It can be predicted that there may be further distances along the line. As another example, if the sensor output indicates that the host vehicle is traveling on a curved path associated with a constant rotation of the steering wheel, the at least one processor determines whether the host vehicle is traveling on a curved path associated with the same rotation of the steering wheel. One can predict what is likely or what may be on the path associated with more rotations of the handle. In such cases, the predicted path may be based on a detected rate of change of the steering controller, or may be based on a detected yaw rate, a detected centripetal acceleration, a detected curvature of the road in front of the vehicle, etc. . In some embodiments, the prediction of the host vehicle's path at the time of the motion prediction may include a target heading of the host vehicle. For example, the target heading may include the direction in which the host vehicle is attempting to travel. As an example, if the output from the sensor indicates that there is an obstacle that the host vehicle must avoid, the target heading may be the direction in which the host vehicle must travel to avoid that obstacle. . The target heading direction can be a direction corresponding to a cardinal orientation or can be based on a predetermined coordinate system. For example, the direction can be north, northeast, east, southeast, south, etc., or can be indicated by the expected heading angle (eg, relative to the current heading angle determined at the time of sensor measurement). In another example, the direction of travel may be expressed based on a degree, yaw rate, etc. relative to a horizontal or vertical axis passing through the body of the host vehicle. In some embodiments, the one or more sensors may include a camera, and the prediction of the path of the host vehicle at the time of the predicted motion may be based on at least one image captured by the camera. For example, a path may include a virtual trajectory through the environment captured in the image, and may include a representation of the path across or on the image. As another example, a route prediction may include, for example, a route between two obstacles detected in an image.
[0795] In some embodiments, the prediction of the path of the host vehicle at the time of the motion prediction may be based at least on a determined speed of the host vehicle and a target trajectory of the host vehicle included in a map of road segments on which the host vehicle will travel. For example, the predicted path may be generated according to a semantic high-definition mapping technique such as REM discussed above. As an example, the road segment on which the host vehicle is traveling may be associated with multiple trajectories that may be used to navigate the autonomous vehicle on that road segment, and the predicted route may be associated with a goal associated with the road segment. May include a position on one of the orbits. As another example, if the host vehicle is determined to be traveling according to a predetermined target trajectory, the predicted path at the time of the motion prediction may include position, velocity, and/or acceleration along the same target trajectory. In some embodiments, the target trajectory may include a predetermined three-dimensional spline representing a preferred path along at least one lane of a road segment. For example, a three-dimensional spline may include a plurality of landmarks, road features, and other objects that define a target trajectory on a road segment. In this example, the predicted route may include a prediction of which landmarks, road features, or other objects may be near the host vehicle at the time of the motion prediction. For example, if the host vehicle is determined to be on the target trajectory based on the host vehicle's position between the first landmark and the second landmark, the predicted path is on the target trajectory between the landmarks. May include location. If the host vehicle is navigating according to a target trajectory in the REM map, localization along the target trajectory at a particular point in time (e.g., time t(0)) can be used to predict the path at a later point in time. can. For example, if the speed of the host vehicle is known at time t(0) and the host vehicle's position relative to the REM target trajectory is known, then a later time point (e.g., time t In (5), the position of the host vehicle can be estimated with respect to the REM target trajectory.If the position on the REM target trajectory is known, then (to continue following the REM target trajectory, the host vehicle (Since the direction of travel can correspond to the direction in which the vehicle is traveling), the direction of travel at that location can be clarified from the map.
[0796] In some embodiments, the prediction of at least one aspect of the host vehicle's behavior (e.g., speed) includes a determined brake pedal position, a determined throttle position, a determination that opposes the host vehicle's behavior. The host vehicle may be based on at least one of applied air resistance, friction, or the slope of the road segment on which the host vehicle travels. For example, the prediction may be based on the pedal position defined within the estimation function and other equations disclosed above. The guess function can be determined empirically, for example, by observing the resulting vehicle speed as a function of the position of the brake pedal and/or throttle pedal. For example, a vehicle may decelerate according to a particular deceleration profile that corresponds to a particular brake pedal position. Similarly, a vehicle may accelerate according to a particular acceleration profile that corresponds to a particular throttle pedal position.
[0797] As another example, if the output of the sensor includes the speed and/or acceleration of the host vehicle, the at least one processor may detect the initial speed and/or acceleration as well as friction or air resistance (or gravity based on road slope). can be used to predict future velocity and/or acceleration. In some embodiments, at least one aspect of the operation of the host vehicle includes determining an overall force on the host vehicle and modifying the current velocity and/or acceleration of the host vehicle based on the overall force. can be revealed by In this example, the overall forces are auxiliary forces (e.g., throttle, accelerator, gravity when driving downhill, etc.) and resisting forces (e.g., wind resistance, friction, driving uphill, etc.). The current speed of the host vehicle can be used to predict future speed using the overall force.
[0798] In some embodiments, prediction of at least one aspect of host vehicle operation may be based on a predetermined function associated with the host vehicle. The predetermined function may enable future speed and acceleration of the host vehicle to be predicted based on the determined current speed of the host vehicle and the determined brake pedal position of the determined throttle position of the host vehicle. For example, predicting at least one aspect of the host vehicle's operation may include predicting the host vehicle's future velocity, acceleration, and/or position using a guess function. For example, as explained above, the positions of the brake and throttle pedals can be defined by p(t)[-1,1], where -1 corresponds to full brake and 1 corresponds to full throttle. and 0 corresponds to no brake or throttle, and the at least one processor may use an inference function to predict aspects of host vehicle operation based on pedal position.
[0799] In some embodiments, the prediction of at least one aspect of the host vehicle's motion at the time of the motion prediction includes a data acquisition rate associated with the one or more sensors and the at least one processor generating navigation commands. The mismatch between the controlled speed and the related controlled speed can be taken into account. For example, one or more sensors may acquire data at a slower rate than the at least one processor can process the data to generate navigation commands based on the data. As a specific example, one or more cameras may acquire images at 10Hz, while at least one processor may generate navigation commands based on the images at a rate of 50Hz. As a result, only one in five processing events may result in a new image on which navigation decisions can be based. In that case, vehicle behavior can be predicted for those processing events (occurring up to 50 Hz) that may occur between image capture events (occurring at 10 Hz).
[0800] Process 5800 may include step 5806 for determining a planned navigation operation of the host vehicle. The planned navigation maneuvers may be based, for example, on predictions of the host vehicle's motion and/or on the host vehicle's navigation goals. Consistent with this disclosure, the at least one processor is configured to: The host vehicle may be programmed to determine a planned navigation operation of the host vehicle based on the generated prediction of at least one aspect of the host vehicle's operation. The planned navigation operation may be any navigation operation described herein, including acceleration, deceleration, turning, changing direction, multiple navigation maneuvers, and the like. For example, the planned navigation operation may include at least one of changing the speed or changing the heading of the host vehicle. As another example, a planned navigation operation may include a desired trajectory or route for the host vehicle to follow. For example, navigation behavior may be determined according to any of the embodiments described herein. However, in certain instances, navigation behavior in embodiments disclosed herein may be determined based on predicted aspects of the host vehicle's behavior at a later time than the time of the sensor measurement. For example, rather than using raw sensing conditions to determine a target trajectory for a host vehicle, the processing device uses those sensing conditions and the resulting vehicle trajectory that is predicted to occur after acquiring those sensing conditions. A target trajectory can be determined based on changes in motion. Using predicted vehicle behavior may have several advantages over other embodiments. For example, referring to FIG. 57C, if the target trajectory is determined based on the sensing conditions at time t(0), host vehicle 5702' comes within range of target vehicle 5706', which may be dangerous (e.g., host vehicle 5702' if the target trajectory is determined based on predicted conditions; travels along a route 5710 that maintains a safe distance from the target vehicle 5706'. In addition, by predicting the speed of the vehicle at the time of the sensor measurement and at the predicted moment of motion that is closer to or equal to the actuator time, the host vehicle can make smoother or less sinusoidal/jittery corrections that may be unpleasant for passengers. It is possible to drive with less or without such correction.
[0801] In some embodiments, the host vehicle's navigation goal may include transitioning from a first location to a second location. For example, the first location may be a starting point and the second location may be a destination. In this example, predicted aspects of the host vehicle's behavior may be used to perform one or more navigation actions that are consistent with the navigation goal (e.g., may move the host vehicle closer to the destination along a selected route or target trajectory). one or more actions). As another example, the first location can be a first location on the target trajectory and the second location can be a second location on the target trajectory. As discussed above with respect to REM maps and related navigation, in this example, the navigation operation may be any navigation operation that causes the host vehicle to travel along a target trajectory toward a second location. In some embodiments, the host vehicle's navigation goal may include a lane change from the current lane occupied by the host vehicle to an adjacent lane. For example, the navigational goal may be the route 5704 described with respect to FIG. 57B. In this example, the processing device may use the predicted aspects of the host vehicle's behavior to determine navigation behavior consistent with the goal, such as by developing a path 5710 for the host vehicle to follow.
[0802] In some embodiments, the host vehicle's navigation goals may include maintaining a proximity buffer between the host vehicle and the detected target vehicle. As discussed above, the proximity buffer zone includes the host vehicle's detected current speed, the host vehicle's maximum braking rate capability, the target vehicle's determined current speed, and the target vehicle's assumed maximum braking rate. can be determined based on the ability of A proximity buffer for the target vehicle may be further determined based on a maximum acceleration capability of the host vehicle, such that the proximity buffer is accelerated at least at the maximum acceleration capability of the host vehicle over a reaction time associated with the host vehicle. The acceleration distance of the host vehicle is determined as the distance traveled by the host vehicle when the host vehicle's acceleration distance is determined as the distance required to reduce the current speed of the host vehicle to zero at the maximum braking rate capability of the host vehicle. It includes the sum of the stopping distance and the stopping distance of the target vehicle determined as the distance required to reduce the current speed of the target vehicle to zero at the capability of the target vehicle's assumed maximum braking rate. As explained above, for example, the host vehicle may have a safety system configured to maintain a proximity buffer between the host vehicle and one or more obstacles. In this embodiment, the proximity buffer may be based on the RSS safety distance. As discussed with respect to FIGS. 52A-52D, in some embodiments, the proximity buffer can be based on the CRSS safety distance. The planned navigation operation in this example may be any navigation operation that maintains a proximity buffer.
[0803] Process 5800 may include step 5808 for generating navigation commands to implement the planned navigation operation. Consistent with this disclosure, at least one processor may be programmed to generate navigation commands to implement at least a portion of the planned navigation operation. A navigation command may be any instruction to cause one or more operations consistent with this disclosure. For example, the commands may include instructions for a braking system to apply the brakes at a certain level or instructions for a throttle system to accelerate at a certain level. In some embodiments, the navigation commands may include at least one of a pedal command to control the speed of the host vehicle or a yaw rate command to control the heading of the host vehicle. For example, the navigation command may include an instruction to depress the throttle pedal by an amount corresponding to an acceleration determined for the planned navigation maneuver. Navigation commands may include any other commands or instructions consistent with this disclosure. For example, commands may be meaningful or non-semantic to cause the host vehicle to change direction, accelerate, decelerate, change speed, coast, stop, etc. may contain commands.
[0804] The examples of navigation operations, navigation goals, and navigation commands described above are exemplary only. It will be appreciated by those skilled in the art with the benefit of this disclosure that any of the embodiments described herein can be modified to use predicted aspects of host vehicle motion to determine planned navigation motions. can be understood. For example, the embodiments discussed with respect to the REM, RSS, and CRSS models, as well as the embodiments discussed with respect to FIGS. 13-15 and 18, can be modified using predicted aspects of host vehicle behavior.
[0805] Process 5800 may include step 5810 for providing navigation commands to at least one actuation system of the host vehicle. Consistent with this disclosure, the at least one processor operates at a second point in time that is after the second point in time and before or about the same as the point in time at which the at least one actuation system component responds to the received command. The at least one actuation system may be programmed to provide the navigation command to the at least one actuation system of the host vehicle such that the at least one actuation system receives the navigation command at the time. For example, with reference to FIG. 56, a navigation command may be provided to the actuation system at a time t(4), which is less than a time t(5) at which the command is implemented by one or more actuators. is also before or at approximately the same time. An actuation system can be any system configured to produce one or more responses to navigation commands. For example, the at least one actuation system may include one or more of a throttle actuation system, a brake actuation system, or a steering actuation system. As specific examples, the throttle actuation system may include a throttle system 220, the brake actuation system may include a brake system 230, and the steering actuation system may include a steering system 240.
[0806] As explained above, the actuation system provides an actuation point t( A navigation response at time t(5) can be generated based on aspects of the motion predicted for 5). This is because aspects of the predicted operation may be more similar to the actual conditions at actuation time t(5) than aspects of the movement detected at time t(0). In some embodiments, the motion prediction time can be after the data acquisition time and before or equal to the actuation time. The closer the predicted operation time point is to the actual operation time point, the more accurate, precise, and safe the navigation operation can be. For example, at actuation time t(5), the one or more actuation systems may cause a navigation operation that corresponds to a planned navigation operation based on predicted aspects of the host vehicle's operation, which include: If the predicted motion time is close to the actuation time, it may correspond to performing a navigation motion that closely matches the planned navigation motion. For example, the motion prediction time point may be approximately similar to the motion time point t(5), and the aspects of the predicted motion associated with the motion prediction time point may be approximately similar to the actual motion time points at the motion time point t(5). It can be similar.
[0807] The embodiments disclosed herein are exemplary and any other means for predicting one or more aspects of host vehicle operation and producing a navigation response based on the prediction may be consistent with this disclosure.
[0808] The above description has been presented for purposes of illustration. The above description is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will become apparent to those skilled in the art from consideration of this specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, it is possible that these aspects may be stored on other types of computer-readable media, such as secondary storage, such as a hard disk or CD ROM, or other forms of storage. Those skilled in the art will understand that storage in RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray or other optically driven media is also possible.
[0809] Computer programs based on the described instructions and disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any technique known to those skilled in the art or designed in conjunction with existing software. For example, program sections or program modules may include .Net Framework, .Net Compact Framework (and related languages such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/AJAX combinations, XML, or Java applets. Can be designed in or with the included HTML.
[0810] Furthermore, while exemplary embodiments have been described herein, the scope of any embodiment is limited to equivalent elements, modifications, abbreviations, and combinations as would be understood by those skilled in the art based on this disclosure. There are adaptations and/or alternatives (eg, of aspects across the various embodiments). Limitations in the claims should be interpreted broadly based on the language utilized in the claims, and are not limited to the examples set forth herein or in the practice of this application. Examples should be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including reordering steps and/or inserting or deleting steps. It is therefore intended that the specification and examples be regarded as exemplary only, with a true scope and spirit being indicated by the following claims and their full range of equivalents.
(Item 1) A system for navigating a host vehicle, the system comprising: receiving at least one image representing an environment of the host vehicle from an image capture device; and a plan for achieving a navigation goal of the host vehicle. determining a planned navigation maneuver based on at least one driving policy; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle; determining a next state distance between the host vehicle and the target vehicle that would occur if the host vehicle were to perform a maximum braking capability of the host vehicle, a maximum acceleration capability of the host vehicle, and a maximum acceleration capability of the host vehicle; and determining the current speed of the host vehicle based on the current maximum braking capacity of the host vehicle, the current maximum acceleration capacity of the host vehicle, and the current speed of the host vehicle. determining a distance of the host vehicle; determining a current speed of the target vehicle; and assuming a maximum braking capability of the target vehicle based on at least one recognized characteristic of the target vehicle; The determined next state is such that the determined current stopping distance is a travel distance of the target vehicle determined based on the current speed of the target vehicle and the assumed maximum braking capacity of the target vehicle. and implementing the planned navigational operation if the distance is less than the sum of the distances.
(Item 2) The current stopping distance of the host vehicle is the distance traveled by the host vehicle over a predetermined period of time at the maximum acceleration capability of the host vehicle, starting from the determined current speed of the host vehicle. The system according to item 1, including an acceleration distance corresponding to the distance obtained.
(Item 3) The system according to item 2, wherein the predetermined period is a reaction time associated with the host vehicle.
(Item 4) The system according to item 1, wherein the maximum braking capacity of the host vehicle is determined based on a detected road surface condition.
(Item 5) The system according to item 1, wherein the maximum braking capacity of the host vehicle is determined based on detected weather conditions.
(Item 6) The system of item 1, wherein the speed of the target vehicle is determined based on analysis of the at least one image.
7. The system of claim 1, wherein the speed of the target vehicle is determined based on an analysis of output from at least one lidar or radar system associated with the host vehicle.
(Item 8) The system according to Item 1, wherein the recognized characteristics of the target vehicle include a type of vehicle.
(Item 9) The system of Item 1, wherein the recognized characteristic of the target vehicle is vehicle size.
(Item 10) The system according to item 1, wherein the recognized characteristics of the target vehicle include a model of the vehicle.
11. The system of claim 1, wherein the recognized characteristic of the target vehicle is determined based on an analysis of the at least one image.
(Item 12) The system according to Item 1, wherein the at least one characteristic of the target vehicle is determined based on at least one of lidar output or radar output.
(Item 13) The system according to item 1, wherein the planned navigation operation includes at least one of a lane change operation, a merging operation, an overtaking operation, a pursuit distance reduction operation, or a throttle maintenance operation.
(Item 14) The at least one processing device determines that the determined current stopping distance of the host vehicle is at least a predetermined distance greater than the travel distance of the target vehicle plus the determined next state distance. is configured to perform the planned navigation operation if the distance traveled by the target vehicle is less than the current speed of the target vehicle and the assumed maximum braking capacity of the target vehicle. The system described in item 1, as determined based on.
15. The system of claim 14, wherein the predetermined minimum distance corresponds to a predetermined separation distance maintained between the host vehicle and other vehicles.
(Item 16) The system of item 15, wherein the predetermined separation distance is at least 1 meter.
(Item 17) A system for navigating a host vehicle, the system comprising: receiving at least one image representing an environment of the host vehicle from an image capture device; and a plan for achieving a navigation goal of the host vehicle. determining a planned navigation maneuver based on at least one driving policy; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle; and determining the planned navigation maneuver based on at least one driving policy; determining a next state distance between the host vehicle and the target vehicle that would occur if determining a speed and assuming a maximum braking rate capability of the target vehicle based on at least one recognized characteristic of the target vehicle; At a predetermined submaximal braking rate that is less than the maximum braking rate capability of the vehicle, the target vehicle's braking rate is determined based on the current speed of the target vehicle and the assumed maximum braking rate capability of the target vehicle. performing the planned navigation operation if the host vehicle can be stopped within a stopping distance of the host vehicle that is less than the travel distance plus the determined next state distance; A system that includes at least one processing device that is programmed to
(Item 18) The predetermined submaximal braking rate is such that the brakes of the host vehicle will always operate at the predetermined submaximum braking rate until the host vehicle is stopped or until it is determined that a braking condition no longer exists. 18. The system of item 17, wherein the system is determined based on a user-selectable braking mode, including a mode to be applied.
(Item 19) The predetermined submaximal braking rate is such that the brakes of the host vehicle are always applied at the predetermined submaximal braking rate for at least a portion of the period during which a braking condition is determined to exist; 18. The system of item 17, wherein the system is determined based on a user selectable braking mode that includes a mode in which the host vehicle's brakes are applied at a maximum braking rate of the host vehicle.
(Item 20) The predetermined submaximal braking rate includes a mode in which the host vehicle's brakes are applied starting at the predetermined submaximal braking rate and increasing gradually to a maximum braking rate of the host vehicle. The system of item 17, determined based on a user-selectable braking mode.
21. The system of claim 17, wherein the predetermined submaximal braking rate is associated with a deceleration rate of up to 50% of a deceleration rate associated with the maximum braking rate capability of the host vehicle.
22. The system of claim 17, wherein the predetermined submaximal braking rate is associated with a deceleration rate of up to 25% of a deceleration rate associated with the maximum braking rate capability of the host vehicle.
23. The system of claim 17, wherein the stopping distance of the host vehicle is determined based on the current speed of the host vehicle and the predetermined submaximal braking rate of the host vehicle.
(Item 24) The stopping distance of the host vehicle is determined by the acceleration distance corresponding to the distance that the host vehicle can travel with the maximum acceleration capability of the host vehicle over a predetermined period and the maximum braking rate capability of the host vehicle. 18. The system of item 17, wherein the system is greater than a maximum braking rate distance corresponding to a distance that the host vehicle can travel while decelerating from the current speed of the host vehicle to zero speed.
(Item 25) The system according to item 24, wherein the predetermined period is a reaction time associated with the host vehicle.
(Item 26) The system according to item 17, wherein the maximum braking rate capability of the host vehicle is determined based on a detected road surface condition.
(Item 27) The system of item 17, wherein the maximum braking rate capability of the host vehicle is determined based on detected weather conditions.
(Item 28) The system of item 17, wherein the speed of the target vehicle is determined based on analysis of the at least one image.
29. The system of claim 17, wherein the speed of the target vehicle is determined based on an analysis of output from at least one lidar or radar system associated with the host vehicle.
(Item 30) The system according to item 17, wherein the recognized characteristic of the target vehicle includes a type of vehicle.
(Item 31) The system of item 17, wherein the recognized characteristic of the target vehicle is vehicle size.
(Item 32) The system of item 17, wherein the recognized characteristics of the target vehicle include a model of the vehicle.
33. The system of claim 17, wherein the recognized characteristic of the target vehicle is determined based on an analysis of the at least one image.
(Item 34) The system according to item 17, wherein the at least one characteristic of the target vehicle is determined based on at least one of lidar output or radar output.
(Item 35) The system according to item 17, wherein the planned navigation operation includes at least one of a lane change operation, a merging operation, an overtaking operation, a pursuit distance reduction operation, or a throttle maintenance operation.
(Item 36) The at least one processing device is configured to determine the determined stopping distance of the host vehicle based on the current speed of the target vehicle and the assumed maximum braking rate capability of the target vehicle. item 17, configured to perform the planned navigation operation if the distance traveled by the target vehicle is less than the distance traveled by the target vehicle plus the distance of the determined next state by at least a predetermined minimum distance; The system described.
37. The system of claim 36, wherein the predetermined minimum distance corresponds to a predetermined separation distance maintained between the host vehicle and other vehicles.
(Item 38) The system of item 37, wherein the predetermined separation distance is at least 1 meter.
(Item 39) The stopping distance of the host vehicle is a first distance at which the host vehicle is braked at a braking rate below the predetermined maximum, and a stopping distance at which the host vehicle is braked at the maximum braking rate of the host vehicle. and a second distance at which the brakes can be applied.
(Item 40) The at least one processing device is configured to reduce the predetermined maximum braking rate over the first distance before applying the brakes of the host vehicle at the maximum braking rate capability of the host vehicle over the second distance. 40. The system of item 39, wherein the system is configured to cause the host vehicle to brake at a braking rate of .
(Item 41) A system for navigating a host vehicle, the system comprising: receiving at least one image representing an environment of the host vehicle from an image capture device; and a plan for achieving a navigation goal of the host vehicle. determining a planned navigation maneuver based on at least one driving policy; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle; determining a next state distance between the host vehicle and the target vehicle that would occur if determining a speed and assuming a maximum braking rate capability of the target vehicle based on at least one recognized characteristic of the target vehicle; For a braking rate profile, the determined next state distance is added to the target vehicle travel distance determined based on the current speed of the target vehicle and the assumed maximum braking rate capability of the target vehicle. carrying out the planned navigation operation if the host vehicle can be stopped within a stopping distance of the host vehicle that is less than the maximum braking rate profile of the host vehicle; a system comprising at least one processing device programmed to perform: incrementally increasing a braking rate from a braking rate to a maximum braking rate;
42. The system of claim 41, wherein the predetermined braking rate profile increases linearly from the submaximal braking rate to the maximum braking rate of the host vehicle.
43. The system of claim 41, wherein the predetermined braking rate profile increases non-linearly from the submaximal braking rate to the maximum braking rate of the host vehicle.
(Item 44) A system for applying brakes of a host vehicle, the system comprising: receiving an output representative of an environment of the host vehicle from at least one sensor; determining a current speed of the host vehicle and a current distance between the host vehicle and the target vehicle; and determining a current speed of the host vehicle and a current distance between the host vehicle and the target vehicle. determining whether a braking condition exists based at least on the current distance between the vehicle and the host vehicle; and applying a braking device associated with the host vehicle according to a predetermined braking profile that progressively increases up to a maximum braking rate of the host vehicle. system.
(Item 45) Once the maximum braking rate of the host vehicle is achieved, the at least one processor causes the braking device of the host vehicle to operate at the maximum braking rate of the host vehicle until the braking condition no longer exists. A system according to item 44 configured to continue to apply.
(Item 46) The system of Item 44, wherein the gradual increase is non-linear.
(Item 47) The system of Item 44, wherein the gradual increase is linear.
(Item 48) An autonomous system for selectively replacing control of a host vehicle by a human driver, the system comprising: receiving at least one image representing an environment of the host vehicle from an image capture device; detecting at least one obstacle in the environment of the host vehicle based on analysis of a plurality of images; and monitoring driver input to at least one of a throttle control, a brake control, or a steering control associated with the host vehicle. determining whether the driver input causes the host vehicle to navigate within a proximity buffer with respect to the at least one obstacle; If the at least one processing device determines that the driver input will not cause the host vehicle to navigate within a proximity buffer, the driver input is transmitted to one or more motion control systems of the host vehicle. the host vehicle navigating within the proximity buffer with respect to the at least one obstacle; at least one processing device programmed to: prevent the driver input from causing the corresponding change in one or more motion control systems of the host vehicle, if the processing device determines; autonomous systems, including;
(Item 49) The at least one obstacle includes a target vehicle, and the proximity buffer to the target vehicle includes a detected current speed of the host vehicle, a maximum braking rate capability of the host vehicle, and a maximum braking rate capability of the host vehicle. the proximity buffer for the target vehicle is further determined based on a maximum acceleration capability of the host vehicle; and whereby the proximity buffer is at least determined as the distance that the host vehicle would travel if accelerated at the maximum acceleration capability of the host vehicle over a reaction time associated with the host vehicle. an acceleration distance of the host vehicle, a stopping distance of the host vehicle determined as the distance required to reduce the current speed of the host vehicle to zero at the maximum braking rate capability of the host vehicle; and the assumed stop distance of the target vehicle. 49. The system of item 48, comprising a sum of stopping distances of the target vehicle determined as the distance required to reduce the current speed of the target vehicle to zero at maximum braking rate capability.
(Item 50) The at least one obstacle includes a target vehicle, and the proximity buffer to the target vehicle includes a detected current speed of the host vehicle, a maximum braking rate capability of the host vehicle, and a maximum braking rate capability of the host vehicle. determined based on a determined current speed of a target vehicle and an assumed maximum braking rate capability of said target vehicle, said proximity buffer for said target vehicle being less than said maximum braking rate capability of said host vehicle. further determined based on a certain predetermined submaximal braking rate, such that the proximate buffer zone at least reduces the current speed of the host vehicle to zero at the predetermined submaximal braking capacity of the host vehicle. a stopping distance of the host vehicle determined as the distance required to reduce the current speed of the target vehicle to zero at the assumed maximum braking capacity of the target vehicle; The system according to item 49, including the sum of the stopping distances of the target vehicles.
(Item 51) The target vehicle is determined to be in front of the host vehicle, and the at least one processing device determines that the driver input is a change in longitudinal distance between the target vehicle and the host vehicle. The system according to item 49, configured to determine that the system will cause
(Item 52) The target vehicle is determined to be in a different lane than the host vehicle, and the at least one processing device determines that the driver input causes a lateral movement of the host vehicle; 50. The system of item 49, configured to determine that the target vehicle will be in front of the host vehicle after the lateral movement.
(Item 53) The target vehicle is determined to be in a different lane than the host vehicle, and the at least one processing device determines that the driver input causes a lateral movement of the host vehicle; 50. The system of item 49, configured to determine that the host vehicle will be in front of the target vehicle after the lateral movement.
(Item 54) The system of item 48, wherein the proximity buffer corresponds to a predetermined lateral distance threshold.
55. The system of claim 49, wherein the proximity buffer for the target vehicle is further determined based on a predetermined minimum distance maintained between the host vehicle and the target vehicle.
(Item 56) The at least one obstacle includes a pedestrian or an object in the road, and the proximity buffer to the at least one obstacle is at least between the host vehicle and the at least one obstacle. The system described in item 48, including the minimum distance maintained between.
(Item 57) The at least one obstacle includes a pedestrian, and the proximity buffer for the pedestrian is determined based on a current speed of the host vehicle, and the proximity buffer includes a pedestrian. The system described in item 48, in which the speed of
(Item 58) The throttle control includes an accelerator pedal, the brake control includes a brake pedal, and the steering control includes a steering wheel, and the driver input includes depressing the accelerator pedal, the brake pedal 49. The system of item 48, comprising at least one of depressing the brake pedal, not depressing the brake pedal, turning the steering wheel, or not turning the steering wheel.
(Item 59) The one or more control systems of the host vehicle include at least one steering actuator for controlling the heading of the host vehicle, a brake actuator for applying a brake device of the host vehicle, or a brake actuator of the host vehicle for applying a brake device of the host vehicle. The system of item 58, including an accelerator actuator for applying the throttle.
(Item 60) Preventing the driver input from causing the corresponding change in one or more motion control systems of the host vehicle comprises: preventing the driver input from causing the corresponding change in one or more motion control systems of the host vehicle; preventing a driver input to the brake pedal from causing a corresponding response by the brake actuator; or preventing a driver input to the accelerator pedal from causing a corresponding response by the accelerator actuator. The system according to item 59, comprising at least one of: preventing from occurring.
(Item 61) To prevent the driver input from causing the corresponding change in one or more motion control systems of the host vehicle, the at least one processing device may be configured to control the throttle control, the brake control, or 49. The system of item 48, configured to prevent operation of at least one of the steering controls.
(Item 62) To prevent the driver input from causing the corresponding change in one or more motion control systems of the host vehicle, the at least one processing device may be configured to control the throttle control, the brake control, or 49. The system of item 48, configured to disable operability of at least one of the steering controls.
(Item 63) In order to prevent the driver input from causing the corresponding change in one or more motion control systems of the host vehicle, the at least one processing device may cause the driver input to cause the according to item 48, configured to prevent operation of at least one of the throttle control, the brake control, or the steering control, and to apply an impulse force to at least one of the throttle control, the brake control, or the steering control. system.
(Item 64) The at least one processing device controls the throttle control until a driver input is received that would not cause the host vehicle to navigate within a proximity buffer to the at least one obstacle. Item 63, configured to continue to prevent operation of at least one of the brake control or the steering control, and to continue to apply the impulse force to at least one of the throttle control, the brake control, or the steering control. system.
(Item 65) The at least one processing device prevents the driver input from causing the corresponding change in one or more motion control systems of the host vehicle during an interval. 49. The system of item 48, configured to autonomously control navigation of the host vehicle.
(Item 66) The at least one processing device causes the host vehicle to receive a driver input that would not cause the host vehicle to navigate within a proximity buffer to the at least one obstacle. A system as described in item 65, configured to return navigation control to the driver.
(Item 67) further comprising a system override control for disabling the autonomous system to selectively replace control by a human driver, the system override control comprising the throttle control, the brake control, and the steering control. The system described in item 48, which differs from .
(Item 68) The at least one processing device is configured to track when the autonomous system for selectively replacing control by a human driver is overridden through operation of the system override control. , the system described in item 67.
(Item 69) further comprising a plurality of image capture devices each having a different field of view with respect to the environment of the host vehicle, wherein the at least one processing device captures one or more images from each of the plurality of image capture devices. and configured to detect the at least one obstacle in the environment of the host vehicle based on analysis of the one or more images received from any of the plurality of image capture devices. The system described in item 48.
70. The system of claim 69, wherein at least one of the plurality of image capture devices is configured to capture an image representing the environment of the host vehicle on a side of the host vehicle.
71. The system of claim 69, wherein at least one of the plurality of image capture devices is configured to capture an image representing the environment of the host vehicle behind the host vehicle.
(Item 72) A navigation system for navigating an autonomous host vehicle according to at least one navigation objective of the host vehicle, the sensor output being indicative of at least one aspect of the host vehicle's operation relative to the host vehicle's environment. from one or more sensors, wherein the sensor output is after the data acquisition point at which the measurement or data acquisition on which the sensor output is based is obtained, and the sensor output is received by the at least one processor. the time between receiving the received sensor output and the data acquisition time and the motion prediction time generated at a first time point that is before the second time point at which the sensor output is received; generating a prediction of the at least one aspect of the behavior of the host vehicle for the predicted time point based at least in part on an estimate of how the at least one aspect of the behavior of the host vehicle changes over an interval; , determining a planned navigation behavior of the host vehicle based at least in part on the at least one navigation objective of the host vehicle and based on the generated prediction of the at least one aspect of host vehicle operation. generating a navigation command to perform at least a portion of the planned navigation operation; the at least one actuation system of the host vehicle such that the at least one actuation system receives the navigation command at a third point in time prior to or substantially the same as the actuation point in response to the received command; providing the navigation command; and the predicted operation time is after the data acquisition time and before or equal to the actuation time. , navigation system.
(Item 73) The navigation system according to item 72, wherein the motion prediction time point substantially corresponds to the third time point.
(Item 74) The navigation system according to item 72, wherein the motion prediction time point substantially corresponds to the second time point.
(Item 75) The navigation system according to item 72, wherein the predicted operation time point substantially corresponds to the actuation time point.
(Item 76) The navigation system according to item 72, wherein the one or more sensors include a speed sensor, an accelerometer, a camera, a lidar system, or a radar system.
(Item 77) The navigation system of item 72, wherein the prediction of at least one aspect of host vehicle motion includes at least one prediction of speed or acceleration of the host vehicle at the time of the motion prediction.
(Item 78) The navigation system of item 72, wherein the prediction of at least one aspect of a host vehicle's operation includes predicting a path of the host vehicle at the time of the operation prediction.
(Item 79) The navigation system according to item 78, wherein the prediction of the route of the host vehicle at the time of the motion prediction includes a target traveling direction of the host vehicle.
(Item 80) The one or more sensors include a camera, and the prediction of the path of the host vehicle at the time of the motion prediction is based on at least one image captured by the camera. Navigation system as described.
(Item 81) The prediction of the path of the host vehicle at the time of the motion prediction is based on a determined speed of the host vehicle and a target trajectory of the host vehicle included in a map of road segments on which the host vehicle travels. A navigation system according to item 78, based at least on
(Item 82) The navigation system according to Item 81, wherein the target trajectory includes a predetermined three-dimensional spline representing a preferred route along at least one lane of the road segment.
(Item 83) The prediction of at least one aspect of the host vehicle's operation includes a determined brake pedal position, a determined throttle position, a determined air resistance, friction, or friction opposing the host vehicle's operation. A navigation system according to item 72, based on at least one of the slopes of the road segment along which the navigation system is based.
(Item 84) The prediction of at least one aspect of the host vehicle's motion at the time of the motion prediction includes a data acquisition rate associated with the one or more sensors and the at least one processor generating the navigation command. Navigation system according to item 72, which takes into account the discrepancy between the speed-related control speed.
(Item 85) The navigation system according to item 72, wherein the planned navigation operation includes at least one of changing the speed or changing the heading of the host vehicle.
(Item 86) The navigation system of Item 72, wherein the at least one actuation system includes one or more of a throttle actuation system, a brake actuation system, or a steering actuation system.
(Item 87) The navigation system according to item 72, wherein the navigation goal of the host vehicle includes transition from a first position to a second position.
(Item 88) The navigation system of Item 72, wherein the navigation goal of the host vehicle includes a lane change from a current lane occupied by the host vehicle to an adjacent lane.
(Item 89) The navigation goal of the host vehicle includes maintaining a proximity buffer between the host vehicle and a detected target vehicle, and the proximity buffer includes a detected current target vehicle of the host vehicle. the proximity buffer zone to the target vehicle; is further determined based on a maximum acceleration capability of the host vehicle, such that the proximity buffer is at least when accelerated at the maximum acceleration capability of the host vehicle over a reaction time associated with the host vehicle. an acceleration distance of the host vehicle determined as the distance traveled by the host vehicle, determined as the distance required to reduce the current speed of the host vehicle to zero at the maximum braking rate capability of the host vehicle; the sum of the stopping distance of the host vehicle and the stopping distance of the target vehicle determined as the distance required to reduce the current speed of the target vehicle to zero at the capability of the assumed maximum braking rate of the target vehicle; The navigation system described in item 72, including:
(Item 90) The prediction of at least one aspect of the host vehicle's operation is based on a predetermined function related to the host vehicle, the predetermined function including a determined current speed of the host vehicle and a predetermined current speed of the host vehicle. 73. The navigation system of item 72, wherein the navigation system enables prediction of future speed and acceleration of the host vehicle based on a determined brake pedal position or a determined throttle position of the host vehicle.
(Item 91) The navigation system according to item 72, wherein the motion prediction time is at least 100 milliseconds after the data acquisition time.
(Item 92) The navigation system according to item 72, wherein the motion prediction time is at least 200 milliseconds after the data acquisition time.
(Item 93) The navigation system according to item 72, wherein the navigation command includes at least one of a pedal command for controlling the speed of the host vehicle or a yaw rate command for controlling the traveling direction of the host vehicle. .
360 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75 Sheet 76 Sheet 77 Sheet 78 Sheet 79 Sheet 80 Sheet 81 Sheet 82 Sheet 83 Sheet 84 Sheet 85 Sheet 86 Sheet 87 Sheet 88 Sheet 89 Sheet 90 Sheet 91 Sheet 92 Sheet 93 Sheet 94 Sheet 95 Sheet 96 Sheet 97 Sheet 98 Sheet 99 Sheet 100 Sheet 101 Sheet 102 Sheet 103 Sheet 104 Sheet 105 Sheet 106 Sheet 107 Sheet 108 Sheet 109 Sheet 110 Sheet 111 Sheet 112 Sheet 113 Sheet 114 Sheet 115 Sheet 116 Sheet 117 Sheet 118 Sheet 119 Sheet 120 Sheet 121 Sheet 122 Sheet 123 Sheet 124 Sheet 125 Sheet 126 Sheet 127 Sheet 128 Sheet 129 Sheet 130 Sheet 131 Sheet 132 Sheet 133 Sheet 134 Sheet 135 Sheet 136 Sheet 137 Sheet 138 Sheet 139 Sheet 140 Sheet 141 Sheet 142 Sheet 143 Sheet 144 Sheet 145 Sheet 146 Sheet 147 Sheet 148 Sheet 149 Sheet 150 Sheet 151 Sheet 152 Sheet 153 Sheet 154 Sheet 155 Sheet 156 Sheet 157 Sheet 158 Sheet 159 Sheet 160 Sheet 161 Sheet 162 Sheet 163 Sheet 164 Sheet 165 Sheet 166 Sheet 167 Sheet 168 Sheet 169 Sheet 170 Sheet 171 Sheet 172 Sheet 173 Sheet 174 Sheet 175 Sheet 176 Sheet 177 Sheet 178 Sheet 179 Sheet 180 Sheet 181 Sheet 182 Sheet 183 Sheet 184 Sheet 185 Sheet 186 Sheet 187 Sheet 188 Sheet 189 Sheet 190 Sheet 191 Sheet 192 Sheet 193 Sheet 194 Sheet 195 Sheet 196 Sheet 197 Sheet 198 Sheet 199 Sheet 200 Sheet 201 Sheet 202 Sheet 203 Sheet 204 Sheet 205 Sheet 206 Sheet 207 Sheet 208 Sheet 209 Sheet 210 Sheet 211 Sheet 212 Sheet 213 Sheet 214 Sheet 215 Sheet 216 Sheet 217 Sheet 218 Sheet 219 Sheet 220 Sheet 221 Sheet 222 Sheet 223 Sheet 224 Sheet 225 Sheet 226 Sheet 227 Sheet 228 Sheet 229 Sheet 230 Sheet 231 Sheet 232 Sheet 233 Sheet 234 Sheet 235 Sheet 236 Sheet 237 Sheet 238 Sheet 239 Sheet 240 Sheet 241 Sheet 242 Sheet 243 Sheet 244 Sheet 245 Sheet 246 Sheet 247 Sheet 248 Sheet 249 Sheet 250 Sheet 251 Sheet 252 Sheet 253 Sheet 254 Sheet 255 Sheet 256 Sheet 257 Sheet 258 Sheet 259 Sheet 260 Sheet 261 Sheet 262 Sheet 263 Sheet 264 Sheet 265 Sheet 266 Sheet 267 Sheet 268 Sheet 269 Sheet 270 Sheet 271 Sheet 272 Sheet 273 Sheet 274 Sheet 275 Sheet 276 Sheet 277 Sheet 278 Sheet 279 Sheet 280 Sheet 281 Sheet 282 Sheet 283 Sheet 284 Sheet 285 Sheet 286 Sheet 287 Sheet 288 Sheet 289 Sheet 290 Sheet 291 Sheet 292 Sheet 293 Sheet 294 Sheet 295 Sheet 296 Sheet 297 Sheet 298 Sheet 299 Sheet 300 Sheet 301 Sheet 302 Sheet 303 Sheet 304 Sheet 305 Sheet 306 Sheet 307 Sheet 308 Sheet 309 Sheet 310 Sheet 311 Sheet 312 Sheet 313 Sheet 314 Sheet 315 Sheet 316 Sheet 317 Sheet 318 Sheet 319 Sheet 320 Sheet 321 Sheet 322 Sheet 323 Sheet 324 Sheet 325 Sheet 326 Sheet 327 Sheet 328 Sheet 329 Sheet 330 Sheet 331 Sheet 332 Sheet 333 Sheet 334 Sheet 335 Sheet 336 Sheet 337 Sheet 338 Sheet 339 Sheet 340 Sheet 341 Sheet 342 Sheet 343 Sheet 344 Sheet 345 Sheet 346 Sheet 347 Sheet 348 Sheet 349 Sheet 350 Sheet 351 Sheet 352 Sheet 353 Sheet 354 Sheet 355 Sheet 356 Sheet 357 Sheet 358 Sheet 359 Sheet 360
Every citation, both ways
| Document | Relation | Office |
|---|---|---|
| JP201765357A | Cites | Japan |
| JP201735927A | Cites | Japan |
| JP201631660A | Cites | Japan |
| WO2016016980A1 | Cites | World Intellectual Property Organization (WIPO) |
128 members in 8 offices
Priority claims15
| Document | Office | Kind | Date |
|---|---|---|---|
| 62645479 | United States of America | – | |
| 201862645479 | United States of America | P | |
| 62646579 | United States of America | – | |
| 201862646579 | United States of America | P | |
| 62718554 | United States of America | – | |
| 201862718554 | United States of America | P | |
| 62724355 | United States of America | – | |
| 201862724355 | United States of America | P | |
| 62772366 | United States of America | – | |
| 201862772366 | United States of America | P | |
| 62777914 | United States of America | – | |
| 201862777914 | United States of America | P | |
| 2019541236 | Japan | A | |
| 2019000281 | International Bureau of the World Intellectual Property Organization (WIPO) | W | |
| 2022074462 | Japan | A |
Members128
| Document | Office | Kind | |
|---|---|---|---|
| US2019291726A1 | United States of America | A1 | |
| US2019291728A1 | United States of America | A1 | |
| WO2019180506A2 | World Intellectual Property Organization (WIPO) | A2 | |
| US2019299984A1 | United States of America | A1 | |
| US2019337509A1 | United States of America | A1 | |
| KR20190126416A | Republic of Korea | A | |
| IL269828A | Israel | A | |
| IL269828D0 | Israel | D0 | |
| WO2020035728A2 | World Intellectual Property Organization (WIPO) | A2 | |
| WO2019180506A3 | World Intellectual Property Organization (WIPO) | A3 | |
| EP3636507A1 | European Patent Office (EPO) | A1 | |
| EP3640105A1 | European Patent Office (EPO) | A1 | |
| EP3640106A1 | European Patent Office (EPO) | A1 | |
| EP3642092A2 | European Patent Office (EPO) | A2 | |
| WO2020035728A3 | World Intellectual Property Organization (WIPO) | A3 | |
| CN111278704A | China | A | |
| KR102127741B1 | Republic of Korea | B1 | |
| KR20200124148A | Republic of Korea | A | |
| KR20200124149A | Republic of Korea | A | |
| KR20200124150A | Republic of Korea | A | |
| US10843690B2 | United States of America | B2 | |
| US10850728B2 | United States of America | B2 | |
| DE112019001421T5 | Germany | T5 | |
| US10933868B2 | United States of America | B2 | |
| EP3787947A2 | European Patent Office (EPO) | A2 | |
| US2021078570A1 | United States of America | A1 | |
| US2021094577A1 | United States of America | A1 | |
| CN112601686A | China | A | |
| EP3805063A1 | European Patent Office (EPO) | A1 | |
| EP3805064A1 | European Patent Office (EPO) | A1 | |
| KR20210042387A | Republic of Korea | A | |
| CN112721923A | China | A | |
| CN112762951A | China | A | |
| EP3828049A1 | European Patent Office (EPO) | A1 | |
| US2021162993A1 | United States of America | A1 | |
| US2021162994A1 | United States of America | A1 | |
| US2021162995A1 | United States of America | A1 | |
| CN112896190A | China | A | |
| US2021171023A1 | United States of America | A1 | |
| US2021179096A1 | United States of America | A1 | |
| US2021191406A1 | United States of America | A1 | |
| EP3842303A2 | European Patent Office (EPO) | A2 | |
| EP3842304A2 | European Patent Office (EPO) | A2 | |
| US2021200235A1 | United States of America | A1 | |
| JP2021516638A | Japan | A | |
| EP3854646A2 | European Patent Office (EPO) | A2 | |
| US11077845B2 | United States of America | B2 | |
| US2021240197A1 | United States of America | A1 | |
| EP3842303A3 | European Patent Office (EPO) | A3 | |
| EP3842304A3 | European Patent Office (EPO) | A3 | |
| EP3854646A3 | European Patent Office (EPO) | A3 | |
| KR102337011B1 | Republic of Korea | B1 | |
| JP2021536599A | Japan | A | |
| KR102369219B1 | Republic of Korea | B1 | |
| US11345340B2 | United States of America | B2 | |
| JP2022105125A | Japan | A | |
| JP2022106879A | Japan | A | |
| KR102423172B1 | Republic of Korea | B1 | |
| KR20220105682A | Republic of Korea | A | |
| JP7110209B2 | Japan | B2 | |
| US11427192B2 | United States of America | B2 | |
| CN115384486A | China | A | |
| KR102479471B1 | Republic of Korea | B1 | |
| IL269828B1 | Israel | B1 | |
| JP7260231B2 | Japan | B2 | |
| US2023166729A1 | United States of America | A1 | |
| JP7315294B2 | Japan | B2 | |
| CN111278704B | China | B | |
| IL269828B2 | Israel | B2 | |
| US11731617B2 | United States of America | B2 | |
| JP2023139069A | Japan | A | |
| JP7367255B2This record | Japan | B2 | |
| CN116968731A | China | A | |
| CN116968732A | China | A | |
| CN117022255A | China | A | |
| CN117022256A | China | A | |
| CN117022257A | China | A | |
| US11820365B2 | United States of America | B2 | |
| US11840258B2 | United States of America | B2 | |
| JP2023179655A | Japan | A | |
| EP3828049B1 | European Patent Office (EPO) | B1 | |
| US11897508B2 | United States of America | B2 | |
| EP3805063B1 | European Patent Office (EPO) | B1 | |
| EP3805063C0 | European Patent Office (EPO) | C0 | |
| US11932277B2 | United States of America | B2 | |
| CN112762951B | China | B | |
| EP3842303B1 | European Patent Office (EPO) | B1 | |
| EP3805064B1 | European Patent Office (EPO) | B1 | |
| EP3805064C0 | European Patent Office (EPO) | C0 | |
| EP3854646B1 | European Patent Office (EPO) | B1 | |
| US12037018B2 | United States of America | B2 | |
| US12037019B2 | United States of America | B2 | |
| EP3805064B8 | European Patent Office (EPO) | B8 | |
| EP3854646B8 | European Patent Office (EPO) | B8 | |
| DE202019006068U1 | Germany | U1 | |
| CN112721923B | China | B | |
| EP3842304B1 | European Patent Office (EPO) | B1 | |
| EP3842304C0 | European Patent Office (EPO) | C0 | |
| JP7574531B2 | Japan | B2 | |
| US12139132B2 | United States of America | B2 |
6 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 | |
| Explanation of circumstances concerning accelerated examinationJAPANESE INTERMEDIATE CODE: A871A871 | A871 |
Numbers
- Publication
- 7367255
- Application
- 114089
Titles2
- Japanese
- 車両をナビゲートするためのシステム及び方法
- English
- Systems and methods for navigating a vehicle
Classification
- CPC, 80
- B60W30/09
- B60W30/0953
- B60W30/0956
- B60W60/0015
- B60W60/0027
- B60W50/12
- B60W50/087
- B60W30/18163
- B60W50/0097
- B60W50/082
- B60W10/06
- B60W10/18
- B60W10/20
- B60W2050/0031
- B60W2554/00
- B60W2554/402
- B60W2554/4046
- B60W2554/80
- B60W2554/804
- B60W2555/20
- B60W2754/20
- B60W2754/30
- B60W30/18154
- B60W60/00272
- B60W60/00274
- B60T7/18
- B60T7/22
- G01S13/931
- G01S17/931
- B60T2201/02
- B60T2201/022
- G01S2013/9323
- B60W2552/50
- B60W2552/53
- B60W2552/30
- B60W2552/15
- B60W2554/802
- B60W2554/4029
- G01S17/86
- G01S17/58
- G01S13/865
- G01S13/60
- B60W2420/403
- B60W2554/20
- B60W2554/406
- B60W2510/18
- B60W2520/105
- B60W2540/12
- B60W2554/4042
- G01C21/3602
- B60W2520/04
- B60W2530/16
- B60W2710/18
- B60W30/181
- B60W40/06
- B60W2510/20
- B60W2520/14
- B60W2552/00
- B60W30/165
- B60W40/02
- B60W2050/0005
- B60W2510/0604
- B60W2520/10
- B60W2540/10
- B60W2710/20
- B60W30/095
- B60W40/105
- B60T8/86
- B60W2420/408
- B60W40/1005
- B60W40/114
- B60W40/107
- B60W2720/14
- B60Y2300/0952
- B60Y2300/0954
- B60Y2300/09
- B60Y2400/303
- B60Y2400/304
- B60Y2300/18166
- B60Y2300/18158
- IPC, 1
- B60W30 095
