Rendering, viewing and annotating panoramic images, and applications thereof

The present invention generally relates to panoramic images.

Computer-processed mapping systems have traditionally provided top-down presentations of mapping data. Extending the capabilities of mapping systems with street-level images poses various interface challenges, which address navigation within street-level views, including, for example, turning at intersections. In absolute terms (eg, latitude, longitude and direction of travel), and associating a user's location and orientation on a map with a user's location and orientation in a street-level view, as well as city streets and directions. Do it as for landmarks such as city intersections. A9 BlockView (no longer online) and Windows® Live Local Technology Preview, backed by Microsoft® Virtual Earth (http://preview.local.live.com), are recently available on urban streets. Attempted to provide an interface that can be used for level views. A9 BlockView addressed the orientation issue by flattening the image into two strips, so users don't have the freedom to look 360 degrees. Windows® Live Local gives a "car" view of street-level images, and the "car" view is rotated by 90 degrees by manipulating the car avatar on the map view. ..

<p> The present invention relates to rendering and viewing a panoramic image and annotating the panoramic image. In a first embodiment, one method can be used to view a panoramic image. The method includes receiving at least a portion of the image of the first panorama and providing a viewport displaying a portion of the image of the first panorama. The viewport includes a three-dimensional cover that is drawn with the image of the first panorama. When the 3D cover is drawn with the image of the 1st panorama, the orientation of the 3D cover is changed in the 3D space to match the change in the orientation of the image of the 1st panorama. Will be done.</p><p> In a second embodiment, one system can be used to view panoramic images. The system is a panorama viewer that receives at least a portion of the image of the first panorama, the panorama viewer providing a viewport, which is one of the images of the first panorama. Includes a panoramic viewer that displays the part. The viewport includes a three-dimensional cover that is drawn with the panoramic image. When the 3D cover is drawn with the panoramic image, the panorama viewer will use the 3D cover in 3D space to match the change in orientation of the panoramic image in the viewport. Change the orientation of.</p><p> In a third embodiment, one method can be used to draw the surface. The method is to calculate the region on the surface defined by the first transformation from the viewport to the surface and to calculate the second transformation that maps from that region to the viewport. Includes determining a portion of the image that intersects the region. Finally, the second transformation is applied to a portion of the image that intersects the region, resulting in a transformed image. The converted image is drawn in the viewport for display.</p><p> In a fourth embodiment, one system draws a surface. The system includes a server that computes the area on the surface defined by the first conversion from the viewport to the surface. The server also computes a second transformation that maps from that region to that viewport. The system further includes a panoramic viewer that determines a portion of the image that intersects the region on the surface. The panoramic viewer applies the second transformation to a portion of the image that intersects the region on the surface, resulting in a second transformed image. The converted image is drawn in the viewport for display.</p><p> In a fifth embodiment, one method can be used to process annotations on the panorama. The method comprises receiving a first user annotation for a feature in the first panorama and a second user annotation for the feature in the second panorama. The coordinates are determined based on the intersection of the data generated for the first user annotation for the feature and the data generated for the second user annotation for the feature. Finally, the coordinates are stored in relation to the annotation representing the feature.</p><p> In a sixth embodiment, one system can be used to process annotations on the panorama. The system includes a server that receives a first user annotation for a feature in the first panorama and a second user annotation for the feature in the second panorama. The server determines the location of the feature based on the first location of the first user annotation for the feature and the second location for the second user annotation for the feature.</p><p> Further, embodiments, features and advantages of the present invention and the structures and operations of various embodiments of the present invention are described in detail below with reference to the accompanying drawings.</p>

Embodiments of the present invention are described with reference to the accompanying drawings. In multiple drawings, the same reference number may indicate the same element or functionally similar elements.<figref num="1">FIG. 1 is a diagram of an exemplary distributed system suitable for the embodiment of one embodiment.</figref><figref num="2">FIG. 2 is a diagram illustrating an example of how a mapping service can be integrated with a panoramic viewer according to one embodiment.</figref><figref num="3">FIG. 3 depicts an example of a browser display according to one embodiment.</figref><figref num="4">FIG. 4 is a flowchart illustrating an exemplary process performed by the panoramic viewer according to one embodiment.</figref><figref num="5">Figure 5 depicts exemplary Extended Markup Language (XML) configuration information.</figref><figref num="6">FIG. 6 illustrates an example of a panoramic image.</figref><figref num="7A">Figures 7A, 7B and 7C illustrate the user's interaction with the viewport of the panorama viewer.</figref><figref num="7B">Figures 7A, 7B and 7C illustrate the user's interaction with the viewport of the panorama viewer.</figref><figref num="7C">Figures 7A, 7B and 7C illustrate the user's interaction with the viewport of the panorama viewer.</figref><figref num="8">FIG. 8 is a flowchart of processing performed by a renderer according to one embodiment.</figref><figref num="9A">Figures 9A, 9B and 9C illustrate the relationship between the surface and the pre-computed area and viewport.</figref><figref num="9B">Figures 9A, 9B and 9C illustrate the relationship between the surface and the pre-computed area and viewport.</figref><figref num="9C">Figures 9A, 9B and 9C illustrate the relationship between the surface and the pre-computed area and viewport.</figref><figref num="10A">Figures 10A and 10B illustrate a simple example of generating transformation parameters.</figref><figref num="10B">Figures 10A and 10B illustrate a simple example of generating transformation parameters.</figref><figref num="11">FIG. 11 depicts an example of a bent panorama according to an embodiment of the present invention.</figref><figref num="12">FIG. 12 depicts an exemplary transformation based on roll and pitch to form the panorama of FIG.</figref><figref num="13">FIG. 13 depicts an exemplary panoramic image displayed according to an embodiment of the present invention.</figref><figref num="14">FIG. 14 is a diagram illustrating a method of generating coordinates for a user annotation according to one embodiment.</figref><figref num="15">FIG. 15 is a flowchart of processing performed in the generation of user annotation coordinates according to one embodiment.</figref>

The present invention is described with reference to the accompanying drawings. The drawing in which the element first appears is typically represented by the leftmost number in the corresponding reference number or the leftmost number.

The present invention relates to drawing and viewing panoramic images, annotating panoramic images, and its applications. In the embodiments of the specification of the present invention for carrying out the invention, references to "one embodiment", "embodiment", "exemplary embodiment", etc., indicate whether the described embodiment has a specific feature or is specific. It indicates that it may include a structure or a particular feature, but not all embodiments may include a particular feature, structure or feature. Moreover, such terms do not necessarily refer to the same embodiment. Moreover, if a particular feature, structure or feature is mentioned with respect to an embodiment, it will be of one of ordinary skill in the art to bring about such feature, structure or feature with respect to other embodiments, whether explicitly stated or not. Presented within the bounds of knowledge.

FIG. 1 is a decentralized system suitable for implementing the embodiments of the present invention. Client 110 communicates with one or more servers 150 across a network, such as the Internet or a local area network. The client 110 can be a general purpose computer having a processor, local memory, a display, and one or more input devices such as a keyboard or mouse. Alternatively, the client 110 may be specialized for a computing device such as a mobile handset. The server 150 can also be implemented using any general purpose computer capable of supplying data to the client 110.

The client 110 executes the panorama viewer 120, and the operation of the panorama viewer 120 is further described herein.

As illustrated by FIG. 1, the panorama viewer 120 requests configuration information 130 from the server (s) 150. As discussed in more detail herein, the configuration information includes meta information about the panorama being loaded, and that meta information includes information about links to other panoramas within the panorama. In embodiments, the configuration information is given in a format such as Extended Markup Language (XML). The panorama viewer 120 retrieves a visual asset 140 for a panorama, for example in the form of a panoramic image or in the form of a panoramic image tile. In another embodiment, the visual asset contains the configuration information in the relevant file format. As further described herein, the panorama viewer 120 is a visual representation of panoramas and additional user interface elements, such as those generated from configuration information 130 and visual assets 140. representation) is given on the client display. When the user interacts with the input device to manipulate the visual representation of the panorama, the panorama viewer 120 processes the visual representation to update and download additional configuration information and visual assets as needed.

In one embodiment, the Panorama Viewer 120 can be a standalone application, i.e. it can be run within a browser 115 such as Mozilla Firefox or Microsoft® Internet Explorer. The Panorama Viewer 120 may be executed, for example, as a script within the browser 115, as a plug-in within the browser 115, or as a program that runs within a browser plug-in such as the Adobe (Macromedia) Flash plug-in. In one embodiment, the panorama viewer 120 is integrated with a mapping service, as described in US Pat. No. 7,158,878, "DIGITAL MAPPING SYSTEM," which is incorporated herein by reference in its entirety.

FIG. 2 illustrates an example of how the mapping service 210 can be integrated with the panorama viewer 120. The map creation service 210 displays the visual representation of the map, for example, as a viewport to the grid of map tiles. The mapping system 210 is implemented using a combination of markup and scripting elements (eg, using HTML and Java® script). When the viewport is moved, the mapping service 210 will add additional map tiles 220 from the server (s) 150, assuming that the requested map tiles have not been cached in the local cache memory by then. Request. The server (s) that supply the map tile 220 is clearly the same as or distinctly different from the server (s) that supplies the panorama tile 140 or any other data contained herein. obtain.

In one embodiment, the browser 115 downloads the program 250 from the server (s) 150 to the panorama viewer 120 and instantiates all the plug-ins needed to run the program 250. The mapping service 210 may request that it proceed. Program 250 can be in the form of a Flash file or some other executable content. The panorama viewer 120 runs and operates as described above. Alternatively, the configuration information 130 and the equal panorama tile 140 may be retrieved by the mapping service 210 and passed to the panorama viewer 120. The panorama viewer 120 and the mapping service 210 communicate to coordinate the behavior of the user interface elements and allow the user to interact with either the panorama viewer 120 or the mapping service 210. , Communicate so that location changes or orientation changes are reflected in both.

FIG. 3 is an example of a browser display 300, which provides both a mapping service such as the mapping service 210 and an integrated panorama viewer such as the panorama viewer 120. The mapping service provides a button 310 entitled "Street View", which, when selected, preferably changes the appearance of the map of the area where panoramic data is available. For example, in FIG. 3, a street with available panoramic data is highlighted. This highlight is, for example, a colored contour and / or a shaded contour, a colored cover and / or a shaded cover, or a color and / or It could be a shading change. This can be implemented by using a transparent image with the map tiles supplied to the mapping service or by including the effect directly in the map tiles supplied to the mapping service.

The mapping service allows the user to launch a panorama viewer by further selecting points on the map. When a point is selected by the user, the character icon 320 or the avatar icon 320 is displayed at the point on the map. In one embodiment, the avatar icon includes an indicator in which direction the avatar icon faces, which indicator is represented in FIG. 3 as an arrow below the avatar icon.

In one embodiment, when the panorama viewer is instantiated by a mapping service, the panorama viewer is provided in the form of viewport 330 embedded in a balloon window of information associated with the avatar icon 320. The orientation of the visual representation of the panorama in viewport 330 matches the orientation of the avatar icon 320. When the user manipulates the visual representation of the panorama in viewport 330, the panorama viewer notifies the mapping service of all orientation or location changes so that the mapping service updates the orientation and location of the avatar icon 320. Can be done. Similarly, when the user manipulates the orientation or location of the avatar icon 320 within the mapping service, the mapping service may notify the panorama viewer so that the panorama viewer updates its visual representation.

In one embodiment, viewport 330 of the panorama viewer gives a panoramic image of the selected area. The user can click on the image and drag it around the image for a 360 degree view. In the exemplary viewport 330 depicted in Figure 3, various user interface elements are added to the underlying panorama. These elements include, for example, zoom and panning controls on the left side of the viewport (eg navigation buttons) and the left side of annotations in the form of lines / bars, arrows and text provided directly within the panorama itself. Includes navigation inputs such as zoom and panning controls. The annotations are drawn in a three-dimensional style, which roughly matches the three-dimensional landscape depicted in the panorama.

In FIG. 3, for example, the lines / bars in viewport 330 correspond to the streets drawn on the corresponding map and may be drawn in the same color as the streets drawn on the map. Multiple arrows are selectable by the user (by clicking or dragging along the street line), and the arrows go in each direction where another available panorama exists. These allow the user to navigate up and down the street (ie, change position from where the street is viewed). The lines and arrows smoothly follow the underlying image so that when the user looks 360 degrees, the lines remain on the underlying street and the arrows are always visible on the screen. This allows users to navigate along the street while looking straight ahead or by the storefront.

When the user clicks on the arrow to navigate in the viewport, the zoom crossfade effect and other visual cues give the user a sense of movement. When a user arrives at the intersection of two streets, there is one green line and two arrows for each street. By seeing all of these at the same time and displaying them all on the label, the user can know the latest location and can move in any direction. This technique can effortlessly scale to accommodate complex intersections with more than four directions. When the user reaches a "dead end" where the road continues but there are no more available images, there is only one arrow on the street indicating the direction in which the user can navigate. In other directions, symbols and messages embedded in the image may be given to notify the user that the image is not available in this direction.

The user interface is not limited to navigating along a line for walking down the street, and when useful, the user deviates from that line element to a side road (for example, to see something up close). Can be easily extended to allow (crossing across the street). What's more, environments where users can expect to snap off from the street and navigate freely within nearby areas (eg parks, squares, shopping areas or other areas). Pedestrian-friendly public place)) exists in the city. The interface is effortlessly upgraded with "free movement zones" to provide this functionality. The user interface is provided within the context of navigation between separate street-level panoramas, but users compare panoramic data so that navigating along the street can be as smooth as video. It should also be noted that user interfaces can be used as well to allow navigation through a continuous set.

The behavior and implementation of user interface elements is described in more detail below.

FIG. 4 is a typical flowchart of processing performed by a panoramic viewer, for example, panoramic viewer 120, according to an embodiment of the present invention.

In step 402, the panorama viewer receives an identifier for the first panorama given and various viewport parameters such as viewport size and orientation viewed within the panorama. This information can be passed from the mapping service to the panorama viewer, for example by using Flashvars or External Interface between the Flash plug-in and Java® script in the mapping service.

In step 404, the panorama viewer uses the panorama identifier to request configuration information (eg, an XML file) from the server. FIG. 5 depicts a typical XML configuration information 500, which is discussed in more detail below. The XML is parsed and the information is loaded into various data structures for use by the panoramic viewer. In one embodiment, the XML includes information for the panorama viewer, such as the latest panorama data and projection properties, and information about annotations / links within the panorama, including links to other panoramas.

In step 406, the panorama viewer requests a visual asset for the panorama and stores the received visual asset in, for example, local memory / local storage. In one embodiment, the panorama viewer may maintain a cache of visual assets and limit bandwidth usage to searching for visual assets that do not exist in the cache. FIG. 6 illustrates an example of the first panoramic image 600. The full panoramic image 600 can be searched by the panoramic viewer, or the panoramic image 600 can be split into multiple panoramic image tiles, which tiles can only be requested by the panoramic viewer when needed.

At step 408, the panorama viewer processes configuration information and visual assets in preparation for drawing a visual representation of the panorama in the viewport at step 410. With respect to visual assets, the panorama viewer can assemble the panorama image tiles into part of a complete panorama image that overlaps the viewport. As further discussed herein, a panoramic viewer can give an image of a panorama, either as a flat surface or as a texture-mapped three-dimensional surface, such as a cylinder or sphere. Respect covering annotation given within the viewport, Panorama viewer, using the setting information, thereby outline and location for these various elements, such as a given line / bar and arrows in the example viewport to Shon calculate.

In one embodiment, the polygon / outline is modeled in a three-dimensional space that corresponds to the space depicted in the panorama. These polygons / contours can be modeled using, for example, a pinhole camera model (eg, the focal length can be generated by multiplying the height of the viewport by a constant relative depth of the center of rotation). The annotation cover polygon / outline changes the orientation of the annotation cover polygon / outline in three-dimensional space in a way that matches the viewport orientation change. In one embodiment, the polygon / outline is rotated at an angle equal to the difference between the latest orientation of the user's viewpoint in the panorama and the orientation of the annotation as identified in the configuration information. As further described herein, polygons / contours can be further transformed around different spatial axes to take into account non-flat panoramas.

In step 410, the visual representation of the panorama in the viewport is drawn.

In step 412, the panorama viewer receives and manages input by capturing input events, such as mouse and keyboard events. The panorama viewer chooses to zoom (eg, click on the panorama), for example, whether the user has panned the viewport (eg, by dragging the mouse or by selecting the pan control button). Detects by or by moving the zoom slider control on the left side of the viewport with the mouse, or by selecting a link to another panorama (for example, by clicking on the arrow with the mouse). ..

At step 420, a determination is made as to whether the user has panned the viewport. If the user pans the viewport, control moves to step 422. If the user does not pan the viewport, control moves to step 430.

At step 422, the panorama viewer determines if the viewport overlaps with any panoramic image tile that needs to be retrieved from either the server or the cache.

At step 424, as further described herein, the panoramic viewer performs the necessary calculations to allow the viewport to be drawn accurately in different orientations.

In step 426, the panorama viewer informs the mapping service of the new orientation selected by the user, so that the mapping service can update the facing indicator of its avatar icon. The panorama viewer recalculates the outline and location for the viewport element and draws the viewport. To illustrate this point, consider Figure 7A, which depicts the panoramic viewer viewport from Figure 3. FIG. 7B shows the result after the user chooses to pan the panorama to the left. Note that when the panorama viewer changes the orientation of the panorama, the lines / bars corresponding to the roads drawn in the panorama change their orientation.

At step 430, a determination is made as to whether the user has zoomed the viewport. If the user zooms in the viewport, control shifts to step 432. If the user does not zoom the viewport, control shifts to step 440.

In step 432, for example, whether the panorama viewer requests a new higher resolution panoramic image tile from the server (or from the cache), or when there is no such higher resolution tile (eg, there). ) Determine if existing tiles are used with different close-up resolutions.

At step 434, the viewport parameters are modified to reflect different zoom levels. Transitions can be provided between zoom levels to give an overview with the property of zooming vigorously to the next zoom level in the panorama. Figure 7C shows the results after the user chooses to zoom in on a feature in Figure 7A.

At step 440, a determination is made as to whether the user has selected a link to another panorama. If the user chooses to link to another panorama, control moves to step 442. If the user does not select a link to another panorama, control moves to step 412.

At step 442, the panorama viewer proceeds to begin processing the transition between the original panorama and the new panorama. The panorama viewer can zoom the original panorama to a new panorama and crossfade to the new panorama, for example, to give the user a sense of movement. Alternatively, the panorama viewer may reproduce the actual video transition between the two panoramas.

In step 444, the panorama viewer informs the mapping service of the new location selected by the user so that the mapping service can update the location of its avatar icon and scroll the map accordingly.

In one embodiment, the panoramic viewer can be implemented using any lucrative programming language or any lucrative programming style. For example, the panorama viewer can be implemented using object-oriented programming with separate classes prepared to handle XML configuration information, annotations, texture generation, tile management and mesh generation.

Figure 5 specifies typical XML configuration information 500 (eg, metadata). As explained by the example shown in FIG. 5, the schema for the setting information is systematized in "data_properties", "projection_properties" and "annotation_properties".

The subgroup Data_Properties are "pano_id" (eg, a unique identifier for the panorama) and "image_width" and "image_height" (eg, the dimension of the panoramic image before it is tiled) and "tile_width" and "tile_height" (for example, tile_height). Includes attributes such as tile dimensions) and attributes such as "lat" and "lng" (eg, the coordinates of the latest panorama) and "num_zoom_levels" (eg, the number of zoom levels the user can view in the panorama viewer). This subgroup also includes "text" (eg, it can be used to represent the street name of the latest panorama), "copyright" (eg, copyright information) and "street_range" (eg, for a given street). Includes elements such as (number range of).

The subgroup Projection_properties are graded as "Pano_yaw_deg" (eg, the orientation of the vehicle that captured the image that produced the panoramic image) and "tilt_yaw_deg" and "tilt_pitch_deg" (eg, as further described herein). Like the highest gradient line rolls and pitches) and "vertical_scale" (eg, the ratio of images along the y-axis seen at relatively low zoom levels) that are useful for processing panoramas with features. Includes attributes.

The subgroup Annotation_properties is "horizon_height_fraction" (eg, the vertical position (height) from the horizon that can be adjusted to maximize the fit between the annotation and the image of the tile) (as the ratio of visible pieces). Includes attributes such as)) and "annotation_height_fraction" (eg, vertical position (height) from the drawing containing annotations (expressed as the ratio of visible fragments)). This subgroup also includes a subgroup of "pano_link", where the subgroup of "pano_link" describes the properties of the link symbol, which navigates the user to a nearby panorama or another related document. Allows you to gate. The "link" subgroup contains "link_text" (eg, a description of the landing panorama) as an element, and includes the following as attributes. "Yaw_deg" (eg, the direction the link points to), "pano_id" (eg, the identifier for the linked panorama) and "road_argb" (eg, road attributes, such as the color of the road on the map) (shown) Absent). The subgroup may also contain a "floating_text" group or "floating_text" element that identifies any feature in the panorama, and may also include any link (eg, to a local data repository or website) (shown). It has not been).

Note that the above schema for configuration information is merely descriptive and can be designed in any of a number of advantageous ways, including using techniques that do not rely on XML.

FIG. 8 is a flowchart of processing performed by the renderer according to an embodiment of the present invention.

In step 802, the renderer precomputs a pre-image of the viewport by reverse transformation. It defines a portion of the surface, which is referred to herein as a "pre-computed area". FIG. 9A illustrates this in the context of the surface 900 of a cylinder with a rectangular viewport 920, which defines a pre-computed region 910 by the reverse transformation. Note that the viewport does not have to be rectangular and that the technique works for discretized cylinders (based on meshes) or for continuous voxel-pixel mapping. For example, a mesh can be defined on the viewport with the corresponding mesh on the cylinder. These meshes do not have to be uniform and are images of each other as defined by forward or reverse mapping. The mesh on the cylinder often only covers part of the cylinder. In the case of continuous transformation, the viewport's pre-image can define a continuous region of the cylinder.

In step 804, the renderer precomputs the transformation, which maps each pixel from the precomputed area to the pixel in the viewport. In a sense, it is assumed that the cylinder stands still in space. Instead of attaching a texture image to the cylinder to be modified, the texture is supposed to "slide" on the cylinder.

In step 806, the renderer transfers the image / texture in response to user input.

In step 808, the renderer determines a portion of the image / texture, which intersects the precomputed area of the surface. This defines the set of pixels that need to be drawn in the viewport. If the user recently changed their perspective, this needs to be updated. More precisely, in step 806, any pan to the left or right of the viewport can be achieved effortlessly by shifting the texture in the corresponding direction, thereby producing different intersections using precomputed regions. To do. Similarly, any pan up or down is achieved by shifting the texture along these directions. Any direction of panning is achieved by simply transferring the texture in the corresponding direction. New intersections with precomputed regions are always generated. This is illustrated in FIGS. 9B and 9C, where 950 represents a pre-computed area and 960 represents an image / texture.

In step 810, the precomputed transformation is applied to a portion of the image / texture, and the portion of the image / texture intersects the precomputed region.

Finally, in step 812, the converted image is drawn in the viewport.

In one embodiment, the renderer uses the projection properties of a linear configuration to speed up the drawing of a panoramic image. When a surface such as a cylinder is viewed endlessly, it is the group given a G with natural movements (eg, movement along an axis and rotation about an axis). Similarly, when viewed endlessly, the texture is also a group H given movement in the plane. It can be seen that there is a standard homomorphism between G and H. In other words, for example, in the x direction, the rotation of a cylinder about its axis is equal to the movement of the texture. For example, in the y direction, the movement of a cylinder along its axis is equal to the movement of a texture. This allows the renderer to pre-calculate all projection parameters and simulate viewport changes as texture movements. 10A and 10B illustrate examples of how to calculate projection parameters from screen space to texture space. As illustrated by FIGS. 10A and 10B. (1) If the point M on the screen 1020 has coordinates (x, y), it has coordinates (x, y, R) in space, where R is the radius of the cylinder 1010. (2) In this case

<maths num="1"><img file="JP2010531007A_D0001.tif" /></maths>And in the texture space, the point P has the following coordinates.

<maths num="2"><img file="JP2010531007A_D0002.tif" /></maths> Dynamic textures based on the latest zoom levels are generated and can be positioned in image space. This texture changes when the user changes the viewpoint (eg, by zooming or panning). This texture can be obtained by connecting tiles from the pyramids of the tile at the appropriate level and at the appropriate scale. If some tiles are missing, fall back onto the tile towards the level of the parent in the tile's pyramid. back) Get. Textures are modeled to be mapped onto a cylinder. The projection is done over the screen space. This non-linear projection can be piecewise approximated as an affine transformation. More precisely, the cylinder and screen space can be discretized using a triangular mesh. Each triangle can be drawn by linearly (or rather to affine) mapping one of the textures that cover it. This is well-defined as an affine transformation in a two-dimensional plane, as the action of the affine transformation on three points (hence the use of triangles) uniquely determines the affine transformation. The mesh can be uniform in screen space (and not in texture space). The screen mesh is always the same regardless of the zoom level. Different texture meshes can be used depending on the zoom level. For each triangle, the texture mesh corresponds to a unique triangle in the screen mesh and a unique (affine) transformation matrix. Such a matrix can be pre-calculated as the product of the screen matrix and the texture matrix (and vice versa).

When the user pans, all the renderer needs to do is adjust and / or refresh the texture. This is quick because it consists of a memory copy. Copying a large chunk of pixels is usually very optimized in various programming languages.

In one embodiment, zooming in consists of splitting both the horizontal vision field and the vertical vision field in two and using the following zoom levels to produce the texture. When the user zooms in / out, the panorama viewer may pre-cache some bitmap images to smooth the animation. As far as the projection itself is concerned, various sets of transformation matrices can be used for the essential zoom levels. A transformation matrix (still agile) can be linearly interpolated between the previous and next zoom levels for non-essential zoom levels.

Where pixels are assumed to be square, they correspond to uniform, continuous angles. The basic fields of vision for a given pixel are the same in the horizontal and vertical directions. This allows trade-offs to be made. For example, you may choose to keep the straight line tight, but this will result in relatively severe distortion at the sides. Alternatively, it may be decided that the straight line should be slightly curved, thereby reducing the amount of distortion. In one embodiment, the height of the original image is scaled to the height of the viewport. Because the pixels are square, the ratio of viewport width to viewport height determines the horizontal field of vision.

In the case of a sphere rather than a cylinder, the above assumptions no longer apply. Therefore, the above technique alone cannot simulate the original pan up / down, as the vision vector only moves along the axis of the cylinder in a state perpendicular to the axis of the cylinder. .. Nevertheless, the original pan motion up / down can be simulated by pre-computing a series of transformations and linearly interpolating between the transformations.

In one embodiment, the panorama viewer is set to handle non-flat panoramas. Not all panoramas depict flat and horizontal features, for example, consider many streets in San Francisco. For example, a camera mounted on a vehicle used to capture a panorama is parallel to the ground. Running on steep slopes in this way can result in photographs pointed in the wrong direction. Therefore, in such situations, bending the panorama can be advantageous to ensure that vertical buildings in the real world are vertical in textured space. Figure 11 illustrates an example of how the Panorama 1100 can be so bent. As illustrated by Figure 12, the example roughly follows a periodic function, which can be used to guide viewport placement and annotation generation in a way that takes into account the panoramic gradient. ..

As shown herein, the configuration information may include projection properties such as the highest gradient roll and pitch in the panorama. As illustrated by FIG. 12, the panorama viewer may use rolls and pitches in the steepest gradient direction to push the viewport into the sinusoidal strips of the bent panorama. The drawing of the annotation element in the viewport can also be modified to take into account the roll and pitch information of the panoramic gradient. The spatial orientation of lines / bars and arrows can be transformed based on roll and pitch information, or evaluated based on the relationship between the roll of the annotation and the roll of the steepest slope. obtain. FIG. 13 illustrates the result of such processing with respect to the setting information about the panoramic gradient. Panorama accurately places vertical buildings in a panoramic image of a steeply sloping street. Moreover, the lines / bars that draw the road (eg street line metadata) are tilted at an angle that roughly matches the slope of the street.

In one embodiment, the panoramic viewer can also facilitate user annotation to the panoramic image. User annotations to the panorama represent a challenge for how to reference annotations in three-dimensional space.

14 and 15 describe embodiments that address user annotations in three-dimensional space. The process described in FIG. 15 can occur in the panorama viewer (or mapping service), on the server, or in combination of the two.

Referring to FIG. 15, in step 1502, the user inputs an annotation for one panorama. The panorama viewer may receive user input in any of a number of different ways, including by instantly receiving a click event on the panorama that the user wants to annotate. The two-dimensional location of the annotation on the panorama is recorded in some advantageous coordinate systems (eg, by the location of the panoramic image, or by the roll and pitch coordinates).

In step 1504, the user navigates to another nearby panorama in the panorama viewer, finds the same feature to be annotated, and again annotates the second panorama. The panorama viewer or mapping service also offers the ability to add additional metadata related to annotations such as titles, links, graphics, etc., as you might expect.

In step 1506, the coordinates of the annotations on the two panoramas are used to generate the three-dimensional coordinates for the annotation. As illustrated in FIG. 14, drawn in 1450, assuming that the position of the camera that took the image with respect to the panorama 1410, panorama 1420 is known, and the user-entered annotation coordinates associated with the two-dimensional image are known. It is possible to calculate the intersection of the two. The result is three-dimensional coordinates for the annotation.

In step 1508, the three-dimensional coordinates for the annotation are assigned to the annotation and stored in the annotation's database. Annotations can be included in any panorama of the calculated coordinates within some favorable range, including panoramas that were not originally annotated by the user.

Alternatively, if the relevant pitch information is not particularly important to the annotation, it is possible to receive the user annotation as a one-dimensional roll direction on both panoramas, which is a two-dimensional to the annotation. Facilitate the allocation of land code (geocode) (with or without default pitch information).

Although various embodiments of the invention have been described above, it should be understood that they have been given for illustration and that they are not limited. It will be apparent to those skilled in the art that various modifications can be made therein without departing from the scope of the invention. In addition, the embodiments of the invention provided herein (excluding the summary and abstract sections) are intended to be used to interpret the claims. That should be recognized correctly. The summary and summary sections specify one or more exemplary embodiments (but not all) of the invention, as expected by the inventors.

The above description of a particular embodiment is sufficient to reveal the overall essence of the present invention, and the present invention (by applying the knowledge of those skilled in the art) will be as diverse as the particular embodiment. It can be effortlessly modified and / or adapted to an application without undue experimentation and without departing from the overall concept of the invention. Therefore, such adaptations and improvements are intended to be within the meaning and scope of the disclosed embodiments equivalents, based on the teachings and suggestions given herein. The terminology or terminology used herein is for explanatory purposes and is limited so that the terminology or terminology for detailing the present invention can be interpreted by those skilled in the art in terms of teaching and suggestion. It should be understood that it is not for the purpose.