Augmented reality service apparatus using location based data and method the same

Augmented reality service device and method using location-based information

A device and method for implementing an augmented reality service by combining a real camera image with local information provided through a location-based service.

Recently, due to the development of mobile communication, interest in Location Based Service (LBS) is increasing. The location-based service is a service that provides specific information according to a user's location to a user of a wireless Internet or a mobile terminal (eg, a smart phone). The location-based service can generate a wide variety of information in that it provides local information specific to the user, and its utilization potential is limitless. As some examples of location-based services, it is possible to inquire about local information of a nearby location or facility, such as a gas station or a restaurant, and obtain actual useful information, such as finding the location of the other party.

In particular, in the case of mobile terminals, which are currently widely used, since a camera is mounted to take an image and display an image through the screen, the terminal combines the local information provided through the location-based service with the image captured by the camera. As a method of projecting various information on a screen becomes possible, a service in the form of augmented reality (AR) can be implemented. Augmented reality (AR) is a complex virtual reality that fuses the real environment and the virtual environment by using a technology that shows a three-dimensional virtual object overlaid on the camera image that the user is actually looking at. In that augmented reality (AR) grafts the virtual world with the real world, it is possible to provide users with the best advantage of location-based services.

Conventionally, after tracking the current position of a camera based on a base station of a mobile communication network, an augmented reality (AR) service is implemented by combining virtual images in real time, or the current camera position and posture using SLAM (Simultaneous Localization And Mapping) After extracting the feature points (image patches) to find out, the augmented reality (AR) service was implemented by arranging 3D graphics or information at the desired location.

However, the location tracking method based on the base station has a limitation in accuracy because the coverage of the base station is large, and the feature point extraction method using SLAM requires simultaneous localization and mapping of the entire area. If you try to implement augmented reality (AR) targeting (for example, nationwide or worldwide), you will need a huge amount of information, and it will be impossible to store that much information in the terminal.

<p>By dividing the area through the base station and allowing the base station to manage the image patch data for the divided area, the amount of information to be handled in the user terminal is reduced, and both the base station-based location tracking method and the feature point search method using SLAM are used. Disclosed are an augmented reality service apparatus and method using location-based information capable of implementing an augmented reality (AR) service by utilizing the same.</p><p>To this end, the augmented reality service apparatus using location-based information according to an aspect of the present invention is an apparatus for implementing an augmented reality service by using location-based information in a user terminal communicating with a plurality of base stations, camera; It receives the feature point of an image from at least one base station among a plurality of base stations, matches the data of the feature point with the image data of the camera to obtain the position of the camera, and provides location-based information in the same direction as the actual image according to the location of the camera. control unit; and a display unit for implementing an augmented reality service in which an actual image and location-based information are combined under the control of the controller.</p><p>A plurality of base stations divide an area in which a user terminal is located in units of cells and manage feature point information and location-based information for the corresponding cell.</p><p>In addition, the augmented reality service apparatus using location-based information according to an aspect of the present invention further includes a communication unit that communicates with a plurality of base stations, and the control unit searches for a base station connectable through the communication unit among the plurality of base stations, and the searched It receives the location information of all base stations and confirms the location of the user terminal.</p><p>The control unit requests the corresponding base station to transmit image patch data corresponding to the feature point according to the confirmed location of the user terminal.</p><p>The control unit receives the image patch data corresponding to the feature point from the corresponding base station and performs a matching operation of one-to-one correspondence with the current image acquired through the camera.</p><p>When the key points are matched, the control unit tracks the key points and acquires the current position and posture information of the camera.</p><p>In addition, the augmented reality service method using location-based information according to an aspect of the present invention is a method of realizing an augmented reality service using location-based information in a user terminal communicating with a plurality of base stations, a real image through a camera to obtain; searching for a connectable base station among a plurality of base stations to check the location of the user terminal; requesting image feature point data from the corresponding base station according to the location of the user terminal; receiving the key point data from the corresponding base station and matching the key point data with the image data of the camera to obtain a position of the camera; The augmented reality service is implemented by combining location-based information in the same direction as the actual image according to the location of the camera.</p><p>To confirm the location of the user terminal, the location information is requested to all the discovered base stations, and when the location information is transmitted from all the found base stations, the location of the user terminal is confirmed using the location information and the transmission time.</p><p>In requesting the feature point data of the image, it is determined to which cell the user terminal belongs to a plurality of base stations and requests the feature point data of the image from the corresponding base station.</p><p>To obtain the position of the camera, the current position and posture information of the camera are obtained through a matching operation of receiving image patch data corresponding to the feature point from the corresponding base station and matching the current image obtained through the camera one-to-one.</p><p>To implement the augmented reality service, a three-dimensional augmented reality service is implemented by combining location-based information transmitted from a corresponding base station with an actual image obtained from a camera.</p>

<p>According to the disclosed augmented reality service apparatus and method using location-based information, by implementing an augmented reality (AR) service using location-based information by utilizing both a base station-based location tracking method and a feature point search method using SLAM, various It is considered that a location-based service (LBS) can be provided, thereby providing a new type of service. In addition, by providing location-based information directly to the terminal screen that the user is currently viewing, the augmented reality (AR) service can be implemented while moving in real time.</p>

1 is an overall configuration diagram of an augmented reality service system using location-based information according to an embodiment of the present invention. 2 is a control block diagram of an augmented reality service system using location-based information according to an embodiment of the present invention. 3 is a conceptual diagram of an image patch for extracting feature points in an augmented reality service system using location-based information according to an embodiment of the present invention. 4 is a diagram illustrating a basic unit cell allocated to a plurality of base stations applied to an augmented reality service system using location-based information according to an embodiment of the present invention. 5 is a diagram illustrating feature points distributed in an image in an augmented reality service device using location-based information according to an embodiment of the present invention. 6 is an operation flowchart illustrating a location recognition process in an augmented reality service method using location-based information according to an embodiment of the present invention. 7 is an operation flowchart illustrating a process of implementing augmented reality in the augmented reality service method using location-based information according to an embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is an overall configuration diagram of an augmented reality service system using location-based information according to an embodiment of the present invention.

In FIG. 1 , an augmented reality service system using location-based information according to an embodiment of the present invention includes a user terminal 100 that implements an augmented reality (AR) service using location-based information, and an augmented reality (AR) service A plurality of base stations 200 that provide location-based information to the user terminal 100 for implementing and a service operator 300 . The user terminal 100 and the plurality of base stations 200 are connected through a network such as an IP network, an Internet network, a mobile communication network, and a LAN to perform mutual communication.

2 is a control block diagram of an augmented reality service system using location-based information according to an embodiment of the present invention.

In FIG. 2 , a user terminal 100 is a device that implements an augmented reality service by combining real camera images with local information provided through a location-based service, and includes a communication unit 110 , an image acquisition unit 120 , and an input unit 130 . ), a control unit 140 , and a display unit 150 . The user terminal 100 uses a mobile terminal such as a smart phone or a personal digital assistance (PDA).

The communication unit 110 communicates with the plurality of base stations 200 through a network such as an IP network, an Internet network, a mobile communication network, or a LAN under the control of the controller 140 .

The image acquisition unit 120 is a camera that acquires a current image of an actual image by capturing an image of an environment that a user is actually viewing.

The input unit 130 sets an option desired by the user (eg, a location-based augmented reality service request).

The controller 140 requests location-based information from the plurality of base stations 200 according to the location-based augmented reality (AR) service request set through the input unit, and the location-based information of each base station 200 from the plurality of base stations 200 . By combining the received information and the location-based information (virtual image) transmitted to the real image acquired through the image acquisition unit 120 by receiving the information, an augmented reality (AR) type service is implemented.

To this end, the control unit 140 searches for connectable base stations 200 and access points (APs) through the communication unit 110 , and requests transmission of location information from all discovered base stations 200 . When location information is transmitted from all the discovered base stations 200 , the location of the user terminal 100 is confirmed from the location information of each base station 200 and the transmission time (information arrival time).

In addition, the controller 140 estimates the cell location from the confirmed location of the user terminal 100 to determine which cell the current location of the user terminal 100 belongs to, and sends the information to the base station 200 corresponding to the determined cell. Transmission of image patch data corresponding to cell number and feature point is requested. When the image patch feature points corresponding to the cell are transmitted from the corresponding base station 200 , a matching operation is performed in which the received feature points are matched one-to-one with the current image acquired through the image acquisition unit 120 . When the feature point is matched, the feature point is tracked through the Kalman filter to obtain the camera's current position and posture information (Localization).

When the current position and posture information of the camera are obtained, the controller 140 displays an image in which virtual information is combined with the camera position information on the display unit 150 to project virtual information in the form of 3D augmented reality (AR). will implement

The display unit 150 provides the actual image acquired through the image acquisition unit 120 and the location-based information arranged in the same direction under the control of the control unit 140 and displays it to the user through the terminal screen. That is, the display unit 150 provides the user with an augmented reality (AR) type service in which an actual image and location-based information are combined.

The plurality of base stations 200 are configured to always communicate with the user terminal 100 for calls and information transmission, and manage data in an area in which the user terminal 100 is located, that is, an area divided by itself. do.

The plurality of base stations 200 store the location information DB 201 for storing their own location information, and image patch feature point information of an area capable of communicating with the user terminal 100 , that is, an area managed by the corresponding base station 200 . and a location-based area information DB 203 that stores location-based area information of an area capable of communicating with the user terminal 100, that is, an area managed by the corresponding base station 200, respectively. .

The service operator 300 extracts image patch data (feature points) of the region where the user terminal 100 is located in advance and registers it with the plurality of base stations 200 . The service operator 300 extracts feature point information on cells allocated to each of the plurality of base stations 200 from an image obtained through a camera. In general, SLAM (Simultaneous Localization And Mapping) consists of a localization process to find one's own location and a mapping process to register an image. A mapping process by extracting feature points in advance As the service operator 300 pre-registers with the base station 200, the user terminal 100 uses a monocular SLAM technology that only performs a localization process to find its own location.

Feature points should be able to extract quantitative features that do not change with time and observation angle when a certain point is selected from the acquired image. In a normal image, a corner image or image patch is used to find feature points that do not change even when observed from any angle.

3 is a conceptual diagram of an image patch for extracting feature points in an augmented reality service system using location-based information according to an embodiment of the present invention.

In FIG. 3 , a feature point is composed of an image patch of about 15 pixels × 15 pixels or larger, and the feature point extracts data having a high eigenvalue from an image through a salience operator. This method extracts data with high edge characteristics from an image patch, and has a characteristic that is easy to distinguish compared to other images.

4 is a diagram illustrating a basic unit cell allocated to a plurality of base stations applied to an augmented reality service system using location-based information according to an embodiment of the present invention.

In FIG. 4 , a plurality of base stations 200 manage data by dividing an area in which the user terminal 100 is located in a cell form of 20 m × 20 m. The divided area is divided to overlap by about 5 m based on the side of the square, and the user terminal 100 transmits its location to the plurality of base stations 200 based on the basic unit cell.

In addition, the plurality of base stations 200 have feature point information on a cell assigned to them, and usually have 500 to 1000 feature point information per cell. The feature point information is pixel information and 3D data information about an image patch. For 3D data information, monocular localization is performed using a local coordinate system in which the southwest corner of the cell shown in FIG. 4 is (0,0,0). The user terminal 100 uses the base station 200 location information and the transmission time (information arrival time) or the base station 200 ID information obtained from each base station 200 to determine which cell the location of the user terminal 100 is. Find out if you belong When a request for information corresponding to the cell of the current location is received from the user terminal 100 , the characteristic point information of the cell is transmitted to the user terminal 100 . When a user moves and enters a common area where cells and cells overlap (that is, an area that overlaps by about 5 m based on the side of the square), the user terminal 100 connects and transmits the characteristic point information of the moved place and receives all of the common characteristic point information. will use If the user manages the feature point information (unique image patch) of the area in which the terminal 100 is located in the user terminal 100 in this way, only the data of the jurisdiction area can be transmitted to the user terminal 100 without considering the vast map area. Therefore, area information can be delivered quickly. In addition, in estimating the current position and posture through the image, the SLAM algorithm can be performed very quickly because a limited number of features can be compared even when the feature point information newly extracted from the image and the feature point on the database are matched one-to-one.

5 is a diagram illustrating feature points distributed in an image in an augmented reality service device using location-based information according to an embodiment of the present invention.

In FIG. 5 , when the image acquisition unit 120 of the user terminal 100 acquires the current image by capturing an image of the environment the user is actually viewing, the base station 200 that manages the area in which the user terminal 100 is located. The feature points () distributed in the image of the current image obtained from ? are transmitted to the user terminal 100 .

In this case, it is impossible to store all of the large amount of location information in the user terminal 100 , and finding one's own location in the large amount of data requires too much data search time. Therefore, it can be seen that the most efficient method in Monocular SLAM is to manage data in the form of cells of about 20 x 20 m by dividing them into regions through division. Once the location of the user terminal 100 is obtained in units of cells through the base station 200, and a feature point registered in the base station 200 is received in the area where the user terminal 100 is located, monocular localization is performed.

On the other hand, in the case of the base station 200 having a large jurisdiction, it is difficult to determine the accuracy of the positional accuracy in units of 20 m X 20 m. Therefore, 20 x 20m through the time of arrival method that calculates the location as the arrival time of the signal transmitted from three or more base stations 200 by receiving the time it takes to transmit data from three or four nearby base stations 200 to obtain the positional accuracy of

In addition, in the case of the base station 200 with a small jurisdictional area, the location accuracy can be increased up to a radius of 20 m if the wireless AP can be detected because a wireless network can be used anywhere because a lot of wi-fi APs are actually distributed. will be. In this case, the complicated method is excluded and 20m X 20m positional accuracy can be obtained only with the cell ID.

Hereinafter, the operation process and effect of the augmented reality service apparatus using the location-based information configured as described above and the method will be described.

6 is an operation flowchart illustrating a location recognition process in an augmented reality service method using location-based information according to an embodiment of the present invention.

In FIG. 6 , the control unit 140 of the user terminal 100 searches for a base station 200 and an access point (AP) connectable through the communication unit 110 ( 500 ), and provides location information to all the discovered base stations 200 . request the transmission of (502).

In response to the location information transmission request of the user terminal 100 , all the base stations 200 that have received the location information transmission request transmit the location information of the corresponding base station 200 to the user terminal 100 .

When the location information is transmitted from all the discovered base stations 200 ( 504 ), the control unit 140 checks the location of the user terminal 100 from the location information of each base station 200 and the transmission time (information arrival time) ( 506).

Thereafter, the control unit 140 estimates the cell location from the confirmed location of the user terminal 100 to determine which cell the current location of the user terminal 100 belongs to ( 508 ), and the base station ( 508 ) corresponding to the determined cell. 200) to request transmission of image patch data corresponding to the cell number and the feature point (510).

In response to the keypoint information transmission request of the user terminal 100 , the base station 200 , which has been requested to transmit the keypoint information, transmits the image patch keypoints for the cell allocated and stored thereto to the user terminal 100 .

When the key point information is transmitted from the corresponding base station 200 ( 512 ), the control unit 140 must make a one-to-one correspondence with the current image acquired through the image acquisition unit 120 and the key point database. Since it is impossible to know which pixel the pixel of the key point is on the screen, the key point having a similar eigenvalue is matched through an operator. Among them, feature points having similar eigenvalues are matched, and when several feature points are matched, the relationship between the feature points is matched using a geometric relationship.

Once the feature point is matched (516), the feature point is tracked through the Kalman filter (518) and an update is performed to obtain the current position and posture information of the camera (520). After the initial matching is performed, the camera movement model is used Since the user terminal 100 posture correction is performed by repeating the prediction process and the update process, a lot of time is not required and estimation can be performed stably. After matching, the feature point tracking proceeds through the extended Kalman filter.

When the current position and posture of the camera are acquired through the position recognition process of FIG. 6 , virtual information may be displayed on the image of the display unit 150 in addition to the position information. For example, by providing an image obtained through a camera and information arranged in the same direction thereto, an augmented reality (AR) type service can be implemented more visually. For example, when searching for a destination, the direction is also directly suggested by projecting an arrow on the floor. In addition, if you check where the desired item is at a mart or large store, it can provide information about the location on the spot.

7 is an operation flowchart illustrating a process of implementing augmented reality in the augmented reality service method using location-based information according to an embodiment of the present invention.

In FIG. 7 , a user sets a desired option, that is, a location-based augmented reality (AR) service request through the input unit 130 ( 600 ).

Accordingly, the controller 140 requests location-based information from the plurality of base stations 200 according to the location-based augmented reality (AR) service request set through the input unit 130 .

In response to the location-based information transmission request of the user terminal 100 , the base station 200 , which has been requested to transmit the location-based information, enhances the 3D virtual object or information object for the stored location-based information (region information). The real (AR) data and the coordinates (arrangement location) to be located are transmitted to the user terminal 100 at the same time.

When the 3D augmented reality (AR) data for location-based information (region information) and the placement location are transmitted from the base station 200 ( 602 ), the controller 140 controls the actual image acquired through the image acquisition unit 120 . By combining the location-based information (virtual image) transmitted to and projecting the virtualization information to the location of the deployment, an augmented reality (AR) type service is implemented ( 604 ).

On the other hand, since there may be small changes in the environment even in a certain place, when the registered feature points do not converge with a very high possibility, this feature point may be selected as a correction target. When the coincidence rate of the registered feature points is remarkably low, the user terminal 100 creates a candidate group with the error feature points from the base station 200, and when the error signal exceeds a certain level, the feature point is discarded and a new feature point is found and registered. do.

Through an embodiment of the present invention, it is possible to implement an augmented reality (AR) service using location-based information with base station 200-based Localization and Monocular SLAM, thereby providing various location-based services, thereby Therefore, it is considered that a new type of service can be provided. If you look at Google or the following example, you can see a case where huge profits could be created by providing a location-based service in the form of a map. The present invention enables such an augmented reality (AR) service to be implemented while moving in real time, Since location-based information (local information) can be provided directly to the screen being viewed, it seems that an incomparable level of effect can be created.

100 : user terminal 110 : communication unit 120: image acquisition unit 130: input unit 140: control unit 150: display unit 200: base station 300: service operator