Mobile device for augmented reality applications

The present invention relates to a mobile device for an augmented reality application as presented in the preamble of claim 1.

PCT application PCT / CH2006 / 000196 controls mobile devices, including cameras, for associating selected applications with external visual information and for initiating execution of selected applications based on visual background information and mobile device orientation. Disclose the module. Since the orientation of a mobile device with respect to the visual background is related to the way the user positions the mobile device, it can be considered as user-related input information. Prior art mobile devices further include output generators for generating visual output signals in response to said applications and for providing these visual output signals to the display of the mobile device.

Therefore, if the mobile device is, for example, a mobile phone, the prior art solution is to display an augmented reality picture with a real visual background on the display of the mobile phone, enhanced with some virtual objects resulting from that application. It is supposed to be done. This means that this augmented reality is only visible on this small screen of the mobile phone. For gaming applications, small screens can significantly limit the gaming experience. When a laptop computer is used as a mobile device, the display is already much larger, but even then, the user's more active, including some physical movement of the user, which is specific to the game itself, for example. For some applications where participation is more desirable, the user experience may be limited. This is done by users who want to participate more actively, for example, not only by manually controlling some control buttons on the mobile device itself, but also by performing a soccer ball kicking, stopping or welcoming movement. It can be applied to soccer games that you may think about.

<p><patcit num="1"><text>PCT application PCT / CH2006 / 000196</text></patcit></p>

<p> Therefore, an object of the present invention is to provide a mobile device of the above-mentioned known type, which can solve problems related to the prior art device.</p>

<p> To this end, the mobile device according to the invention further includes the features described in the feature portion of the first claim.</p><p> The projector device included in the mobile terminal will thereby be able to project some virtual visual objects onto a real visual background. In the case of the soccer game example described above, the virtual visual object may then be a virtual soccer ball that will be projected onto the floor on which the player is currently standing during play. With a projected ball on the floor that allows the user to kick against it, the player then has the more exciting gaming experience that was the object of the invention: much more active. It is possible to participate in various games and improve the quality of the human / device interface. The projector of the mobile device only needs to project the virtual visual object in the background, and therefore only needs to receive this virtual visual object information from the processing device. Therefore, the processing device is adapted to provide this type of signal to the projector. By projecting this virtual visual object onto the background, the external visual background information will, of course, change in response to the application itself.</p><p> Another unique feature of the present invention is set forth in claim 2. In this case, the user-related input information includes visual information such as an image of the foot of a soccer ball player included in the background information. This user-related information is detected by a motion detector built into the mobile device, and detects movement of the mobile device with respect to the background, for example by determining the orientation of the mobile device with respect to the visual background information. Motion information can also be provided.</p><p> Additional features of the present invention are set forth in claims 4-6. By distinguishing between real visual background information and virtual visual objects from the external visual information received from the visual input device (C) described above, and by continuously extracting metadata information from them, the mobile device. A generic platform is provided that allows you to plug into some of the above applications.</p><p> Claims 7 to 10 are made such that, in certain embodiments, the processing device generates a second visual output signal associated with the virtual visual object for provision to a display provided on the mobile device. Describe. In some of these embodiments, the virtual visual object will then also appear on the screen of the mobile device. In another embodiment, the processing device is further adapted to generate the second visual output signal described above, assuming that it comprises at least one other virtual visual object. This may also apply to the aforementioned soccer games, for example, where the screen of the mobile device can only show the player's score on the virtual scoreboard and the virtual soccer ball is only projected onto the floor. In yet another embodiment, the processing device provides the second visual output signal as comprising said virtual visual object that overlays at least one other virtual visual object generated by the application and / or visual background information. Further done to produce. In the soccer game example described above, the mobile device display then soccer, along with a virtual goal, a virtual scoreboard, and, in some cases, a floor on which the user is standing and a virtual soccer ball is projected. You can show the ball. Other combinations can also be envisioned.</p><p> By referring to the following description of the embodiments with the accompanying drawings, the above and other objects and features of the present invention will be further clarified and the present invention itself will be best understood.</p>

<figref num="1">It is a figure which shows typically one player U who plays a soccer game using the mobile device MD according to this invention.</figref><figref num="2">It is a figure which shows the possible detailed implementation of one Embodiment of the mobile device MD by this invention.</figref><figref num="3">It is a figure which shows an example of the metadata generated by the image analyzer and the application engine of the processing apparatus of FIG.</figref><figref num="4">It is a figure which shows schematic the multiplayer participation situation of two players using the mobile devices MD1 and MD2 by this invention.</figref><figref num="5">It is a figure schematically showing another application running on the mobile device according to this invention.</figref><figref num="6">FIG. 5 is a diagram schematically showing yet another application running on a mobile device according to the present invention.</figref>

The present invention relates to mobile devices for augmented reality applications. These mobile devices can be included in the following limited list, mobile computers such as mobile phones, personal digital assistants, laptop computers, tablet computers, gaming machines, and mobile devices such as GPS terminals. .. In general, any visual information can be collected, this visual information can be processed for use in an application, and a virtual object can be projected into the background as described in the following sections. Portable devices can be used to implement the present invention.

FIG. 1 shows an embodiment of such a mobile device MD operated by user U. The mobile device MD is equipped with a visual input device C such as a camera, a digital image system, or a CCD sensor, which can collect external visual information itself having real visual background information, which is abbreviated in VBI. .. This real visual background information is generally a virtual visual object in which the visual information is abbreviated in VVO. It is referred to as information coming from the "real" world, as opposed to the visual information generated by software applications, etc., which will be indicated by objects). In the situation shown in Figure 1, where User U is playing a game on his / her mobile device MD, this real visual background information may point towards the floor on which the user is standing. It has some visual information collected by a camera. The real visual background information VBI will then include an image of the floor, along with the leg L and / or foot F portion of the user U. This information on the user's foot F and leg L is user-related input information, so in this case real visual background information that will be used by the application running on the mobile device, as described in the next section. Is the part of.

Figure 1 also shows that the camera is not only collecting this "real" background information, but also the projected image of the soccer ball B. This projected soccer ball B is not a projected image of a "real" soccer ball, but is itself the result of an application such as a game, running on the processing device PD and inside the mobile device MD, therefore. It is a projected image of a virtual soccer ball, not shown in FIG. In the illustrated example, the game is therefore projected, consisting of real visual background information VBI, which also comprises external visual information, in this case user-related information, as well as a projected soccer ball in this case. A soccer game that uses both virtual object information as input. The position of the user's foot with respect to the projected ball B will update the next state and output as produced by the game. In the soccer game example, this is only displayed on the screen S of the mobile device in the embodiment shown in FIG. 1, whether the user has scored or kicked the ball out of the virtual goal. It can mean that the game provides the following states, which may indicate either: This virtual goal is itself another virtual visual object, as it also does not correspond to a "real" physical goal. Therefore, the processing device continuously updates the game parameters, calculates the new position of the ball in the game, and transfers the updated visual information to the projection device P, which itself depends on the collected external visual information. It is necessary to send back so that the latter can project this virtual ball to the updated position. This updated projection position also depends, in this case, on the relative position of the mobile device with respect to the real visual background. This relative position is also related to user-related input information. Such location information can be detected by a motion detector built into the mobile device, but otherwise this motion information is also performed on external visual information by comparing previously collected images with each other. It can also be extracted by various imaging techniques. This is further explained in the next section.

The projection device P incorporated in the mobile device MD may be a mini-projector currently known in the art. In any case, the updated picture or image of the virtual ball will be projected to the updated position on the floor.

In one embodiment of the invention, only projector device P displays one or more virtual visual objects. In another embodiment, the virtual visual object is not only projected by the projector device P, but also displayed on the screen S of the mobile device. In yet another embodiment, the screen S will display some virtual visual objects, as shown in the example shown in FIG. In this embodiment, a virtual ball, as well as a virtual goal, as well as a player-generated and thus virtual, image are displayed on the screen. In yet another embodiment, information from real visual backgrounds such as legs and feet collected by the camera, as well as one or several virtual visual objects, will also be displayed on the screen.

FIG. 2 shows a more detailed view of possible embodiments of the processing device PD incorporated in the mobile device MD. Many other embodiments of the processor PD are possible, and this FIG. 2 will only provide a schematic of some of the key blocks and processing steps that will be performed in this particular embodiment. Please note. Such processor PDs can be implemented using dedicated processors, with embedded software on general purpose processors, or via ASICs, as is well known to those skilled in the art. Therefore, the different blocks are for illustration purposes only. Other embodiments are also possible in which only a single processor performs all steps.

Figure 2 also shows a very schematic representation of the different building blocks of a mobile device: a visual input device C, shown as a camera in Figure 2, a projector device, shown as a "traditional" projector P, in Figure 2. Display S shown as the screen of a mobile phone in 2. The mobile device MD processor PD embodiment shown in FIG. 2 further includes several functional blocks: reality abstractor 2, image analyzer 3, application engine 4, virtual image animator 5. , Dual output generator 6 and virtual image buffer 9. This dual output generator exists only in these embodiments, one producing dual output to the projector P and the other to the screen S. One embodiment of the mobile device incorporates a single output generator designed to only generate a single image that will be transmitted to the projector P.

The reality abstractor, block 2, is a module designed to divide the input external visual information provided by the camera C indicated by the "image" into the real visual background information VBI and the virtual visual object VVO. The notation VVO (n) is used for previously generated and previously projected to distinguish between previously generated virtual visual objects by the application engine and those that will be generated in the current cycle of the application. And VVO (n + 1) will be used for the updated ones that will be generated and projected in the current period. In some embodiments, no virtual visual object information is available at the start. In that case, the input image is completely composed of real visual background information. In order to split the input external visual information, the reality abstractor needs to know what the virtual object or the projected image of the object will look like. Apart from the initial input image, which cannot initially contain any virtual image, this projected image P (n) during all other image captions was created by the dual output generator in the previous cycle. , Temporarily saved, or buffered in virtual image buffer 9. After obtaining the stored projected image from that buffer, the reality abstractor attempts to find the same pattern in the camera's image. This can be done via image recognition techniques, as is known to those of skill in the art. When finding these patterns, the reality abstractor can also summarize some information about how the mobile device is positioned in relation to the background (via scaling the projected image), but this Is optional. When it recognizes the pattern and in some cases derives scaling information, it subtracts the entire virtual image from the entire image, thereby deriving the real visual background information, and the virtual projection input image as collected by the camera. VVO (n) and when this is already relevant to new and actual information , Real background information VBI (n + 1) and both are transmitted to the image analyzer 3 as separate information. At the start, the virtual image buffer is generally empty, so the input image corresponds to the real visual background information, so the VVO signal does not contain any relevant information at that moment.

The next module in the processing apparatus embodiment, the image analyzer 3, receives the collected virtual visual object information VVO of the input image and the collected real visual information VBI as provided by the reality abstractor 2. Will be done. The main function of the image analyzer is to extract and / or generate metadata from these images. For this purpose, some iterations may be needed, such as to improve the results. This is shown in Figure 2 using the feedback arrow of the image analyzer. Techniques for extracting metadata are known to those of skill in the art and may include pattern recognition, digital geometry and / or signal processing.

In this context, the metadata to be understood herein is a parameter with some semantic meaning. An example of a soccer game is: Object = Ball, Background = Floor ..., shown in Figure 3. Therefore, in order to generate such metadata, the image analyzer can also be made to analyze the background changes and derive from them the specific behavior that occurred in front of the camera. These movements may be the movements of the user's legs / feet in the case of soccer games. Alternatively, this motion information can be provided by a motion detector. The Image Analyzer 3 is then further adapted to generate such metadata in a format understood by the application that can run on this processing device, such as the XML format shown in FIG. Therefore, the metadata output shown by the thick arrow and M (n) in Figure 2 and generated by the image analyzer is valuable to the application, for example, information indicating that the foot has moved 7 degrees towards the ball. And contains comprehensible information.

The metadata M (n) generated by the image analyzer is then provided to the application engine 4. This application engine can run several applications. Such applications generally include pre-compiled software, which may be written by a third-party developer in some cases. This application is designed to understand the format of the input information coming from the image analyzer. Alternatively, some conversion module can be provided to convert certain input formats into formats that are understandable by the application.

In the soccer game example described above, therefore, this soccer game corresponds to an application. Some other applications are listed in the sections below. Since these applications are generally widespread and known to those of skill in the art, their internal behavior is not further discussed herein.

The output produced by the application typically contains updated metadata M (n + 1) with information related to the updated virtual visual object that eventually needs to be projected onto the real background. Figure 3 shows an example of metadata input to an application and metadata generated by the application.

In the embodiment shown in FIG. 2, this updated metadata M (n + 1) as generated and provided by the application engine 4 is first provided to the next module, which is the virtual image animator 5. The engine.

This virtual image animator 5 is adapted to receive the updated metadata information signal M (n + 1) as provided by the application engine and corresponds to the application's actual state or period. It is further made to generate a adapted virtual visual object or object VVO (n + 1) from which it will be displayed according to what is determined by the engine. In the case of the soccer game described above, only one virtual object is projected. However, in many other applications, including soccer games, some virtual visual objects, such as virtual balls and virtual goals, can be projected. In this embodiment, which comprises an image in the shape of a ball, the virtual object that will be projected onto the floor is then further passed to a dual or single output generator, depending on the particular embodiment of the mobile device. .. However, in the embodiment shown in FIG. 2, although the projector projects a virtual soccer ball image in the background, the display of the mobile device itself also displays another virtual visual object, a virtual scoreboard in this embodiment. Indicates the mobile device MD. For this purpose, an additional virtual visual object VVOS (n + 1) will be generated by the virtual image animator, and in this embodiment this additional virtual visual object is therefore in the form of a scoreboard. A dual output generator is included in the embodiment of FIG. 2 because the image that will be displayed on the screen is different from the one that will be projected. Because even if the same virtual visual object is both shown on the screen and projected into the background, two separate images need to be generated: one to be projected and one to be projected. The dual output generator will also be included in the processing device to be displayed on the screen of the mobile device.

Thus, the dual output generator 6 is responsible for generating output signals both for the screen or display S of the mobile device MD itself, and for the projector or projection device P, on the other hand. The image generated by the virtual image generator simply contains basic shape information such as the basic shape of the ball, the output generator can adjust this image to the correct size and the projector can project it correctly onto the background. It is responsible for positioning it in its correct position so that it can be done. This is because, in this embodiment, the dual output generator receives the latest virtual objects VVO (n + 1) and VVOS (n + 1) and the latest visual background information VBI (n + 1) on the display. They must be further processed to provide a first output signal S (n + 1) to be transmitted and a second output signal P (n + 1) to provide to the projector P. Means that. The illustrated embodiment S (n + 1) is primarily associated with VVOS (n + 1) and P (n + 1) is primarily associated with VVO (n + 1). However, in other embodiments, the output signal that will be transmitted to the display is VVO (n + 1) and other virtual as a virtual scoreboard that is not projected onto the real background and is only shown on the display. It can consist of objects. Alternatively, in addition to some virtual soccer ball stadium, virtual goals and virtual scoreboards, some information about the actual background, such as the user's feet or legs, can also be shown on the screen. As is known to those skilled in the art, a number of other possibilities can be envisioned, primarily depending on the application itself.

This dual output generator 6 is always up to date directly from the reality abstractor 2 to generate the output P (n + 1) that will be sent to the projector P for the projection of the virtual object onto the background itself. Get the real background VBI (n + 1) of. This means that the background input is near real time. For the projected augmented reality virtual object image P (n + 1), the dual output generator needs to make sure that the correct virtual object is projected in the correct size and in the correct location on the real background. It can do this by comparing the latest background with the previous background. The previous view can be stored in the internal buffer of the dual output generator and is not shown in the drawing. Since the device must ensure that the coordinates and size of the projected image are accurate, this comparison step can be performed with the calculation of the motion performed by the camera. This is because, for example, due to the possible movement of the device, its size and coordinates must be constantly adjusted so that the projected virtual image is not too large when the mobile device stays closer to the real object. This is because it does not (note the projection start angle). Alternatively, this motion information can be obtained from an image analyzer or from a separate motion or orientation detector.

Of course, alternative embodiments are possible. Instead of feeding back the updated virtual object image to the buffer, the generated metadata can also be fed back to the alternate buffer. In that case, the reality abstractor then incorporates some of the features of the virtual image animator and dual output generator, such as to generate the latest projected virtual objects for further comparison with the input image. There is a need. Still other embodiments are possible, as known to those of skill in the art.

Figure 4 shows two users U1 and U2 now playing this soccer game on their respective mobile devices MD1 and MD2, and therefore in this embodiment the display of their respective mobile terminals. Shows the situation of a multiplayer experience trying to kick a virtual projected ball to the goal. Alternatively, this virtual goal can also be projected onto the floor. In either case, the mobile device needs to be further equipped with communication tools that enable multiplayer applications, as is well known to those skilled in the art.

Apart from the soccer game example described above, other applications may use the method and embodiments. Among these, there is an application in which a projector can project footprints in the background to indicate, for example, the direction in which he / she must go when a person tries to find a train station. In this case, the method does not improve the gaming experience, but is used to improve and clarify the human-computer interface. Mobile devices can then guide people more intuitively. This is schematically shown in FIG.

Another application involves highlighting only items on a user's shopping list in a store shelf with a mobile phone projection device. This is schematically shown in FIG.

Yet another example could consist of projecting a virtual keyboard on a table that the user can then "hit" or "press" to form text on the screen. The keyboard also relates to a piano keyboard that the user can "play" to produce the appropriate associated sound.

Yet another example consists of projecting a virtual table tennis ball onto a playing field, such as a table. By tilting a mobile device, the user can "hit" a virtual ball with the mobile device. In this case, the input information includes movement and orientation information of a mobile device such as a mobile phone with respect to a fixed background and a projected ball. This table tennis application can also be realized as a multiplayer game, of course.

Although the principles of the invention have been described above for a particular device, it is clear that this description is made by way of example only and is not a limitation to the scope of the invention as defined in the appended claims. I want you to understand. Within the scope of the claims herein, any element described as a means of performing a particular function is intended to include any method of performing that function. This includes, for example, a) a combination of electrical or mechanical elements that perform the function, or b) firmware, microcode combined with the appropriate circuitry to execute the software to perform the function. It may include any form of software, including as a result, as well as mechanical elements, if any, coupled with software control circuitry. The present invention as defined by such claims resides in the fact that the functions provided by various enumerated means are combined and combined in the manner required by the claims. Unless otherwise defined as such, any physical structure is of little importance to the novelty of the claimed invention. Therefore, the applicant considers any means capable of providing those functions to be equivalent to those set forth herein.