Game program and game device

The present invention relates to a game program and a game device capable of advancing a game based on image data obtained by imaging a user.

Conventionally, a game program that images a user's face and advances the game based on the image data obtained by imaging the user's face has been known (see, for example, Non-Patent Document 1). In this game program, a user is imaged at a timing when a predetermined time has elapsed from the start of the game, and the image of the captured user is displayed together with the game result at the end of the game.<nplcit num="1"><text>"Adult DS Face Training" Instruction Manual, Nintendo Co., Ltd., released on August 2, 2007</text></nplcit>

<p> In the above-mentioned prior art, the user is imaged at a timing when a predetermined time has elapsed from the start of the game, and the facial expression of the user at that timing is only displayed at the end of the game.</p><p> The present invention is to provide a new game program and a game device that advances a game based on image data obtained by imaging a user and displays a facial expression of the user who is playing the game.</p>

<p> According to the first aspect of the present invention, the imaging means (23: reference code corresponding to the embodiment; the same shall apply hereinafter) and the display means (12, 22) are executed on the computer of the available game device (100). Provides a game program to be played. Here, the imaging means is adapted to image the user. The game program captures a computer with a game progressing means (110,120,130,132) that advances the game based on a user image included in image data acquired by the imaging means, at least when the user image satisfies predetermined conditions. It functions as a display control means (136) for displaying an image based on image data acquired by the means on the display means.</p><p> According to this first aspect, it is possible to acquire image data showing the appearance of the user at a certain timing when the user is playing the game provided by the game program. That is, the user who is playing the game can be imaged at the timing when the user image included in the image data obtained by imaging the user satisfies a predetermined condition. Then, by displaying these image data after the game is over, the user who is playing the game can enjoy seeing an unexpected or interesting facial expression. The user image means at least an image including an area in which the user is captured as a subject in the image data acquired by the imaging means.</p><p> According to the preferred second aspect, the display control means acquires image data by the imaging means when the game by the game progress means is not in progress and at least the user image satisfies a predetermined condition. The image based on is displayed on the display means.</p><p> According to this second aspect, while the user is not playing the game, he / she can enjoy seeing an unexpected or interesting facial expression of the user who is playing the game.</p><p> According to the preferred third aspect, the display control means uses the image based on the image data acquired by the imaging means as the display means at least when the user image satisfies a predetermined condition after the end of the game. Display it.</p><p> According to this third aspect, after the game is over, the user who is playing the game can enjoy seeing an unexpected or interesting facial expression.</p><p> According to the preferred fourth aspect, the game progress means determine whether the game is successful or not based on the user image, and the display control means is at least the imaging means when it is determined that the game is successful. The display means displays an image based on the image data acquired by.</p><p> According to this fourth aspect, it is possible to capture the facial expression of the user around the timing when the game is successful. Therefore, it is possible to display the facial expression of the user when the game is successful.</p><p> According to the preferred fifth aspect, the game progress means determines whether the game is successful or not based on the position of the user image and the position of the object displayed according to the game effect.</p><p> According to this fifth aspect, it is possible to capture the facial expression of the user around the timing when the game is successful. Therefore, after the game is over, the facial expression of the user when the game is successful can be displayed.</p><p> Further, according to the fifth aspect, when the success condition associated with the position of the object displayed according to the game effect is satisfied, the position of the user's face, hand, etc. according to the success condition is satisfied. Since it is considered that the image data is almost certainly present, it is possible to acquire image data having almost the same composition regardless of which user plays the game. As a result, when an image object or the like prepared in advance is superimposed on the image data and displayed, the content intended by the developer can be expressed more reliably.</p><p> According to a more preferred sixth aspect, the game program further functions the computer as a user image as a detection means (126) for detecting a user area indicating a user area in the image data. The game progress means determines whether or not the game is successful based on the position of the object displayed according to the game effect and the position of the user area detected by the detection means.</p><p> According to this sixth aspect, since the area in which the user is captured can be accurately identified, erroneous detection by a subject other than the user can be prevented.</p><p> According to a more preferred seventh aspect, the game program further functions the computer as a user image as a detection means for detecting a user area indicating a user area in the image data, and the game progress means for game production. Whether or not the game is successful is determined based on the position of the object displayed accordingly and the position of the user area detected by the detection means.</p><p> According to a more preferable eighth aspect, the recognition result reflecting means displays the user area in a manner different from the image captured by the imaging means.</p><p> According to this eighth aspect, while the user is playing the game, the facial expression of the user acquired by the imaging means is not displayed, so that the user plays the game without being aware that the image is being captured. be able to. As a result, it is possible to acquire image data that captures the natural facial expression of the user who is playing the game.</p><p> According to a more preferable ninth aspect, the detecting means includes the skin color area detecting means for detecting the skin color area in the image data, and the game progressing means includes the position of the object displayed according to the game effect and the skin color area. Whether or not the game is successful is determined based on the position of the skin color region detected by the detection means.</p><p> According to this ninth aspect, since it is possible to more reliably determine that the imaging means is imaging the skin of the user, it is possible to prevent erroneous detection by a subject other than the user.</p><p> According to a more preferable tenth aspect, the skin color region detecting means sequentially detects the skin color region on the image data acquired by the imaging means at a predetermined cycle, and the game program uses the computer to perform the skin color while the game is in progress. It further functions as a recognition result reflecting means for displaying the skin color area detected by the area detecting means.</p><p> According to this tenth aspect, the user can play the game while confirming the skin color region detected with respect to the image data obtained by imaging the user.</p><p> According to the eleventh aspect, which is more preferable, the recognition result reflecting means displays the skin color region in a manner different from the image captured by the imaging means.</p><p> According to this eleventh aspect, while the user is playing the game, the facial expression of the user acquired by the imaging means is not displayed, so that the user plays the game without being aware that the image is being captured. be able to. As a result, it is possible to acquire image data that captures the natural facial expression of the user who is playing the game.</p><p> According to a preferred twelfth aspect, the game program further causes the computer to function as a user image as a face area detecting means (122) for detecting a human face area in the image data. The game progress means determines whether or not the game is successful based on the position of the object displayed according to the game effect and the position of the human face area detected by the face area detecting means.</p><p> According to this twelfth aspect, since the human face region is specified, it can be more reliably determined that the imaging means is imaging the user's face. This makes it possible to prevent erroneous detection by a subject other than the user.</p><p> According to the preferred thirteenth aspect, the game program causes the computer to further function as a presence detection means for detecting the presence of the user, and the game progress means when the presence of the user is detected by the presence detection means. , Determine if the game was successful.</p><p> According to this thirteenth aspect, when the user is not present in front of the game device, it is possible to prevent the game from progressing due to an erroneous determination.</p><p> According to the preferred fourteenth aspect, from the start of the game to the end of the game, the display control means obtains image data each time a plurality of different conditions are satisfied, including a condition for determining that the game is successful. Each is stored, and the display control means switches and displays a plurality of image data captured by the imaging means in a predetermined order.</p><p> According to this 14th aspect, not only the facial expressions of the user in the vicinity where the game is successful but also the facial expressions of a series of users who are playing the game can be appropriately imaged. Therefore, after the game is finished, a series of facial expressions of the user who is playing the game can be displayed like a moving image, and the user can enjoy seeing unexpected or interesting facial expressions.</p><p> According to a preferred fifteenth aspect, the display control means superimposes and displays a predetermined object on the image data captured by the image pickup means.</p><p> According to this fifteenth aspect, after the end of the game, an object having some meaning is superimposed on the image data and displayed, so that the facial expression of the user is used instead of the facial expression of the user who is playing the game. You can enjoy it with various expressions.</p><p> According to a more preferable 16th aspect, the display control means superimposes and displays an object corresponding to the game effect when the image data is stored on the image data.</p><p> According to this 16th aspect, after the game is finished, the game can be enjoyed with an image that reproduces the game effect when the image data is stored.</p><p> Alternatively, according to a more preferable 17th aspect, the display control means superimposes and displays an object independent of the game effect when the image data is stored on the image data.</p><p> According to this 17th aspect, the facial expression of the user who is playing the game can be used to give an expression having a completely different meaning.</p><p> According to a more preferred eighteenth aspect, the game program is as an object display means that causes the display means to display an object while the game is in progress and an image is displayed based on image data captured by the imaging means. The object display means replaces the object with another object at the position of the object displayed during the progress of the game, and displays the object during the display of the image based on the image data acquired by the imaging means.</p><p> According to this eighteenth aspect, an object different from the object displayed during the game is given to the object that was meant during the game by displaying it together with the image data acquired during the game. It is possible to display an image having a meaning different from the meaning of the object.</p><p> According to a more preferred nineteenth aspect, the game program serves as an object display means that causes the display means to display an object while the game is in progress and an image is displayed based on image data captured by the imaging means. To function, the object display means displays an object that is not related to the object displayed while the game is in progress while displaying an image based on the image data captured by the imaging means.</p><p> According to this nineteenth aspect, an object that is not related to the object displayed during the game is given to the object that was meant during the game by displaying it together with the image data acquired during the game. It is possible to display an image having a different meaning that is not related to the meaning of the object.</p><p> According to the more preferable twentieth aspect, the game progressing means advances the game by judging the success of the game based on the success conditions associated with each stage in a plurality of predetermined stages, and the game progresses. The program further causes the computer to function as an object display means that switches the objects associated with the success conditions corresponding to each stage and displays them on the display means as the game progresses.</p><p> According to this twentieth aspect, a plurality of objects having various meanings can be displayed at any time according to the progress of the game.</p><p> According to the more preferable 21st aspect, the game progress means progresses and displays the game by judging the success of the game based on the success conditions associated with each stage in a plurality of predetermined stages. The control means includes a plurality of image data acquired in each of the plurality of stages during the game and captured by the imaging means, and an object associated with the success condition associated with each stage in which the image data was acquired. , Is displayed as a composite image.</p><p> According to the 21st aspect, a plurality of image data of a user acquired during the play of a series of games are expressed by adding an object according to the situation in which each image is acquired. Can be done.</p><p> According to the more preferable 22nd aspect, the game progress means progresses and displays the game by judging the success of the game based on the success conditions associated with each stage in a plurality of predetermined stages. The control means continuously displays a plurality of image data acquired and stored in the storage means at each of the plurality of stages during the progress of the game.</p><p> According to this 22nd aspect, a plurality of image data of the user's appearance acquired during the play of a series of games can be displayed like a moving image, and the user's unexpected or interesting facial expression can be displayed. You can enjoy watching such things.</p><p> According to the 23rd aspect of the present invention, a game apparatus (100) including an imaging unit (23) adapted to image a user, a display unit (12,22), and an arithmetic processing unit (31). I will provide a. The arithmetic processing unit converts the progress of the game based on the user image included in the image data acquired by the imaging unit, and at least the image data acquired by the imaging unit when the user image satisfies a predetermined condition. It is possible to display the image based on the display on the display unit.</p><p> According to the 23rd aspect, it is possible to acquire image data showing the appearance of the user at a certain timing when the user is playing the game provided by the game device. That is, the user who is playing the game can be imaged at the timing when the result of the human body recognition satisfies the predetermined condition with respect to the image data obtained by imaging the user. Then, by displaying these image data after the game is completed, it is possible to enjoy seeing an unexpected or interesting facial expression of the user who is playing the game.</p><p> In the above description, reference numerals and supplementary explanations for showing the correspondence with the embodiments described later have been added to help the understanding of the present invention, but these are not intended to limit the present invention in any way.</p>

<p> According to the present invention, the user can advance and enjoy the game based on the image data obtained by imaging himself / herself, and can also enjoy watching his / her facial expression while playing the game.</p>

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated.

As a typical example of the game device (computer) according to the present invention, the portable game device 100 will be described below. The game device 100 can use an imaging means (or an imaging unit) and a display means (or a display unit). Further, as an example of the game program according to the present invention, a game program executed by the game device 100, which is a computer, will be described. The computer according to the present invention is not limited to the game device, and may be realized as a personal computer capable of executing various applications. Further, the game program according to the present invention may be incorporated as a part of a function of various applications executed on a personal computer.

&lt;Terminology&gt; As used herein, "available" means that devices such as imaging means (or imaging units) and display means (or display units) are connected to a computer (typically a gaming device 100) by wire or wirelessly. It means that data communication is possible. At this time, the devices such as the image pickup means (or the image pickup unit) and the display means (or the display unit) may be integrally configured with the game device 100 or may be separate bodies.

&lt;Appearance&gt; 1 and 2 are external views of a game device 100 according to an embodiment of the present invention.

The game device 100 according to the present embodiment with reference to FIGS. 1 and 2 is a foldable portable game device. FIG. 1 shows a game device 100 in an open state (open state), and FIG. 2 shows a game device 100 in a closed state (closed state). The game device 100 is sized so that the user can hold it with both hands or one hand even when it is open.

The game device 100 has a first housing 11 and a second housing 21. The first housing 11 and the second housing 21 are connected so as to be openable and closable (foldable). In the example shown in FIG. 1, the first housing 11 and the second housing 21 are each formed in a rectangular plate shape, and are rotatably connected by hinges at long side portions of each other.

Normally, when the user uses the game device 100, the game device 100 is left in the open state. On the other hand, when the user does not use the game device 100, the game device 100 is kept in the closed state. In the game device 100, the angle formed by the first housing 11 and the second housing 21 can be maintained between the closed position (approximately 0 degrees) and the open position (approximately 180 degrees), if necessary. That is, the first housing 11 can be rested at an arbitrary angle with respect to the second housing 21. At this time, the frictional force generated at the connecting portion between the first housing 11 and the second housing 21 can be used. In addition to or in place of the frictional force, a latch mechanism may be adopted at the connecting portion between the first housing 11 and the second housing 21.

The first housing 11 is provided with a first LCD (Liquid Crystal Display) 12 as a display unit (display means). The first LCD12 has a rectangular shape, and is arranged so that its long side direction coincides with the long side direction of the first housing 11. In the present embodiment, a case where an LCD is adopted as a display unit (display means) is shown, but another appropriate display device such as a display device using EL (Electro Luminescence) is adopted. May be good. Further, the resolution of the display unit (display means) can be appropriately designed according to the application to be executed and the like.

The first housing 11 is provided with buttons 14A to 14K as input units (input means) for performing various operations on the game device 100. Of the buttons 14A to 14K, the direction input button 14A, operation button 14B, operation button 14C, operation button 14D, operation button 14E, power button 14F, start button 14G, and select button 14H are the first housing 11 and the second housing. It is provided on the inner main surface of the first housing 11, which is inside when the 21 is folded.

That is, in the arrangement example shown in FIG. 1, the direction input button 14A and the power button 14F are on one of the left and right sides (on the left side of the paper in FIG. 1) with respect to the first LCD12 provided near the center of the inner main surface of the first housing 11. It is provided on the main surface. The buttons 14B to 14E, the start button 14G, and the select button 14H are provided on the inner main surface of the first housing 11 which is on the left and right opposite sides of the first LCD12 (on the right side of the paper in FIG. 1).

The L button 14I is provided at the left end of the paper surface of the first housing 11, and the R button 14J is provided at the right end of the paper surface of the first housing 11. The volume button 14K is provided on the left side surface of the first housing 11.

The direction input buttons 14A, L button 14I, and R button 14J are used, for example, for selection operations and the like. Buttons 14B to 14E are used, for example, for a decision operation and a cancel operation. The power button 14F is used to turn on / off the power of the game device 100. The volume button 14K is used to adjust the volume of the speaker mounted on the game device 100.

The game device 100 further has a touch panel 13 as an input unit (input means) separate from the buttons 14A to 14K. The touch panel 13 is mounted so as to cover the screen of the first LCD12, and detects the coordinates when an input operation is performed by the user. That is, the touch panel 13 is arranged in association with the display surface of the first LCD12.

As the touch panel 13, for example, a resistance film type touch panel can be adopted, but the touch panel 13 is not limited to the resistance film type, and various pressing type touch panels can be adopted. Further, the resolution (detection accuracy) of the touch panel 13 is preferably about the same as the resolution (display accuracy) of the first LCD12. However, the resolution of the touch panel 13 and the resolution of the first LCD 12 do not necessarily have to match.

An insertion slot for the touch pen 27 (indicated by a broken line in FIG. 1) is provided on the right side surface of the first housing 11 on the paper surface. A touch pen 27 used for performing an input operation on the touch panel 13 can be stored in this insertion slot. The input operation to the touch panel 13 is usually performed using the touch pen 27, but instead of the touch pen 27, the input operation to the touch panel 13 can be performed with a user's finger or the like.

An insertion slot (indicated by a chain double-dashed line in FIG. 1) for accommodating the memory card 28 is further provided on the side surface of the first housing 11 on the right side of the paper surface. Inside the insertion slot, a connector (not shown) for electrically connecting the game device 100 and the memory card 28 is provided. The memory card 28 is detachably attached to this connector. The memory card 28 is used for reading game programs and image data acquired from other information processing devices and game devices, and for storing (storing) image data shot and / or image-processed by the game device 100. .. The memory card 28 is made of a non-volatile storage medium such as an SD (Secure Digital) card.

An insertion slot (indicated by a alternate long and short dash line in FIG. 1) for accommodating the memory card 29 is provided on the side surface of the first housing 11 on the upper side of the paper surface. Inside the insertion slot, a connector (not shown) for electrically connecting the game device 100 and the memory card 29 is provided. The memory card 29 is detachably attached to this connector. The memory card 29 stores a game program or the like.

Three LEDs 15A to 15C are attached to the left side of the paper surface of the connection between the first housing 11 and the second housing 21. As will be described later, the game device 100 can perform wireless communication with other devices, and the first LED 15A lights up when the power of the game device 100 is on. The second LED 15B lights up according to the state of the battery of the game device 100 (charging, low remaining capacity, etc.). The third LED 15C lights up according to the state of wireless communication. Therefore, the three LEDs 15A to 15C can notify the user of the power on / off status, battery status, and wireless communication status of the game device 100.

The second housing 21 is provided with a second LCD 22 as a display unit (display means). The second LCD 22 has a rectangular shape, and is arranged so that the long side direction thereof coincides with the long side direction of the second housing 21. As with the first LCD12, another suitable display device may be used instead of the LCD. The game device 100 employs a configuration in which a touch panel as an input means (input unit) is mounted so as to cover the screen of the first LCD12, but another touch panel is mounted on the screen of the second LCD22. May be good.

The second housing 21 is further provided with two cameras (inner camera 23 and outer camera 25) as imaging means (imaging apparatus). As shown in FIG. 1, the inner camera 23 is mounted on the inner main surface near the connection of the second housing 21. On the other hand, as shown in FIG. 2, the outer camera 25 has a surface opposite to the inner main surface to which the inner camera 23 is attached, that is, the outer main surface of the second housing 21 (the game device 100 is in the closed state). It is attached to the outer surface in some cases). In FIG. 1, the outer camera 25 is shown by a broken line.

With such an arrangement position, the inner camera 23 can image the direction in which the inner main surface of the second housing 21 faces. That is, the inner camera 23 is adapted to image the user himself who holds and plays the game device 100. On the other hand, the outer camera 25 can take an image in the direction opposite to the image taking direction of the inner camera 23, that is, the direction in which the outer main surface of the second housing 21 faces.

A microphone (microphone 43 shown in FIG. 3) is housed as a voice input device inside the connecting portion of the game device 100. Then, on the inner main surface near the connecting portion of the game device 100, a microphone hole 16 is provided for the microphone 43 to detect the sound around the game device 100. The position for accommodating the microphone 43 and the position of the microphone hole 16 do not have to be the connecting portion of the game device 100. For example, the microphone 43 is housed in the first housing 11 and the microphone hole 16 is first. It may be provided at a position corresponding to the storage position of the microphone 43 on the housing 11.

A fourth LED 26 (FIG. 2) is provided on the outer main surface of the second housing 21 at a position close to the outer camera 25. The fourth LED 26 lights up according to the imaging state of the outer camera 25. That is, the fourth LED 26 notifies the person existing in the field of view and the people around it that the image is being taken by the game device 100. More specifically, the fourth LED 26 is lit while the image is being taken by the outer camera 25. Further, the fourth LED 26 may be blinked while the moving image is acquired by the outer camera 25 (the captured image data is continuously stored). In order to prevent the light emitted by lighting from being reflected on the screen to be imaged, after the camera is instructed to take a picture, until the image data acquired by the camera is stored in the memory or the like according to the instruction. During that time, the 4th LED 26 may be turned off.

Sound vent holes 24 are provided on the left and right main surfaces of the second LCD 22 provided near the center of the inner main surface of the second housing 21, respectively. A speaker (speaker 45 shown in FIG. 3) is housed as an audio output device inside the second housing 21 that communicates with the sound extraction hole 24. That is, the sound extraction hole 24 guides the sound emitted from the speaker 45 to the outside of the game device 100.

As described above, the first housing 11 has an input unit (touch panel 13 and buttons 14A to 14K) for performing operation input to the game device 100, and a first LCD12 which is a display means for displaying various images. And are provided. Further, the second housing 21 is provided with an inner camera 23 and an outer camera 25 for acquiring image data, and a second LCD 22 as a display means for displaying various images.

&lt;Internal configuration of game device&gt; FIG. 3 is a block diagram showing an example of the internal configuration of the game device 100 according to the embodiment of the present invention.

With reference to FIG. 3, the game apparatus 100 includes a CPU 31, a main memory 32, a memory control circuit 33, a storage data memory 34, a preset data memory 35, and a memory card interface (memory card I / F). It has electronic components such as 36 and 37, a wireless communication module 38, a local communication module 39, a real-time clock (RTC) 40, a power supply circuit 41, and an interface circuit (I / F circuit) 42. These electronic components are mounted on an electronic circuit board and housed in a first housing 11 (or a second housing 21).

The CPU 31 is an arithmetic processing unit that executes various programs. The CPU 31 expands and executes a game program stored in any of the memory (typically, storage data memory 34), the memory card 28, and the memory card 29 inside the game device 100 in the main memory 32. By executing the game program of the CPU 31, various processes according to the present embodiment described later are provided. As will be described later, the game program according to the present embodiment is typically supplied to the game device 100 from a network-connected distribution server device or the like via a wired or wireless communication line. The game program supplied to the game device 100 is stored in the storage data memory 34.

Further, the CPU 31 has a VRAM (Video Random Access Memory) (not shown) for exclusively controlling the display on the first LCD 12 and the second LCD 22. Image data for displaying various images, which will be described later, is temporarily stored in this VRAM. The data stored in the main memory 32 is transferred to this VRAM, or the file (data) stored in the storage data memory 34 is directly read and the contents are written. ..

FIG. 4 is a diagram showing a form of providing a game program to the game device 100 according to the embodiment of the present invention.

With reference to FIG. 4, a typical system for distributing a game program according to the present embodiment mediates wireless communication between a distribution server device SRV connected to a network NW such as the Internet and the network NW of the game device 100. Includes access point APs. The distribution server device SRV holds a plurality of game programs including the game program according to the present embodiment in a downloadable state, and is requested after undergoing a predetermined procedure according to the access from the game device 100 or the like. Start sending (downloading) the game program.

The access point AP is connected to the network NW by wire and establishes a wireless connection with the wireless communication module 38 (Fig. 3). The game device 100 accesses the distribution server device SRV via this access point AP. Then, as described above, when the game program distributed from the distribution server device SRV is received, the CPU 31 of the game device 100 stores the received game program in the storage data memory 34 or the like. As a general rule, the game program once stored in the storage data memory 34 of the game device 100 is prohibited from moving to the outside of the game device 100. Therefore, since the game program once incorporated into the game device 100 is executed only in the game device 100, various settings and parameters are used according to the usage record and usage pattern of the owner of the game device 100. Can be customized or arranged independently of the game device 100.

Instead of the network distribution configuration as shown in FIG. 4, the game program may be provided by selling the memory card 28 or the memory card 29 in which the game program is stored. In this case, the storage medium for storing the game program according to the present invention is not limited to a semiconductor storage device such as a memory card 28 or a memory card 29, but may be an optical storage medium such as a CD-ROM or a DVD.

With reference to FIG. 3 again, the main memory 32, the memory control circuit 33, and the preset data memory 35 are connected to the CPU 31. Further, a storage data memory 34 is connected to the memory control circuit 33.

The main memory 32 is a storage means used as a work area or a buffer area of the CPU 31. That is, the main memory 32 temporarily stores data used for various processes, or expands and temporarily stores a game program read from a storage data memory 34, a memory card 28, a memory card 29, or the like. To do. In this embodiment, for example, PSRAM (Pseudo-SRAM) is used as the main memory 32.

The storage data memory 34 is a storage means for storing a game program executed by the CPU 31, image data captured by the inner camera 23 and the outer camera 25, and the like. The storage data memory 34 is made of a non-volatile storage medium such as a NAND flash memory, for example. The memory control circuit 33 is a circuit that controls data reading from the storage data memory 34 and data writing to the storage data memory 34 according to instructions from the CPU 31.

The preset data memory 35 is a storage means for storing data (preset data) such as various parameters preset in the game device 100. As the preset data memory 35, a flash memory connected to the CPU 31 by an SPI (Serial Peripheral Interface) bus can be adopted.

Memory cards I / F 36 and 37 are connected to CPU 31, respectively. The memory card I / F 36 reads and writes data to the memory card 28 mounted on the connector in response to an instruction from the CPU 31. Further, the memory card I / F 37 reads and writes data to the memory card 29 mounted on the connector according to the instruction of the CPU 31.

In the present embodiment, the image data acquired by the inner camera 23 and the outer camera 25 and the image data acquired from other devices are written to the memory card 28 via the memory card I / F 36, and are written to the memory card 28. The image data stored in is read from the memory card 28 via the memory card I / F 36 and written as a file in the storage data memory 34. Further, various programs stored in the memory card 29 are read from the memory card 29 via the memory card I / F 37 and written to the main memory 32.

The wireless communication module 38 has a function of connecting to a wireless LAN by, for example, a method compliant with the IEEE802.11.b / g standard. Further, the local communication module 39 has a function of performing wireless communication with a game device of the same type by a predetermined communication method. The wireless communication module 38 and the local communication module 39 are connected to the CPU 31. The CPU 31 uses the wireless communication module 38 to send and receive data to and from other devices via a network line such as the Internet, and the local communication module 39 to send and receive data to and from other game devices of the same type. You can send and receive.

Further, the RTC 40 and the power supply circuit 41 are connected to the CPU 31. RTC40 counts the time and outputs it to CPU31. For example, the CPU 31 can also calculate the current time (date) and the like based on the time clocked by the RTC 40. The power supply circuit 41 controls the electric power supplied from the electric power (typically a battery, which is housed in the first housing 11) of the game apparatus 100, and supplies electric power to each component of the game apparatus 100.

The game device 100 further has an I / F circuit 42 connected to the CPU 31. A microphone 43, an amplifier 44, and a touch panel 13 are connected to the I / F circuit 42. The I / F circuit 42 includes a voice control circuit that controls the microphone 43 and the amplifier 44 (and the speaker 45), and a touch panel control circuit that controls the touch panel 13.

The microphone 43 detects the user's voice and the like uttered toward the game device 100, and outputs a voice signal indicating the detected voice to the I / F circuit 42. The amplifier 44 amplifies the audio signal from the I / F circuit 42 and outputs it from the speaker 45. That is, the voice control circuit included in the I / F circuit 42 performs A / D conversion on the voice signal detected by the microphone 43, outputs the result to the CPU 31, and D for the signal generated by the CPU 31 or the like. Performs / A conversion and outputs the result to amplifier 44. Further, the voice control circuit converts the voice signal into voice data in a predetermined format suitable for storage.

Further, the touch panel control circuit included in the I / F circuit 42 generates touch position data according to the detection signal from the touch panel 13 and outputs the touch position data to the CPU 31. For example, the touch position data includes coordinate values (hereinafter, also referred to as input coordinates) indicating a position where input is performed on the input surface of the touch panel 13. The touch panel control circuit cyclically executes reading of a signal from the touch panel 13 and generation of touch position data at a predetermined cycle. The CPU 31 can detect the input coordinates in which the user has performed an input operation on the touch panel 13 by acquiring the touch position data via the I / F circuit 42 (touch panel control circuit).

Button 14 is a generic name for the above-mentioned buttons 14A to 14K, and is connected to the CPU 31. Operation data indicating the input status (whether or not the button is pressed) for the buttons 14A to 14K is output from the button 14 to the CPU 31, respectively. By acquiring the operation data from the button 14, the CPU 31 executes the process according to the user operation for the button 14.

The inner camera 23 and the outer camera 25 are each connected to the CPU 31. The inner camera 23 and the outer camera 25 take an image according to the instruction of the CPU 31, and output the image data acquired by the image to the CPU 31. Each of the inner camera 23 and the outer camera 25 includes an image sensor such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor), and a peripheral circuit for reading image data acquired by the image sensor. For example, the CPU 31 instructs one of the inner camera 23 and the outer camera 25 to take an image, and outputs the image data acquired by the camera receiving the instruction of the image taking to the CPU 31.

Further, both the first LCD12 and the second LCD22 are connected to the CPU 31. The first LCD12 and the second LCD22 each display an image according to an instruction from the CPU 31. In one form, the CPU 31 displays the image acquired by the inner camera 23 or the outer camera 25 on one of the first LCD12 and the second LCD22, and accepts the operation from the user and / on the other of the first LCD12 and the second LCD22. Alternatively, a screen (image) for explaining the operation to the user is displayed.

&lt;Application overview&gt; An outline of the application provided by executing the game program according to the present embodiment will be described. The game device 100 according to the present embodiment provides an application including a game process and a display process after the game is completed. As the game processing, the image pickup unit (typically, the inner camera 23) is used to mainly image the user's face and hands, and the game progresses based on the user image included in the image data obtained by this image pickup. Will be done. Here, the user image means at least an image including an area in which the user is captured as a subject in the image data acquired by the imaging unit. More specifically, in order to identify the user image, a human body recognition process (typically, a skin color region detection process and / or a face recognition process, which will be described later) is executed on the acquired imaging data. Then, based on this recognition result (user image), it is sequentially determined whether or not a predetermined condition according to the progress of the game is satisfied, and how to proceed with the game is determined based on this determination result. To. Further, as will be described later, in addition to the user image included in the image data acquired by the imaging unit, the game may be advanced based on the user's voice data collected by the microphone 43. In addition, in this specification, a "game" includes a case where it is a series of games according to a predetermined scenario, and a case where it is an aggregate of a plurality of mini games.

In the game device 100 according to the present embodiment, during the progress of such a series of games, the user plays the game by the imaging unit in relation to or independently of the condition determination result for progressing the game. The figure is imaged as appropriate. An image based on the image data obtained by such imaging is displayed on the display unit (typically, the second LCD22) when the game is not in progress (typically after a series of games is completed). Is displayed. The user and those around him can enjoy seeing the displayed facial expression of the user during play.

FIG. 5 is a schematic view showing a state of a user playing the game device 100 according to the embodiment of the present invention. As shown in FIG. 5, typically, the game device 100 is placed on a desk or the like, that is, after fixing the imaging range of the inner camera 23, the user himself / herself has his / her head, body, hands, etc. Move according to the game production. As a result, the recognition result in the game device 100 (that is, the user image included in the acquired image data) changes, and it is sequentially determined whether or not the game progress condition is satisfied based on the changing recognition result. To.

FIG. 6 is a diagram for temporally explaining an example of the processing content executed by the game device 100 according to the embodiment of the present invention. With reference to FIG. 6, in the game device 100 according to the present embodiment, when the game progress is started, the image data is acquired by the imaging unit (inner camera 23), and the human body recognition process based on the acquired image data is performed. The execution is repeatedly executed at a predetermined cycle. Then, the game progresses based on the recognition result acquired (updated) periodically. That is, when the recognition result sequentially satisfies the conditions 1, 2, ..., The game progresses in sequence. If all the conditions (or the success conditions of the game) are not satisfied within the predetermined time, the game may be judged as "failed". In this specification, a state in which all the conditions prepared in advance for the progress of the game are satisfied may be referred to as "success" of the game.

In parallel with the progress of this game, the image pickup unit captures the image of the user playing. The image data acquired for performing the human body recognition process is basically discarded without being saved. On the other hand, the image data showing the user playing is saved in the main memory 32 or the like. Therefore, from the viewpoint of clarifying the difference in the processing, the processing of storing (preserving) the image data of the user's playing is also referred to as "capture".

In both the human body recognition process and the capture, the image data acquired by the imaging unit is commonly used, but the timing of using the image data in each process may be different. That is, in the human body recognition process, the image data is processed at regular intervals, whereas in the capture, the image data is stored at an event-like timing such as the satisfaction of a predetermined condition.

As shown in FIG. 6, typically, the image data is captured at the timing when the recognition result satisfies the conditions 1, 2, ... While the game is in progress. That is, based on the user image included in the image data acquired by the imaging unit (inner camera 23), it is determined whether or not the game is successful, and at least when it is determined that the game is successful, the image is captured. The image data acquired by the unit is saved. Note that FIG. 6 illustrates a case where the conditions used for the progress of the game and the conditions for determining the execution timing of the capture are the same, but it is not always necessary to match the two conditions, depending on the game scenario and the like. , You may set conditions independent of each other.

After that, when the series of games is completed, the image data 1, 2, ... captured during the game play is displayed (replayed). After the end of this game, image data is displayed in various modes according to the content of the game.

The game device 100 according to the present embodiment typically provides the following three types of games. Hereinafter, the contents of these games and an example of the display mode of the image displayed after the game is over will be described. It should be noted that some or all of these games are selectively executed according to the user's instruction. Further, as the display timing, the image data may be replayed at an arbitrary timing specified by the user even if the game is not finished.

&lt;1st game&gt; First, with reference to FIGS. 7 to 10, a first game provided by the game device 100 according to the present embodiment and a display mode of images after the end of the first game will be described. During the play of this first game, from the start of the game to the end of the game, the user plays each time a plurality of different conditions are satisfied, including a condition for determining that the first game is "successful". Each figure is captured. Then, after the end of the first game, the plurality of captured image data are displayed by switching (continuously) in the order of the captured time or in a random order. Therefore, after the end of the first game, the appearance of the user during the play of the first game is displayed in a moving image-like manner. That is, as an image display based on the stored image data, in the first game, a moving image-like image that switches a plurality of image data at a predetermined time is displayed. By replacing the display order of the image data with one different from the original imaging order, it is possible to provide a more enjoyable image display. Further, by making the display speed (switching speed of image data) shorter than the actual imaging interval, it is possible to display a more dynamic image.

7 to 9 are diagrams showing an example of a scene of the first game according to the embodiment of the present invention. FIG. 10 is a diagram showing a display example of an image after the end of the first game according to the embodiment of the present invention.

In the first game according to the present embodiment, the user makes a face on the game device 100 (inner camera 23) so that the recognition result for the image data acquired by the inner camera 23 exceeds the size of the predetermined determination area. It's a game that brings you closer.

First, when the user instructs the start of the first game, screens as shown in FIG. 7 are displayed on the first LCD12 and the second LCD22. For convenience of explanation, FIGS. 7 to 9 show the entire images (screenshots) displayed on the first LCD12 and the second LCD22 side by side. The same applies to the following figures.

More specifically, in the scene shown in FIG. 7, a notification image INF for notifying the user of what kind of movement should be displayed is displayed on the first LCD12. That is, in the example shown in FIG. 7, an image showing a message "Try it!" Notifying the user to bring his / her face closer is displayed on the first LCD 12. On the other hand, on the second LCD22, an image object for game production (in the example shown in FIG. 7, an image of the human mouth) is displayed, and the image data acquired by the inner camera 23 is obtained by human body recognition processing. The recognition display REC indicating the range of the user's face is reflected. As a general rule, this recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

FIG. 7 shows a state in which the user is shown only in a relatively lower position in the image data acquired by the inner camera 23. Therefore, the recognition display REC corresponding to the position of the captured user's face is reflected only on the lower side of the second LCD 22.

After that, when the user brings his / her face close to the game device 100 and the range in which the user is captured in the image data acquired by the inner camera 23 expands, the screen transitions to the screen shown in FIG. In the state shown in FIG. 8, an image object of the human mouth that is enlarged as compared with FIG. 7 is displayed, and a wider range of recognition display REC is reflected.

Further, when the user brings his / her face closer to the game device 100 and the range in which the user is captured in the image data acquired by the inner camera 23 is further expanded, the screen transitions to the screen as shown in FIG. FIG. 9 shows a state in which the first game is "successful". In the "successful" state of this first game, a larger image object of the human mouth is displayed compared to Fig. 8, and an image object with a small bird appearing between the teeth is displayed. Will be done.

In the series of games shown in FIGS. 7 to 9 as described above, the user sets the relative position with the game device 100 according to the progress of the game while referring to the game effect and the recognition display REC displayed on the second LCD 22. Change. During the play of the first game, the appearance of such a user in the game is captured multiple times. Then, after the end of the first game, the plurality of captured image data are sequentially displayed. FIG. 10 shows an example of how this image data is displayed.

In FIG. 10 (a), the face of the user who played the first game is shown smaller than that in FIG. 10 (b). This corresponds to the state shown in FIG. 8 described above, and shows the case where the user's face is located at a position relatively far from the game device 100. On the other hand, in FIG. 10 (b), the user's face is reflected in a larger range than in FIG. 10 (a). This corresponds to the state shown in FIG. 9 described above, and shows a case where the user's face is relatively close to the game device 100.

By continuously switching and displaying such images, it is possible to display a user playing the first game in a moving image-like manner. In other words, the state in which the user is playing the first game can be replayed and displayed.

Note that FIG. 10 typically shows an example in which two image data are sequentially switched, but even if more image data is acquired during the first game and these are sequentially switched and displayed. Good.

&lt;Second game&gt; Next, with reference to FIGS. 11 to 13, a second game provided by the game device 100 according to the present embodiment and a display mode of images after the end of the second game will be described. This second game is typically composed of a plurality of mini-games, and during the play of each mini-game, at least one image of the user playing at a timing when a predetermined condition is satisfied is captured. Then, after the end of a series of mini-games, that is, after the end of the second game, for each of these captured image data, an object related to the corresponding mini-game or an object having a meaning different from that of the corresponding mini-game. Are displayed in layers. Therefore, after the end of the second game, the appearance of the user at a certain timing during the play of the second game (typically around the timing when the mini-game is "successful") is expressed together with the object corresponding to the game production. Or, it is expressed with an object (unrelated) independent of the game production. It is preferable that the image data captured in each of the plurality of mini games is displayed by switching at predetermined time intervals, that is, displayed like a so-called slide show.

When the start of the second game is instructed by the user, it may be possible to randomly determine which of the plurality of mini-games prepared in advance is to be executed.

11 to 13 are diagrams showing an example of a scene during and after the game of the second game according to the embodiment of the present invention.

(1) 1st mini game In the first mini-game included in the second game shown in FIG. 11 (a), the object OBJ10 of "glasses" is displayed at a predetermined position of the second LCD22, and the user recognizes his / her own eyes. Adjust its relative position to the game device 100 (inner camera 23) so that the position matches the object OBJ10.

In the scene shown in FIG. 11A, the image object (object OBJ10) for the game production is displayed on the second LCD22, and the user's body recognition process for the image data acquired by the inner camera 23 is performed. The recognition display REC indicating the range of the face is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later. In addition, an image object indicating "eyes" is displayed at a position in the pupil of the user who is recognized as a human body.

When the user adjusts his / her relative position with respect to the game device 100, it is determined that the position of the human-recognized user's pupil in the image data acquired by the inner camera 23 matches the position of the object OBJ10. , The first mini-game will be "success". In this way, based on the result of human body recognition, it is determined whether or not the success condition associated with the position of the object displayed according to the game effect is satisfied.

With reference to FIG. 11 (b), after the end of the second game, the image data captured during the play of the first mini-game shown in FIG. 11 (a) is displayed on the second LCD 22. This displayed image is typically overlaid with the "glasses" object OBJ10 # that was displayed during the play of the first mini-game. In the example shown in FIG. 11 (b), the same "glasses" object as the "glasses" displayed in the corresponding first mini-game is displayed in an overlapping manner, but another form of "glasses" is superimposed. It may be displayed. Alternatively, an image object completely different from the "glasses" may be displayed in an overlapping manner.

The first LCD12 displays a button image or the like for controlling the display (slide show) of the image data captured in each of the plurality of mini games. More specifically, the first LCD 12 displays an order notification image 202, a stop button image 204, a back button image 206, a forward button image 208, and an end button image 210. A touch panel 13 is provided on the first LCD 12, and when the user touches any position on the first LCD 12, the touched coordinates are detected by the touch panel 13. Then, when any of the above-mentioned button images is displayed at the detected coordinates, the process assigned to the corresponding button image is executed as described below.

The order notification image 202 notifies which mini-game the currently displayed image data was captured. In the example shown in FIG. 11 (b), the case where the five mini-games are played first is shown, and the currently displayed image data is shown to have been captured in the first mini-game. Stop button The image 204 receives a stop instruction for switching the display of image data (slide show). When the slide show is in the stopped state, a play button image for accepting a slide show restart instruction is displayed instead of the stop button image 204. Back button The image 206 accepts an instruction to return one of the displayed image data from the currently displayed image data in the series of image data. In addition, the forward button image 208, among a series of image data, accepts an instruction to advance the displayed image data by one from the currently displayed one. End button The image 210 receives an instruction to end the display of image data.

(2) Second mini game Next, in the second mini-game included in the second game shown in FIG. 12 (a), the object OBJ11 indicating "mosquitoes" is moved and displayed on the second LCD22, and the user can move the object by the recognized movement of his / her hand. Clap your hands so that OBJ11 is "crushed".

In the scene shown in FIG. 12 (a), the image (object OBJ11) on the game effect is displayed on the second LCD22 at different positions in time, and the object OBJ12 that prompts the user to clap their hands is displayed. To. Further, the second LCD 22 reflects the recognition display REC indicating the range of the user's hand and face obtained by the human body recognition process on the image data acquired by the inner camera 23. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later. On the other hand, on the first LCD12, a notification image INF indicating a message "Take!" Notifying the user to clap his / her hand is displayed.

When the user claps his / her hand after adjusting his / her relative position with respect to the game device 100, and the human body recognition process for the image data acquired by the inner camera 23 recognizes that the user has clap his / her hand at the position of the object OBJ11. , The second mini-game will be "successful", and the effect will be such that "mosquitoes" fall to the bottom of the screen.

With reference to FIG. 12 (b), after the end of the second game, the image data captured during the play of the second mini-game shown in FIG. 12 (a) is displayed on the second LCD 22. This displayed image is overlaid with an object OBJ13 that has a different meaning and is independent of the "mosquito" object OBJ11 that was displayed during the play of the second mini-game. That is, the movement of the user hitting the "mosquito" is replaced with a cute expression of the user and displayed.

As in FIG. 11B described above, the first LCD 12 displays a button image or the like for controlling the display (slide show) of the image data captured in each of the plurality of mini games.

As shown in FIG. 12, during the display of the image based on the image data captured during the play of the second mini-game after the end of the second mini-game, the object displayed while the second mini-game is in progress. The image is displayed by replacing the object with an object different from the object at the position. In particular, in the example shown in FIG. 12, an object not related to the object displayed while the second mini-game is in progress (mosquito in the example shown in FIG. 12 (a)) (in the example shown in FIG. 12 (b)). "Heart mark" etc.) is displayed along with the image data captured during the play of the second mini-game.

(3) 3rd mini game Next, in the third mini-game included in the second game shown in FIG. 13A, two users can play the game. That is, in the third mini-game shown in FIG. 13 (a), two "totem pole" objects OBJ14 and OBJ15 are displayed on the second LCD22, and the image data acquired by the inner camera 23 is obtained by human body recognition processing. The recognition display REC showing the range of each user is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later. These "totem pole" objects OBJ14 and OBJ15 are partially provided with a circular cutout, and each user recognizes his or her face position to match the corresponding cutout. As described above, the relative position with respect to the game device 100 (inner camera 23) is adjusted.

In addition, the first LCD12 displays a notification image INF indicating a message "together!" Notifying the user that his / her face is aligned with a predetermined position.

Both users adjust their relative positions with respect to the game device 100, and determine that the position of the human-recognized face in the image data acquired by the inner camera 23 matches the cutout portion of the corresponding objects OBJ14 and OBJ15. Then, the third mini-game becomes "success".

With reference to FIG. 13 (b), after the end of the second game, the image data captured during the play of the third mini-game shown in FIG. 13 (a) is displayed on the second LCD 22. This displayed image is overlaid with objects OBJ16 and OBJ16, which have different meanings and are independent of the objects OBJ14 and OBJ15 of the "totem pole" displayed during the play of the third mini-game. That is, the appearance of both users located in the respective cutouts provided in the "totem pole" is replaced with an expression in which both users are protruding from the large "nose".

As in FIG. 11B described above, the first LCD 12 displays a button image or the like for controlling the display (slide show) of the image data captured in each of the plurality of mini games.

As described above, the figure of the user captured during the play of the second game (the mini-game included in it) is independent of the object corresponding to the game production of the corresponding mini-game or the game production of the corresponding mini-game. The objects (having different meanings) are superimposed. As a result, the user can enjoy a more enjoyable image using the appearance of playing the mini game.

As shown in FIG. 13, during the display of the image based on the image data captured during the play of the third mini-game after the end of the third mini-game, the object displayed while the third mini-game is in progress is displayed. Is replaced with another object and the image is displayed. In particular, in the example shown in FIG. 13, an object (example shown in FIG. 13 (b)) that is not related to the object displayed while the third mini-game is in progress (totem pole in the example shown in FIG. 13 (a)). Then, "nose" etc.) is displayed together with the image data captured during the play of the third mini-game.

&lt;Third game&gt; Next, with reference to FIGS. 14 to 19, a third game provided by the game device 100 according to the present embodiment and a display mode of images after the end of the third game will be described. The third game is composed of a series of a plurality of mini-games, and during the play of each mini-game, at least one image of the user playing at a timing when a predetermined condition is satisfied is captured. Then, after the end of a series of mini-games, that is, after the end of the third game, objects having a specific meaning are superimposed on each of the captured image data, so that the scenario is prepared in advance. A cartoon-like expression is provided. The cartoon-like image data is sequentially switched and displayed at predetermined time intervals.

14 to 18 are diagrams showing an example of a scene of the third game according to the embodiment of the present invention. 19 and 20 are diagrams showing an example of displaying an image after the end of the third game according to the embodiment of the present invention. In the following description, a case where the third game is composed of five mini-games and a "5-frame cartoon" is displayed using a total of five image data captured in each mini-game will be illustrated.

(1) 4th mini game In the fourth mini-game included in the third game shown in FIG. 14, the object OBJ17 representing the human head and back is displayed at a predetermined position on the second LCD22, and the user recognizes himself / herself. Move your hand as if you "slap" the part of object OBJ17 that corresponds to the "shoulder".

In the scene shown in FIG. 14, the image (object OBJ17) on the game effect is displayed on the second LCD22, and the range of the user's hand obtained by the human body recognition process on the image data acquired by the inner camera 23 is displayed. The recognition display REC shown is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

After the user adjusts the relative position of the hand with respect to the game device 100, the user raises and lowers the hand, and the user corresponds to the predetermined position (shoulder"" of the object OBJ17 by the human body recognition process for the image data acquired by the inner camera 23. If the position) is judged to be "struck" a predetermined number of times, the fourth mini-game becomes "success". If it is determined that the "shoulder" is "struck" while the game is in progress, the object OBJ18 indicating that is displayed. In addition, the second LCD22 also displays the object OBJ19, which indicates the number of remaining "slaps" on the shoulder required for the game to be "successful".

(2) 5th mini game In the fifth mini-game included in the third game shown in FIG. 15, the object OBJ20 representing the farewell scene is displayed on the second LCD22, and the user moves his / her hand left and right to perform a bye-bye operation.

In the scene shown in FIG. 15, the image (object OBJ20) on the game effect is displayed on the second LCD22, and the range of the user's hand obtained by the human body recognition process on the image data acquired by the inner camera 23 is displayed. The recognition display REC shown is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

The user adjusts the relative position of the hand with respect to the game device 100 and then moves the hand left and right, and the human body recognition process for the image data acquired by the inner camera 23 recognizes that the user has moved the hand left and right a predetermined number of times. Then, the 5th mini game becomes "success". While the game is in progress, the object OBJ21 that prompts the user to wave is displayed.

(3) 6th mini game In the sixth mini-game included in the third game shown in FIG. 16, the object OBJ22 representing the "nose" is displayed on the second LCD22, and the user can recognize the object whose index finger and middle finger represent the "nose". Move your hand so that it is "inserted" into the position corresponding to the "hole" of OBJ22.

In the scene shown in FIG. 16, the image (object OBJ22) on the game effect is displayed on the second LCD22, and the range of the user's hand obtained by the human body recognition process on the image data acquired by the inner camera 23 is displayed. The recognition display REC shown is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

The user adjusts the position of the hand relative to the game device 100, and then the human body recognition process for the image data acquired by the inner camera 23 causes the user's index finger and middle finger to become the "hole" of the object OBJ22. The sixth mini-game is "successful" when it is recognized that it has been plugged into the corresponding position.

(4) 7th mini game In the 7th mini-game included in the 3rd game shown in FIG. 17, the object OBJ23 representing the "character wearing a cloak" is displayed on the 2nd LCD22, and the user brings his / her face closer to the inner camera 23 and the inner camera. Breathe into the microphone holes 16 (and the microphones 43 housed in them) located close to 23.

In the scene shown in FIG. 17, the image (object OBJ23) on the game effect is displayed on the second LCD22, and the range of the user's hand obtained by the human body recognition process on the image data acquired by the inner camera 23 is displayed. The recognition display REC shown is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

The user adjusts the relative position of his / her face with respect to the game device 100 and then blows, so that the recognition area of the user's face exceeds a predetermined threshold value and the sound pressure at the time of blowing is predetermined. If the threshold is exceeded, the 7th mini-game is judged to be "successful". If it is determined that the player has blown while the game is in progress, the displayed object OBJ23 is "blown away" according to the determination.

(5) 8th mini game In the 8th mini-game included in the 3rd game shown in FIG. 18, the object OBJ24 representing the "crying baby" is displayed on the 2nd LCD22, and the user can perform an action like covering the face with both hands and both hands from the face. The action of releasing the object is repeated, so-called "Peek-a-boo".

In the scene shown in FIG. 18, the image (object OBJ24) on the game effect is displayed on the second LCD22, and the range of the user's hand obtained by the human body recognition process on the image data acquired by the inner camera 23 is displayed. The recognition display REC shown is reflected. This recognition display REC is sequentially updated at each processing cycle of the skin color region detection processing described later.

When it is recognized that the user repeats the action of covering his / her face with both hands and the action of releasing both hands for a predetermined number of times, the eighth mini-game is judged to be "successful".

(6) Display example after the game is over After the fourth to eighth mini-games (third games) as described above are completed, cartoon-like expressions are provided according to the scenarios prepared in advance as shown in FIGS. 19 and 20.

After the end of the third game, the second LCD 22 displays a title screen prepared in advance as shown in FIG. 19 (a), and subsequently displays a screen as shown in FIG. 19 (b). In the screen shown in FIG. 19 (b), an image object according to a prepared scenario is superimposed on the image data captured while the user is playing the fourth mini game (FIG. 14) included in the third game. It is a thing. Then, following the screen shown in FIG. 19 (b), the screen shown in FIG. 19 (c) is displayed. In the screen shown in FIG. 19 (c), an image object according to a prepared scenario is superimposed on the image data captured while the user is playing the fifth mini game (FIG. 15) included in the third game. It is a thing.

In the same manner thereafter, following the screen shown in FIG. 19 (c), the screen shown in FIG. 20 (a), the screen shown in FIG. 20 (b), and the screen shown in FIG. 20 (c) are sequentially arranged on the second LCD 22. Is displayed. The screen shown in FIG. 20 (a), the screen shown in FIG. 20 (b), and the screen shown in FIG. 20 (c) are the sixth mini-game (FIG. 16) and the seventh mini-game (FIG. 16) included in the third game, respectively. This is an overlay of the image data captured while the user is playing the eighth mini-game (FIG. 17) and the eighth mini-game (FIG. 18) with each image object according to a scenario prepared in advance.

As shown in FIGS. 19 and 20, objects according to a prepared scenario are superimposed on the user captured during the play of a series of mini-games. As a result, the user who played the third game can be enjoyed with a cartoon-like image as the main character.

As shown in FIGS. 14 to 18 described above, in the third game, in a plurality of predetermined stages (fourth to eighth mini-games), the success conditions associated with each stage (mini-game) are met. The game progresses by judging the success of the game based on each. At this time, in each mini-game, the objects associated with the corresponding success conditions are switched and displayed.

Further, as shown in FIGS. 19 and 20 described above, in the third game, success associated with each stage (mini game) in a plurality of predetermined stages (fourth to eighth mini games). The game progresses by judging the success of the game based on the conditions. After the end of this third game, a plurality of image data acquired and stored in the image storage unit 116 in each of a plurality of stages (mini games) in which the game is in progress, and a stage in which each image data is acquired (mini game). ) Is associated with the object associated with the success condition, and the composite image is displayed.

&lt;Control structure&gt; Next, a control structure for providing the above-mentioned processing will be described.

FIG. 21 is a block diagram showing a control structure of the game device 100 according to the present embodiment. With reference to FIG. 21, the game device 100 has, as its control structure, a voice data buffer 102, a voice recognition unit 104, a determination unit 110, an image data buffer 112, a capture controller 114, and an image storage unit 116. , Human body recognition unit 120, game image controller 130, game image DB (Data Base) 132, recognition result reflection unit 134, replay controller 136, replay image DB 138, and rendering engine 140. The control structure shown in FIG. 21 typically provides that the CPU 31 executes a game program using hardware such as the main memory 32. Alternatively, all or part of it may be embodied by hardware circuits.

The voice data buffer 102 stores a digital-format voice signal obtained by D / A conversion of the voice acquired by the microphone 43 (FIG. 3) by the I / F circuit 42 (FIG. 3) for a predetermined time. The audio data buffer 102 is provided as a part of the main memory 32.

The voice recognition unit 104 performs known voice recognition on the voice data stored in the voice data buffer 102. Typically, the voice recognition unit 104 determines whether or not the user in the seventh mini-game included in the third game shown in FIG. 17 described above is breathing. It should be noted that it may be determined whether or not the user is breathing based on whether or not the sound acquired by the microphone 43 is equal to or higher than a predetermined sound pressure. The recognition result by the voice recognition unit 104 is sent to the determination unit 110.

The image data buffer 112 stores image data acquired from the cameras (inner camera 23 and outer camera 25) for a predetermined number of frames. Since the inner camera 23 is basically used in each of the above-mentioned games, the image data captured by the inner camera 23 is accumulated in the image data buffer 112 at any time. It is provided as a part of the main memory 32. Further, the image data buffer 112 is connected to the capture controller 114 and the human body recognition unit 120.

The capture controller 114 is arranged between the image data buffer 112 and the image storage unit 116, and stores the image data stored in the image data buffer 112 in the image storage unit 116 in response to a capture command from the determination unit 110 described later. Output to. That is, during game play (while the inner camera 23 is enabled), the image data captured by the inner camera 23 is periodically written to the image data buffer 112, and among these image data, What is needed is selectively stored in the image storage unit 116.

The image storage unit 116 stores image data to be displayed after the end of the game according to the present embodiment. The image storage unit 116 is provided as a part of the main memory 32 or the storage data memory 34. The image data stored in the image storage unit 116 may be stored in the memory card 28 mounted on the game device 100 from the outside.

The human body recognition unit 120 recognizes the human body with respect to the image data acquired by the inner camera 23, and outputs the recognition result to the determination unit 110. More specifically, it includes a face recognition unit 122, a color coordinate conversion unit 124, and a skin color region detection unit 126.

The face recognition unit 122 executes face recognition processing on the image data stored in the image data buffer 112. That is, the face recognition unit 122 detects a human face region from the captured image data as a user image. At this time, the face recognition unit 122 also extracts facial feature points and the like included in the face region. The face recognition unit 122 outputs these face recognition results to the determination unit 110. More specifically, the coordinate values indicating the range specified as the face area and the center coordinates such as the pupil are output as the detection result. Since the face recognition process in the face recognition unit 122 is a known technique, detailed description thereof will not be given here.

The color coordinate conversion unit 124 converts the image data described in the RGB color system into the HVC color system as a preprocessing in the skin color area detection unit 126 described later. That is, in the image data captured by the camera, the color information of each pixel is represented by the gradation values of R (red), G (green), and B (blue), and the color coordinate conversion unit 124 uses these gradation values. The color information of each pixel is converted into a coordinate system of hue (Hue), brightness (Value), and saturation (Chroma).

The skin color area detection unit 126 detects the skin color area in the image data acquired by the inner camera 23 based on the color information of the HVC coordinate system output from the color coordinate conversion unit 124 (skin color area detection process). That is, the skin color region detection unit 126 detects the skin color region in the image data acquired by the inner camera 23 as a user image. As a result, the skin color area detection unit 126 detects the user area indicating the user area in the image data. More specifically, the skin color region detection unit 126 determines whether or not the hue, brightness, and saturation are included in the corresponding threshold range for each pixel, and is included in which threshold range. It is determined that the pixel is "skin color". Since the color information obtained by imaging the skin part of the user varies depending on the surrounding lighting environment, the user is imaged once or multiple times in advance, and based on the image data obtained by these imagings. It is preferable to dynamically determine the threshold range to be the skin color. The skin color region detection unit 126 outputs the skin color region (coordinate value and / or number of pixels of each pixel determined to be skin color) detected in this way to the determination unit 110.

The determination unit 110 satisfies the conditions uniquely set for each game during the progress of the game based on the recognition results from the voice recognition unit 104 and the human body recognition unit 120 (face recognition unit 122 and skin color area detection unit 126). It is judged sequentially whether or not it is. That is, the determination unit 110 determines whether or not the condition associated with the position of the object displayed according to the ongoing game effect is satisfied based on the recognition result. More specifically, the determination unit 110 stores a condition group 110a composed of a plurality of conditions in advance in association with each game, and sets the corresponding conditions according to the game selected according to the user command. Select to determine whether the input recognition result satisfies the condition. Then, the determination unit 110 determines that the game is successful when the condition (success condition) for determining that each game is successful is satisfied.

The determination unit 110 may determine whether or not the game is successful only when the presence of the user is detected by the skin color area detection unit 126 and / or the face recognition unit 122. Here, as a criterion for determining whether or not the presence of the user is detected, the skin color area detection unit 126 detects the skin color area equal to or larger than a predetermined area, or the face recognition unit 122 detects the face area. Including cases and so on. By adding such a condition, it is possible to avoid a situation in which the game progresses due to an erroneous determination even though the user does not exist in front of the game device 100.

Further, the determination unit 110 determines whether or not the condition for outputting the capture command to the capture controller 114 is satisfied in addition to the success condition for determining whether or not the game is successful. To do.

Further, the determination result by the determination unit 110 is output to the game image controller 130 and the recognition result reflection unit 134. Further, the determination unit 110 outputs the recognition result from the human body recognition unit 120 to the recognition result reflection unit 134.

The game image controller 130 and the game image DB 132 advance the game based on the recognition result from the determination unit 110. More specifically, the game image DB 132 stores image objects corresponding to a plurality of games prepared in advance, and the game image controller 130 uses the game image based on the recognition result from the determination unit 110. Output the required image object from DB132 to the rendering engine 140 at an appropriate timing. In this way, the determination unit 110, the game image controller 130, and the game image DB 132 advance the game based on the result of human body recognition.

The recognition result reflection unit 134 displays the skin color area (user image) detected by the human body recognition unit 120 during the progress of the game as an area recognized as the user's face or hand (recognition display shown in FIG. 7 or the like). REC). As shown in FIG. 7 and the like described above, the recognition result reflecting unit 134 displays the detected skin color region as a shadow. That is, the recognition result reflecting unit 134 displays the skin color region (user image) detected during the progress of the game in a mode different from the image captured by the inner camera 23.

The replay controller 136 and the replay image DB 138 perform the above-mentioned image display (replay operation) after the end of the game. More specifically, the replay image DB 138 stores an image object to be superimposed on the image data after the game ends, and the replay controller 136 reads out the image data stored in the image storage unit 116 after the game ends. At the same time, if necessary, the corresponding image object is read from the replay image DB138 and both are output to the rendering engine 140. As shown in FIG. 10, when displaying a plurality of image data without overlapping any image objects, the replay controller 136 transfers only the image data stored in the image storage unit 116 to the rendering engine 140. Output.

The rendering engine 140 controls the image display on the first LCD12 and the second LCD22. More specifically, the rendering engine 140 superimposes and displays an image object from the game image controller 130 and an image showing a recognized area from the recognition result reflecting unit 134 while the game is in progress. Further, the rendering engine 140 superimposes and displays the image data and the image object from the replay controller 136 after the game is completed.

FIG. 22 is a diagram conceptually showing the processing in the rendering engine shown in FIG. Note that FIG. 22 shows an example in which the display on the second LCD 22 shown in FIG. 11 (b) is realized.

With reference to FIG. 22, in the rendering engine 140, the video signal for displaying an image on the first LCD12 and the second LCD22 is generated by using a plurality of layers. More specifically, the rendering engine 140 has at least a first layer LYR1 for displaying the image data IMG and a second layer LYR2 for displaying the image object . The image data IMG is expanded on the first layer LYR1, and the object OBJ10 # is expanded on the second layer LYR2. Each of the layers LYR1 and LYR2 is provided by forming a memory area large enough for display on the first LCD12 or the second LCD22 in VRAM (not shown) in the CPU31. A video signal is generated by synthesizing these layers LYR1 and LYR2.

In this way, by expanding the objects required for the second layer LYR2, any object can be switched and displayed while the game is in progress and the image is displayed based on the image data stored in the image storage unit 116. Can be done.

&lt;Processing content and processing procedure of the first game&gt; Next, the processing contents and the processing procedure for providing the above-mentioned first game (FIGS. 7 to 10) will be described. In the first game, the game is mainly advanced based on the position of the object displayed according to the game effect and the position and size of the skin color area detected by the skin color area detection unit 126 (FIG. 22).

FIG. 23 is a diagram for explaining the conditions for the progress of the game according to the embodiment of the present invention. With reference to FIG. 23, in the first game, the reference point RP1 set in the image data acquired by the inner camera 23 is centered in association with the position of the object displayed on the second LCD 22 shown in FIGS. 7 to 9. As, a plurality of determination areas AREA1 to AREA5 divided into concentric circles are set. Then, among the pixels included in these determination areas AREA1 to AREA5, the number of pixels (ratio) detected as "skin color" is used as a condition for the progress of the game.

More specifically, in the first game, the game progress is divided into the first to fifth stages, and the number of pixels detected as "skin color" among the pixels included in the determination areas AREA1 to AREA5, respectively. The ratio to the total number of pixels included in the corresponding determination area for is calculated, and the steps are sequentially advanced on condition that the calculated ratio exceeds the corresponding threshold value. This is because in the first game, it is assumed that the user gradually brings his face closer to the game device 100, and it is assumed that the pixels detected as "skin color" gradually expand due to such user movements. Because it is done.

That is, after the start of the first game, first, the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA1 to the total number of pixels included in the determination area AREA1 is the predetermined threshold value Th1. When it exceeds, it proceeds to the second stage. In the second stage, whether or not the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA2 to the total number of pixels included in the determination area AREA2 exceeds the predetermined threshold value Th2. Is continuously judged, and when the threshold Th2 is exceeded, the process proceeds to the third stage.

Hereinafter, in the same manner, in the third stage, the ratio of the number of pixels detected as skin color among the pixels included in the determination area AREA3 to the total number of pixels included in the determination area AREA3 is a predetermined threshold value Th3. It is continuously judged whether or not it exceeds the threshold value, and when the threshold value Th3 is exceeded, the process proceeds to the fourth stage. In the subsequent fourth stage, whether or not the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA4 to the total number of pixels included in the determination area AREA4 exceeds the predetermined threshold value Th4. Is continuously judged, and when the threshold value Th4 is exceeded, the process proceeds to the fifth stage. In the subsequent fifth step, the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA5 to the total number of pixels included in the determination area AREA5 exceeds the predetermined threshold value Th5. Whether or not it is continuously judged, and when the threshold value Th5 is exceeded, the first game becomes "success".

The threshold values Th1 to Th5 are preferably set to 50% or more. Further, in the above description, the case where each determination region is determined by the ratio of the number of pixels detected as "skin color" is illustrated, but the determination is made by the absolute number of pixels detected as "skin color". You may do so.

As described above, in each stage, it is determined whether or not the condition is satisfied based on the pixels included in the corresponding determination area. A detection result different from the originally expected movement of the user, for example, immediately after the start of the first game, even though the number of pixels detected as "skin color" in the inner judgment area is small, the outer judgment is made. When the number of pixels detected as "skin color" in the area is large, it is possible to avoid a false judgment that the first game is "successful" suddenly.

In addition, the game effect is changed according to the stage of the game progress described above. For example, in the first to third stages, the image object as shown in FIG. 7 is displayed, and at the timing when the third to fourth stages are advanced, the display is switched to the image object as shown in FIG. Further, when the conditions in the fifth stage are satisfied and the first game is judged to be "successful", the display is switched to the image object as shown in FIG.

In each game according to the present embodiment, the appearance of the user during play is captured at the timing when a predetermined condition is satisfied. In the first game, the image data acquired by the inner camera 23 is captured at the timing when the above-mentioned conditions in the first to fifth stages are satisfied. That is, the captured image data includes the user's appearance around the timing when the stage of the game progress changes.

The positions of the above-mentioned reference point RP1 and the determination areas AREA1 to AREA5 are appropriately changed depending on the positions of the objects displayed on the game effect.

As described above, in the first game, the game progresses by determining the success of the game based on the success conditions associated with each stage in a plurality of predetermined stages. After the end of the first game, a plurality of image data acquired and stored in the image storage unit 116 at each of the plurality of stages during the game are continuously displayed.

FIG. 24 is a flowchart showing a processing procedure of the first game according to the embodiment of the present invention. The flowchart shown in FIG. 24 is executed when the start of the first game is instructed by the user.

With reference to FIG. 24, first, in step S100, the CPU 31 acquires image data captured by the inner camera 23. In the following step S102, the CPU 31 detects the skin color region in the acquired image data. In the subsequent step S104, the CPU 31 has a threshold value Th1 of the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA1 shown in FIG. 23 to the total number of pixels included in the determination area AREA1. Judge whether or not it exceeds. If the ratio does not exceed the threshold Th1 (NO in step S104), the process of step S100 is repeated. On the other hand, if the ratio exceeds the threshold value Th1 (YES in step S104), the process proceeds to step S106.

In step S106, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. Then, the process proceeds to step S108. That is, the progress of the first game progresses from the first stage to the second stage.

In step S108, the CPU 31 acquires image data captured by the inner camera 23. In the following step S110, the CPU 31 detects the skin color region in the acquired image data. In the subsequent step S112, the CPU 31 has a threshold value Th2 of the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA2 shown in FIG. 23 to the total number of pixels included in the determination area AREA2. Judge whether or not it exceeds. If the ratio does not exceed the threshold Th2 (NO in step S112), the process of step S108 is repeated. On the other hand, if the ratio exceeds the threshold value Th2 (YES in step S112), the process proceeds to step S114.

In step S114, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. Then, the process proceeds to step S116. That is, the progress of the first game progresses from the second stage to the third stage.

In step S116, the CPU 31 acquires image data captured by the inner camera 23. In the following step S118, the CPU 31 detects the skin color region in the acquired image data. In the subsequent step S120, the CPU 31 has a threshold value Th3 of the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA3 shown in FIG. 23 to the total number of pixels included in the determination area AREA3. Judge whether or not it exceeds. If the ratio does not exceed the threshold Th3 (NO in step S120), the process of step S116 is repeated. On the other hand, if the ratio exceeds the threshold value Th3 (YES in step S120), the process proceeds to step S122.

In step S122, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. In the subsequent step S124, the CPU 31 switches the image object to be displayed from the image object shown in FIG. 7 to the image object shown in FIG. Then, the process proceeds to step S126. That is, the progress of the first game progresses from the third stage to the fourth stage.

In step S126, the CPU 31 acquires image data captured by the inner camera 23. In the following step S128, the CPU 31 detects the skin color region in the acquired image data. In the subsequent step S130, the CPU 31 has a threshold value Th4 of the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA4 shown in FIG. 23 to the total number of pixels included in the determination area AREA4. Judge whether or not it exceeds. If the ratio does not exceed the threshold value Th4 (NO in step S130), the process of step S126 is repeated. On the other hand, when the ratio exceeds the threshold value Th4 (YES in step S130), the process proceeds to step S132.

In step S132, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. Then, the process proceeds to step S134. That is, the progress of the first game progresses from the fourth stage to the fifth stage.

In step S134, the CPU 31 acquires image data captured by the inner camera 23. In the following step S136, the CPU 31 detects the skin color region in the acquired image data. In the subsequent step S138, the CPU 31 has a threshold value Th5 of the ratio of the number of pixels detected as "skin color" among the pixels included in the determination area AREA5 shown in FIG. 23 to the total number of pixels included in the determination area AREA5. Judge whether or not it exceeds. If the ratio does not exceed the threshold value Th5 (NO in step S138), the processing of step S134 or lower is repeated. On the other hand, if the ratio exceeds the threshold value Th5 (YES in step S138), the process proceeds to step S140.

In step S140, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. In the subsequent step S142, the CPU 31 switches the image object to be displayed from the image object shown in FIG. 8 to the image object shown in FIG. Then, the process proceeds to step S144. That is, the first game is judged to be "successful", and the progress of the first game ends.

If the series of processes from steps S100 to S142 described above is not completed within a predetermined time, it may be determined that the first game has ended in "failure". In this case, instead of the processing after step S144, a screen indicating that the first game is "failed" is displayed.

In step S144, the CPU 31 reads the captured plurality of image data from the main memory 32 or the storage data memory 34, sequentially switches to the second LCD 22 in a predetermined order, and displays the captured image data. That is, the CPU 31 replays and displays the appearance of the user during game play. The display order of the image data may be the time order in which the images were captured, or may be a random order independent of the time order in which the images were captured.

In the following step S146, the CPU 31 determines whether or not the end has been instructed by the user. If the user has not instructed the end (NO in step S146), the process of step S144 is repeated. On the other hand, when the user instructs the end (YES in step S146), the process for the first game ends.

&lt;Processing content and processing procedure of the second game&gt; As described above, the second game according to the present embodiment includes a plurality of mini games. In each mini-game, the user's playing appearance is captured in parallel with the progress of the game. Then, when a series of mini-games is completed, various objects are superimposed and displayed on the image data captured in each mini-game.

(1) Overall processing FIG. 25 is a flowchart showing a processing procedure of the second game according to the embodiment of the present invention. The flowchart shown in FIG. 25 is executed when the start of the second game is instructed by the user.

With reference to FIG. 25, first, in step S20, the CPU 31 executes a mini game. In this executed mini-game, game progress and image data capture are executed in parallel. In the following step S30, the CPU 31 determines whether or not all the prepared mini-games have been executed. If all the mini-games have not been executed (NO in step S30), the process of step S20 is repeated. That is, the process of step S20 is repeated for the number of mini games prepared in advance. When all the mini-games have been executed (YES in step S30), that is, after the end of the second game, the process shifts to the replay process in step S32 or lower.

In step S32, the CPU 31 reads out one of the image data captured in the previous mini-game. In the following step S34, the CPU 31 reads an object according to the read image data. In the subsequent step S36, the CPU 31 superimposes and displays the object read in the step S34 on the image data read in the step S32. After the elapse of the predetermined time, the process proceeds to step S38.

In step S38, the CPU 31 determines whether or not the termination has been instructed by the user. If the user has not instructed the end (NO in step S38), the process of step S32 is repeated. On the other hand, when the user instructs the end (YES in step S38), the process for the second game ends.

Next, processing contents and processing procedures for providing the first to third mini-games (FIGS. 11 to 13) included in the above-mentioned second game will be described.

(2) 1st mini game In the first mini-game included in the second game, the game is mainly based on the position of the object displayed according to the game effect and the position of the face feature amount detected by the face recognition unit 122 (Fig. 22). To proceed. More specifically, among the face feature quantities recognized by the face recognition unit 122, it is determined whether or not the conditions related to the progress of the game are satisfied based on the center coordinates of each "pupil".

FIG. 26 is a diagram for explaining a game progress condition of the first mini-game included in the second game according to the embodiment of the present invention. With reference to FIG. 26, in the first mini-game, the image data acquired by the inner camera 23 is associated with the center point of the glasses object OBJ10 displayed on the second LCD22 shown in FIG. 11 (a). The reference point RP2 is set. Then, the determination areas DTL and DTR are set symmetrically with the reference point RP2 as the center. These determination areas DTL and DTR are set to correspond to the lens portion of the "glasses" object OBJ10. Although the determination regions DTL and DTR are rectangular in FIG. 26, the determination regions may be circular or elliptical. Further, the degree to which the value of the center coordinate of the recognized "pupil" is included in each of these determination areas DTL and DTR is used as a condition for the progress of the game.

More specifically, in the first mini-game, it is determined whether or not the recognized center coordinate DEP of the "pupil" is included in the determination areas DTL and DTR for each determination cycle, and if it is included, it is determined. A predetermined score is added, and if it is not included, a predetermined score is subtracted. As a result, when the cumulative points exceed a predetermined threshold value, the first mini-game becomes "successful".

In addition, in the first mini-game included in the second game, a plurality of intermediate threshold values are preset between the initial value (typically zero) and the threshold value determined as "success". When the cumulative points exceed these intermediate thresholds for the first time, the user's appearance during play is captured. Typically, five thresholds of Th11, Th12, Th13, Th14, Th15 (however, Th11 &lt;Th12 &lt;Th13 &lt;Th14 &lt;Th15) are preset, and the cumulative value is the first for each threshold. When the image data is captured at the timing exceeding the threshold value and the cumulative value exceeds the threshold value Th5, the first mini-game becomes "successful". Therefore, the captured image data includes the user's appearance at the timing when the cumulative value exceeds the threshold value.

The position of the reference point RP2 and the positions and sizes of the determination areas DTL and DTR are appropriately changed depending on the type and position of the object displayed on the game effect.

FIG. 27 is a flowchart showing a processing procedure of the first mini-game included in the second game according to the embodiment of the present invention.

With reference to FIG. 27, first, in step S200, the CPU 31 initializes the cumulative value S (sets the cumulative value S to zero). In the following step S202, the CPU 31 initializes the threshold value ThS (sets the threshold value ThS to the smallest threshold value Th11).

In step S204, the CPU 31 acquires image data captured by the inner camera 23. In the following step S206, the CPU 31 recognizes the center coordinates of the left and right "pupils" in the acquired image data. Further, in step S208, it is determined whether or not the center coordinates of the recognized left and right "pupils" are included in the determination areas DTL and DTR (FIG. 26). When the center coordinates of the left and right "pupils" are included in the determination areas DTL and DTR (YES in step S208), a constant value α is added to the cumulative value S (step S210). On the other hand, when the center coordinates of the left and right "pupils" are not included in the determination areas DTL and DTR (NO in step S208), the constant value β is subtracted from the cumulative value S (step S212). After that, the process proceeds to step S214.

In step S214, the CPU 31 determines whether the cumulative value S exceeds the threshold ThS. If the cumulative value S does not exceed the threshold value ThS (NO in step S214), the processing of step S202 or lower is repeated. On the other hand, when the cumulative value S exceeds the threshold value ThS (YES in step S214), the process proceeds to step S216.

In step S216, the CPU 31 stores the most recently acquired image data in the main memory 32 or the storage data memory 34. That is, the CPU 31 captures the user's playing appearance. In the following step S218, the CPU 31 determines whether or not the threshold value Th15, which is the largest threshold value, is set in the current threshold value ThS. If the current threshold ThS is not set to the threshold Th15 (NO in step S218), CPU31 is the next largest threshold after the threshold currently set to the threshold ThS. Is set (step S220), and the processing of step S202 and subsequent steps is repeated.

On the other hand, if the current threshold ThS is set to the threshold Th15 (YES in step S218), the first mini-game is judged to be "successful" and the progress of the first mini-game ends. ..

If the series of processes from steps S200 to S220 described above is not completed within a predetermined time, it may be determined that the first mini-game has ended in "failure". Further, it is preferable to set a minimum value (for example, "zero") of the cumulative value S.

(3) Second mini game In the second mini-game included in the second game, it is mainly based on the position of the object displayed according to the game effect and the position and size of the skin color area detected by the skin color area detection unit 126 (FIG. 22). The game progresses.

FIG. 28 is a diagram for explaining a game progress condition of the second mini-game included in the second game according to the embodiment of the present invention. With reference to FIG. 28, in the second mini-game, the image data acquired by the inner camera 23 is associated with the center point of the mosquito object OBJ11 displayed on the second LCD22 shown in FIG. 12 (a). The reference point RP3 is set. A determination area DT31 including the reference point RP3 is set around the reference point RP3, and a determination area DT32 including the reference point RP3 is set in a wider range than the determination area DT31. Further, starting from the reference point RP3, the judgment lines DL1 to DL8 are set in a total of eight directions in the vertical, horizontal, and diagonal directions. Then, among the pixels included in these determination areas DT31 and DT31 and the determination lines DL1 to DL8, the number of pixels (ratio) detected as "skin color" is used as a condition for the progress of the game.

More specifically, in the second mini-game, the game is judged to be "successful" when all of the following conditions (i) to (iii) are satisfied.

(i) The ratio of the number of pixels detected as "skin color" among the pixels included in the determination area DT32 to the total number of pixels included in the determination area DT32 exceeds the predetermined threshold value Th32. (ii) The ratio of the number of pixels detected as "skin color" among the pixels on the judgment lines DL1 to DL8 to the total number of pixels on the judgment lines DL1 to DL8 exceeds the predetermined threshold value Th33. (iii) The ratio of the number of pixels detected as "skin color" among the pixels included in the determination area DT31 to the total number of pixels included in the determination area DT31 does not exceed the predetermined threshold value Th31. The above (i) and (ii) are conditions for determining that the user's "hand" is in the vicinity of the displayed "mosquito" object OBJ11. Further, (iii) is a condition for avoiding erroneous recognition of the user's face or the like as a "hand".

Further, in the second mini-game included in the second game, image data is periodically acquired, and a predetermined number of images are accumulated from the latest one. Then, at the timing when the second mini-game is "successful", the acquisition (update) of the image data is stopped. Therefore, from the timing when the second mini-game is "successful", it is possible to capture the user's playing appearance before a predetermined period.

The position of the above-mentioned reference point RP3 and the sizes of the determination areas DT31 and DT32 are appropriately changed depending on the position and number of objects displayed on the game effect.

FIG. 29 is a flowchart showing a processing procedure of the second mini-game included in the second game according to the embodiment of the present invention.

With reference to FIG. 29, first, in step S240, the CPU 31 acquires image data captured by the inner camera 23. In the following step S242, the CPU 31 determines whether or not a predetermined number of image data has already been stored in the main memory 32 or the storage data memory 34. If the specified number of image data has not been stored (NO in step S242), the process proceeds to step S246. When the specified number of image data has already been stored (YES in step S242), the CPU 31 deletes the oldest acquired image data among the stored image data (step S244) and newly stores the image data. The image data acquired in the above is stored in the main memory 32 or the storage data memory 34 (step S246). Then, the process proceeds to step S248.

In step S248, the CPU 31 detects a skin color region in the acquired image data. In the following step S250, the CPU 31 determines whether or not all of the above-mentioned determination conditions (i) to (iii) are satisfied. If any of the above-mentioned determination conditions (i) to (iii) is not satisfied (NO in step S250), the process of step S240 or lower is repeated. If all of the above judgment conditions (i) to (iii) are not satisfied (YES in step S250), the second mini-game is judged to be "successful", and the progress of the second mini-game ends. ..

(4) 3rd mini game In the third mini-game included in the second game, the game progresses mainly based on the position of the object displayed according to the game effect and the position of the face area detected by the face recognition unit 122 (FIG. 22). Will be done.

FIG. 30 is a diagram for explaining a game progress condition of the third mini-game included in the second game according to the embodiment of the present invention. With reference to FIG. 30, in the third mini-game, it is acquired by the inner camera 23 in association with the cutouts of the totem pole objects OBJ14 and OBJ15 displayed on the second LCD22 shown in FIG. 13 (a). Reference points RP41 and RP42 are set for the image data, respectively. The determination area DT41 including the reference point RP41 is set around the reference point RP41, and the determination area DT42 including the reference point RP42 is set around the reference point RP42. Then, it is determined whether or not the recognized face areas DFA1 and DFA2 are included in these determination areas DT41 and DT42, and if they are included, a predetermined score is added, and if they are not included, a predetermined score is added. Subtract points. As a result, when the cumulative points exceed a predetermined threshold value, the third mini-game becomes "successful".

That is, if it can be determined that the user's face is located in any of the cutouts of the objects OBJ14 and OBJ15 of the "totem pole", the third mini-game is determined to be "successful".

Further, in the third mini-game included in the second game, image data is periodically acquired, and a predetermined number of images are accumulated from the latest one. Then, at the timing when the third mini-game is "successful", the acquisition (update) of the image data is stopped. Therefore, from the timing when the third mini-game is "successful", it is possible to capture the user's playing appearance before a predetermined period.

The positions of the above-mentioned reference points RP41 and RP42 and the sizes of the determination areas DT41 and DT42 are appropriately changed depending on the position and number of objects displayed on the game effect.

Since the processing procedure of the third mini-game included in the second game is the same as the processing procedure of the first mini-game shown in FIG. 27 above, detailed description will not be repeated.

&lt;Processing content and processing procedure of the 3rd game&gt; As described above, the third game according to the present embodiment includes a plurality of mini games executed in series. In each mini-game, at least one image of the user playing at the timing when a predetermined condition is satisfied is captured in parallel with the progress of the game. Then, when a series of mini-games is completed, objects with specific meanings are superimposed and displayed on each of these captured image data, so that a cartoon-like expression according to a prepared scenario can be created. Provided.

In the game device 100 according to the present embodiment, a plurality of scenarios are prepared in advance, and a plurality of mini games are executed in a predetermined order according to one scenario selected for each execution of the third game. To.

In each mini-game, it is determined whether or not the conditions for game progress are satisfied based on the results of the human body recognition process (skin color area detection process and face recognition process). Since these game progress conditions are the same as or similar to the conditions in each mini-game included in the second game described above, detailed description will not be repeated.

FIG. 31 is a flowchart showing a processing procedure of the third game according to the embodiment of the present invention. The flowchart shown in FIG. 31 is executed when the start of the third game is instructed by the user.

With reference to FIG. 31, first, in step S300, the CPU 31 randomly selects one scenario from the plurality of scenarios prepared in advance. In the following step S302, the CPU 31 identifies a plurality of mini-games to be executed according to the scenario selected in the step S300, and determines the execution order thereof. Then, the process proceeds to step S304.

In step S304, the CPU 31 executes the first set mini-game. The CPU 31 advances the mini game based on the result of the human body recognition process (skin color area detection process and face recognition process). In the following step S306, the CPU 31 determines whether or not the first mini-game is "successful". If the first mini-game is "successful" (yes in step S306), the CPU 31 captures the image data from the first mini-game (step S308). On the other hand, if the first mini-game did not "success" (NO in step S306), the CPU 31 sets a flag indicating that the first mini-game ended in "failure". (Step S310).

Hereinafter, the same processing as in steps S304 to S310 is repeatedly executed for each of the second to fifth mini-games. When the series of mini-games is completed, the process proceeds to step S312.

In step S312, the CPU 31 reads and displays the title image corresponding to the selected scenario (see FIG. 19 (a)). In the following step S314, the CPU 31 reads and displays the image data acquired in the first mini-game and the corresponding image object (see FIG. 19 (b)). If a flag indicating that the mini-game has ended in "failure" is set for the corresponding mini-game, the CPU 31 displays an image indicating that the game has failed.

In the following step S316, the CPU 31 reads and displays the image data acquired in the second mini-game and the corresponding image object (see FIG. 19 (c)). In the same manner thereafter, in steps S318 to S322, the CPU 31 sequentially reads and displays the image data and the corresponding image objects acquired in the third to fifth mini-games (FIGS. 20 (a) to (c)). )reference).

In the following step S324, the CPU 31 determines whether or not the user has instructed the termination. If the end is not instructed by the user (NO in step S324), the process of step S312 or lower is repeated. On the other hand, when the user instructs the end (YES in step S324), the process for the third game ends.

As described above, it is possible to display all the plurality of image data acquired by performing imaging when a predetermined condition is satisfied during the game, or a plurality of images captured during the game. The image data of the above may be sequentially acquired, and after the game is over, a plurality of acquired image data satisfying a predetermined condition may be extracted and displayed.

Further, in the above-described embodiment, the amount of information contained in the image data acquired by the imaging unit is used as it is for display, but the amount of information per pixel of the acquired image data is reduced to reduce the image. It may be displayed.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

<figref num="1">It is an external view (open state) of the game apparatus according to the embodiment of this invention.</figref><figref num="2">It is an external view (closed state) of the game apparatus according to the embodiment of this invention.</figref><figref num="3">It is a block diagram which shows an example of the internal structure of the game apparatus according to embodiment of this invention.</figref><figref num="4">It is a figure which shows the form of providing the game program to the game device which follows the embodiment of this invention.</figref><figref num="5">It is a schematic diagram which shows the state of the user who is playing the game apparatus which follows the embodiment of this invention.</figref><figref num="6">It is a figure for temporally explaining an example of the processing content executed by the game apparatus according to the Embodiment of this invention.</figref><figref num="7">It is a figure (the 1) which shows an example of the scene of the 1st game which follows the embodiment of this invention.</figref><figref num="8">It is a figure (the 2) which shows an example of the scene of the 1st game which follows the embodiment of this invention.</figref><figref num="9">It is a figure (the 3) which shows an example of the scene of the 1st game which follows the embodiment of this invention.</figref><figref num="10">It is a figure which shows the display example of the image after the end of the 1st game according to the Embodiment of this invention.</figref><figref num="11">It is a figure (the 1) which shows an example of the scene in the game and after the game of the second game according to the embodiment of this invention.</figref><figref num="12">It is a figure (the 2) which shows an example of the scene in the game and after the game of the second game according to the embodiment of this invention.</figref><figref num="13">It is a figure (the 3) which shows an example of the scene in the game and after the game of the second game according to the embodiment of this invention.</figref><figref num="14">It is a figure (the 1) which shows an example of the scene of the 3rd game according to the embodiment of this invention.</figref><figref num="15">It is a figure (the 2) which shows an example of the scene of the 3rd game according to the embodiment of this invention.</figref><figref num="16">It is a figure (the 3) which shows an example of the scene of the 3rd game according to the embodiment of this invention.</figref><figref num="17">It is a figure (the 4) which shows an example of the scene of the 3rd game according to the embodiment of this invention.</figref><figref num="18">It is a figure (the 5) which shows an example of the scene of the 3rd game according to the embodiment of this invention.</figref><figref num="19">It is a figure (the 1) which shows the display example of the image after the end of the 3rd game according to the Embodiment of this invention.</figref><figref num="20">It is a figure (the 1) which shows the display example of the image after the end of the 3rd game according to the Embodiment of this invention.</figref><figref num="21">It is a block diagram which shows the control structure of the game apparatus which follows this embodiment.</figref><figref num="22">It is a figure which conceptually shows the processing in the rendering engine shown in FIG.</figref><figref num="23">It is a figure for demonstrating the condition of the game progress of the 1st game according to the embodiment of this invention.</figref><figref num="24">It is a flowchart which shows the processing procedure of the 1st game according to embodiment of this invention.</figref><figref num="25">It is a flowchart which shows the processing procedure of the 2nd game according to embodiment of this invention.</figref><figref num="26">It is a figure for demonstrating the condition of the game progress of the 1st mini-game included in the 2nd game according to the embodiment of this invention.</figref><figref num="27">It is a flowchart which shows the processing procedure of the 1st mini-game included in the 2nd game which follows embodiment of this invention.</figref><figref num="28">It is a figure for demonstrating the condition of the game progress of the 2nd mini-game included in the 2nd game according to the embodiment of this invention.</figref><figref num="29">It is a flowchart which shows the processing procedure of the 2nd mini-game included in the 2nd game which follows embodiment of this invention.</figref><figref num="30">It is a figure for demonstrating the condition of the game progress of the 3rd mini-game included in the 2nd game according to the embodiment of this invention.</figref><figref num="31">It is a flowchart which shows the processing procedure of the 3rd game according to embodiment of this invention.</figref>

Code description

11 1st housing, 12 1st LCD, 13 touch panel, 14A ~ 14K button, 15A 1st LED, 15B 2nd LED, 15C 3rd LED, 16 microphone holes, 21 2nd housing, 22 2nd LCD, 23 inner camera, 24 sound removal Hole, 25 outside camera, 26 4th LED, 27 touch pen, 28,29 memory card, 32 main memory, 33 memory control circuit, 34 storage data memory, 35 preset data memory, 36,37 memory card I / F, 38 Wireless communication module, 39 local communication module, 41 power supply circuit, 42 interface circuit (I / F circuit), 43 microphone, 44 amplifier, 45 speaker, 100 game device, 102 voice data buffer, 104 voice recognition unit, 110 judgment unit, 110a condition group, 112 image data buffer, 114 capture controller, 116 image storage unit, 120 human body recognition unit, 122 face recognition unit, 124 color coordinate conversion unit, 126 skin color area detection unit, 130 Game image controller, 132 game image DB, 134 recognition result reflection unit, 136 replay controller, 138 replay image DB, 140 rendering engine, AP access point, NW network, SRV distribution server device.