Program, information storage medium, and image pickup and display device

The present invention causes a computer to control the display of a finder image captured by an imaging means on a display device, and synthesizes a character image with the captured image captured by performing a shutter operation to control the display. The present invention relates to a program or the like for performing display control as if a character exists in space.

Image recognition technology that recognizes the contents of captured images by performing image recognition is used in various fields. For example, as an application example of the image recognition technology, Patent Document 1 discloses a game device for playing a card game and a fortune-telling device for performing tarot fortune-telling. In such a device, various cards such as playing cards and tarot cards and recognition objects such as dice are imaged by using a camera, and the arrangement and the like are recognized by a pattern recognition process for performing image recognition on the captured image.<patcit num="1"><text>Japanese Unexamined Patent Publication No. 2002-49909</text></patcit>

<p> The technology of Patent Document 1 described above defines the shape and contents of the recognition object in advance, but the recognition target is not particularly limited, and a new interest in imaging is realized by using the image recognition technology. Can't you? The present invention has been made in view of the above circumstances.</p>

<p> The first invention for solving the above problems is to display a finder image captured by an image pickup means (for example, the camera unit 7 in FIG. 1 and the image pickup unit 200 in FIG. 7) on a computer (for example, FIG. 1). Is there a character in the real space by controlling the display on the display 1 and the image display unit 410) in FIG. 7 and controlling the display by synthesizing the image of the character with the captured image captured by the shutter operation? It is a program for performing display control such as, and has a finder image feature analysis means that captures a finder image when a predetermined condition generation instruction operation is input and analyzes a feature portion of the captured image. Condition generation storage that generates and stores a composite image condition, which is an image condition for determining whether or not to synthesize a character image with an captured image, based on a feature portion analyzed by the finder image feature analysis means. Means (for example, firewood feeding execution unit 320 and storage unit 500 in FIG. 7, steps S3 to S11 in FIG. 16), It has an image pickup image feature analysis means for analyzing a feature portion of an image captured by the shutter operation when a shutter operation is performed, and the feature portion analyzed by the image pickup image feature analysis means is the condition generation storage means. Judgment means for determining whether or not the composite image conditions memorized by (for example, imaging execution unit 330 in FIG. 7 and steps S15 to S19 in FIG. 16) are satisfied, and when it is determined that the determination means is satisfied, A program for operating the computer as a display control means (for example, imaging execution unit 330 in FIG. 7 and step T113 in FIG. 17) that synthesizes and displays a character image with an image captured by the shutter operation (for example). For example, the game program 510) in FIG.</p><p> Further, the tenth invention relates to an image pickup means (for example, the camera unit 7 in FIG. 1 and the image pickup unit 200 in FIG. 7) and a display device (for example, FIG. 1) for displaying a finder image captured by the image pickup means. It is equipped with a display 1 and an image display unit 410) in FIG. 7, and by synthesizing the image of the character with the captured image captured by the shutter operation and controlling the display, it is as if the character exists in the real space. An image imaging display device (for example, the portable game device 1000 shown in FIGS. 1 and 7) that performs various display controls, captures a finder image when a predetermined condition generation instruction operation is input, and features the captured image. It has a finder image feature analysis means for analyzing a part, and based on the feature part analyzed by the finder image feature analysis means, it is an image condition for determining whether or not to synthesize a character image with an captured image. A condition generation storage means for generating and storing a certain composite image condition, It has an image pickup image feature analysis means for analyzing a feature portion of an image captured by the shutter operation when a shutter operation is performed, and the feature portion analyzed by the image pickup image feature analysis means is the condition generation storage means. A determination means for determining whether or not the composite image condition stored in the above is satisfied, and a determination means for determining whether or not the composite image condition is satisfied, and when the determination means is determined to be satisfied, the image of the character is combined with the captured image captured by the shutter operation and displayed. It is an image imaging display device including a display control means for the operation.</p><p> According to the first or tenth invention, a finder image is captured when a predetermined condition generation instruction operation is input, and a composite image condition is generated and stored based on the analyzed feature portion of the captured image. Will be done. Further, when the shutter operation is performed, it is determined whether or not the analyzed feature portion of the captured image captured by the shutter operation satisfies the stored composite image condition, and it is determined that the shutter operation is satisfied. In addition, the image of the character is combined with this captured image and displayed.</p><p> That is, if the captured image captured by the shutter operation is substantially the same as the finder image when the condition generation instruction operation is input, it is determined that the analyzed feature portion of this captured image satisfies the stored composite image condition. Will be done. Therefore, when a condition generation instruction operation is input while a certain place or subject is being photographed and then the place or subject is imaged, the image of the character is compositely displayed on the captured image, as if the character exists in the real space. It is possible to realize a new interest in imaging as if the image was taken. Further, by inputting the compositing instruction operation in advance, when the desired place or the subject is imaged, the character image can be combined and displayed on the captured image.</p><p> Further, as a second invention, the program of the first invention attracts a character by synthesizing a given attracting object with the finder image and controlling the display when the condition generation instruction operation is input. It may be a program for operating the computer as an attraction effect processing means (for example, the firewood feeding execution unit 320 in FIG. 7 and step S13 in FIG. 16) that performs the effect processing as if it were performed.</p><p> According to the second invention, when a condition generation instruction operation is input, a given attraction object is combined with a finder image and display control is performed to perform an effect process as if the character is attracted. It is said. Therefore, as an object, for example, by sprinkling the food that the character likes to the place where the character is shooting or the position of the subject when the condition generation instruction operation is input, a game-like effect such as attracting the character is realized. it can.</p><p> Further, as a third invention, the finder image feature analysis means analyzes the shape of an object shown in the finder image and extracts the program of the first or second invention as a feature portion. (For example, the sowing feeding execution unit 320 in FIG. 7, step S9 in FIG. 16), and the condition generation storage means sets the shape condition based on the shape of the object analyzed by the finder image shape analysis means. It has a shape condition generating means (for example, a firewood feeding execution unit 320 in FIG. 7 and step S11 in FIG. 16) to generate as a composite image condition, and the captured image feature analysis means captures the shape of an object in the captured image. An object having an image pickup image shape analysis means (for example, an image pickup execution unit 330 in FIG. 7 and step S17 in FIG. 16) that is analyzed and extracted as a feature portion, and the determination means is analyzed by the image pickup image shape analysis means. It has a shape determining means (for example, an image capturing execution unit 330 in FIG. 7 and step S19 in FIG. 16) for determining whether or not the characteristic portion of the shape satisfies the shape condition generated by the shape condition generating means. It may be a program for operating the computer.</p><p> According to the third invention, the shape of the object shown in the finder image is analyzed and extracted as a feature portion, and a shape condition based on the analyzed shape of the object is generated as a composite image condition. Then, the shape of the object shown in the captured image is analyzed and extracted as a feature portion, and it is determined whether or not the feature portion of the analyzed object shape satisfies the generated shape condition. That is, it is determined whether or not the character image is compositely displayed on the captured image depending on whether or not the captured object shape satisfies the shape condition of the captured object when the condition generation instruction operation is input. To.</p><p> Further, as the fourth invention, the program of any one of the first to third inventions is provided with the condition generation instruction operation, which includes a character instruction operation for instructing a character to be combined with the captured image, and the condition generation storage. The means stores the composite image condition generated when the condition generation instruction operation is input in association with the character instructed by the character instruction operation included in the condition generation instruction operation, and the display control means causes the display control means. The program may be used to make the computer function so that the image of the character associated with the composite image condition determined to be satisfied by the determination means is combined with the captured image and the display is controlled.</p><p> According to the fourth invention, the condition generation instruction operation includes a character instruction operation for instructing a character to be combined with the captured image, and the composite image condition generated when the condition generation instruction operation is input is a composite image condition. It is stored in association with the character instructed by the character instruction operation included in this condition generation instruction operation. Then, the image of the character associated with the composite image condition determined that the feature portion of the captured image is satisfied is combined with the captured image and the display is controlled. Therefore, by inputting the character instruction operation for instructing the character together with the condition generation instruction operation, when the subject or place captured at the time of inputting the condition generation instruction operation is imaged, the desired character is added to the captured image. It can be displayed compositely.</p><p> Further, as a fifth invention, the computer is stored so that the program of any of the first to fourth inventions is stored in association with the generated composite image condition by the condition generation storage means. It is made to function, and it is determined whether or not the elapsed time from the generation time associated with the composite image condition determined to be satisfied by the determination means to the time when the shutter operation is performed satisfies the given time condition. The computer is made to function as an elapsed time determining means (for example, ghost appearance determining unit 334 in FIG. 7, steps T9 to T11 in FIG. 17), and it is determined that the display controlling means is satisfied by the determining means, and the elapsed time is determined. When it is determined by the time determination means that the computer is satisfied, the program may be used to make the computer function so as to synthesize the image of the character with the captured image and control the display.</p><p> According to the fifth invention, the generated composite image condition is stored in association with the generation time. Then, it is determined that the characteristic portion of the captured image satisfies the composite image condition, and that the elapsed time from the generation time corresponding to the composite image condition to the time when the shutter operation is performed satisfies the given time condition. When this is done, the character image is combined with the captured image and the display is controlled.</p><p> That is, even when the place or subject that was photographed when the condition generation instruction operation is input is imaged, the elapsed time from the input of the condition generation instruction operation to the input of the shutter operation does not satisfy the time condition. The character image is not compositely displayed on the captured image. Therefore, for example, by assuming that the time condition is satisfied when a predetermined time elapses from the input of the condition generation instruction operation, it is possible to realize an effect as if the character is moving to the place or the subject.</p><p> Further, as the sixth invention, it was determined that the program of the fifth invention was satisfied by the determination means, but it was determined that the given time condition was not satisfied because the elapsed time was long. In this case, the trace effect processing means (for example, the ghost appearance determination unit 334 of FIG. 7) that performs the effect processing as if the character existed by synthesizing the given trace object with the captured image and controlling the display. It may be a program for operating the computer as step T15) of 17.</p><p> According to the sixth invention, when it is determined that the characteristic portion of the captured image satisfies the composite image condition, but it is determined that the characteristic portion of the captured image does not satisfy the time condition due to the long elapsed time, it is given. By synthesizing the trace object with the captured image and controlling the display, the effect processing as if the character existed is performed. That is, it is possible to realize an effect as if the character had already left by the trace object indicating that the character existed.</p><p> Further, as a seventh invention, the program of the fifth or sixth invention is displayed and controlled by synthesizing a given attracting object with the finder image when the condition generation instruction operation is input. An attraction effect processing means (for example, the firewood feeding execution unit 320 in FIG. 7 and step S13 in FIG. 16) that performs an effect process as if to attract a character, and the determination means determined to be satisfied, but the elapsed time When it is determined that the given time condition is not satisfied due to the short period of time, the attracting object is combined with the captured image and the display is controlled to produce an effect as if the character has not yet arrived. It may be a program for operating the computer as an unarriving effect processing means (for example, the image capturing execution unit 330 in FIG. 7 and step T15 in FIG. 17).</p><p> According to the seventh invention, when a condition generation instruction operation is input, a given attraction object is compositely displayed on the finder image to perform an effect process as if the character is attracted. Further, when it is determined that the characteristic portion of the captured image satisfies the composite image condition, but it is determined that the time condition is not satisfied because the elapsed time is short, this attracting object is compositely displayed on the captured image. As a result, the effect processing is performed as if the character has not arrived yet.</p><p> Further, as an eighth invention, the finder image determining means, the finder, which analyzes a feature portion of a finder image and determines whether or not the analyzed feature portion satisfies the composite image condition stored by the condition generation storage means. To make the computer function as a notification control means that performs a predetermined notification process for notifying whether or not a character image is combined with an captured image based on a determination result by the image determination means as whether or not a character exists. It may be the program described in.</p><p> According to the eighth invention, it is determined whether or not the analyzed feature portion of the finder image satisfies the stored composite image condition, and based on this determination result, whether the character image is combined with the captured image. A predetermined notification process for notifying whether or not the character exists is performed. That is, based on the finder image, whether or not the character image is combined with the captured image is notified as whether or not the character exists. Therefore, for example, even if you remember the place or subject you were shooting when you entered the condition generation instruction operation, you can reliably combine the character image with the captured image by inputting the shutter operation with reference to the notification content. It becomes possible to display it.</p><p> Further, the ninth invention is a computer-readable information storage medium (for example, the storage unit 500 of FIG. 7) that stores the program of any one of the first to eighth inventions.</p><p> The "information storage medium" here is a storage medium such as a hard disk, MO, CD-ROM, DVD, memory card, or IC memory in which a computer can read the stored information. Therefore, according to the ninth invention, by causing the computer to read the information stored in the information storage medium and execute the arithmetic processing, the same effect as that of the first to eighth inventions can be obtained. Can play.</p>

<p> According to the present invention, when a condition generation instruction operation is input while a certain place or subject is being photographed and then the place or subject is imaged, the character image is compositely displayed on the captured image, as if in real space. It is possible to realize a new interest in imaging as if an existing character was imaged. Further, by inputting the compositing instruction operation in advance, when the desired place or the subject is imaged, the character image can be combined and displayed on the captured image.</p>

Hereinafter, embodiments suitable for the present invention will be described with reference to the drawings. In the following, a case where the present invention is applied to a portable game device will be described, but the application of the present invention is not limited to this.

[appearance]

FIG. 1 is an external view showing an example of the portable game device 1000 according to the present embodiment. FIG. 6A shows a front view of the portable game device 1000, and FIG. 6B shows a rear view. According to the figure, the portable game device 1000 includes a display 1, a speaker 3, and an operation key 5 in front of a substantially rectangular housing. Further, the camera unit 7 is provided on the upper side surface of the housing, and an insertion slot for inserting the game cartridge 10 is provided on the lower side surface.

The display 1 is a backlit color liquid crystal display such as a TFT display, and displays various information, images, and the like. Further, the display 1 functions as an electronic viewfinder when the camera unit 7 operates, and displays a finder image (an image captured by the photographing lens 8) by the camera unit 7.

The operation key 5 is for the player (user) to input the game operation, the shutter key 5a for inputting the shutter operation, the firewood food key 5b for inputting the food selection operation and the firewood food execution operation, and the cross key. 5c is included.

The camera unit 7 has a photographing lens 8 formed of glass or the like provided on the back side thereof and an image pickup element such as CMOS or CCD, and digitally digitally images image data formed by the image pickup element through the image pickup lens 8. Convert to data and output. Although the camera unit 7 is integrally configured in the housing, it may be detachably configured.

Further, the portable game device 1000 includes a reading device for reading game information (game program and game data) stored in the game cartridge 10 inserted into the insertion slot, and an external device such as another portable game device 1000. It is equipped with a communication device for connecting and performing wireless / wired communication, and a control device equipped with a CPU, IC memory, etc. The CPU mounted on the control device executes various processes based on the programs and data read from the IC memory and the game cartridge 10, the operation signals of the operation keys 5, the image data input from the camera unit 7, and the like. , Generates image signals of game screens and sound signals of game sounds. Then, the generated image signal is output to the display 1 to display the game screen, and the sound signal is output to the speaker 3 to output the game sound.

Information (system program, game program, game data, etc.) necessary for the portable game device 1000 to execute a game is stored in an IC memory or a game cartridge 10 mounted on the control device. More specifically, the system program is stored in the IC memory of the control device, and the game program and the game data are stored in the game cartridge 10. Therefore, the player can cause the portable game device 1000 to execute a different game by exchanging the game cartridge 10.

[Overview]

In the game of the present embodiment, the game is produced as if a ghost (character) exists in the space around the player, which is a real space.

2 to 6 are diagrams for explaining the outline of the game of the present embodiment. As shown in FIG. 2A, when the camera unit 7 of the portable game device 1000 is activated, the image (finder image) captured by the photographing lens 8 is displayed on the display 1. When the shutter operation is performed in this state (shutter ON), the ghost image is combined with the captured image captured by the shutter operation and displayed on the display 1. That is, a ghost "appears". The shutter operation can be input by operating the shutter key 5a.

FIG. 3A shows a case where the shutter is turned on while the photographing lens 8 is capturing the subject OB1, that is, a case where the subject OB1 is imaged, and the ghost GS1 appears. Then, the ghost can be "captured" by performing a predetermined capture process on the ghost that has appeared. It should be noted that the ghost does not always appear when the shutter is turned on, and it is possible that the ghost does not appear.

At this time, in the portable game device 1000, when the shutter operation is performed, a predetermined image recognition process for analyzing the feature portion of the captured image is performed, and the feature data obtained as a result is prepared in advance. One of the plurality of ghosts is selected, combined with the captured image, and displayed on the display 1.

Which of the multiple ghosts appears depends on the subject. That is, as shown in FIG. 6B, when a subject OB2 different from the subject OB1 is imaged, a ghost GS2 different from the ghost GS1 appears. Further, as shown in FIG. 3, when the same subject OB1 is imaged at a place different from that in FIG. 2 (a), the same ghost GS1 appears.

Further, in the present embodiment, it is possible to sprinkle "food" which is an attracting object in the real space, and perform "maki food" as if to attract (attract) the ghost by this food. That is, as shown in FIG. 4A, when the food selection operation is performed with the camera unit 7 activated, the display 1 displays a list LS of the types of food that can be sown. If you move the cursor CS on this list LS to select the type of food and then perform the sowing food execution operation, the image corresponding to the selected type of food will be combined with the finder image and displayed. That is, the food is "sown".

FIG. 3A shows a case where the feeding lens 8 is capturing the subject OB3, that is, the subject OB3 is being photographed and the food A is sown. The food selection operation can be input by operating the firewood food key 5b, and the movement operation of the cursor CS can be input by operating the cross key 5c. In addition, the firewood feeding execution operation can be input by operating the firewood feeding key 5b again.

Then, as shown in FIG. 6B, when the same subject is imaged after a predetermined time has elapsed from the input of the sowing food execution operation, the ghost image is combined with the captured image and displayed on the display 1. To. That is, a ghost appears. Figure (b) shows the case where the subject OB3 is imaged, and the ghost GS3 appears.

The ghost that appears at this time is determined by the type of food to be sown, and does not depend on the subject being photographed. That is, as shown in FIG. 5 (a), the food selection operation is performed while the subject OB3 is being photographed as in the case of FIG. 4 (a), and the displayed food type list LS cursor CS Move to select food B and perform the sowing food execution operation. Then, the image of the food B different from the food A is combined with the finder image and displayed. That is, food B is sown. Then, as shown in FIG. 5B, when the subject OB3 is imaged after a lapse of a predetermined time, a ghost GS4 different from the ghost GS3 is displayed.

In other words, the type of bait that appears determines the ghost that appears when the same subject is imaged. That is, by specifying the type of bait to sow, it is possible to specify the type of ghost that appears. Therefore, the player can select and sow the type of food according to the ghost that he / she wants to appear (attract).

In addition, there is a time condition for the appearance of ghosts due to the execution of firewood food. That is, the ghost appears only when the elapsed time from the execution of the sowing bait is within the predetermined time range and the same subject as the subject taken at the time of inputting the sowing bait execution operation is imaged. In the case of, the ghost does not appear.

For example, as shown in FIG. 4, when the elapsed time after feeding the food A (from the execution of the feeding of the food A) is within a predetermined time range (for example, after 10 minutes have passed and before 20 minutes have passed). , When the same subject OB3 is imaged, the ghost GS1 corresponding to the food A is displayed.

However, as shown in FIG. 6A, when the elapsed time from the feeding execution of the feed A does not reach the above-mentioned predetermined time range (for example, the elapsed time is 5 minutes), the same subject OB3 is imaged. , The image of the complete food A immediately after being sown as shown in Fig. 4 (a) is displayed, and no ghost appears. That is, the effect is made as if the ghost has not arrived yet.

Further, as shown in FIG. 6 (b), when the elapsed time from the feeding execution of the feed A exceeds the above-mentioned predetermined time range (for example, the elapsed time is 30 minutes), the same subject OB3 is imaged. The ghost does not appear, and an image of the wreckage of food A, which is a trace object representing the trace that the ghost existed, is displayed. That is, the effect is as if the ghost attracted to food A had already left.

[Functional configuration]

FIG. 7 is a block diagram showing an example of the functional configuration of the portable game device 1000. According to the figure, functionally, the portable game device 1000 includes an operation input unit 100, an imaging unit 200, a processing unit 300, an image display unit 410, a sound output unit 420, and a storage unit 500. , The communication unit 600 is provided.

The operation input unit 100 receives an operation instruction from the player and outputs an operation signal corresponding to the operation to the processing unit 300. In particular, in the present embodiment, the input of the feed selection operation, the firewood feed execution operation, and the shutter operation is detected and output to the processing unit 300. In FIG. 1, the operation key 5 corresponds to this.

The image sensor 200 is an image sensor such as CMOS or CCD, a photographing lens that forms an image of a subject's light beam on the image sensor, and an A / D conversion that converts an electric signal (analog image signal) output from the image sensor into a digital signal. The circuit is included, and the captured image data of the subject is output to the processing unit 300. In FIG. 1, the camera unit 7 corresponds to this.

The processing unit 300 performs various arithmetic processes such as overall control of the portable game device 1000, game progress control, and image generation. This function is realized by, for example, an arithmetic unit such as a CPU (CISC type, RISC type), an ASIC (gate array, etc.) and its control program. In FIG. 1, the CPU and the like mounted on the built-in control device correspond to this.

Further, the processing unit 300 includes a game calculation unit 310 that mainly performs arithmetic processing related to game execution, and an image generation unit 340 that generates a game image based on various data obtained by the processing of the game calculation unit 310. It includes a sound generation unit 350 that generates game sounds such as sound effects and BGM, and a time measurement unit 360 that measures the time.

The game calculation unit 310 performs various game processing based on the game information (program and data) read from the storage unit 500, the operation signal input from the operation input unit 100, the image data input from the imaging unit 200, and the like. Run. Further, in the present embodiment, the game calculation unit 310 includes a firewood feeding execution unit 320 and an imaging execution unit 330.

The firewood feeding execution unit 320 executes a predetermined firewood feeding process. Specifically, when the food selection operation is input from the operation input unit 100, the food types prepared in advance are displayed in a list with reference to the food information 522.

Food information 522 is information about food that can be sown. FIG. 8 shows an example of the data structure of the food information 522. According to the figure, the food information 522 stores the name (food name) 522a, the ghost name 522b, and the appearance period 522c in association with each other for a plurality of types of food. The ghost name 522b is the name of the ghost that can appear when the corresponding food is sown. The appearance period 522c is a period during which a ghost can appear when the corresponding food is sown, and is set as a time range based on the time when the food is sown (the execution time of the sowing food).

After that, when the firewood feed execution operation is input from the operation input unit 100, the firewood food execution unit 320 captures the finder image at the time of input of this execution operation, and analyzes the feature portion of the captured image. Perform processing to extract feature data.

There are various image recognition methods performed here, but in the present embodiment, feature data is extracted for each image recognition method by using any one or more of shape recognition, color recognition, and pattern recognition. Specifically, in shape recognition, a Hough transform is performed on the target image data to detect the linear component and its strength, the shape of the subject is recognized by the combination of the detected straight lines, and the feature based on the recognized shape is recognized. Extract the part. Further, in the color recognition, the color of the entire image or the feature portion recognized by the shape recognition is recognized. Further, in pattern recognition, for example, a Fourier transform is performed on the target image data to extract a frequency component, and the pattern of the entire image or a feature portion recognized by shape recognition is recognized.

The image recognition method performed by the firewood feeding execution unit 320 is specified by the image recognition method information 526. FIG. 9 shows an example of the data structure of the image recognition method information 526. According to the figure, the image recognition method information 526 stores the image recognition method 526a and the weight 526b in association with each other. The weight 526b is a value indicating the importance of the corresponding image recognition method, and based on this value, the feature data of the captured image, the feature data of the finder image captured at the time of feeding the firewood, and the feature data prepared in advance are prepared as described later. The total value of the similarity with the characteristic data of each ghost is calculated.

When the feature data of the imaged data was extracted, the firewood feed execution unit 320 selected the extracted feature data at the input time of the firewood feed execution operation and the type of the sowed food (at the time of inputting the firewood feed execution operation). It is added to the sowing food execution information 532 in association with the type of food).

The firewood food execution information 532 is information related to the execution contents of the firewood food. FIG. 10 shows an example of the data structure of the firewood feed execution information 532. According to the figure, the firewood food execution information 532 stores the execution time 532b, the food name 532c, and the feature data 532d in association with each different execution No. 532a, and one execution No. 532a The data corresponding to is the data related to the execution of one sowing bait. The execution time 532b is the time at the time when the firewood feed execution operation is input. The food name 532c is the type of food sown. The feature data 532d is feature data extracted from the finder image when the sowing feed execution operation is input.

Further, when the sowing food execution operation is input, the sowing food execution unit 320 synthesizes and displays the image of the food corresponding to the selected type of food as the finder image for a predetermined time. As a result, it is possible to produce an effect as if the food was sown in the place (real space) where the image is being taken. Images of these foods are prepared for each type of food and stored in the food image information 523.

The imaging execution unit 330 includes a similarity calculation unit 332 and a ghost appearance determination unit 334, and executes a predetermined imaging process. Specifically, when a shutter operation is input from the operation input unit 100, a predetermined image recognition process for analyzing the feature portion of the captured image by the shutter operation is performed, and the feature data is extracted. Here, the image recognition method performed by the image pickup execution unit 330 is specified by the image recognition method information 526 shown in FIG. 9 as an example.

Further, the feature data extracted by the imaging execution unit 330 is stored in the captured image recognition result information 534. FIG. 11 shows an example of the data structure of the captured image recognition result information 534. According to the figure, the captured image recognition result information 534 stores the image recognition method 534a and the feature data 534b calculated by the image recognition method in association with each other. The image recognition method 534a is a method specified by the image recognition method information 526. For example, in the same figure, the extracted feature data for shape recognition and color recognition specified in the image recognition method information 526 shown in FIG. 9 is stored.

The similarity calculation unit 332 calculates the similarity between the feature data of the extracted captured image and the feature data of the finder image captured at the time of executing the sowing feed and the feature data of each ghost prepared in advance.

Specifically, the degree of similarity between the feature data of each execution number and the feature data of the extracted captured image is calculated with reference to the firewood feed execution information 532. That is, for each execution No., the degree of similarity between the corresponding feature data and the extracted feature data is calculated for each image recognition method. Then, with reference to the image recognition method information 526, each calculated similarity is multiplied by a weight corresponding to the image recognition method and added to calculate the total value of the similarity.

The similarity calculated here is stored in the calculated similarity information 536 for sowing food. FIG. 12 shows an example of the data structure of the calculated similarity information 536 for sowing food. According to the figure, the calculated similarity information 536 for the firewood bait stores the calculated similarity 536b for each execution No. 536a stored in the firewood food execution information 532. The similarity degree 536b includes the similarity degree for each image recognition method specified in the image recognition method information 526 and the total value of these similarity degrees. For example, in the same figure, the similarity degree for shape recognition and color recognition specified in the image recognition method information 526 shown in FIG. 9 and the total value of the similarity degree are stored.

Further, the similarity calculation unit 332 refers to the ghost information 524 and calculates the similarity between the feature data of each ghost and the feature data of the extracted captured image.

Ghost information 524 is information about ghosts prepared in advance. FIG. 13 shows an example of the data structure of the ghost information 524. According to the figure, the ghost information 524 stores the name (ghost name) 524a and the feature data 524b in association with each other for a plurality of ghosts. The feature data 524b stores a value corresponding to each image recognition method specified in the image recognition method information 526. For example, in the figure, the values of "shape" and "color" are set corresponding to "shape recognition" and "color recognition" specified in the image recognition method information 526 shown in FIG.

The similarity calculation unit 332 calculates the similarity between the corresponding feature data and the extracted feature data for each image recognition method for each ghost stored in the ghost information 524. Then, with reference to the image recognition method information 526, each calculated similarity is multiplied by a weight corresponding to the image recognition method and added to calculate the total value of the similarity.

The similarity calculated here is stored in the calculation similarity information 537 for normal use. FIG. 14 shows an example of the data structure of the calculation similarity information 537 for normal use. According to the figure, the calculated similarity information 537 for normal use stores the calculated similarity 537b in association with each ghost 537a specified in the ghost information 524. Similarity 537b includes the similarity for each image recognition method specified in the image recognition method information 526 and the total value of these similarities, as in the calculated similarity information 536 for sowing bait. For example, in the same figure, the similarity degree for shape recognition and color recognition specified in the image recognition method information 526 shown in FIG. 9 and the total value of the similarity degree are stored.

The similarity calculated by the similarity calculation unit 332 is represented by a decimal value in the range of "0.0" to "1.0", the minimum value "0.0" indicates a complete mismatch, and the maximum value "1.0" is complete. Represents a match. Further, since the method for calculating the degree of similarity is a known method using, for example, pattern matching, detailed description thereof will be omitted.

The ghost appearance determination unit 334 determines the appearance of a ghost based on the similarity calculated by the similarity calculation unit 332. Specifically, first, the appearance of a ghost for the fired food that has been executed so far is determined. That is, the execution No. having the maximum total similarity value is selected with reference to the calculated similarity information 536 for sowing bait. Then, it is determined whether or not the total value of the similarity of the selected execution numbers is equal to or higher than the predetermined appearance threshold value for sowing bait.

The appearance threshold value for sowing food is a threshold value that serves as a criterion for determining whether or not a ghost appears for the execution of sowing food, and is set to a value close to "1.0" indicating an exact match. That is, when it is equal to or higher than the appearance threshold value, the feature data of the captured image almost matches the feature data of the finder image when the sowing feed was executed in the past, that is, the place imaged this time executed the sowing food in the past. It corresponds to the case where it can be regarded as a place (where food is sown). Therefore, in this case, it is determined that a ghost appears.

The value of the appearance threshold value for this sowing bait is stored in the appearance threshold value information 528. FIG. 15 shows an example of the data structure of the appearance threshold information 528. According to the figure, the appearance threshold information 528 stores the appearance threshold 528a for sowing food and the appearance threshold 528b for normal use. As will be described later, the normal appearance threshold value 528b is a threshold value that serves as a criterion for determining whether or not a ghost (stored in the ghost information 524) prepared in advance is to appear. In addition, the appearance threshold value 528b for normal use is set lower than the appearance threshold value 528a for sowing food.

Then, when the ghost appearance determination unit 334 determines that the total value of the similarity of the selected execution numbers is equal to or higher than the appearance threshold value for the sowing bait, the ghost appearance determination unit 334 refers to the sowing bait execution information 532 and performs this selected execution. Calculate the elapsed time from the No. sowing feed execution time to the input time of the shutter operation. Next, with reference to the food information 522, the appearance period corresponding to the type of food of the selected execution number is determined, and it is determined whether or not the calculated elapsed time is within the determined appearance period.

If it is within the appearance period, it is determined that a ghost corresponding to the type of food of the selected execution No. appears by referring to the food information 522, and the image of the ghost is combined with the captured image to display the image. Display on 410. If the elapsed time is not within the determined appearance period, an image corresponding to the elapsed time before and after the appearance period is combined with the captured image and displayed on the image display unit 410. That is, if the elapsed time has not reached the appearance period, the complete image of the food of the selected execution number is combined with the captured image and displayed. On the other hand, if the elapsed time exceeds the appearance period, the image of the food debris of the executed execution No. is combined with the captured image and displayed. The ghost image is stored in the ghost image information 525, and the complete image of the food and the image of the wreckage are stored in the food image information 523.

In addition, if the total value of the similarity of the execution No. selected from the calculated similarity information 536 for the sowing bait is not equal to or more than the appearance threshold for the sowing bait, the ghost appearance determination unit 334 uses the sowing bait in the past. It is assumed that it is not the place where it was executed, and the process shifts to the process of determining the appearance of the ghost prepared in advance.

That is, the ghost having the maximum total similarity value is selected with reference to the normal calculation similarity information 537. Then, it is determined whether or not the total value of the similarity of the selected ghosts is equal to or higher than the predetermined normal appearance threshold value, and if it is equal to or higher than the normal appearance threshold value, it is determined that the selected ghost appears. , The ghost image is combined with the captured image and displayed on the image display unit 410. If it is not equal to or higher than the normal appearance threshold, it is determined that the ghost does not appear. Here, the normal appearance threshold is stored in the appearance threshold information 528. The ghost image is stored in the ghost image information 525.

The image display unit 410 displays a game image based on the image signal from the image generation unit 340. In particular, in the present embodiment, the finder image by the imaging unit 200 is displayed on the monitor in real time, and the image of the food sown by the input of the sowing food execution operation is combined with the finder image and displayed, and it is determined that the image appears. The image of the ghost is superimposed on the captured image and displayed. This function is realized by hardware such as CRT, LCD, and ELD. In FIG. 1, display 1 corresponds to this.

The sound generation unit 350 generates game sounds such as sound effects and BGM used during the game, and outputs the sound signal of the generated game sound to the sound output unit 420. The sound output unit 420 outputs game sounds such as BGM and sound effects based on the sound signal from the sound generation unit 350. This function is realized by, for example, a speaker or the like. In FIG. 1, the speaker 3 corresponds to this.

The storage unit 500 stores system programs for realizing various functions for the processing unit 300 to control the portable game device 1000 in an integrated manner, programs and data necessary for executing the game, and the like. It is used as a work area of the processing unit 300, and temporarily stores the calculation result executed by the processing unit 300 according to various programs, the input data input from the operation input unit 100, and the like. This function is realized by, for example, various IC memories, hard disks, CD-ROMs, DVDs, MOs, RAMs, VRAMs, and the like. In FIG. 1, the ROM, RAM, etc. mounted on the control device correspond to this.

Further, the storage unit 500 stores the game program 510 and the game data for causing the processing unit 300 to function as the game calculation unit 310. The game data includes food information 522, food image information 523, ghost information 524, ghost image information 525, image recognition method information 526, appearance threshold information 528, firewood food execution information 532, and captured image recognition. The result information 534, the calculated similarity information 536 for sowing food, and the calculated similarity information 537 for normal use are included.

The communication unit 600 connects to a predetermined communication line and performs data communication with an external device such as another portable game device 1000. This function is realized by wireless communication modules such as Bluetooth (registered trademark) and IrDA, modems, TAs, jacks and control circuits for wired communication cables. In FIG. 1, this corresponds to the built-in communication device.

[Processing flow]

FIG. 16 is a flowchart for explaining the flow of game processing in the present embodiment. This process is realized by the game calculation unit 310 executing the process according to the game program 510.

According to the figure, first, the game calculation unit 310 causes the image display unit 410 to display a finder image based on the image data input from the image pickup unit 200 (step S1).

Next, the firewood feeding execution unit 320 determines whether or not the feeding selection operation has been input from the operation input unit 100, and if not (step S3: NO), the process proceeds to step S15. If the food selection operation is input (step S3: YES), the prepared food types are listed in the finder image with reference to the food information 522. Then, the type of food is selected according to the operation input from the operation input unit 100 (step S5).

Subsequently, the firewood food execution unit 320 determines whether or not the firewood food execution operation has been input from the operation input unit 100, and if not (step S7: NO), proceeds to step S15. If the firewood feeding operation is input (step S7: YES), the finder image at the time of inputting the firewood feeding operation is captured, and the image recognition method specified in the image recognition method information 526 is used for this captured image. Performs image recognition processing in step S9, and extracts feature data for each image recognition method (step S9). Then, the extracted feature data is added to the sowing bait execution information 532 in association with the input time (execution time) of the sowing bait execution operation and the type of food selected at the time of inputting the execution operation ( Step S11).

After that, the firewood feeding execution unit 320 performs the firewood feeding effect processing on the finder image. That is, the complete image of the food according to the selected type of food is combined with the finder image for a certain period of time and displayed, and the effect is performed as if the food was sown in the place where the food is being taken (step). S13).

Subsequently, the imaging execution unit 330 determines whether or not the shutter operation has been input from the operation input unit 100, and if it has not been input (step S15: NO), the process proceeds to step S25. If the shutter operation is input (step S15: YES), the finder image at the time of inputting the shutter operation is captured and used as an captured image, and the image recognition method specified in the image recognition method information 526 is used for this captured image. Performs image recognition processing in step 2 and extracts feature data for each image recognition method (step S17). Next, the ghost appearance determination process is executed (step S19).

FIG. 17 is a flowchart for explaining the flow of the ghost appearance determination process. According to the figure, in the ghost appearance determination process, first, the loop A process is executed for each execution number stored in the firewood feed execution information 532 in order. That is, in loop A, the similarity calculation unit 332 calculates the similarity between the feature data of the target execution No. and the feature data of the extracted captured image for each image recognition method (step T1). Next, each calculated similarity is multiplied by a weight corresponding to the image recognition method specified in the image recognition method information 526 and added to calculate the total value of the similarity (step T3).

When the processing of the loop A for all the execution numbers is completed, the ghost appearance determination unit 334 determines the ghost to appear. That is, among each execution No., the execution No. having the maximum calculated similarity total value is selected (step T5), and the total similarity value of the selected execution No. is equal to or higher than the appearance threshold value for sowing food. Judge whether or not.

If it is equal to or higher than the appearance threshold value for sowing food (step T7: YES), the ghost appearance determination unit 334 calculates the elapsed time from the execution of the selected execution number to the input time of the shutter operation (step T9). Next, in the food information 522, the appearance period associated with the type of food of the selected execution number is determined, and it is determined whether or not the calculated elapsed time is within this appearance period.

If the elapsed time is within the appearance period (step T11: YES), the ghost appearance determination unit 334 determines in the food information 522 that the ghost associated with the food type of the selected execution number appears. Then, the image of this ghost is combined with the captured image and displayed on the image display unit 410 (step T13).

On the other hand, if the elapsed time is not within the appearance period (step T11: NO), it is determined that the ghost does not appear, and the image of the food in the situation according to the context of the elapsed time and the appearance period is combined with the captured image. Is displayed on the image display unit 410. That is, if the elapsed time has not reached the appearance period, the complete image of the food of the selected execution No. is combined with the captured image and displayed, and if the appearance period is exceeded, the selected execution No. The image of the food debris is combined with the captured image and displayed (step T15).

Further, in step T7, if the total value of the similarity of the selected execution numbers is not equal to or higher than the appearance threshold value for sowing food (step T7: NO), the normal appearance determination process is executed (step T17).

FIG. 18 is a flowchart for explaining the flow of the normal appearance determination process. According to the figure, in the normal appearance determination process, the loop B process is executed for all the ghosts specified in the ghost information 524 in order. That is, in the loop B, the similarity calculation unit 332 is the image recognition method. For each, the similarity between the feature data of the target ghost and the feature data of the extracted captured image is calculated (step P1). Then, each calculated similarity is multiplied by a weight corresponding to the image recognition method specified in the image recognition method information 526 and added to calculate the total value of the similarity (step P3).

When the processing of loop B for all ghosts is completed, the ghost appearance determination unit 334 selects the ghost having the highest total calculated similarity value from all ghosts (step P5), and selects this ghost. It is determined whether or not the total value of the similarity of ghosts is equal to or higher than the normal appearance threshold value.

If it is equal to or higher than the normal appearance threshold value (step P7: YES), the ghost appearance determination unit 334 determines that the selected ghost appears, synthesizes the image of this ghost with the captured image, and displays it on the image display unit 410. Let it (step P9). On the other hand, if it is not equal to or higher than the normal appearance threshold value (step P7: NO), it is determined that the ghost does not appear (step P11). When the above processing is performed, the normal appearance determination processing is terminated.

When the above processing is performed, the ghost appearance determination process is terminated, and subsequently, the game calculation unit 310 determines whether or not the ghost has appeared, and if it has appeared (step S21: YES), the operation input unit. A predetermined ghost capture process is executed based on an operation signal or the like from 100 (step S23). After that, it is determined whether or not to end the game, and if it does not end (step S25: NO), the game returns to step S1 and if it ends (step S25: YES), the game process ends.

[Action / Effect]

As described above, according to the present embodiment, when the camera unit 7 of the portable game device 1000 is activated, the food selection operation is input to select the food, and the firewood food execution operation is input to select the food. The image of the food of the same type is combined and displayed in the finder image. That is, food is sown. Then, after a lapse of a predetermined time, when the subject captured at the time of inputting the sowing food execution operation is imaged, the ghost image corresponding to the sown food is combined with the captured image and displayed. That is, a ghost appears.

At this time, in the portable game device 1000, when the sowing food execution operation is input, the feature data of the captured finder image is extracted, the extracted feature data, the input time of the sowing food execution operation, and the selected food. It is added to the sowing feed execution information 532 in association with the type of. Then, when the shutter operation is input, the similarity between the feature data of the captured image and the feature data of each execution number of the firewood feed execution information 532 is calculated, and the ghost to appear is determined based on the calculated similarity. are doing. That is, the execution No. having the largest total value of the calculated similarity is selected, and the total value of the similarity of the selected execution No. is equal to or higher than the predetermined appearance threshold for sowing bait, and the execution No. When the elapsed time from the execution time of the sowing food is within the appearance period of the sown food, the image of the ghost associated with the type of food of this execution No. is combined with the captured image in the food information 522. Is displayed.

As a result, it is possible to realize a new interest in imaging as if the character existing in the real space was imaged. Further, when the player inputs the sowing bait execution operation, the ghost can appear when the desired place or subject is imaged. In addition, the ghost that appears at the time of imaging is determined by the type of food selected when inputting the sowing food execution operation, that is, the type of food that has been sown. Therefore, by appropriately selecting the type of food, the desired ghost appears. It becomes possible to make it.

In addition, by setting the appearance period during which ghosts can appear for each type of food, it is possible to realize an effect as if the ghosts were attracted to the sowed food in the real space. Furthermore, in this case, if the elapsed time from the execution of the sowing food (input of the sowing food execution operation) has not reached the appearance period, the complete image of the food is compositely displayed on the captured image and the appearance period has been exceeded. In this case, by compositing and displaying the image of the wreckage of the food on the captured image, it is possible to realize an effect as if the ghost had been attracted to the food and then ate the food and left.

[Modification example]

The application of the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. For example, it may be transformed as follows.

(A) Ghosts that appear due to sowing food In the above-described embodiment, the ghosts that appear are determined by the type of sowing food, but the type of sowing food may not be selected. In this case, the ghost to appear by the sowing bait is determined as follows, for example.

(A-1) Randomly determined Specifically, when a sowing feed execution operation is input, the feature data of the captured finder image is stored in the sowing feed execution information. Then, when the shutter operation is input, the similarity between the feature data of the captured image and the feature data stored in the sowing bait execution information is calculated, and the total value of the calculated similarity appears for the predetermined sowing bait. If it is equal to or higher than the threshold value, it is determined that a ghost will appear, and for example, an image of a ghost randomly selected from a plurality of prepared ghosts for sowing food or ghosts stored in ghost information 524. Is combined with the captured image and displayed (appeared).

(A-2) Make the ghost according to the place where the food is sown. Specifically, the feature data of the finder image captured when the sowing food execution operation is input and each stored in the ghost information 524. The total value of the similarity with the feature data of the ghost is calculated, and the ghost having the largest (highest) total value of the calculated similarity is stored in the sowing food execution information in association with the feature data of the captured image. Then, when the shutter operation is input, the similarity between the feature data of the captured image and the feature data stored in the sowing feed execution information is calculated, and the total value of the calculated similarity is the predetermined sowing feed. If it is equal to or higher than the use appearance threshold, it is determined that a ghost will appear, and the image of the ghost associated with the feature data in the sowing food execution information is combined with the captured image and displayed.

(B) Number of Corresponding Ghosts In the above-described embodiment, in the food information 522 (see FIG. 8), the type of food and the ghost are associated with each other on a one-to-one basis. Is also good. Specifically, in the food information, a plurality of ghosts are associated with each type of food. Then, when the shutter operation is input, an image of, for example, a randomly selected ghost among the ghosts according to the type of food sown is combined with the captured image and displayed.

Further, as shown in FIG. 19, in the food information, a plurality of ghosts having different appearance periods may be associated with each type of food. In this case, when the shutter operation is input, the image of the ghost in the appearance period in which the elapsed time is satisfied among the ghosts corresponding to the sown food is combined with the captured image and displayed. By doing so, even when the same place where the sowing food was executed in the past is imaged, different ghosts can appear depending on when the image is taken.

(C) Elapsed time is displayed as a timer Also, the elapsed time from the execution time of the firewood feed (input time of the firewood feed execution operation) may be displayed as a timer. When the sowing bait is executed a plurality of times, the elapsed time from each execution time may be displayed in parallel. By doing so, the player can take an image at an appropriate timing and make the ghost appear without forgetting that the sowing feed has been executed.

(D) Ghost Sensor Further, the effect may be performed as if a ghost sensor for detecting the presence of a ghost in the surrounding space of the portable game device 1000 is provided. This ghost sensor notifies whether or not to synthesize a ghost image with the captured image when a shutter operation is input, as the presence of the ghost in the surrounding space of the device.

Specifically, a finder image is captured at predetermined time intervals, image recognition processing is performed on the captured image, and feature data is extracted. Then, the similarity between the extracted feature data and the feature data of each execution number stored in the firewood feed execution information 532 is calculated, and based on the calculated similarity, the detection result (ghost reaction) for the existence of the ghost is obtained. ) Is displayed. For example, as in the case of inputting the shutter operation, it is determined whether or not a ghost appears based on the calculated similarity of each execution number, and this determination result is used as a predetermined display indicating, for example, whether or not a ghost appears. Notify by showing / hiding. Alternatively, the largest value among the calculated total values of the similarity of each execution number is displayed on the meter as the strength of the ghost reaction. As a result, even if the place where the food is sown is remembered, it is possible to surely make the ghost appear by referring to the displayed ghost reaction.

<figref num="1">An example of the appearance of the portable game device of the present embodiment.</figref><figref num="2">Schematic diagram of this embodiment.</figref><figref num="3">Schematic diagram of this embodiment.</figref><figref num="4">Schematic diagram of this embodiment.</figref><figref num="5">Schematic diagram of this embodiment.</figref><figref num="6">Schematic diagram of this embodiment.</figref><figref num="7">Functional configuration example of a portable game device.</figref><figref num="8">Data structure example of food information.</figref><figref num="9">Data structure example of image recognition method information.</figref><figref num="10">Data structure example of firewood feed execution information.</figref><figref num="11">Data configuration example of captured image recognition result information.</figref><figref num="12">Data structure example of calculated similarity information for firewood bait.</figref><figref num="13">An example of ghost information data structure.</figref><figref num="14">Data structure example of calculation similarity information for normal use.</figref><figref num="15">Data structure example of appearance threshold information.</figref><figref num="16">The flow diagram of the game processing of this embodiment.</figref><figref num="17">A flow chart of the ghost appearance determination process executed during the game process.</figref><figref num="18">The flow chart of the normal appearance judgment processing executed during the ghost appearance judgment processing.</figref><figref num="19">A modified example of the data structure of food information.</figref>

Code description

1000 Portable game device 100 Operation input unit 200 Imaging unit 300 Processing unit 310 Game calculation unit 320 Firewood feeding execution unit 330 Imaging execution unit 332 Similarity calculation unit 334 Ghost appearance judgment unit 340 Image generation unit 350 Sound generation unit 360 Time measurement unit 410 Image display unit 420 Sound output unit 500 Storage unit 510 Game program 522 Feed information 523 Feed image information 524 Ghost information 525 Ghost image information 526 Image recognition method information 528 Appearance threshold information 532 Fire feed execution information 534 Captured image recognition result information 536 Fire feed Calculation similarity information for normal use Calculation similarity information for normal use 600 Communication unit