Method and system for facilitating wireless, full-body, real-time user interaction with digitally generated text data

0001This patent application claims priority under 35 U.S.C. §119 of provisional patent application, Ser. No.: 60/178,228, filed Jan. 26, 2000, which is hereby incorporated by reference as if set forth in its entirety herein (including the source code).

COMPUTER PROGRAM LISTING APPENDIX

0002This application incorporates by reference in its entirety a computer program listing appendix containing the source code-listings which were described in the patent application at the time of filing and which have now been reproduced on compact disk. The compact disk contains the file entitled “APPENDIX (SOURCE CODE),” was created on Jun. 18, 2003, and encompasses 812 kilobytes.

BACKGROUND OF THE INVENTION

00031. Field of the Invention

0004The present invention relates to interactive computer systems, and in particular to a method and apparatus for facilitating wireless, full-body interaction between a human participant and a computer-generated graphical environment including textual data.

00052. Description of Related Art

0006So-called “virtual reality” (“VR”) systems enable users to experience computer-generated environments instead of merely interacting with them over a display screen. Such systems typically require the user to don goggles, through which he or she perceives the virtual environment, as well as sensors that encode the user's gestures as electrical signals. The user reacts naturally to the changing virtual environment, generating signals that the computer interprets to determine the state and progress of the presented environment.

0007In order to encode a sufficiently board spectrum of gestures to facilitate natural interaction, VR systems ordinarily require the user to wear, in addition to the goggles, at least one “data glove” to detect hand and finger movements, and possibly a helmet to detect head movements. Full-body systems, which encode movements from numerous anatomical sites to develop a complete computational representation of the user's overall body action, require many more sensors; however, such systems, would be capable of projecting the user fully into the virtual environment, providing the user with greater control and a heightened sense of participation ideally suited to interactive simulations.

0008Unfortunately, numerous practical difficulties limit the capacity of current VR systems to achieve this goal. The nature of the interaction currently offered, even with full-body sensor arrays, is limited. The computational demands placed on a system receiving signals from many sensors can easily overwhelm even large computers, resulting in erratic “jumps” in the visual presentation that reflect processing delays. Moreover, no matter how many sensors surround the user, that cannot “see” the user, and therefore cannot integrate the user's true visual image into the virtual environment.

0009Economic and convenience factors also limit sensor-type VR systems. As the capabilities of VR systems increase, so do the cost, awkwardness and inconvenience of the sensor array. The sensors add weight and heft, impeding the very motions they are intended to detect. They must also ordinarily be connected, by means of wires, directly to the computer, further limiting the user's movement and complicating equipment arrangements.

0010In order to overcome the limitations associated with sensor-based VR systems, researchers have devised techniques to introduce the user's recorded image into a virtual environment. The resulting composite image is projected in a manner so that it may be viewed by the user, enabling the user to observe his or her appearance in and interaction with the virtual environment.

0011Three such approaches include the VideoPlace system (see, e.g., M. Krueger, Artificial Reality II (1991) and U.S. Pat. No. 4,843,568), the Mandala system (see, e.g., Mandala VR News, Fall/Winter 1993; Vincent, “Mandale: Virtual Village” and Stanfel, “Mandela: Virtual Cities,” Proceedings of ACM SIGGRAPH 1993 at 207-208 (1993)), and MIT's system disclosed in U.S. Pat. No. 5,563,988. Unfortunately, these systems exhibit various limitations. For example, Krueger's VideoPlace requires a special background and ultraviolet lamps, and extracts and represents only the user's silhouette. The Mandala system can integrate the user's full image within the virtual environment it creates, but requires a chroma-key blue background. The system developed by MIT provides a 3-Dimensional spatial video interaction with a user and is directed to detecting and analyzing gestural information of the user to cause predetermined responses from graphically generated objects within the virtual environment (e.g., if a virtual user pets a graphically generated dog within the virtual environment, the dog may wag its tail). Regardless of the systems, all of the above-described VR systems employ complex video analysis and tracking techniques to detect and follow the boundaries of the user's image within the virtual environment.

0012The disclosure of U.S. Pat. Nos. 5,563,988 of Maes et al, and 4,843,568 of Krueger et al. are hereby incorporated by reference as if set forth in their entirety herein.

0013Applicant has recognized the need for a simple 2-dimensional video-generated real-time interaction system wherein text data is easily introduced into a virtual environment, as viewed on a monitor or on a projection screen. The inputted text data is manipulated in the virtual environment by video representation of a human participant (a “virtual user”). Through movements in the real world, the human participant or virtual user freely interacts with the inputted text data in the virtual environment as the human participant (and other people watching the monitor or projection screen) absorbs the teachings of the text data as he or she interacts with the text data in the virtual environment.

OBJECTS OF THE INVENTION

0014It is an object of the present invention to provide a wireless VR system that easily accepts text data from a text input to be displayed onto a monitor (or projection screen) so that the font, size, shape, and/or movements of words and/or characters (e.g, letters, symbols and numbers) of the inputted text data may be altered by a virtual user (a real-time video representation of a human participant).

0015It is another object of the invention to enable the virtual user to interact with computer-generated, visually represented text data in such a manner that allows the inputted text to be readable within the virtual environment so that the virtual user may learn, understand, or otherwise experience the meaning of the textual data as he or she interacts with the text within the virtual environment.

0016It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment to promote marketing, advertisement, sales, and entertainment.

0017It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data viewed within the virtual environment may be changed in realtime in response to actions by the user or by others.

0018It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is imported from a local text file.

0019It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is imported from a remote site over the Internet.

0020It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is inputted directed from a keyboard.

0021The foregoing and other objects of this invention can be appreciated from the Summary of the Invention, the drawing Figures and Detailed Description.

SUMMARY OF THE INVENTION

0022In accordance with the invention, a controlling software program instructs an interactive display system. The interactive display system includes an electronic camera (e.g., charge-coupled device or “CCD”) which intermittently records (in realtime) a human participant's (a user) image and inputs the video image into a computer. The computer creates a digitized version of the inputted image (e.g., a bitmap of pixels) and thereby creates a “virtual user image”. According to one embodiment, the camera is positioned behind a large viewing screen which faces the user and records the user's image through an opening provided within the screen. The computer further receives a data record including text data and text controlling parameters, preferably including at least a fall rate value. The data record may be supplied to the computer using any of several conventional memory mediums, such as a hard drive memory, disc memory, compact disc, or the data record may only include a text string which is supplied to the computer directly from the Internet through a connected modem, for example, or through the use of a connected keyboard. Alternately, the text may be inputted to the computer directly from a word processing program, such as Corel's WordPerfect®, or Microsoft Word® as a text file or by “copying” selected text from an open text file directly into the present software program.

0023According to the invention, upon receiving a captured video image of the user, the software program (see appendix for source code) first parses the text data into lines of text. The program determines which line or lines of text should be displayed on the screen and further determines the destination of each character within any line that is to be displayed on the screen, according to the fall rate assigned to each character. The program then compares the pixel color value of “destination pixels” (pixels located at the destination of each particular character) with a threshold color value. If the color value of the destination pixels is below the threshold value (i.e., lighter), then the particular character will be positioned at a destination according to its prescribed fall rate. If, however, the color value of the destination pixels exceeds the threshold value (i.e., darker), then it is determined that the virtual user's image is located at the destination of the particular character of the line of text. In such instance, the program locates a suitable new destination positioned vertically above the original destination which is lighter in color value than the threshold value.

0024Once all the characters of all the lines of text intended to be projected onto the screen are provided with suitable destinations, the program instructs the computer to display all of the characters on the screen at their respective destinations with the particular captured video frame. The process continues where a new image frame is available from the camera.

0025The program analyzes each captured image as it acquires at a prescribed sampling rate. The sampling rate is dependent on the speed of the video card components and drivers, the speed of the computer, and light sensitivity of the digital camera.

0026According to the invention, as the individual characters of a single line make their way down along the projection screen, the program keeps track of when a predetermined number of characters (e.g., at least half of them) travels a predetermined distance from the top of the viewing screen (such as the halfway point). If this is the case, the program causes the entire line of text to fade at a prescribed fade rate. When one line fades at the lower portion of the screen, another line appears at the top of the screen, with its characters beginning to fall. It is preferred that at least three lines of text be displayed on the screen at any given time.

0027Another aspect of the invention displays each line of text data as a separate color so that viewers may easily discern the letters or characters from different lines of inputted text, allowing the lines of text to be read relatively easily, as the letters fall towards and rest on various darker elements located within the virtual environment, such as the arm of a virtual user and perhaps a “virtual umbrella image” carried by the virtual user.

0028According to another aspect of the invention, each character of an inputted text string is horizontally held so that each character is restricted to vertically movement as it descends on the viewing screen at its prescribed fall rate or ascends upwardly with similar upward movement by the virtual user.

0029According to another embodiment of the invention, the program generates a shadow zone of dark pixels which are projected onto the viewing screen. The shadow zone is aligned with the optical input of the digital camera so that the camera will be protected from the relatively bright light of the projector. The computer generated shadow zone helps prevent the images recorded by the camera from becoming “washed out” by the light from the projector shining on the camera's lens.

BRIEF DESCRIPTION OF THE DRAWINGS

0030The foregoing discussion will be understood more readily from the following detailed description of the invention, when taken in conjunction with the accompanying drawings, in which:

0031<figref idref="DRAWINGS">FIG. 1</figref> is a schematic illustrating an interactive display system including a projection screen, a pickup camera, a projector, a computer, and a text input, according to the invention;

0032<figref idref="DRAWINGS">FIG. 2</figref> is a perspective view of the inventive display system of <figref idref="DRAWINGS">FIG. 1</figref>, showing the field of view of the camera, according to the invention;

0033<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram of the interactive display system, according to the invention;

0034<figref idref="DRAWINGS">FIG. 4</figref> is a illustration of an exemplary text string to be inputted into the interactive display system, according to the invention;

0035<figref idref="DRAWINGS">FIG. 5</figref> is a illustration of an exemplary data record which includes the text string of FIG. <b>4</b> and further includes text-controlling parameters, according to the invention;

0036<figref idref="DRAWINGS">FIG. 6</figref> is a flow diagram showing steps of obtaining and combining video and text data, according to the invention;

0037<figref idref="DRAWINGS">FIG. 7</figref> is a front view of the viewing screen, illustrating an exemplary displayed virtual image including a virtual user holding a virtual umbrella, and further including falling text, according to the invention;

0038<figref idref="DRAWINGS">FIG. 8</figref> is an enlarged view of a portion of the virtual umbrella of FIG. <b>7</b> and select letters of the text, according to the invention; and

0039<figref idref="DRAWINGS">FIG. 9</figref> is a enlarged partial front view of the viewing screen showing details of a central opening and a computer generated shadow zone, according to the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

0040Referring to <figref idref="DRAWINGS">FIGS. 1</figref>, <b>2</b>, and <b>3</b> the interactive display system <b>10</b>, according to the present invention, is shown including a large display screen <b>12</b> having a generally central and small opening <b>14</b>, an image projector <b>16</b> positioned to project an image (along projection beam <b>17</b>) onto the front surface of screen <b>12</b>, and a camera <b>18</b>, preferably a mini black and white CMOS-type video camera (alternately a Connectrix QuickCam® by Logitech® with appropriate platform drivers may be used). The camera <b>18</b> is positioned behind screen <b>12</b> with its image-receiving lens aligned with opening <b>14</b>, so that a person (a user) standing in front of screen <b>12</b> (in the area represented by reference numeral <b>15</b>, in <figref idref="DRAWINGS">FIGS. 1 and 2</figref>) and within the field of view <b>19</b> of the camera <b>18</b>, is recorded by the camera. The camera <b>18</b> is preferably sufficiently light sensitive to “capture” repetitive images of the user in apparent realtime (i.e., generating a sufficient number of consecutive images so that when the sampled images are eventually projected, the motions of the user appear smooth and continuous). It is preferred, although not shown, that a rear surface or wall be located opposite the screen <b>12</b>, behind the user, and that this rear surface be white and illuminated. Although the rear surface is not necessary for the present system to operate, it does allow the present program to quickly and easily distinguish the virtual user from the background.

0041The present interactive display system <b>10</b> further includes a computer <b>20</b> having a memory <b>21</b> and a text input <b>22</b>, as shown in FIG. <b>1</b>. Computer <b>20</b> may be conventional, and preferably includes a fast central processing unit (CPU), such as Intel's Pentium II® or higher. The video card of the computer preferably includes enough VRAM to display 640 X 480 resolution at high color (16 bit). If a CMOS type camera <b>18</b> is being used behind the large screen <b>12</b>, then the computer <b>20</b> further requires a video capture card <b>24</b> to create a bitmap of the analog image of the CMOS camera <b>18</b>. If a Connectix®-type camera is used (or equivalent), no capture card <b>24</b> is necessary because the Connectix®E-type camera uses software to digitize a recorded image. The computer <b>20</b> preferably uses Microsoft's Window 98® or Microsoft's Windows NT® operating systems.

0042As is understood by those skilled in the art, the video capture card <b>24</b> is used to receive analog image data from the camera <b>18</b> and digitize the image by generating a bitmap (the present program can control the rate of sampling). During each sampling, as described below, the video card <b>24</b> captures a frame of image data from camera <b>18</b> as the camera records the user and digitizes the frame into array of picture elements, or “pixels,” called a bitmap. Depending on the format, each pixel within a bitmap is identified by an RGB color code (a three number code which identifies the exact color of the pixel). The bitmap representing the received frame of analog image data is stored in memory for analysis and manipulation by the software program.

0043The computer receives a data record including text data and text-controlling parameters from text input <b>22</b>. As described below the present software program manipulates both the digitized image of the received frame of analog image data and the text data from text input <b>22</b> and combines the video and text data onto the projection screen <b>12</b> in a generally smooth and continuous manner.

0044Text input <b>22</b> may include any of a variety of conventional input devices, such as a hard drive memory, disc memory, compact disc, or a conventional Internet connection (such as through a connected modem), or through the use of a connected keyboard. Alternately, the text may be supplied to the computer directly from a word processing program, such as Corel's WordPerfect®, or Microsoft Word® as a text file or by copying selected text from an open text file directly into the program. Text data is preferably inputted into the memory <b>21</b> of computer <b>20</b> for later retrieval by the present program and displayed in accordance with the invention, or alternately, the text is inputted in realtime for immediate projection onto screen <b>12</b>.

0045An example of text data <b>30</b> for use with the present invention is shown in <figref idref="DRAWINGS">FIG. 4</figref> (shown displayed on a computer monitor <b>32</b>): “When it rains, it pours. When it stops, its wet.” This exemplary text data <b>30</b> is inputted into the computer <b>20</b> by any conventional means and is parsed into appropriately-sized lines <b>31</b> (depending on the size of the projection screen <b>12</b>). If the text data <b>30</b> exceeds a certain number of characters <b>36</b>, the program will parse the text data <b>30</b> into separate lines of text <b>31</b> so that each line may fit onto the screen <b>12</b> (the wider the screen <b>12</b>, the longer each line of text <b>31</b>). The program will form a line data record to keep track of which lines of text <b>31</b> are on the screen <b>12</b>, are fading from the screen <b>12</b>, and should be introduced at the top of the screen. Each line of text <b>31</b> located on the line data record will be introduced in a prearranged order and every line of text <b>31</b> which is being projected on the screen <b>12</b> will be electronically marked or flagged in the line data record. In this manner, as one line of text <b>31</b> fades from the screen <b>12</b>, its electronic mark is removed from the line data record and a subsequent line of text <b>31</b> will be introduced onto the screen <b>12</b>.

0046Each line located in the line data record, is also stored in a character data record <b>34</b>, shown in FIG. <b>5</b>. According to one aspect of the invention, the character data record <b>34</b> may further include various parameters which will control how each character <b>36</b> of each line of text <b>31</b> moves and appears within the virtual environment, as viewed on screen <b>12</b>. The parameters may vary depending on the desired effect of the characters <b>36</b> of each line of text <b>31</b> within the virtual environment and depending on the complexity and speed of the interaction system.

0047The controlling parameters may include a fall rate value which controls the speed each character <b>36</b> of each line of text <b>31</b> appears to fall on the screen <b>12</b>. The fall rate value may be defined in a variety of ways, such as the number of pixels traveled in one second or between captured frames of video.

0048Another character parameter may be a fall delay value which determines the length of time each character <b>36</b> will remain at the top of the screen <b>12</b> prior to falling.

0049A color code may be assigned to each character <b>36</b> or all of the characters within each line of text <b>31</b>.

0050Another parameter for each character <b>36</b> is a horizontal hold condition which, when set, restricts each character to vertical movement when projected within the virtual environment.

0051A fade parameter controls whether a character <b>36</b> fades when reaching the bottom of the screen <b>12</b> within the virtual environment.

0052An edge-detect parameter determines whether a character will recognize the virtual image of a user within the virtual environment (i.e., determines if a virtual user interacts with the falling characters).

0053The distance the characters <b>36</b> “fall” along the screen <b>12</b> within the virtual environment may also be controlled using a maximum fall distance parameter.

0054The above-listed character parameters are provided to illustrate that each character <b>36</b> may be controlled in a variety of ways independently of the other characters <b>36</b> within a line of text <b>31</b>. Although a variety of parameters are possible, as exemplified above, it is preferred that most, if not all, include default values which simplify the setup and use of the present program. It is preferred that an operator merely be required to input the text data <b>30</b> into computer <b>20</b> without further setup.

0055In operation, referring to <figref idref="DRAWINGS">FIG. 6</figref>, the present program first receives input text data <b>30</b> at step <b>100</b>. The text data <b>30</b> is first divided up into lines of text <b>31</b> at step <b>110</b>. The number of characters <b>36</b> in each line of text <b>31</b> depends, in part, on the dimensions of the viewing screen <b>12</b>. The lines of text <b>31</b> may be divided into any number of characters, following, for example, the length of each line of a poem. A fall rate value (and perhaps other variables) is assigned to each character <b>36</b> of each line of text <b>31</b> at step <b>120</b>. As described above, it is preferred that all the characters <b>36</b> of any given line of text <b>31</b> have the same fall rate. At step <b>130</b>, a frame of video from the camera is received and digitized by the video capture card <b>24</b>, generating a bitmap wherein an RGB color value is assigned to each pixel defined in the captured image frame. Once the program starts capturing frames of video data, the program enters into a callback mode whereby every time a new frame of video data is received, select pixels of the bitmap are analyzed, as described below. Since only select pixels are analyzed by the present program, the speed of the program is much faster than prior art VR systems which rely on analyzing each pixel in the bitmap for each frame of image data received.

0056At step <b>140</b>, the program determines which lines of text <b>31</b> are to be displayed on the screen <b>12</b>. It is preferred that three lines of text <b>31</b> be displayed on the screen <b>12</b> at any given time. The number of lines of text <b>31</b> may vary, of course, depending on the particular application, the size of the viewing screen <b>12</b>, and the desired effect. At step <b>150</b>, in <figref idref="DRAWINGS">FIG. 6</figref>, the program determines the destination for a selected character <b>36</b> among the characters of the “to be displayed” lines of text <b>31</b>, according to the predetermined fall rate of the selected character <b>36</b>. At step <b>160</b>, the color value of the “destination pixels” or those pixels which are located at the destination of the selected character <b>36</b> at step <b>150</b>. The program, at step <b>170</b>, then compares the read color value of the destination pixels with a threshold color (or contrast) value. At step <b>180</b>, if it is determined at the comparison step <b>80</b> that the destination pixel color value is greater than the threshold color value, then the destination of the selected character <b>36</b> is raised by a predetermined number of pixels, preferably a single pixel, at step <b>190</b>. In this instance, the pixel color value at the newly determined destination of the selected character <b>36</b> is again read at step <b>160</b>, its value compared again with the threshold value at step <b>170</b> and a decision is again made at step <b>180</b> until the read pixel value of the destination pixel is less than or equal to the threshold color value.

0057At this point, the determined destination of the selected character <b>36</b> is stored in memory at step <b>200</b>. At step <b>210</b>, it is determined if any more characters <b>36</b> remain within the “to be displayed” lines of text <b>31</b>. If there are more characters <b>36</b> remaining within the “to be displayed” lines of text <b>31</b>, another character <b>36</b> is selected at step <b>220</b> and again the destination of the newly selected character <b>36</b> is determined at step <b>150</b> and the above-described process is repeated until the destination of each character <b>36</b> of the “to be displayed” lines of text <b>31</b> have been determined. When this occurs, the captured video image frame together with each character <b>36</b> of the “to be displayed” lines of text <b>31</b> are displayed onto screen <b>12</b> (at step <b>230</b>), with each character <b>36</b> located at each respective destination, as determined by the program and as stored at step <b>200</b>.

0058The program then returns to step <b>130</b> to acquire another image frame of video from camera <b>18</b> and repeats the above described steps, as shown in the flow diagram of FIG. <b>6</b>.

0059The end result is that the individual characters <b>36</b> of their respective “to be displayed” lines of text <b>31</b> will appear to fall (at their prescribed fall rate) from the top of the projection screen <b>12</b>, as illustrated in <figref idref="DRAWINGS">FIGS. 7 and 8</figref>. In <figref idref="DRAWINGS">FIG. 7</figref>, an exemplary projected image on screen <b>12</b> is shown to have a shoulder <b>240</b>, and an arm <b>242</b> of a virtual user <b>244</b>. The actual user (not shown) and the virtual user <b>244</b> are holding an umbrella <b>246</b> which appears on the screen <b>12</b> in this example, as a virtual umbrella <b>246</b>. As is shown, the program has displayed the video image of the virtual user (shown in part) <b>244</b> holding the virtual umbrella <b>246</b> and the inputted lines of text <b>31</b>: “When it rains, it pours”. In this exemplary scene, a new line of text <b>31</b>: “When it rains, it pours” has just been introduced at the top portion of the screen <b>12</b>, generally indicated by arrow <b>248</b> of FIG. <b>7</b>. The characters “h” and “e” of the word “When” have already started to fall at their quicker fall rates. Various characters <b>36</b> from previously introduced lines of text <b>31</b> (in this example, all the lines of text <b>31</b> shown on the screen <b>12</b> are the same: “When it rains, it pours”) have been drifting downwardly from the top portion <b>248</b> of the screen <b>12</b> to towards the bottom portion of the screen, generally indicated by arrow <b>250</b> in FIG. <b>7</b>.

0060Following the steps described above, as the computer <b>20</b> determines the destination of each character <b>36</b>, it quickly reads the pixel color values at that destination of the captured image and determines whether a virtual obstacle is already located there. If no obstacle is there (i.e., the read pixels are less than or equal to a threshold value), then the particular character <b>36</b> appears on the screen <b>12</b> at the destination and, as subsequent images are displayed on screen <b>12</b>, the particular character <b>36</b> will appear to slowly fall, like snow. If however, a virtual obstacle is determined to be located at the destination of a particular character <b>36</b>, then a new destination located higher on the screen <b>12</b> is selected so that the character <b>36</b> appears to interact with the virtual obstacle located in the virtual environment. This interaction is illustrated in <figref idref="DRAWINGS">FIG. 8</figref>, wherein an enlarged partial section of the umbrella <b>246</b> is shown “capturing” the characters “r”, “a”, and “i” of the word “rains”. These characters <b>36</b> will remain at the first “light” pixel (below th pixel color threshold value) located above the relatively dark pixels which make up the umbrella <b>246</b> so that the characters <b>36</b> appear to have fallen onto and are supported by the umbrella <b>246</b>. If the user raises the umbrella <b>246</b>, the program will quickly detect this upward movement because the “lighter” pixel between the character <b>36</b> and the umbrella <b>246</b> will become a dark value causing the computer to respond by raising the affected characters <b>36</b> upwardly with the umbrella movement, so that the characters <b>36</b> appear to have been lifted by the umbrella <b>246</b>. The characters <b>36</b> can move upwardly very quickly, but will preferably always fall at the predetermined fall rate, as discussed above.

0061As subsequent video images are sampled, each character <b>36</b> of a line of text <b>31</b> appear to gradually descend on the screen <b>12</b> until the character <b>36</b> confronts a dark pixel, signifying a virtual obstacle within the virtual environment. Although the present program may be setup so that the characters <b>36</b> fall at any fall rate, it is preferred that the characters <b>36</b> fall in a manner which is somewhat familiar and pleasing to the user, similar, for example, to the graceful movements of an airborne leaf slowing falling from a tree. A user can “catch” the falling leaf and quickly push it back into the wind currents, but the leaf will always fall back down at its leisurely pace. By “training” the introduced characters <b>36</b> to respond in this manner within the virtual environment, the user is likely to remain comfortable and will be encouraged to further interact and explore within the virtual environment.

0062The text data <b>30</b> may be the lines of a poem, song, or even the tag-line of an advertisement of a particular product and may be projected in any size, shape or font (system fonts are preferably used so that the processing is faster). As the user interacts with the introduced lines of text <b>31</b>, he or she must physically manipulate the falling characters <b>36</b> and perhaps capture several of them to read them and understand the actual text, but the fun of combining such a physical activity with reading only encourages the user to succeed in reading the textual information. Whatever message the textual information is to convey is more likely to reach the viewer or user when the user gets to participate with the message in an interactive game-like environment.

0063The textual information introduced into the present program may be continuously changed, such as in the case of weather reports, or news stories. Alternately, the textual information may be altered in realtime, perhaps in response to the particular actions of the user. In this instance, a user may be asked questions in a text format with the question appearing on the screen <b>12</b> together with the image of the user. When the user reads the question on the screen <b>12</b>, perhaps after capturing the characters <b>36</b>, he or she can audibly answer the question. The response to the question can be heard by the operator of the system. who may then ask another question (again in text format) which relates or corresponds to the user's response to the previous question. This type of interaction between the text-input operator and the user may further entertain and teach the user and any viewer of the system.

0064According to another aspect of the invention, text data may be introduced one line at a time allowing a first line of text <b>31</b> to be introduced at the top of the of the viewing screen <b>12</b> so that each character <b>36</b> begins to fall (as viewed on the viewing screen <b>12</b>), either at similar fall rates or at different ones. When a predetermined number of falling characters <b>36</b> (e.g., at least half of them) travels a predetermined distance from the top of the viewing screen <b>12</b>, a second line (or select characters of the second line) of text <b>31</b> will appear at the top of the screen <b>12</b>. The characters <b>36</b> of the newly introduced second line of text <b>31</b> will begin to fall along the viewing screen <b>12</b> in a similar manner to those of the first line of text. Third and other lines of text <b>31</b> may be displayed in a similar and repeating manner.

0065Another aspect of the invention causes inputted lines of text <b>31</b> to fade (at a prescribed fade rate) when a predetermined number of characters (e.g., at least half of them) reach a predetermined position on the viewing screen (e.g., enter the lower half of the screen). In a preferred embodiment, at any given moment, three lines of text <b>31</b> are displayed on the viewing screen <b>12</b>, one line of text <b>31</b> being introduced at the top of the screen, a second one falling in the middle of the screen (an “interactive region”), and a third line of text <b>31</b> fading out at the lower portion of the screen. Of course, some of the characters <b>36</b> (or letters) of earlier introduced lines of text <b>31</b> may remain on the screen, “captured” by virtual obstacles, such as the above-described umbrella <b>246</b> of FIG. <b>7</b>. If one particular character <b>36</b> from a first line of text <b>31</b> remains held in vertical space by a virtual obstacle (e.g., umbrella <b>246</b>), the same character <b>36</b> from subsequent lines of text <b>31</b> will eventually overlay the first “held” character <b>36</b> so that similar characters <b>36</b> held by virtual obstacles will not accumulate in number over time. However, it is also contemplated that as characters <b>36</b> descend onto each other on the screen <b>12</b>, the program can be altered so that any higher character <b>36</b> will consider a lower “held” character to be part of the virtual obstacle so that the characters will pile up (stack on top of each other), until the obstacle is cleared (i.e., the user moves).

0066According to another aspect of the invention, each line of text <b>31</b> is displayed as a separate color. In this arrangement, viewers may easily discern the characters from different lines of text <b>31</b> so that the lines of text <b>31</b> to be read relatively easily even as the individual characters <b>36</b> fall towards and rest on various darker elements located within the virtual environment, such as the arm of a virtual user and perhaps a “virtual umbrella image” carried by the virtual user.

0067According to another aspect of the invention each character <b>36</b> of inputted text data <b>30</b> is horizontally-held, so that each character <b>36</b> is restricted to vertically movement as it descends on the viewing screen <b>12</b> at its prescribed fall rate, or ascends with upward movement of a virtual obstacle.

0068The techniques described herein for integrating a user's image and inputted textual data within a digitally represented environment can be applied to numerous applications including fun and educational communication applications for use in schools to help kids learn to read or just have fun. In this application, Kids could be asked, for example, to interact with a falling sentence by “capturing” a verb located within the falling sentence.

0069The present system can also be installed at a store front, in which case a display monitor may be positioned within the store so that it may be viewed by passing pedestrians. In this application, the appearance of a pedestrian on the monitor will invariably attract the attention of the pedestrian, who will stop and interact with the textual message falling all around his or her image being displayed on the screen. The message, in this example, would help lure customers inside the store to purchase products.

0070Other useful applications for this technology include performance art, either used in connection with an artist or an audience member. The present invention can be setup within an art gallery, exhibit, or museum as an interesting and alluring way to convey any textual information either as a piece of art work, or as a source of information, such as an interactive information kiosk.

0071According to another aspect of the invention, the present software is connected to the Internet through appropriate modems, T<b>1</b>, or DSL communication lines for the purpose of allowing Internet users to introduce text data <b>30</b> to a particular interactive display shown over the Internet. Also, the software program may be used with a local PC whereby the computer's monitor becomes the viewing screen, and a local “webcam” type camera (or any other appropriate camera) may be used to record the user, who in this case, would be positioned in front of the computer. If a CRT or LCD type monitor is used as the screen <b>12</b>, the camera may be positioned on top of (or adjacent to) the monitor.

0072Referring to <figref idref="DRAWINGS">FIG. 9</figref>, an enlarged partial screen <b>12</b> showing the central opening <b>14</b> is shown, according to another embodiment of the invention. Applicant has discovered that light from projector <b>16</b> directed to the optical input lens of camera <b>18</b> causes the images captured by the camera to appear “washed out” because the camera automatically compensates for correct exposure. To overcome this problem, the present program generates a shadow zone <b>13</b> of dark pixels which are projected onto the viewing screen <b>12</b>. The shadow zone <b>13</b> is aligned with the optical input of the digital camera so that the camera will be protected from the relatively bright light of the projector. The computer-generated shadow zone <b>13</b> helps prevent the images recorded by the camera from becoming “washed out” by the light from the projector shining on the camera's lens. The shadow zone <b>13</b> is shown as a darkened square, however, the shadow zone <b>13</b> may take on any useful shape to cover opening <b>14</b>, and also may be any shade of gray. The program includes means to project the shadow zone <b>13</b> when each image and lines of text <b>31</b> are projected onto screen <b>12</b>. Also, the projected position of the shadow zone <b>13</b> on screen <b>12</b> may be easily moved by the program, using an appropriate input device, such as a mouse or a keyboard (not shown).

0073The present software program (see appendix for actual source code) is written in C++ programming language using standard Microsoft® and Video-for-Windows® Libraries.

0074Although the above detailed description of the invention assumes the presence of a single user, the invention is capable of simultaneously supporting multiple users. These users can occupy the same space in front of a single camera <b>18</b>, or images of each user can be inputted to a single computer <b>20</b> using separate cameras at physically disparate locations. In the latter case, remote image data reaches the invention by means of suitable network connections, and the invention sends picture signals to the user's remote monitor over the same facility. In this way, all users interact in the same virtual world.

0075The terms and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed.