Copy protection for digital motion picture image data

This invention generally relates to an apparatus for displaying a time-varying copy-deterrent pattern when projecting a digital motion picture, the copy-deterrent pattern not visible to a viewing audience but visible in a recording of the motion picture made using a video capture device such as a video camera.

Movie piracy is a cause of substantial revenue loss to the motion picture industry. Illegally copied movies, filmed during projection with video cameras or camcorders and similar devices, are a significant contributing factor to revenue loss. Even the questionable quality of copies pirated in this fashion does not prevent them from broad distribution in the "black market", especially in some overseas markets, and on the Internet. As video cameras improve in imaging quality and become smaller and more capable, the threat of illegal copying activity becomes more menacing to motion picture providers. While it may not be possible to completely eliminate theft by copying, it can be advantageous to provide display delivery techniques that frustrate anyone who attempts to copy a motion picture using a portable video camera device.

It is known to provide a distinct symbol or watermark to an original still image as a means of image or copy identification, such as in order to authenticate a copy. As examples, U.S. Patent No. 5,875,249 (Mintzer et al.), U.S. Patent No. 6,031,914 (Tewfik et al.), U.S. Patent No. 5,912,972 (Barton), and U.S. Patent No. 5,949,885 (Leighton) disclose methods of applying a perceptually invisible watermark to image data as verification of authorship or ownership or as evidence that an image has not been altered. However, where such methods identify and validate image data, they provide no direct means of protection against copying an image, such as using a conventional scanner and color printer. In contrast, U.S. Patent No. 5,530,759 (Braudaway et al.) discloses providing a visible, color correct watermark that is generated by altering brightness characteristics but not chromaticity of specific pixels in the image. But the approach used in U.S. Patent No. 5,530,759 could be objectionable if used for a motion picture, since the continuing display of a watermark on film could annoy an audience and adversely affect the viewing experience.

The above examples for still-frame images illustrate a key problem: an invisible watermark identifies but does not adversely affect the quality of an illegal copy, while a visible watermark can be distracting and annoying. With video and motion picture images, there can be yet other problems with conventional image watermarking. For example, U.S. Patent No. 5,960,081 (Vynne et al.) discloses applying a hidden watermark to MPEG data using motion vector data. But this method identifies and authenticates the original compressed data stream and would not provide identification for a motion picture that was copied using a camcorder. Other patents, such as U.S. Patent No. 5,809,139 (Girod et al.), U.S. Patent No. 6,069,914 (Cox), and U.S. Patent No. 6,037,984 (Isnardi et al.) disclose adding an imperceptible watermark directly to the discrete cosine transform (DCT) coefficients of a MPEG-compressed video signal. If such watermarked images are subsequently recompressed using a lossy compression method (such as by a camcorder, for example) or are modified by some other image processing operation, the watermark may no longer be detectable.

The invisible watermarking schemes disclosed in the patents listed above add a watermark directly to the compressed bit stream of an image or image sequence. Alternatively, there are other watermarking schemes that add the watermark to the image data itself, rather than to the compressed data representation. An example of such a scheme is given in U.S. Patent No. 6,044,156 (Honsinger et al.), which discloses a spread spectrum technique using a random phase carrier. However, regardless of the specific method that is used to embed a watermark, there is always a concern that a watermarking method be robust, that is, able to withstand various "attacks" that can remove or alter the watermark. Some attacks may be deliberately aimed at the underlying structure of a given watermarking scheme and require detailed knowledge of watermarking techniques applied. However, most attack methods are less sophisticated, performing common modifications to the image such as using lossy compression, introducing lowpass filtering, or cropping the image, for example. Such modifications can be made when a video camera is used to capture a displayed motion picture. These methods present a constant threat that a watermark may be removed during the recording process.

The watermarking schemes noted above are directed to copy identification, ownership, or authentication. However, even if a watermarking approach is robust, provides copy control management, and succeeds in identifying the source of a motion picture, an invisible watermark may not be a sufficient deterrent for illegal copying.

As an alternative to watermarking, some copy deterrent schemes used in arts other than video or movie display operate by modifying a signal or inserting a different signal to degrade the quality of any illegal copies. The modified or inserted signal does not affect playback of a legally obtained manufactured copy, but adversely impacts the quality of an illegally produced copy. As one example, U.S. Patent No. 5,883,959 (Kori) discloses deliberate modification of a burst signal to foil copying of a video. Similarly, U.S. Patent No. 6,041,158 (Sato) and U.S. Patent No. 5,663,927 (Ryan) disclose modification of expected video signals in order to degrade the quality of an illegal copy. As yet another example of this principle, U.S. Patent No. 4,644,422 (Bedini) discloses adding a degrading signal to discourage copying of audio recordings. An audio signal having a frequency at and above the high threshold frequency range for human hearing is selectively inserted into a recording. The inserted signal is not detectable to the listener. However, any unauthorized attempt to copy the recording onto tape obtains a degraded copy, since the inserted audio signal interacts adversely with the bias oscillator frequency of a tape recording head.

The above-mentioned copy protection schemes disclose the use of a deliberately injected signal introduced in order to degrade the quality of an electronic copy. While such methods may be effective for copy protection of data from a tape or optical storage medium, these methods do not discourage copying of a motion picture image using a video camera.

As a variation of the general method where a signal is inserted that does not impact viewability but degrades copy quality, U.S. Patent No. 6,018,374 (Wrobleski) discloses the use of a second projector in video and motion picture presentation. This second projector is used to project an infrared (IR) message onto the display screen, where the infrared message can contain, for example, a date/time stamp, theater identifying text, or other information. The infrared message is not visible to the human eye. However, because a video camera has broader spectral sensitivity that includes the IR range, the message will be clearly visible in any video camera copy made from the display screen. The same technique can be used to distort a recorded image with an "overlaid" infrared image. While the method disclosed in U.S. Patent No. 6,018,374 can be effective for frustrating casual camcorder recording, the method has some drawbacks. A more sophisticated video camera operator could minimize the effect of a projected infrared watermark using a filter designed to block infrared light. Video cameras are normally provided with some amount of IR filtering to compensate for silicon sensitivity to IR. With a focused watermark image, such as a text message projected using infrared light, retouching techniques could be applied to alter or remove a watermark, especially if the infrared signal can be located within frame coordinates and is consistent, frame to frame. A further drawback of the method disclosed in U.S. Patent No. 6,018,374 relates to the infrared light source itself. Since an infrared lamp can generate significant amounts of heat, it may not be practical to project a watermark or copy deterrent image over a large area of the display screen using only an IR source.

Motion picture display and video recording standards have well-known frame-to-frame refresh rates. In standard motion picture projection, for example, each film frame is typically displayed for a time duration of 1/24 second. Respective refresh rates for interlaced NTSC and PAL video recording standards are 1/60 second and 1/50 second. Video camera capabilities such as variable shutter speeds allow close synchronization of a video camera with film projection, making it easier for illegal copies to be filmed within a theater. Attempts to degrade the quality of such a copy include that disclosed in U.S. Patent No. 5,680,454 (Mead). U.S. Patent No. 5,680,454, which discloses use of a pseudo-random variation in frame rate, causing successive motion picture frames to be displayed at slightly different rates than nominal. Using this method, for example, frame display periods would randomly change between 1/23 and 1/25 second for a nominal 1/24 second display period. Timing shifts within this range would be imperceptible to the human viewer, but significantly degrade the quality of any copy filmed using a video camera. Randomization, as used in the method of U.S. Patent No. 5,680,454, would prevent resynchronization of the video camera to a changed display frequency. While the method of U.S. Patent No. 5,680,454 may degrade the image quality of a copy made by video camera, this method does have limitations. As noted in the disclosure of U.S. Patent No. 5,680,454, the range of frame rate variability is constrained, since the overall frame rate must track reasonably closely with accompanying audio. Also, such a method does not provide a mechanism for including any type of spatial pattern or watermark in each frame, which could be used to provide a human-readable warning message or to trace the individual copy of the film that was illegally recorded.

U.S. Patent No. 5,959,717 (Chaum) also discloses a method and apparatus for copy prevention of a displayed motion picture work. The apparatus of U.S. Patent No. 5,959,717 includes a film projector along with a separate video projector. The video projector can be used, for example, to display an identifying or cautionary message or an obscuring pattern that is imperceptible to human viewers but can be recorded using a video camera. Alternately, the video camera may even display part of the motion picture content itself. By controlling the timing of the video projector relative to film projector timing, a message or pattern can be made that will be recorded when using a video camera, but will be imperceptible to a viewing audience. The method of U.S. Patent No. 5,959,717, however, has some drawbacks. Notably, this method requires distribution of a motion picture in multiple parts, which greatly complicates film replication and distribution. Separate projectors are required for the film-based and video-based image components, adding cost and complexity to the system and to its operation. Image quality, particularly for large-screen environments, may not be optimal for video projection and alignment of both projectors to each other and to the display surface must be precisely maintained.

Conventional methods such as those described above could be adapted to provide some measure of copy deterrence and watermarking for digital motion pictures. However, none of the methods noted above is wholly satisfactory, for the reasons stated. None of the existing copy protection or watermarking methods takes advantage of key characteristics of the digital motion picture environment that would prevent successful recording using a video camera.

While the capability for encoding "passive" invisible digital watermarks within digital images data has been developed, there is a need for more aggressive copy-deterrence techniques that can be embedded within digital motion picture data content and can take full advantage of digital projector technology.

Image aliasing is a well-known effect that results from a difference between the scan line or frame refresh rate of an electronic display or motion picture and the sampling rate of a video camera. Inherently, image aliasing imposes some constraints on the image quality of a video camera recording made from a display screen. Thus it is known that simply varying a scan or refresh rate may result in increased levels of aliasing. For example, video projectors from Silicon Light Machines, Sunnyvale, CA, use a high scan rate and complex segmented scanning sequence that can corrupt a video-taped copy by producing vertical black bars in the captured image. Similar effects are also observed when one tries to capture an image from a computer screen with a camcorder. These are the result of differences in scan rates between the display and the video camera systems. These techniques, however, offer a somewhat limited capability for protection, since scan synchronization of video camera apparatus makes it feasible to override this protection. Moreover, aliasing caused by simple scan rate differences does not provide a suitable vehicle for display of a warning message or other pattern in a taped copy or for digital watermarking in order to identify the source of the original image.

In a fully digital motion picture system, the spectral content and timing of each displayed pixel is known and can be controlled for each frame. While there can be a standard refresh rate for screen pixels (corresponding to the 1/24 or 1/30 second frame rate used for motion picture film or video displays), there may be advantages in altering the conventional "frame-based" model for motion picture display. Each displayed pixel on the screen can be individually addressed within any frame, and its timing characteristics can be modified as needed. This capability has, however, not been used for displaying a copy-deterrent pattern.

Therefore, it can be seen that there is clearly a need for a method that allows embedding of a copy-deterrent pattern within motion picture content, where the content is projected from digital data. It would be most advantageous for such a pattern to be invisible to a viewer but recordable using a video camera. Further, it can be seen that there is a need for a method that uses the opportunity for control of timing and of individual screen pixel content that digital motion picture technology offers in order to discourage movie piracy using a video camera.

With the above description in mind, it is an object of the present invention to provide a method and apparatus for displaying, within a projected frame of a digital motion picture, said frame comprising an array of pixels, a copy-deterrent pattern, said pattern comprising a plurality of pixels selected from said frame, said pattern not visible to a human viewer but perceptible when sampled and displayed using a video capture device. In a preferred embodiment, the pattern is modulated at a modulation rate and modulation scheme selected to maximize signal aliasing when said digital motion picture is sampled by said video capture device.

It is another object of the present invention to provide a copy-deterrent projection apparatus for projecting a digital motion picture onto a display screen, said digital motion picture comprising a sequential plurality of frames, each of said frames comprising an array of pixels, each said pixel assigned to be projected at a predetermined intensity for the duration of each said frame, said apparatus comprising: <ul id="ul0001" list-style="none" compact="compact"><li>(a) a pattern generator capable of specifying a pattern of pixels within each said frame and capable of modulating said pattern at a modulating frequency and using a modulation timing capable of being varied so as to produce a modulated pattern, said modulating frequency chosen for aliasing with the displayed digital motion picture when sampled by a video capture device, said pattern generator also capable of specifying a variable intensity level for each said pixel within said pattern of pixels;</li><li>(b) an image forming assembly capable of accepting said modulated pattern specified by said pattern generator and of projecting onto said display screen, within each said frame, said modulated pattern.</li></ul>

A feature of the present invention is the deliberate use of modulation frequency and modulation timing in order to obtain aliasing of the projected image when sampled using a video capture device. At the same time, however, modulation effects are not perceptible to a human viewer.

It is an advantage of the present invention that it provides an apparatus and method for obscuring an illegal copy of a projected digital motion picture, where said apparatus and method apply copy protection at the time of projection.

It is a further advantage of the present invention that it provides a method for displaying a copy-deterrent effect that is imperceptible to a viewing audience.

It is a further advantage of the present invention that it allows digital watermarking of projected digital motion picture frames using modulation of projected pixels at frequencies that are not perceptible to a viewing audience.

These and other objects, features, and advantages of the present invention will become apparent to those skilled in the art upon a reading of the following detailed description when taken in conjunction with the drawings wherein there is shown and described an illustrative embodiment of the invention.

While the specification concludes with claims particularly pointing out and distinctly claiming the subject matter of the present invention, it is believed that the invention will be better understood from the following description when taken in conjunction with the accompanying drawings, wherein: <ul id="ul0002" list-style="none" compact="compact"><li>Figure 1 is a graph showing relative sensitivity of the human eye to flicker;</li><li>Figure 2 is a graph showing the timing arrangement used for double-shuttering with conventional film projection;</li><li>Figure 3 is a graph showing the timing arrangement for a single pixel in digital motion picture projection;</li><li>Figure 4 is a graph showing a time-domain representation of a raised cosine having an oscillation frequency of 10 Hz;</li><li>Figure 5 is a graph showing a frequency-domain representation of a visible signal, flickering at 10 Hz;</li><li>Figure 6 is a graph showing a frequency-domain representation of a signal oscillating at 80 Hz;</li><li>Figure 7 is a graph showing a time-domain representation of a function oscillating at 10 Hz and sampled at 80 Hz;</li><li>Figure 8 is a graph showing a frequency-domain representation of the sampled function of Figure 7;</li><li>Figure 9 is a graph showing a frequency-domain representation of the function of Figure 7 sampled at 30 Hz;</li><li>Figure 10 is a graph showing a frequency-domain representation of an aliasing condition for a sampled, oscillating function;</li><li>Figure 11a is a graph showing a frequency-domain representation of a sinusoidally oscillating function truncated to contain only 10 cycles;</li><li>Figure 11b is a graph showing a frequency-domain representation of a sinusoidally oscillating function truncated to contain only 5 cycles;</li><li>Figure 12 is a graph showing a time-domain representation of an introduced flicker effect for a pixel;</li><li>Figure 13 is a graph showing changing pixel intensity values for n consecutive frames of a digital motion picture;</li><li>Figure 14 is a graph showing a frequency-domain representation of the changing pixel intensity values of Figure 13;</li><li>Figure 15 is a graph showing a frequency-domain representation of the intensity values of Figure 13, modulated;</li><li>Figure 16 is a graph showing a frequency-domain representation of the time-varying function of Figure 13, not modulated, but sampled;</li><li>Figure 17 is a graph showing a frequency-domain representation of the time-varying function of Figure 13, modulated and sampled;</li><li>Figure 18 is a plane view showing a digital motion picture frame having a pattern;</li><li>Figure 19 is a plane view showing a digital motion picture frame with a message as a pattern;</li><li>Figure 20 is a plane view showing a digital motion picture frame with multiple regions defined for modulation of patterns;</li><li>Figure 21 is a flow chart showing the decision process followed for the method of the present invention.</li><li>Figure 22 is a schematic block diagram showing major components of the copy-deterrent projection apparatus of the present invention;</li><li>Figure 23 is a block diagram showing the signal conversion path for a digital projection system;</li><li>Figure 24 is a schematic block diagram showing major components of an alternative embodiment of the copy-deterrent projection apparatus of the present invention; and</li><li>Figures 25-28 are tables 2, 3, 4 and 5, respectively, as referred to in the specification.</li></ul>

The present description is directed in particular to elements forming part of, or cooperating more directly with, apparatus in accordance with the invention It is to be understood that elements not specifically shown or described may take various forms well known to those skilled in the art.

The present invention provides a method and apparatus capable of providing copy protection and watermarking for digital motion picture display. The present invention accomplishes this purpose by introducing, as part of the displayed images, a copy-deterrent pattern that is imperceptible to a human observer, but that is clearly perceptible when captured using a video camera or related image capture device that uses sampling for image capture. In order to adequately disclose an implementation for practice of the present invention, it is first necessary to describe specific boundaries within which the method and apparatus of the present invention operate.

Sensitivity of the Human Visual System to Time-Varying Stimuli

The first boundary of interest relates to the flicker sensitivity of the human visual system. Flicker sensitivity refers to the perception of a light source with time-varying intensity (e.g., a strobe light) as a steady illumination source.

For a time-varying stimulus at a given temporal frequency and under a given set of viewing conditions, such as image display size and adaptation level, an average threshold amplitude can be identified at which the time-varying stimulus is no longer perceived as flickering (that is, the flicker-fusion threshold). Studies show that sensitivity of the human visual system to sinusoidal intensity oscillations decreases dramatically at higher temporal frequencies. (Reference is made to Kelly, D. H., "Visual Responses to Time-Dependent Stimuli: Amplitude Sensitivity Measurements" in Journal of the Optical Society of America, Volume 51, No. 4, p. 422 and to Kelly, D. H., "Visual Responses to Time-Dependent Stimuli: III Individual Variations" in Journal of the Optical Society of America, Volume 52, No. 1, p. 89). Referring to Fig. 1, which shows the flicker-fusion threshold as a function of temporal frequency, human visual system sensitivity to flicker is maximized near the 10-30 cycles/sec range, drops off rapidly at just above 30 cycles/sec, and continues to drop as temporal frequency increases. For temporal frequencies above a cutoff frequency, there is essentially no perception of flicker regardless of the stimulus amplitude. This cutoff frequency occurs somewhere around 50-70 Hz for the light adaptation levels that occur in typical display systems.

Although the flicker sensitivity results shown in Fig. 1 refer to sinusoidal intensity oscillations, it is possible to derive similar curves for other types of time-varying stimuli. Of particular interest are squares waves, which have distinct "ON" and "OFF" phases, as opposed to the continuously varying characteristics of sinusoids. For square waves, it is possible to compute a "duty cycle" which is the proportion of the ON interval in a full ON/OFF cycle. A typical duty cycle is 50%, meaning that the ON and OFF intervals are equal. However, it is also possible to create stimuli with longer duty cycles, so that the ON duration is greater than the OFF duration within a single cycle. In addition, other stimuli might have a non-zero light intensity during the OFF phase, so that the stimuli are never completely dark. In general, studies have shown that increasing the duty cycle and/or increasing the intensity during the OFF phase results in reduced flicker sensitivity. It has been demonstrated that the sensitivity variations for many different stimuli can be explained by considering the amplitude of the fundamental frequency component of the stimulus waveform. (Reference is made to Kelly, D. H., Flicker Fusion and Harmonic Analysis" in Journal of the Optical Society of America, Volume 51, p. 917.)

Relevant to the present invention, when a sequence of motion picture frames is displayed at a sufficiently high temporal frequency, a human observer does not detect flicker but instead integrates the sequence of frames to perceive the effect of images in smooth motion. However, video cameras do not use the same detection mechanisms as the human visual system. Thus, it is entirely possible for a time-varying illumination to be captured by a video camera while the human observer detects only a steady illumination.

The object of the present invention is to provide, utilizing this inherent sensitivity of the human visual system and using the ability of a digital motion picture projection system to control timing and intensity levels at each individual pixel, an apparatus and method for frustrating illegal filming of a digital motion picture using a video camera. The present invention operates by inserting a time-varying pattern within successive projected digital motion picture frames, where the time-varying pattern cannot be detected by the unaided eye but is clearly visible from a video camera.

Conventional versus Digital Motion Picture Projection

Another boundary of interest relates to the nature of motion picture projection as it has evolved using film during the past century, and to new capabilities inherent to digital motion picture projection. It is instructive to distinguish the mode of operation used by display projectors for digital motion pictures from the mode used for film projectors with conventional motion picture films.

A conventional film projection system consists generally of a high brightness arc lamp and a lens assembly that are used to illuminate and project film frames onto the display screen. Film frames are typically captured at 24 frames/sec, but projection at this same rate is undesirable as a 24Hz frequency is within the region of high flicker sensitivity as noted in the preceding discussion. Therefore, in order to reduce the perceptible flicker of projected films, a technique known as double shuttering is used. Double shuttering increases the effective display rate to 48 frames/sec by alternatively blocking and unblocking the projected light twice during the projection of each frame. This concept is shown in Fig. 2 for an ideal case of two consecutive frames, n and n+1 at intensity levels I_n and I_n+1, respectively. Double shuttering increases the presentation rate to a frequency at which flicker sensitivity is greatly reduced as compared to the sensitivity at 24 Hz, which allows a film to be viewed without flicker. It is also noted that double shuttering further improves the quality of the projected images by allowing a new frame to settle (during the first "OFF" time) before it is projected on the screen. In a practical situation, there is a finite transition time between the "ON" and "OFF" periods and therefore, the square-shaped blocks of Fig. 2 may be replaced by saw-tooth functions, for example.

In a conventional film projection system, each film frame is illuminated by a light source that has approximately constant intensity across the full extent of the frame. Moreover, each frame is sequentially projected from a film reel, and the average illumination intensity is held constant from frame to frame, as controlled by the shutter. In contrast, digital motion picture display projectors are capable of controlling, for each pixel in a two-dimensional array of pixels, multiple characteristics such as intensity, color, and refresh timing. With digital motion picture projection, the "image frame" presented to the viewer is a projection of this two-dimensional pixel array.

In a digitally projected movie, there is no need for shuttering. The projected frames consist of individual pixels, typically made up of three primary component colors (Red, Green, and Blue, abbreviated RGB) and having variable intensity, where frames are refreshed at regular intervals. This refresh rate may be 1/24 of a second or higher. The transition time for the display of new pixel values, indicated as a pixel transition period in Fig. 3, is short enough that no perceptible flicker artifacts are produced. Fig. 3 shows a typical timing arrangement for a single pixel having two different intensity values, I₁ and I₂ for frames n and n+1, respectively.

Because motion pictures are typically captured at 24 frames/sec, the description that follows uses a 24 Hz frame refresh rate as the fundamental rate to be used for digital motion picture projection. However, the actual refresh rate could vary. The present invention is capable of adaptation to any standard refresh rate selected. As mentioned, the object of the present invention is to provide an apparatus and method for frustrating illegal filming of a digital motion picture using a video camera, by using the ability of a digital motion picture display system to control timing and intensity levels at each individual pixel.

Sampling of Movie Content by Video Camera

A video camera operates by sampling a scene at regular time intervals. By sampling at a fast enough rate, a video camera can reproduce time-varying scenes with sufficient accuracy for the human visual system to perceive the temporally sampled data as continuous movement.

However, the complication with video camera sampling of a motion picture is that the motion picture display is not truly continuous, as is noted above. Thus, attempting to capture a motion picture using a video camera introduces the complexity of sampling a time-varying image display using time-varying sampling apparatus. Intuitively, it can be seen that some synchronization of sampling rate to refresh rate would be most likely to yield satisfactory results.

It may be possible to adjust the sampling rate of a capturing device to provide synchronization between the video camera capture frequency and the motion picture projector frequency. Frame-to-frame synchronization of a video camera capture frequency to a motion picture projector frequency then enables illegal filming of a displayed motion picture with few, if any, imaging anomalies due to timing differences. The method and apparatus of the present invention is intended to prevent any type of adequate synchronization, thereby deliberately causing interference due to frequency differences to obscure or mark any copy of a motion picture obtained using a video camera.

The baseline sampling rates for video cameras can vary over a range of discrete values. Typical sampling rates for most video cameras commercially available are in a range between 60-120 Hz. For example, the NTSC and PAL video standards, conventionally used for commercially available video cameras, use discrete rates of 50 and 60 fields per second, respectively. Optionally, in some of the so-called flickerless video cameras, multiples of these base rates can be used, allowing higher sampling rates of 100 or 120 Hz, respectively. These rates are, in turn, easily convertible to the 50 and 60 fields per second replay rates that are used in most TVs and VCRs.

It must be noted that the present invention is not constrained to any assumption of video camera sampling rate being at a specific value. However, for the purpose of description, a standard, discrete sampling rate within the 50-120 Hz range is assumed. In subsequent description, sampling rate is represented as ξ_s.

With these bounds of human visual system flicker sensitivity, pixel refresh rate of the display, and video camera sampling rate as outlined above, it is next instructive to describe the tools and techniques used for analyzing and describing frequency-related phenomena in general.

Time Domain and Frequency Domain

As is well known in the signal processing arts, it is possible to describe and quantify a time-varying signal in either a time domain or in a frequency domain. The frequency domain is assuredly the less intuitive of the two. However, in order to clearly disclose the functions performed by the apparatus and method of the present invention, it is most illustrative to utilize the descriptive tools and representation of the frequency domain. (The following discussion will highlight those features most pertinent to description of the present invention. A more detailed theoretical description can be found in an upper-level undergraduate or graduate text in linear systems analysis, from which the following description can be derived. The nomenclature used in the subsequent description substantially follows the conventions used in a standard upper-level text, Linear Systems, Fourier Transforms, and Optics, by Jack D. Gaskill, published by John Wiley &amp; Sons, New York, NY, 1978.

The frequency domain representation is also sometimes called the signal "spectrum". Within certain constraints, the mathematical transformation between the time and frequency domains is accomplished via Fourier Transformation. Using this transformation tool, the relationship between the function in time domain, f(t), and the function in frequency domain, F(ξ), may be written as:<maths id="math0001" num=""><img file="EP1237369A2_D0001.tif" /></maths><maths id="math0002" num=""><img file="EP1237369A2_D0002.tif" /></maths>

Here, f(t) and F(ξ) are referred to as Fourier Transform pairs. It can be seen from the above equations that Fourier transformation is reversible or invertible. In other words, if F(ξ) is the transform of f(t), then f(ξ) will be the Fourier transform of F(t).

There have been a number of corresponding pairs of invertible functions derived in working between the two domains. Some of the more useful Fourier transform pairs for the present discussion are given in Table 1. <tables id="tabl0001" num="0001"><img file="EP1237369A2_D0003.tif" /></tables>

Note that multiplication in one domain corresponds to convolution in the other domain. A shift in one domain corresponds to a linear phase multiplier in the other domain (e^-j2πbξ represents the phase).

The functions listed in Table 1 above are ideal, mathematical functions. Such idealized functions are rarely found under actual, measured conditions. However, functions such as these are useful for modeling and for high-level assessment of actual conditions, as will be apparent in subsequent description.

Raised Cosine as Modulation Model

It was noted above that the present invention takes advantage of differences between human eye sensitivity to a flickering pattern and video camera sensitivity to a flickering pattern. In order to describe the present invention clearly, it is beneficial to consider a model type of oscillation that is conceptually simple. For this purpose, the raised cosine function, as shown in Fig. 4, is a suitable model that illustrates how the present invention uses a modulation function to induce flicker in motion image sequences. Peak values of the raised cosine waveform would correspond to periods of highest intensity and valleys would correspond to the periods of lowest intensity on the screen.

The following is the time-domain equation for a "raised cosine" function:<maths id="math0003" num="(3)"><math display="block"><mrow><msub><mrow><mtext>f(t) = a[1 + cos(2πξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext>t)]</mtext></mrow></math><img file="EP1237369A2_D0004.tif" /></maths>

This function has an oscillation frequency ξ_m and an average or DC level equal to "a". The Fourier transform of f(t) may be calculated as:<maths id="math0004" num="(4)"><math display="block"><mrow><msub><mrow><mtext>F(ξ) = a[δ(ξ) + ½ [δ(ξ-ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><msub><mrow><mtext>) + δ(ξ+ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext>)]</mtext></mrow></math><img file="EP1237369A2_D0005.tif" /></maths>

Figs. 4 and 5 show f(t) and its spectrum, F(ξ), respectively, for a raised cosine function. For convenience, the constant amplitude multiplier, a, is omitted from the figures. The function, f(t), may be used to represent an intensity pattern on a movie screen, varying sinusoidally with respect to time. As indicated in Fig. 5, the spectrum of a raised cosine contains three components: the DC component, which is represented by a delta function at the origin, and two discrete frequency components at ±ξ_m (ξ_m=10 Hz in Fig. 5). A region of flicker perceptibility is also indicated in Fig. 5 by the shaded region, with a range ± ξ_c where ξ_c is the cutoff threshold frequency for flicker visibility as in Fig. 1. As the example of Figs. 4 and 5 shows, a flicker frequency, approximated by a sinusoid at 10 cycles per second, or 10 Hz, as shown in Fig. 4, has spectral frequencies ± ξ_m that are well within the perceptible range.

In contrast to the conditions of Figs. 4 and 5 where intensity flicker occurs at a 10 Hz oscillation rate, Fig. 6 shows what happens when the rate of intensity flicker is increased, to 80 Hz in this example. Notice that, for the conditions of Fig. 6, spectral frequencies ± ξ_m are not within the perceptible range. In contrast to what occurs under the conditions of Figs. 4 and 5, a viewer cannot perceive any flickering in pixel intensity if this flickering occurs at a rate of 80 Hz. In terms of Fig. 6, the left and right sidebands that represent spectral frequencies ± ξ_m have moved outside the perceptible range. The viewer perceives only a constant (that is, DC) intensity level.

The spectrum representation shown in Figs. 5 and 6 are idealized. In practice, actual measured modulated signals do not exhibit a spectrum that is as easily visualized. However, as will be seen subsequently, the representation given in Figs. 5 and 6 is sufficiently close for illustrating actual modulation behavior.

Sampling Frequency Considerations

As noted above, a video camera operates by periodically sampling an image, unlike the human eye. The rate at which this sampling occurs, that is, the sampling frequency, affects how the video camera responds to a flicker pattern having a specified flicker frequency. The interplay of video camera sampling frequency and display flicker frequency must be considered in order to make effective use of the present invention.

In order to reproduce a time-varying signal, a capture system acquires samples of that signal at given instants in time. Intuitively, if the samples are "close enough" to each other, the time varying signal function may be reproduced with great accuracy. The Shannon-Whittaker sampling theorem quantifies the above statement by indicating how close the samples have to be in order to exactly reproduce a signal function. According to this theorem, if a function, f(t), is band limited (i.e., frequency components of F(ξ) are contained within a limited range of frequencies, |<maths id="math0005" num=""><math display="inline"><mrow><mfrac><mrow><mtext mathvariant="italic">W</mtext></mrow><mrow><mtext>2</mtext></mrow></mfrac></mrow></math><img file="EP1237369A2_D0006.tif" /></maths>|), then it is only necessary to sample f(t) at <i>t</i>_<i>N</i> = <maths id="math0006" num=""><math display="inline"><mrow><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext mathvariant="italic">W</mtext></mrow></mfrac></mrow></math><img file="EP1237369A2_D0007.tif" /></maths> intervals, or higher, in order to perfectly reproduce f(t). This theorem is sometimes referred to as the Nyquist Theorem and ξ_N=<maths id="math0007" num=""><math display="inline"><mrow><mfrac><mrow><mtext>2</mtext><mtext mathvariant="italic">W</mtext></mrow><mrow><mtext>2</mtext></mrow></mfrac></mrow></math><img file="EP1237369A2_D0008.tif" /></maths> is referred to as the Nyquist Frequency. Mathematically, sampling of a function in time and frequency domains can be described as follows:<maths id="math0008" num="(5)"><math display="block"><mrow><msub><mrow><mtext>f</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext>(t) = f(t) . |1/t</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext>|. comb(t/t</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><mtext>)</mtext></mrow></math><img file="EP1237369A2_D0009.tif" /></maths><maths id="math0009" num=""><img file="EP1237369A2_D0010.tif" /></maths> where t_s is the sampling interval, ξ_s=1/t_s is the sampling frequency, and "*" indicates convolution. It should be noted that an idealized "zero-width" sampling function, |1/t_s|.comb(t/t_s), is used in the following description to facilitate the understanding of the underlying concepts. In practice, rectangular functions of finite width are used for sampling. The ramifications of such sampling functions will be discussed below.

According to the above equations, sampling of a function, f(t), results in the replication of its spectrum, F(ξ), at intervals ξ_s along the frequency axis. The replicates of F(ξ) are referred to as the spectral orders of f_s(t), with F(ξ- nξ_s) known as the nth spectral order. As a straightforward first approximation for initial analysis, if f(t) is selected to be a raised cosine function, then the sampled function and its spectrum will take the following forms:<maths id="math0010" num="(7)"><math display="block"><mrow><msub><mrow><mtext>f</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext>(t) = a[1 + cos(2πξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><msub><mrow><mtext>t)]. |1/t</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext>|. comb(t/t</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><mtext>)</mtext></mrow></math><img file="EP1237369A2_D0011.tif" /></maths><maths id="math0011" num=""><img file="EP1237369A2_D0012.tif" /></maths>

Figures 7 and 8 depict f_s(t) and E_s(ξ), respectively, for a sampling frequency ξ_s of 80 Hz. The constant multipliers are once again omitted for convenience. Note that in Fig. 8, because of high sampling rate, all spectral orders other than the 0^th order are outside the region of flicker visibility. In contrast, Fig. 9 shows the spectrum of the same raised cosine function that is sampled at a lower rate of ξ_s= 30 Hz. It is evident that some of the frequency components of the ±1^st orders have moved into the region of flicker visibility when using this lower sampling rate. If the temporally sampled version of f(t) with ξ_s= 30 Hz were to be subsequently displayed and viewed by a human observer, extraneous frequency artifacts (i.e., flickers) would be observed in addition to the inherent flicker of the raised cosine at 10 Hz. It should be noted that where oscillation and sampling frequencies interact as is shown in Fig. 9, there can be some remedial steps taken to minimize frequency artifacts in video camera recording, such as using temporal low-pass filters, for example.

Aliasing

Recalling that the purpose of the present invention is to cause frequency artifacts, it can be seen that there would be advantages in causing signal aliasing, where such aliasing could not then be remedied using low-pass filter techniques. Aliasing is often the cause of visual artifacts in the display of sampled images that contain high frequency components (a familiar example resulting from aliasing is the effect by which carriage or locomotive wheels appear to rotate backwards in early movies).

If the sampling rate, ξ_s, is below the Nyquist frequency, the components from higher order harmonics will overlap the 0^th order components. This phenomenon is known as aliasing. This can be readily visualized by examining Figs. 8,9 and 10 in progression. In Fig. 8, only the fundamental or "0 order" frequency components are within the perceptible region; none of the +1 or -1 order components will be perceptible to a viewer. Moving from Fig. 8 to Fig. 9, the affect of a change in sampling frequency ξ_s is illustrated. Fig. 9 shows how reducing the sampling frequency can cause lower order frequency components (shown in dashed lines) to creep toward the origin, into the perceptible region. In Fig. 9, components from the +1 and -1 orders now are within perceptible range.

Fig. 10 shows an example in which a raised cosine intensity pattern has an oscillation frequency of 50 Hz. This results in spectral components at ± 50 Hz. As noted above, the eye does not easily perceive the flicker of this pattern since this oscillation frequency ξ_m is near the visible threshold. Now, if the displayed pattern is captured by a camcorder that has a sampling rate ξ_s= 30 Hz, aliased components from higher order harmonics now fall within the region of perceptibility and degrade the viewed image. This is depicted in Fig. 10 with only the ±1^st order harmonics present. It is evident that the ±2^nd spectral orders will also introduce some additional frequency artifacts into the perceptible region. For clarity, however, these orders are not shown to avoid cluttering the figure.

Aliasing occurs whenever any portion of the higher order components overlap the 0 order frequency components, as is illustrated in Fig. 10. It was noted above that, given the conditions shown in Fig. 9, a temporal low-pass filter could be employed to isolate only those frequency components of the fundamental order. However, it must be pointed out that aliasing, as illustrated in Fig. 10, does not permit a straightforward remedy using filtering techniques.

At this point, it is instructive to re-emphasize that aliasing, as is used by the present invention, occurs as a result of sampling over discrete intervals, as performed by a video camera. The human eye does not "sample" a motion picture image in the same manner. Thus, aliasing effects as illustrated in Fig. 10 occur only with respect to video camera sampling; a human viewer would not perceive any such aliasing effect in watching the displayed motion picture itself.

The preceding discussion has focused on the interaction of oscillation and sampling frequencies from an idealized theoretical perspective. With the principles of this interaction in mind, it is now instructive to describe some more practical aspects of actual sampling conditions, in order to provide a framework for understanding how to implement the present invention.

Effects of a Finite Sampling Duration

The above description of spectral frequencies and aliasing used an ideal "zero-width" comb sampling function |1/t_s|comb(t/t_s) as a straightforward first approximation to show the interaction of oscillation and sampling frequencies. The comb function is familiar to those skilled in the digital signal processing arts. In practice, however, sampling functions have a finite integration time duration. For the purpose of this disclosure, this is termed a "finite sampling duration".

A good approximation of a practical function with some finite width is a rectangular function of width (i.e., duration) "d" that is used to sample the function f(t) at fixed intervals t_s. In such a case, the sampled time-varying function takes the following form:<maths id="math0012" num=""><img file="EP1237369A2_D0013.tif" /></maths>    that is, equivalent to f(t) times a sampling function. Its corresponding spectrum equivalent takes the following form:<maths id="math0013" num=""><img file="EP1237369A2_D0014.tif" /></maths>

Here, the result in the frequency domain can be interpreted again as series of F(ξ) functions that are replicated at ξ_s intervals. But, as equation (10) above shows, the amplitude of each nth spectral order is attenuated by a constant value, that is, by a sinc( ) function evaluated at ξ= dnξ_s. Thus, the amplitude of the ±1^st orders is attenuated by sinc(dξ_s), the amplitude of the ±2^nd orders is attenuated by sinc(2d ξ_s), etc. It is evident that if d&lt;&lt;t_s (that is, the width of the rectangular sampling function is much smaller than the interval between samples), the attenuation due to the sinc( ) function is negligible for the first few spectral orders. However, if the value of d becomes comparable to T_s, all spectral orders (other than the 0^th order) will undergo attenuation dictated by the sinc( ) envelope. The ramification of this phenomenon is reduced visibility of the higher spectral orders if they fall in the region of flicker perceptibility.

Time-varying functions of Finite Extent

Referring back to Figs. 4 and 5, function f(t) was represented as a raised cosine extending from negative infinity to positive infinity. In practice, any time-varying function f(t) must be limited to some finite time interval . In the case of motion pictures this finite time interval corresponds to some portion or the entire presentation length of the movie. This truncation can be mathematically represented by multiplying f(t) by a rectangular function for the duration of interest. Here, the duration (i.e., width of the rectangle) is selected to be D=nt_f,, where n is an integer typically much greater than 1 and t_f is an arbitrary time interval, usually representing the duration of one movie frame. Then the finite time-varying function f_f(t) and its spectrum F_f(ξ) can be represented by the following:<maths id="math0014" num=""><img file="EP1237369A2_D0015.tif" /></maths><maths id="math0015" num=""><img file="EP1237369A2_D0016.tif" /></maths>    where ξ_D=1/D.

Note that 2ξ_D is the width of the main lobe of the sinc( ) function. The main lobe contains the majority of the energy (i.e., area) of the sinc( ). Examination of the above equations reveals that if the spectral width of the sinc( ) is small compared to the spectral width of F(ξ) (which means that the width, D, of the rectangular function is large), the sinc( ) function effectively acts as a delta function. For example, the spectrum of the raised cosine intensity pattern shown in Fig. 5 would contain narrow sinc( ) functions in place of delta functions. In this case, the pure spectral components at ±ξ_m are replaced by broadened components at ±ξ_m that have the shape of a sinc( ) function. Fig. 11a shows this broadening effect for the raised cosine function truncated to contain 10 cycles (i.e., ξ_D=10ξ_m). Fig. 11b shows the spectrum for a raised cosine function truncated to contain 5 cycles (i.e., ξ_D=5ξ_m).

As ξ_D becomes comparable to the spectral extent of F(ξ) (i.e., ξ_D ≈ ξ_m), the spectral components overlap. This has no impact on the concepts developed so far, but makes visualization of the spectrum plots somewhat difficult. In a practical implementation of the present invention, the target sequence of images usually contain a large number of frames, and the spectral broadening effects are negligible.

Realistic Intensity Function

Using the raised cosine function as a model, the above description illustrates basic concepts underlying pixel modulation, frequency sampling, and aliasing, all viewed with respect to the frequency domain. The next part of this description considers a more realistic time-varying pixel intensity function f(t). Referring to Fig. 13, there is shown an example intensity function for a pixel over n consecutive image frames. (Pixel transition time, assumed to be small relative to frame duration, is not shown.)

For the example represented in Fig. 13, t_f is the frame duration and parameters a₀, a₁, ..., a_n are used to indicate the height of each rectangular function (i.e., the amplitude of the intensity function for duration of frames 0, 1, ... n, respectively). Mathematically, this train of rectangular functions may be represented as follows:<maths id="math0016" num=""><img file="EP1237369A2_D0017.tif" /></maths><maths id="math0017" num=""><img file="EP1237369A2_D0018.tif" /></maths><maths id="math0018" num="(15)"><math display="block"><mrow><mtext>= [</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>0</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext>)+</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext> - </mtext><msub><mrow><mtext mathvariant="italic">t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)+</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext> - 2·</mtext><msub><mrow><mtext mathvariant="italic">t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)+...+</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext mathvariant="italic">n</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext> - </mtext><mtext mathvariant="italic">n</mtext><mtext>·</mtext><msub><mrow><mtext mathvariant="italic">t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)]*</mtext><mtext mathvariant="italic">h</mtext><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>).</mtext></mrow></math><img file="EP1237369A2_D0019.tif" /></maths>

Equations 13 and 14 are mathematically equivalent. Insight can be gained from examination of equation 14, which presents the time-varying function f(t) as the convolution of a series of weighted and shifted delta functions with a rectangular function of width t_f. In the context of digital projection of pixels, each delta function can represent the pixel intensity value for the duration of one frame. Convolution with the rect( ) function effectively spreads (that is, interpolates) the value of each delta function over its frame duration, t_f. The rect( ) function, described in this manner, can be considered to be an interpolation or spread function. This description uses rectangular spread functions to represent projected movie pixels. It is significant to note that other forms of spread functions can be used in place of rectangles without compromising the validity of this analysis. Equation (15) describes the projected frame pixels in their most generic form.

The Fourier transform of f(t) from Equation (15) may be written generally as:<maths id="math0019" num="(16a)"><math display="block"><mrow><mtext mathvariant="italic">F</mtext><mtext>(ξ) = [</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>0</mtext></mrow></msub><mtext> + </mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><mtext> · </mtext><msup><mrow><mtext mathvariant="italic">e</mtext></mrow><mrow><mtext>-</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">j</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext>2π</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><msub><mrow><mtext mathvariant="italic">t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext>ξ</mtext></mrow></msup><mtext> + </mtext><mtext mathvariant="italic">a</mtext><mtext>2 · </mtext><msup><mrow><mtext mathvariant="italic">e</mtext></mrow><mrow><mtext mathvariant="italic">-j</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext>2π2</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">t</mtext></mrow></msup><msup><mrow><mtext mathvariant="italic">f</mtext></mrow><mrow><mtext mathvariant="italic">ξ</mtext></mrow></msup><mtext> +...+ </mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext mathvariant="italic">n</mtext></mrow></msub><mtext> · </mtext><mtext mathvariant="italic">e</mtext><msup><mrow><mtext></mtext></mrow><mrow><mtext>-</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">j</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext>2π</mtext></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><msub><mrow><mtext mathvariant="italic">nt</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub></mrow></msup><msup><mrow><mtext>​</mtext></mrow><mrow><mtext>ξ</mtext></mrow></msup><mtext>]·</mtext><mtext mathvariant="italic">H</mtext><mtext>(ξ)</mtext></mrow></math><img file="EP1237369A2_D0020.tif" /></maths>    and, in this particular case:<maths id="math0020" num=""><img file="EP1237369A2_D0021.tif" /></maths>    where ξ_f = 1/t_f and H(ξ) is the Fourier transform of the interpolation function h(t). For the example of Fig. 13, the Fourier transform of a rect() function is a sinc() function, as presented in Equation (16b) above. The term in square brackets is a series of weighted linear phase functions that contains both real and imaginary parts. In order to produce the spectrum plot of F(ξ), amplitude of the term in square brackets must be calculated. This will result in a complicated mathematical amplitude term, A(ξ), that is multiplied by a sinc( ) function:<maths id="math0021" num=""><img file="EP1237369A2_D0022.tif" /></maths>

It is possible to numerically or symbolically evaluate A(ξ) using scientific software packages, as is well known in the linear systems analysis arts. Fig. 14 shows a plot of |F(ξ)| for the case where n=9, t_f=1/24 and a_n coefficients that are arbitrarily selected to be: (a₀, a₁, a₂, a₃, a₄, a₅, a₆, a₇, a₈, a₉) = (1, 0.7, 0.8, 0.7, 0.6, 0.4, 0.5, 0.3, 0.4, 0.2). The plot of Fig. 14 indicates that the spectrum of this signal is nearly band limited to within ±24 Hz, but the energy of the signal becomes negligible beyond roughly ±12 Hertz.

For the purposes of the present invention, it is not necessary to know the exact form of |F(ξ)| in equation 17, as this quantity depends on varying parameters such as n and a_n. Instead, it is beneficial to examine some of the general characteristics of F(ξ). One point to note is that if A(ξ) is evaluated at different values of ξ, the highest value for A(ξ) is obtained at ξ=0, namely A(0) = a₀+a₁+...+a_n. Thus one property of A(ξ) is that its highest value occurs at the origin (that is, the DC value). This property plus the fact that A(ξ) is multiplied by a sinc( ) function, limits the spectral spread of |F(ξ)| to within the first few side lobes of the sinc( ) function. In other words, regardless of the exact form of A(ξ), the spectrum plot of F(ξ) is band limited and follows a sinc( ) envelope (more generally, follows the envelope of the Fourier transform of the spread function, h(t)). The sinc( ) envelope is shown in dashed lines in Fig. 14.

Modulating the Projected Pixel

There are a number of possible modulation schemes that can be employed in order to introduce a flickering pattern that is not perceptible to a human viewer, but that degrades a video camera copy by using the interaction of frequencies described here. For the pixel represented (without flicker) in Fig. 3, the basic concept for adding a flicker effect is illustrated in Fig. 12. Here, a frame interval of 1/24 second is further subdivided into 8 segments which are then used to provide flickering on/off intervals.

It is important to note, as shown in the example of Fig. 12, that average pixel intensity must be maintained in whatever modulation scheme is used. For example, using the timing arrangement of Fig. 12, a pixel is on for ½ of the total time when compared against Fig. 3. This represents a 50% duty cycle. To provide the same average intensity for the pixel, then, the pixel intensity during an on-time must be doubled.

The pulse-width modulation (PWM) technique shown in Fig. 12 gives a flicker frequency of 96 Hz, outside the perceptible range of the human eye. However, such a PWM modulated pixel appears to flicker when recorded and viewed from a typical video camera.

In Fig. 12, an idealized square wave modulation scheme is used. More generally, given a modulation function m(t), the modulated time-varying signal can be written as:<maths id="math0022" num="(18)"><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">f</mtext></mrow><mrow><mtext mathvariant="italic">m</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>)</mtext><mtext mathvariant="italic">= f</mtext><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>)</mtext><mtext mathvariant="italic">·m</mtext><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>) = {[</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>0</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext>)+</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext>1</mtext></mrow></msub><mtext>δ(</mtext><mtext mathvariant="italic">t</mtext><mtext> - </mtext><msub><mrow><mtext mathvariant="italic">t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)</mtext><msub><mrow><mtext mathvariant="italic">+a</mtext></mrow><mrow><mtext>2</mtext></mrow></msub><mtext mathvariant="italic">δ</mtext><mtext>(</mtext><mtext mathvariant="italic">t -</mtext><mtext> 2</mtext><msub><mrow><mtext mathvariant="italic">·t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)+...+</mtext><msub><mrow><mtext mathvariant="italic">a</mtext></mrow><mrow><mtext mathvariant="italic">n</mtext></mrow></msub><mtext mathvariant="italic">δ</mtext><mtext>(</mtext><msub><mrow><mtext mathvariant="italic">t - n·t</mtext></mrow><mrow><mtext mathvariant="italic">f</mtext></mrow></msub><mtext>)]*</mtext><mtext mathvariant="italic">h</mtext><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>)}</mtext><mtext mathvariant="italic">·m</mtext><mtext>(</mtext><mtext mathvariant="italic">t</mtext><mtext>).</mtext></mrow></math><img file="EP1237369A2_D0023.tif" /></maths>

For the time-varying function of Fig. 13, with a rectangular interpolation function, this becomes:<maths id="math0023" num=""><img file="EP1237369A2_D0024.tif" /></maths>

In practice, m(t) may be a sinusoidal function (such as the raised cosine discussed earlier) or a train of square waves with 50% duty cycle (as depicted in Fig. 12) or some other time-varying function. In order to simplify the analysis, a raised cosine modulation function is used. This modulation function is shown as dashed lines in Fig. 12. The modulated time-varying function then becomes:<maths id="math0024" num=""><img file="EP1237369A2_D0025.tif" /></maths>

The Fourier transform of this signal becomes:<maths id="math0025" num=""><img file="EP1237369A2_D0026.tif" /></maths>

The modulated spectrum, F_m(ξ), consists of three replicates of the unmodulated signal spectrum, F(ξ), given by equation 17 and plotted in Fig. 14. One of these replicates is located at the origin and the other two, with one-half the amplitude of the original spectrum, are centered respectively around ±ξ_m. In conventional linear systems analysis terminology, F(ξ), (ξ-ξ_m) and F(ξ+ξ_m) are sometimes referred to as the main band, right sideband and left sideband, respectively. The plot of F_m(ξ) is shown in Fig. 15 for the parameters n=9, t_f=1/24, t_m=1/48 and a_n coefficients (a₀, a₁, a₂, a₃, a₄, a₅, a₆, a₇, a₈, a₉) = (1, 0.7, 0.8, 0.7, 0.6, 0.4, 0.5, 0.3, 0.4, 0.2).

Note that in the absence of modulation, the signal spectrum only contains the first term, F(ξ). In effect, by introducing modulation into the system, we have advantageously increased the bandwidth (that is, the spectral spread) of the original signal by adding the components F(ξ ± ξ_m). It must be stressed once again that if a high enough modulation frequency, ξ_m, is selected, the left and right sidebands cannot be seen by the observer. However, the interaction of the modulation frequency with the video camera sampling frequency, ξ_s, will result in an aliased signal that is perceptible in the video camera copy.

Sampling the Projected Pixel

As a summary and wrap-up of the theoretical material provided above, it is instructive to illustrate, for a realistic example, the spectral characteristics of a sampled un-modulated signal and to compare these against the spectral characteristics of a sampled modulated signal.

As has been described above, if an un-modulated signal, f(t), is sampled at t_s intervals (t_s&lt; t_f), spectral orders appear in the spectrum plot. Referring to Fig. 16, this is shown for the time varying function of Fig. 13, using a sampling frequency of 180 Hz. Mathematically, the Fourier transform of the <b>un-modulated and sampled</b> signal can be written as follows:<maths id="math0026" num=""><img file="EP1237369A2_D0027.tif" /></maths>

If, instead of an un-modulated signal, sinusoidally modulated pixel values are sampled at a rate ξ_s=1/t_s (as with a video camera, for example), the spectrum contains broader spectral orders. This is depicted in Fig. 17, using a sampling frequency of 180 Hertz and modulation frequency of 48 Hertz. Mathematically, the spectrum of the <b>sinusoidally modulated and sampled</b> signal can be represented by the following:<maths id="math0027" num=""><img file="EP1237369A2_D0028.tif" /></maths>

The fundamental order is given by the terms on the second line of equation 23, the ±1^st orders appear in the third line, etc. According to the Nyquist Theorem, in order to produce aliasing, the sampling frequency must be less than twice the bandwidth of the signal. If spectral spread, W, of F(ξ) is taken into account, this criterion becomes ξ_s&lt; 2(ξ_m+ W/2).

It is instructive to note that in the equations derived above, care has been taken to express intensity, modulation, and interpolation functions in a generic form, using f(t), m(t), and h(t). This allows these equations to be applied to any type of real-world function as it may be encountered in digital projection of frames and pixels. In addition, parametric representation of quantities such as modulation frequency ξ_m, sampling frequency ξ_s, and bandwidth W allow straightforward manipulation using the underlying concepts outlined above.

Therefore, in a practical application for deliberately causing aliasing effects in digital motion picture projection, it is sufficient to employ best approximations of f(t), m(t), h(t), and other variables and, with the aid of linear systems and Fourier analysis tools as developed above, to determine the best conditions for maximum aliasing.

Exemplary Tables and Calculations

For the purpose of describing the method and apparatus of the present invention in detail, it is instructive to provide a summary listing of those variables that must be considered in order to deliberately cause aliasing artifacts when using a video camera to record a digital motion picture. Using the naming conventions employed in the above description, the following parameters must be considered: <ul id="ul0003" list-style="none" compact="compact"><li>(a) Sampling frequency ξ_s. This parameter is not in the control of any copy protection method or apparatus. The method of the present invention will be effectively implemented by making some assumptions on likely sampling frequencies that could be used by a video camera when attempting to make a copy of a displayed motion picture. Standard sampling rates widely used by commercially available video cameras include 1/50 second (the standard NTSC rate) and 1/60 second (the standard PAL rate), and so-called "flickerless" speeds of 1/100 second and 1/120 second. Other rates are possible; the method of the present invention be employed where image aliasing appears at the most widely used sampling frequencies as well as in cases where non-standard sampling rates are utilized.</li><li>(b) Bandwidth, W, of the unmodulated signal or the one-sided bandwidth, W/2, as is used in some instances throughout this application to denote the spectral extent in either the positive or negative frequency domains. As an upper limit, the bandwidth of the unmodulated signal is the displayed frame rate. As noted in the above description, the frame rate has a likely value of 24 Hz. However, the present invention can be used with other frame rates. It must be noted that the "unmodulated" signal, as this term is used here, refers to continuous pixel intensity as is shown in Fig. 13. The actual bandwidth value is not controlled by the present invention; instead, W is dependent on the frame rate and on the scene content, frame to frame. Thus, a practical limit of bandwidth less than or equal to the frame rate is sufficient for purposes of the present invention.</li><li>(c) Modulation frequency ξ_m. This parameter is controlled using the method of the present invention, taking into consideration parameters (a) and (b) above and also considering practical constraints imposed by intensity limitations of the projection system. For any individual pixel or set of pixels, a modulation frequency ξ_m is selected so that the effective bandwidth of the modulated signal is such that aliasing will be perceptible when sampled using a video camera.</li></ul>

The method of the present invention, then, consists in careful selection of a modulation frequency that is not visible to the unaided eye, but causes aliasing when sampled by a video camera.

As was noted with respect to Fig. 14, a result of using a finite sampling duration is the attenuation of higher order spectral components. For this reason, it is generally considered most practical to consider the first order (± 1 order) spectral components for creating the aliasing effects of the present invention. Second order spectral components may also have sufficient energy to cause visible aliasing in some instances.

Referring to the examples illustrated by Tables 2, 3, 4, and 5, (see Figures 25-28, respectively) there are shown values of spectral spread for various modulation frequencies, using "flickerless" conditions of sampling frequency (i.e., ξ_s=120 Hz), varied for modulated signal bandwidth.

Recall from the above discussion that the spectral spread of the zero order modulated signal with modulation frequency ξ_m and bandwidth W ranges from -(ξ_m + W/2) to (ξ_m + W/2). Table 6 below summarizes the spectral spread values for different spectral orders. In the example of Fig. 15, ξ_m = 48 Hz and W/2 = 24, so that the spectral spread of the zero order component, according to Table 6, is -60 to +60 Hz.

Aliasing due to the 1^st spectral order begins as soon as the following is true:<maths id="math0028" num="(24)"><math display="block"><mrow><msub><mrow><mtext>[ξ</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext> - (ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><msub><mrow><mtext>+W/2)] &lt; [(ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext>+W/2)]</mtext></mrow></math><img file="EP1237369A2_D0029.tif" /></maths>

Solving the inequality for ξ_m results in the lower limit for the modulation frequency:<maths id="math0029" num=""><img file="EP1237369A2_D0030.tif" /></maths>

Aliasing continues until the following condition holds:<maths id="math0030" num="(26)"><math display="block"><mrow><msub><mrow><mtext>[ξ</mtext></mrow><mrow><mtext>s</mtext></mrow></msub><msub><mrow><mtext>+ (ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><msub><mrow><mtext>+W/2)] &gt; - (ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext>+W/2)</mtext></mrow></math><img file="EP1237369A2_D0031.tif" /></maths>

Solving inequality (26) for ξ_m results in the upper limit for the modulation frequency. Thus, the condition for modulation frequency to produce first order aliasing can be expressed as follows:<maths id="math0031" num=""><img file="EP1237369A2_D0032.tif" /></maths>

Similar calculations can be carried out to determine the range of modulation frequencies that can produce aliasing for other orders. Table 7 summarizes these results for the first few orders, using the spectral spread values of Table 6. <tables id="tabl0002" num="0002"><table frame="all"><title>Table 6.</title><tgroup cols="3" colsep="1" rowsep="1"><colspec colnum="1" colname="col1" colwidth="52.50mm" /><colspec colnum="2" colname="col2" colwidth="52.50mm" /><colspec colnum="3" colname="col3" colwidth="52.50mm" /><thead valign="top"><row><entry namest="col1" nameend="col3" align="left"><b>Spectral spread of the fundamental, ±1</b>^<b>st</b><b> and ±2</b>^<b>nd</b><b> orders for a signal with bandwidth W and modulation frequency ξ</b>_<b>m</b><b>.</b></entry></row><row><entry namest="col1" nameend="col1" rowsep="0" /><entry namest="col2" nameend="col3" align="left"><b>Spectral Range</b></entry></row><row><entry namest="col1" nameend="col1" /><entry namest="col2" nameend="col2" align="left"><b>From</b></entry><entry namest="col3" nameend="col3" align="left"><b>To</b></entry></row></thead><tbody valign="top"><row><entry namest="col1" nameend="col1" align="left"><b>0</b>^<b>th</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">-(ξ_m+W/2)</entry><entry namest="col3" nameend="col3" align="left">+(ξ_m+W/2)</entry></row><row><entry namest="col1" nameend="col1" align="left"><b>+1</b>^<b>st</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">ξ_s - (ξ_m+W/2)</entry><entry namest="col3" nameend="col3" align="left">ξ_s + (ξ_m+W/2)</entry></row><row><entry namest="col1" nameend="col1" align="left"><b>-1</b>^<b>st</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">-ξ_s - (ξ_m+W/2)</entry><entry namest="col3" nameend="col3" align="left">-ξ_s + (ξ_m+W/2)</entry></row><row><entry namest="col1" nameend="col1" align="left"><b>+2</b>^<b>nd</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">2ξ_s - (ξ_m+W/2)</entry><entry namest="col3" nameend="col3" align="left">2ξ_s + (ξ_m+W/2)</entry></row><row rowsep="1"><entry namest="col1" nameend="col1" align="left"><b>-2</b>^<b>nd</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">-2ξ_s- (ξ_m+W/2)</entry><entry namest="col3" nameend="col3" align="left">-2ξ_s+ (ξ_m+W/2)</entry></row></tbody></tgroup></table></tables><tables id="tabl0003" num="0003"><table frame="all"><title>Table 7.</title><tgroup cols="2" colsep="1" rowsep="1"><colspec colnum="1" colname="col1" colwidth="78.75mm" /><colspec colnum="2" colname="col2" colwidth="78.75mm" /><thead valign="top"><row><entry namest="col1" nameend="col2" align="left"><b>Modulation Frequency range that can produce aliasing for different orders</b></entry></row><row><entry namest="col1" nameend="col1" align="left"><b>Aliasing order</b></entry><entry namest="col2" nameend="col2" align="left"><b>Modulation Range</b></entry></row></thead><tbody valign="top"><row><entry namest="col1" nameend="col1" align="left"><b>+1</b>^<b>st</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">(ξ_s-W)/2 &lt; ξ_m &lt; (ξ_s+W)/2</entry></row><row><entry namest="col1" nameend="col1" align="left"><b>-1</b>^<b>st</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">(-ξ_s-W)/2 &lt; ξ_m &lt; (-ξ_s+W)/2</entry></row><row rowsep="0"><entry namest="col1" nameend="col1" align="left"><b>+2</b>^<b>nd</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">(2ξ_s-W)/2 &lt; ξ_m &lt;</entry></row><row><entry namest="col1" nameend="col1" /><entry namest="col2" nameend="col2" align="left">(2ξ_s+W)/2</entry></row><row rowsep="0"><entry namest="col1" nameend="col1" align="left"><b>-2</b>^<b>nd</b><b> order</b></entry><entry namest="col2" nameend="col2" align="left">(-2ξ_s-W)/2 &lt; ξ_m &lt; (-</entry></row><row rowsep="1"><entry namest="col1" nameend="col1" /><entry namest="col2" nameend="col2" align="left">2ξ_s+W)/2</entry></row></tbody></tgroup></table></tables>

It must be noted that the aliasing conditions summarized in Table 7 are necessary---but not sufficient---conditions for visible aliasing. In order to obtain the necessary and sufficient conditions, it is necessary to look more closely at the actual frequency spread within each of the spectral orders. This is because it is unlikely that the entire spectral range of the modulated signal (i.e., ±(ξ_m+W/2)) is populated with spectral components. To illustrate this point, re-examine Figs. 14 and 15. If W/2 and ξ_m are selected to be 12 and 48 Hz respectively, the modulated signal of Fig. 15, by definition, has a bandwidth of ±(ξ_m+W/2) = ±60 Hz, as was noted above. But, as evident from the figure, the energy of the spectrum in not uniformly distributed within the band of ±60 Hz. In fact, most of the energy is only concentrated around three spectral frequencies of 0, +48 and -48 Hz. Thus, if this signal is sampled, only the components around the main band, the left sideband and the right sideband of the higher orders have sufficient energy to create strong aliasing effects. In a practical system design, care must be taken to ensure that aliasing is caused by the portions of the spectrum with a significant amount of energy (i.e., by the main, left and right sidebands).

Tables 2-5 provide spectral spread calculations for different values of modulation frequencies, ξ_m. Entries in each table indicate the spectral spread of the main, left and right bands of the first and second spectral orders for the modulated and sampled signal. For example, in Table 2, the spectral spread of the left sideband of the +1^st order for a signal with W/2=30, ξ_s=120 and ξ_m=50 Hz is calculated to be from 40 to 100 Hz. The results for the corresponding negative spectral orders can be obtained by reversing the signs of the entries. Also note that aliasing only becomes visible if the spectral spread of these higher orders falls somewhere within the frequency range ±40 Hz.

Using the straightforward calculations described above, any number of tables for a given sampling frequency ξ_s and bandwidth can be derived, over any number of modulation frequencies, ξ_m. It is important to note that Tables 2-5, each tabulated frequency spread values for a different estimate of W/2 namely, W/2 of 30, 24, 12, 5, respectively. Alternatively, an exact value of W/2 could have been obtained by examining the Fourier transform of the time-varying function if the intensity values of the projected pixels within the target frames were precisely known. The selection of suitable modulation frequencies for causing aliasing can be accomplished by inspection of such tables. Table 2 lists spectral spread values for 1st and 2nd spectral orders given a one-sided bandwidth W/2 of 30 Hz, sampling frequency ξ_s of 120 Hz, for modulation frequencies from 50 Hz to 210 Hz.

Table 3 lists spectral spread values for 1st and 2nd spectral orders given a one-sided bandwidth W/2 of 24 Hz, sampling frequency ξ_s of 120 Hz, for modulation frequencies from 50 Hz to 210 Hz.

Table 4 lists spectral spread values for 1st and 2nd spectral orders given a one-sided bandwidth W of 12 Hz, sampling frequency ξ_s of 120 Hz, for modulation frequencies from 50 Hz to 210 Hz.

Table 5 lists spectral spread values for 1st and 2nd spectral orders given a one-sided bandwidth W of 5 Hz, sampling frequency ξ_s of 120 Hz, for modulation frequencies from 50 Hz to 210 Hz.

For the purpose of finding a suitable modulation frequency ξ_m under given conditions, the preferred approach is to consider worst-case conditions. Among factors to be considered is bandwidth. A narrow bandwidth is a worst-case condition, since it is more difficult to cause aliasing with a narrow bandwidth signal. Table 5, with values for a one-sided bandwidth W/2 of 5, represents worst-case conditions among Tables 2-5. Using just one example from Table 5, it can be seen that aliasing at frequencies within the visible range can be caused using modulation frequencies ξ_m between 70 and 165 Hz. The optimum values appear in the middle of this range. For example, with a modulation frequency ξ_m of 110 Hz, the left sideband of the 1^st order is centered at roughly around 10 Hz. This creates a distinctly visible aliasing condition, no matter how narrow the bandwidth. Other modulation frequency ξ_m values near this 110 Hz frequency also are likely candidates for causing aliasing.

Note that negative spectral orders (-1 order, -2 order) are not listed in Tables 2 - 5. However, these values can be expressed simply by changing the sign for each spectral value listed.

Simple arithmetic calculations are all that is needed to obtain the values that populate successive rows of Tables 2-5. For the 1^st and 2^nd spectral orders in Tables 2-5, the location of the main band is determined by sampling frequency ξ_s. Since sampling frequency is 120 Hz, the 1^st spectral order main band is centered at 120 Hz (ξ_s). The 2^nd spectral order main band is centered at 240 Hz (2ξ_s) respectively. The From/To spread of the main band and of both left and right side bands is set by the bandwidth W, which differs for each of Tables 2-5. The left and right side bands are centered using the following simple calculation:<maths id="math0032" num="(28)"><math display="block"><mrow><msub><mrow><mtext>Center of main band - Modulation frequency ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext> = Center of left band</mtext></mrow></math><img file="EP1237369A2_D0033.tif" /></maths><maths id="math0033" num="(29)"><math display="block"><mrow><msub><mrow><mtext>Center of main band + Modulation frequency ξ</mtext></mrow><mrow><mtext>m</mtext></mrow></msub><mtext> = Center of right band</mtext></mrow></math><img file="EP1237369A2_D0034.tif" /></maths>

It should be noted that the previous analysis of aliasing assumed the simplest case, which is the use of a sinusoidal modulation frequency. In actual practice, it may prove advantageous to employ a different type of modulation signal waveform, as we will discuss in the next section.

Method of Preferred Embodiment

The present invention temporally modulates the displayed pixel intensities in such a way that objectionable patterns will be produced when the displayed pixel intensities are recorded with a video camcorder. More specifically, the displayed pixel intensities are modulated so that an observer of a displayed movie will not see any degradation in image quality, but any attempt to capture the displayed movie with a camcorder will result in aliased temporal frequency components that will be readily visible when the camcorder copy is subsequently viewed. According to the present invention, individual pixels or groups of pixels are modulated in various ways to produce specific spatial patterns in the video copy and to prevent a video pirate from circumventing the degradations that are produced by the patterns. The key aspects of the present invention are 1) the spatial arrangements of pixels that undergo temporal modulation; 2) the temporal modulation signal waveform; and 3) the temporal modulation frequency (or frequencies). We now discuss each of these aspects.

Referring to Fig. 18, there is represented an arbitrary frame 100 of a digital motion picture. Frame 100 comprises an array of pixels 102 that display the scene content. To display scene content in color, each individual pixel 102 actually comprises a red pixel 102R, a green pixel 102G, and a blue pixel 102B, where the red, green, and blue color pixels are visually overlapped and intensities are varied using color representation techniques well known in the imaging arts. However, for simplicity of description, the model of a single, generic pixel 102 is used here as a generalization. The description provided here refers to individual RGB color components only when necessary.

As Fig. 18 shows, a spatial pattern 104 of pixels 102 within frame 100 can be identified for modulation using the method of the present invention. There are numerous options for selecting and modulating one or more patterns 104 during display of a digital motion picture. Chief among selection options for specifying pattern 104 composition are the following, including combinations of the following: <ul id="ul0004" list-style="none" compact="compact"><li>(1a) random arrangement of pattern 104. In a preferred embodiment, a random selection of pixels 102 is used to produce a pattern 104 that is optimal for obscuring a video camera copy of the digital motion picture. Pattern 104 can be changed for each frame 100 or for each set of n frames. The use of a random modulated pattern 104 that is constantly changing obviates the use of temporal filters or other image processing techniques for re-creating an acceptable image from the degraded video copy.</li><li>(1b) arrangement of pattern 104 as a text message. In another preferred embodiment, pattern 104 could be employed as a "bitmap" for a message 106 comprising text and symbols, as shown in Fig. 19. Message 106 could be the same for each frame 100, such as a message that displays the name of the theater where the projection apparatus is located, but concurrently part of message 106 could be changed every n frames to display time-varying information such as the date and time. Message 106 could also display a simple message such as "ILLEGAL COPY" or "STOLEN," or a more detailed message could be displayed, such as a reward notice or other incentive for return of the illegal copy.</li><li>(1c) arrangement of pattern 104 as a pictorial image, possibly as part of an animation. Pattern 104 could be embodied as an image intended to obscure or provide information. Animation techniques could be employed to generate an informational or annoying pattern 104.</li><li>(1d) arrangement of pattern 104 as a watermark message. A watermark pattern 104 could comprise a plurality of modulated pixels 102 within various parts of frame 100. For example, an algorithm could be used for determining which pixels 102 are used to create a digital watermark, assigning a spatial distribution to the watermark. Cryptographic methods could be employed, in conjunction with the algorithm used for pixel 102 selection, to securely encode watermark information. In order to recover a message contained within a watermark pattern 104, a decryption key may be required for deciphering contents in accordance with the watermark embedding algorithm.</li><li>(1e) entire frame 100 considered as pattern 104. In certain implementations, it may be advantageous to modulate every pixel 102 within frame 100, typically with the same modulation frequency for every pixel. However, if such a method were used, it would then be preferable to change modulation frequency periodically or randomly, to prevent synchronization of video camera timing with modulation frequency.</li></ul>

The above techniques for pattern 104 selection are just some of the more likely techniques that could be used, all within the scope of the present invention.

There are some practical limitations that may constrain the modulation of an individual pixel within the chosen pattern 104. This is because the process of modulating the pixel intensity inherently results in a lower average intensity for a given peak intensity value. For example, if pixel 102 corresponds to a very bright displayed value, it may not be possible to modulate pixel 102 because the peak intensity that is needed to maintain the average intensity for pixel 102 may exceed the projector capabilities. Thus, it may be necessary to modulate other nearby pixels 102 or other areas of frame 100, where the peak intensity requirements can be met. Thus, pixels in an image frame may be analyzed according to a peak intensity criterion and pixels meeting the criterion further determine the pattern that is subject to the temporal modulation. This criterion may be pixels not exceeding a certain brightness level. Additionally, there may be some types of scene content or locations within the scene for which induced flicker may be perceptible at higher modulation frequencies than average. This may necessitate applying modulation to other regions of a displayed frame.

Some of these limitations are addressed in the present invention by choosing the proper mode for the temporal modulation. In a preferred embodiment, the modulation signal waveform is a sinusoid, as we described previously in the analysis of aliasing. A sinusoidal function will minimize the spectral extent of the frequency sidebands that are produced by the modulation, which makes it somewhat easier to place the sidebands at the desired frequency position. However, many projection systems may not be capable of modulating the pixel intensities according to a pure sinusoidal waveform, so another preferred embodiment is to approximate a sinusoidal waveform using a rectangular or sawtooth modulation signal waveform. In a preferred embodiment using a rectangular modulation waveform, a 50% duty cycle is used, which requires a doubling of the peak pixel intensity to maintain the same average intensity as an unmodulated signal. In still another preferred embodiment, the rectangular waveform may not be completely dark during the OFF period, in order to reduce the peak pixel intensities that are required to maintain a desired average intensity. This potentially allows for more pixels to be modulated when a display device has a limited maximum intensity. The tradeoff is that the resulting aliased components will not be as severe as a full ON/OFF modulation. Another useful variation of the modulation waveform is to use a duty cycle of more than 50%, which also lessens the peak intensity that is required to maintain the same average intensity. Any number of other possible modulation waveforms can be employed, all within the scope of the present invention.

In addition to the basic shape of the modulation signal waveform, there are a number of temporal modulation options for pattern 104 as part of the present invention. These temporal modulation options include the following, and combinations of the following: <ul id="ul0005" list-style="none" compact="compact"><li>(2a) using a single modulation frequency. In a preferred embodiment, a single modulation frequency is used to modulate the pixels that comprise pattern 104. It is advantageous to change the modulation frequency periodically or randomly, in order to frustrate attempts to synchronize video camera recording equipment to the modulation rates in use.</li><li>(2b) using multiple modulation frequencies. In another preferred embodiment, two or more different modulation frequencies are used in two or more regions 108 of frame 100, as is shown in Fig. 20. The use of different modulation frequencies within a frame 100 allows for video cameras with different sampling rates to be simultaneously affected to the maximum extent. Even with a single camera operating at a single sampling rate, the use of two or more modulation frequencies in different regions will introduce patterns that flicker at different rates, which can be highly objectionable. When multiple modulation frequencies are used within the same frame, it is extremely difficult to synchronize video camera recording equipment to all modulation rates simultaneously. However, it again may be advantageous to change the modulation frequencies periodically or randomly, in order to further frustrate attempts to synchronize the video camera to the modulation rates in use.</li><li>(2c) modulation of pixels 102R, 102G, 102B in pattern 104. Pixels in each R, G, and B color plane can be modulated independently of one another or synchronously, within the scope of the present invention. It may be advantageous to modulate pixels within only one or another color plane to simplify apparatus design, for example.</li><li>(2d) using frequency modulation techniques as a form of digital watermarking. In addition to obscuring any video camera copy, modulation of pattern 104 can also be used to encode an information signal. Changes in modulation frequency can thereby be used as an encoding technique for copy watermarking and for information such as location, time and date of projection, identifying number of the film copy sent, and related information.</li><li>(2e) using pulse-width modulation. This variation on a frequency modulation technique uses manipulation of the duty cycle of pixels 102. Pulse width modulation can be particularly useful where intensity levels needed for pixel 102 modulation may not be possible for a projector to achieve at 2X, as is needed for a 50% duty cycle modulation. Using this technique, however, the intensity must be preserved for each modulated pixel 102.</li><li>(2f) using amplitude modulation. This modulation technique can employ the inherent capability for digital cinema to control intensity of each pixel 102 over a range. An amplitude modulation scheme would enable encoding of watermark information within a sequence of variable intensity values.</li></ul>

It would also be feasible to combine or to alternate any of methods (2a) through (2f) to implement a hybrid modulation scheme for digital watermarking, within the scope of the present invention.

Once selections have been made for the spatial arrangement of pattern 104 (as in methods (1 a) - (1 e)) and the temporal modulation waveform and mode (as in methods (2a)-(2f)), it is necessary to choose the specific modulation frequency (or frequencies). In general, a reasonable design approach is to select the modulation frequency so that one of the first-order side bands of the sampled signal is centered in the frequency range of approximately 10 to 30 Hz. This produces an aliased component in the sampled signal that is in the peak sensitivity range of the human visual system as shown in Fig. 1. From equations 28 and 29, we can see that this design approach is equivalent to satisfying the following equation:<maths id="math0034" num="(30)"><math display="block"><mrow><msub><mrow><mtext>10 Hz ≤ (Sampling frequency ξ</mtext></mrow><mrow><mtext mathvariant="italic">s</mtext></mrow></msub><mtext> - Modulation frequency </mtext><msub><mrow><mtext mathvariant="italic">ξ</mtext></mrow><mrow><mtext mathvariant="italic">m</mtext></mrow></msub><mtext> | ≤ 30 Hz.</mtext></mrow></math><img file="EP1237369A2_D0035.tif" /></maths> For example, if the sampling frequency ξ_s of the specified camcoder is 120 Hz, then a modulation frequency of either 100 Hz or 140 Hz will place the center of a first-order side band at 20 Hz, thus producing a strong aliased component in the sampled video signal. Since many camcorders use a sampling frequency of either 60 Hz or 120 Hz, the selection of a modulation frequency at 90 Hz will produce an aliased component at 30 Hz for either sampling frequency. In this way, a single modulation frequency can produce the desired effect regardless of the particular camcorder that is used. However, as described in methods (2a) - (2b), it may be advantageous to use different modulation frequencies, either for different regions within a single frame or for a given region across multiple frames, to provide an even greater deterrent to video piracy. For example, using modulation frequencies of 80 Hz and 100 Hz for different regions in a frame 100 will produce aliased components at 20 Hz and 40 Hz regardless of whether the sampling frequency is 60 Hz or 120 Hz. In this way, there is always an aliased component in the camcorder video at 20 Hz, which is near the peak flicker sensitivity, thus producing a highly objectionable pattern. In other applications, it may desirable to select a modulation frequency that moves the aliased component to very low temporal frequencies, say less than 10 Hz. The visual appearance of a slowly varying pattern may not be as visually objectionable as a rapidly varying pattern, but if pattern 104 represents a text message, the slowly varying pattern may be more easily comprehended. Finally, when changing the modulation frequency (or frequencies) periodically or randonly to prevent the synchronization of video recording equipment, the amount of change from the preferred frequency (or frequencies) does not need to be extremely large. Even small changes, such as ± 5Hz, will be sufficient to prevent continuous synchronization.

Summary of steps for implementation of method

The basic steps for implementing the method of the present invention are as follows: <ul id="ul0006" list-style="none" compact="compact"><li>1. Identify pattern 104 to be used. Decide upon a strategy for displaying pattern 104, using options (1a)-(1e) given above, or some other technique for pattern 104. This decision depends, in large part, on the purpose for which this invention is to be applied. For example, when maximally obscuring any copied movie content is the goal, as in the preferred mode of the present invention, a randomly changing pattern 104 has advantages.</li><li>2. Select the appropriate mode for temporal modulation. Options (2a)-(2f) given above provide the preferred options for temporal modulation available in implementing the present invention.</li><li>3. Select the appropriate modulation frequency or frequencies. Suitable modulation frequencies can be identified using the techniques described above and those used to generate Tables 2-5. As was noted, the best option is generally to employ first order frequencies for aliasing; however, there can be applications where second order aliasing has advantages.</li></ul>

The general flow diagram for implementation of the copy-deterrent pattern is shown in Fig. 21. In a Frame Selection step 200, n consecutive frames 100 are selected for implementation of the copy-deterrent pattern. Next, in a Pattern Selection step 202, specific pixel 102 locations within copy-deterrent pattern 104 are selected. As mentioned above, pixels 102 for modulation may be simply selected at random or may comprise a text or encrypted message. In a Pixel Value Calculation step 204, the intensity values of selected pixels 102 (that make up copy-deterrent pattern 104 and are spanned over n frames 100) are calculated. This calculation may simply entail computation of individual R, G and B values or may include computation of an average or hybrid luminance or chrominance measure. In a Decision step 206, it is made certain that output intensity characteristics/limitations of the projection system do not prevent the selected pixels 102 from being modulated and projected. If projection is not possible, new pixels 102 must be selected. In a Bandwidth Calculation step 208, the bandwidth of the selected pixel intensity pattern (that is spanned over n frames) is calculated and/or estimated. As noted earlier, this bandwidth is less than the frame 100 projection rate but its exact value depends on the movie content (i.e., number of selected frames 100 and the pixel 102 intensity values within the selected frames 100). Since the number of frames 100 and the intensity values of the selected pixels 102 are precisely known, bandwidth calculations may be carried out by taking the Fourier transform of the selected pixels 102 and examining its spectrum. However, since this process may require a considerable amount of computation, a simple estimate of the bandwidth (e.g., W/2 ∼ 5 Hz for a very conservative estimation) may be used.

The next step is a Sampling Frequency Selection step 210. This can be selected to be one of several standard sampling rates (e.g., 50, 60, 100, 120 Hz, etc.) used in today's camcorders. It is also possible to select one sampling frequency value for the first n₁ set of frames 100 and select a different sampling rate for the next n2 set of frames 100. Alternatively, or in conjunction with above, different sampling rates for different regions 108 within the frame 100 may be used to carry out the calculations. This way, copy-deterrent pattern 104 would affect a wider variety of camcorders. In a Modulation Selection step 212, the appropriate modulation scheme (or a combination of them) as is outlined in (2a) - (2e) above is used at the appropriate modulation frequency to produce aliasing for the chosen sampling rate. In order to broaden the effectiveness of this technique, a range of modulation schemes and frequencies may be used to affect the same set of pixels 102 but in different set of "n" frames 100. Finally, once the appropriate modulation scheme and frequency is selected, the necessary information is supplied to image-forming assembly 16 in a Modulate step 214 in order to affect the projection of pixels 102.

Apparatus of Preferred Embodiment

Referring to Fig. 22, there is shown a block diagram of a copy-deterrent projection apparatus 10 with an arrangement of components for implementing the present invention in a preferred embodiment. Image data for projection of each frame 100 is typically provided in compressed form and may be transmitted to the projecting site or provided on storage media. In any event, compressed image data is input to a decompression circuit 12. Decompression circuit 12 decompresses frame 100 data and provides this data to a display logic assembly 14. Display logic assembly 14 formats the image data for frame 100 into pixel array format and provides the necessary color correction, image resizing, and related functions. Then the data is passed to an image forming assembly 16 for display onto a display screen 20. The decompression and display sequence and apparatus just described are familiar to those skilled in the digital motion picture projection arts and may be embodied using a number of different types of components.

Pattern data and control parameters are provided to a pattern generator/modulator assembly 18 for generating and modulating pattern 104. Pattern data and control parameters can be provided from a number of sources. For example, pattern data and control parameters can be provided by the motion picture supplier, provided along with the compressed image data. This arrangement would put the movie supplier in control of pattern 104 generation for copy protection. At the other extreme, copy protection could be solely in the domain of the projection site itself. In such a case, pattern data and control parameters can be provided by optional pattern logic circuitry 22, indicated by a dashed box in Fig. 22, or can be generated on the fly at the time of projection, based on known characteristics of the movie that must be provided to the projection site. Other possibilities include some combination of control by the movie supplier and local control by the projection site.

The object is to provide the location of pixels 102 as well as modulation frequency and intensity variation data. As is described previously for methods(1a) - (1e) and (2a) - (2f), plaintext or encrypted information can be sent by selection of specific location, frequency, and intensity data for modulation of pixels 102. Significantly, pattern generator/modulator assembly 18 provides this location, frequency, and intensity data for modulation, assigned to existing pixels 102 in frame 100. That is, pattern generator/modulator assembly 18 does not provide "new" pixels 102 relative to the scene content for frame 100. Instead, the output from pattern generator/modulator assembly 18 can be considered as control signals, sent to image forming assembly 16, in order to manipulate the scene content that is sent from display logic assembly 14.

Pattern generator/modulator assembly 18 provides modulation information for pixels 102 within pattern 104 that are sent to the display logic assembly 14 and the image forming assembly 16 for projection.

It is instructive to note that, in addition to image forming assembly 16, decompression circuit 12, and display logic assembly 14, pattern generator/modulator assembly 18 can be part of projector 10 as in the preferred embodiment or can be separate components, such as components running on a separate computer or other processor.

Image forming assembly 16 may employ any one of a number of display technologies for projection of a sequence of frames 100 onto display screen 20. In a preferred embodiment, image forming assembly 16 comprises a transmissive Liquid-Crystal Device (LCD) spatial light modulator and support components for projection of frames 100. Pattern 104 data may be used to modulate individual pixels within the LCD for each color. Or, a separate LCD may be employed for modulation of pattern 104. Other types of modulator could alternately be used, including a digital micromirror device (DMD) or reflective LCDs, for example.

Alternate techniques for applying modulation:

The apparatus of the present invention can employ any of a number of different techniques for applying modulation to pattern 104. Referring to Fig. 23 which shows the signal path that applies for projection of each pixel 102 in frame 100, methods for applying modulation include the following: <ul id="ul0007" list-style="none" compact="compact"><li>(a) Modulation at digital control circuitry. In the preferred embodiment, modulation is applied to a digital signal 30 within the digital control circuitry used for generation of the array of pixels 102 in each frame 100. With this method, the digital values that correspond to pixels 102 in each frame 100 are modulated in accordance with the modulation scheme. An example of a 50% duty-cycle square wave modulation is depicted in Fig 23 as an "AND" operation that can be readily implemented in digital domain.</li><li>(b) Analog modulation. Analog modulation is possible, using some method for modulation of an analog signal 32 that controls intensity of the light source used for pixels 102 within pattern 104, for example. Analog modulation techniques, as opposed to the digital techniques of part a, manipulate continuous-time analog signals. Nevertheless, similar concepts of modulation can be carried over from the digital domain to the analog domain.</li><li>(c) Optical modulation. Optical modulation could be implemented using techniques of projection optics 34 such as shuttering, masking, or controlled pixel emission. Optical modulation could be applied within image forming assembly 16 as well as at display screen 20, using masking or filtering techniques. Referring to Fig. 23, the optical modulation mask 216, for example, can be used either in front of the projection optics or on top of the display screen to selectively manipulate the projected pixel locations. Liquid crystal display material, with electrically controlled transmission characteristics, may be used to construct such a spatial light modulation mask; the opacity of different regions of such a mask may be controlled by changing the applied electrical signal to that region of the mask.</li></ul>

As was noted above, modulation can be differently applied for each color component of frame 100.

Alternative Embodiments

Referring to Fig. 24, there is shown an alternate embodiment for copy-deterrent projection apparatus 10, in which image forming assembly 16 does not project onto display screen 20. Instead, image forming assembly 16 communicates with an emissive display panel 24. Emissive display panel 24 may be in the form of a panel that comprises an LED array 26, similar to the display panels widely used in large sports facilities, for example. Using display panel 24 of this type, image forming assembly 16 directly controls the intensity and modulation of red, green, and blue LEDs, 28R, 28G, and 28B, respectively.

While the invention has been described with particular reference to its preferred embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements in the preferred embodiments without departing from the scope of the invention. For example, copy-deterrent pattern 104 can comprise any of a number of arrangements of pixels 102 to form messages 106 or regions 108 anywhere within frame 100. Copy-deterrent patterns 104 can be provided primarily in order to obscure a recording or to provide digital watermarking, or both. The present invention is adaptable to a number of possible configurations of digital motion projector and display screen apparatus, such as using micromirror technology or a light valve array, for example.

Therefore, what is provided is a copy-deterrent projection apparatus for digital motion pictures and a method for applying modulation to pixels within a displayed motion picture frame in order to discourage recording of the image using a video camera.