System and method for interactive projection

FIELD OF THE INVENTION

[0001] The disclosure relates to systems and methods for projecting images. In particular, the disclosure relates to a system and method for projection of interactive areas and mapping of the interactive areas onto complex surfaces.

BACKGROUND OF THE INVENTION

[0002] Active product displays are useful in demonstrating advanced features of various products. Prior art systems are typically passive display systems, in that content is presented to a viewer on a screen with little opportunity for feedback. In other prior art systems, a touch screen computer is employed to allow the viewer to select content for review and to provide feedback. However, these systems require extensive custom set-up or are prohibitively expensive. Further, these systems require extensive maintenance due to frequent wear on the touch screen.

[0003] For example, U.S. Patent No. 7,598,942 to Underkoffler, et al. discloses a gestural interface to visually presented elements on a display screen. Multiple cameras capture movement and detect location and orientation and generate output signals to processors connected to a computer. The processors translate the camera outputs into gestural signals which are interpreted as input information. The computer uses the input information to generate commands to control computer functions.

[0004] U.S. Patent No. 7,034,807 toMaggioni discloses a system for interaction with a display. The system includes a recording unit for recording a pointer object in the vicinity of the display. A computer is connected to the recording unit and is used to determine a position of the pointer object. The system requires multiple cameras and multiple reflective surfaces.

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

[0005] U.S. Patent Publication No. 2012/0212413 to Plage mam, et al. discloses a system for receiving image information and translating it into computer instructions. Image information is received for a single predetermined action space to identify motion. A camera combined with mirrors, prisms, or optic cables is used to gather the image information. However, only one action area is monitored at any given time.

[0006] U.S. Patent Publication No. 2012/0162077 to Sze, et al. discloses an input device used to detect locations and motions of objects in a virtual working area. A camera is directed to a region of interest. A region of interest is illuminated by a “flat” beam of light. An object is placed in the region of interest and illuminated. The camera captures an image of the object and sends it to the processor. The processor processes the image to obtain locations and movements. Based on the movements, the processor produces computer instructions.

[0007] Therefore, there is a need for a system and method for projecting an interactive environment for a display. The system should provide options for the user to actively select the content to view. The system should be easy to set up and should be capable of being easily transported to other locales.

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192

SUMMARY OF THE INVENTION

[0008] The system disclosed projects a “touchable” image on a three-dimensional object which enables user interaction. The touchable image is defined by a trigger area and a content area.

[0010] The system includes an infrared camera and a projector connected to a computer processor. In one embodiment, a content server and content database are also connected to the computer via a network. Content to be projected is stored in the database. During set-up, the system captures a baseline image of a three-dimensional target surface. The system then transfonns the content to correct for projector position and for target curvature. During operation, the corrected content image is projected onto the three-dimensional target surface along with similarly transformed image of the trigger area. The infrared camera then monitors the trigger area for a trigger event and reports it to the server when it occurs. A trigger event occurs when an object is placed inside the trigger area for a predetermined period of time. Upon report of a trigger event, the system fetches the additional stored content, transforms it, and projects it onto the target surface. The computer processor uses both affine and a non-affine transforms to translate and “warp” the content image to match the surface of the target object.

[0011] In an alternate embodiment, the system uses facial recognition software to project predefined content associated with a particular recognized user on the surface of the object.

[0012] The action areas can be comprised of a plurality of separately defined actions areas each corresponding to different content. When a user interacts with an action area, the system recognizes the presence of a user and projects the associated content.

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

The affine transformations can be described through linear algebra. In linear algebra, linear transformations can be represented by matrices. If Τ’ is a linear transformation mapping R to R' and x is a column vector with n entries, then T(x) = Ax

[0013] For some m*n matrix A, called the transformation matrix of T. There are alternative expressions of transformation matrices involving row vectors that are preferred by some authors.

[0014] Matrices allow arbitrary linear transformations to be represented in a consistent format, suitable for computation. This also allows transformations to be concatenated easily (by multiplying their matrices).

[0015] Linear transformations are not the only ones that can be represented by matrices. Some transformations that are non-linear on a n-dimensional Euclidean space R, can be represented as linear transformations on the //-1-dimensional space R^-1. These include both affine transformations (such as translation) and projective transformations. For this reason, 4*4 transformation matrices are widely used in 3D computer graphics. These //+1-dimensional transformation matrices are called, depending on their application, affine transformation matrices, projective transformation matrices, or more generally non-linear transformation matrices. With respect to an «-dimensional matrix, an n+1-dimensional matrix can be described as an augmented matrix.

[0016] In the physical sciences, an active transformation is one which actually changes the physical position of a system, and makes sense even in the absence of a coordinate system whereas a passive transformation is a change in the coordinate description of the physical system (change of basis). The distinction between active and passive transformations is important. By default, by transformation, mathematicians usually mean active transformations.

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192 while physicists could mean either.

[0017] Put differently, a passive transformation refers to description of the same object as viewed from two different coordinate frames.

[0018] If one has a linear transformation T(x) in functional form, it is easy to determine the transformation matrix A by transforming each of the vectors of the standard basis by T, then inserting the result into the columns of a matrix. In other words,

A = [7(¾) 7(¾) ... 7(¾)] Eq. 1

[0019] For example, the function T(x) = 5x is a linear transformation. Applying the above process (suppose that n = 2 in this case) reveals that

7(x) = 5x = 5Ix = β θ] x Eq. 2

Lv D J

[0020] It must be noted that the matrix representation of vectors and operators depends on the chosen basis; a similar matrix will result from an alternate basis. Nevertheless, the method to find the components remains the same.

[0021] Mathematically, vector v can be represented in basis vectors, E = (¾¾ ... e_n]with coordinates [v]_£ = [1^2 — ^vn]^T· v = + v₂e₂ 4-----h v_ne_n = £ = E[v]_£ Eq. 3

[0022] Now, express the result of the transformation matrix A upon v, in the given basis:

A(v) = A(ZViêi) = ZViAied = [4(¾)^¾) ..Μ(4)][^]ε = Eq. 4

<td></td><td>^ai,i</td><td> ^1,2</td>

<td> = A-[v]_E = [êië2-ên]</td><td>^a2,l</td><td>^a2,2</td>

<td></td><td> ,°n.l</td><td> «n.2</td>

Eq. 4(cont.)

[0023] The a^·elements of matrix A are determined for a given basis E by applying

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

A to every ëj = [00... (Vj = 1)... 0]^T, and observing the response vector Ae_} = a_{l i}e₁ + a₂jë₂ 1-----b ^an.j^n = Σ ^aij This equation defines the elements, a^j, of j-th column of the matrix A.

[0024] To represent affine transformations with matrices, we can use homogeneous coordinates. This means representing a 2-vector (x, y) as a 3-vector (x,y, 1), and similarly for higher dimensions. Using this system, translation can be expressed with matrix multiplication. The functional form x' = x + t_x; y' = y + t_y becomes:

10t_x lt_y y

LiJ LooiJ^Li^J

Eq. 5

[0025] All ordinary linear transformations are included in the set of affine transformations, and can be described as a simplified form of affine transformations. Therefore, any linear transformation can also be represented by a general transformation matrix. The latter is obtained by expanding the corresponding linear transformation matrix by one row and column, filling the extra space with zeros except for the lower-right comer, which must be set to 1. For example, the clockwise rotation matrix from above becomes:

’ cos6 — sin6 . 0 sin0 cos0 0

0

1.

Eq. 6

[0026] Using transformation matrices containing homogeneous coordinates, translations can be seamlessly intermixed with all other types of transformations. The reason is that the real plane is mapped to the w = 1 plane in real projective space, and so translation in real Euclidean space can be represented as a shear in real projective space. Although a translation is a non-linear transformation in a 2-D or 3-D Euclidean space described by Cartesian coordinates, it becomes, in a 3-D or 4-D projective space described by homogeneous coordinates, a simple

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192 linear transformation (a shear).

[0027] More affine transformations can be obtained by composition of two or more affine transformations. For example, given a translation T with vector (t'_x, t'_y), a rotation R by an angle Θ counter-clockwise, a scaling S with factors (s_x,s_y)and a translation T of vector (t>, t_y), the result M of T'RST is:

s_x cos Θ —Sy sin Θ t_xs_x cos Θ — t_ys_y sin Θ + t’_x s_x sin Θ Sy cos Θ t_xs_x sin Θ + t_ys_y cos Θ + t'y .00 1

Eq. 7

[0028] When using affine transformations, the homogeneous component of a coordinate vector (normally called w) will never be altered. One can therefore safely assume that it is always 1 and ignore it. However, this is not true when using perspective projections.

[0029] Another type of transformation, of particular importance to this disclosure, is the perspective projection. Whereas parallel projections are used to project points onto the image plane along parallel lines, the perspective projection projects points onto the image plane along lines that emanate from a single point, called the center of projection. This means that an object has a smaller projection when it is far away from the center of projection and a larger projection when it is closer. This transformation is used to correct for the position of the projector relative to the three-dimensional target object.

[0030] The simplest perspective projection uses the origin as the center of projection, and z = 1 as the image plane. The functional form of this transformation is then x' = x/z, y' = y/τ. . We can express this in homogeneous coordinates as:

y_cz_cy^c.

0 0] Γ^ΧΊ

0100

0010

LO 0 1 0J ^LwJ

Eq. 8

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

[0031] After carrying out the matrix multiplication, the homogeneous component w_cwill, in general, not be equal to 1. Therefore, to map back into the real plane we must perform the homogeneous divide or perspective divide by dividing each component by w_c:

<img file="CA2971280C_D0001.tif" />

Eq. 9

[0032] More complicated perspective projections can be composed by combining this one with rotations, scales, translations, and shears to move the image plane and center of projection wherever they are desired.

[0033] Non-affine transformations such as transformation of a two-dimensional surface to a three-dimensional surface can be accomplished, in theory, by treating the threedimensional surface as a distortion or, in other words, a deviation from true rectilinear projection. This type of transformation is used to correct for the complex shape of the threedimensional target object.

[0034] Although distortion can be irregular or follow many patterns, the most commonly encountered distortions are radially symmetric, or approximately so. Radial distortions can usually be classified as either “barrel” distortions or “pincushion” distortions.

[0035] In barrel distortion, image magnification decreases with distance from the optical axis. The apparent effect is that of an image which has been mapped around a sphere (or barrel). Fisheye lenses, which take hemispherical views, utilize this type of distortion as a way to map an infinitely wide object plane into a finite image area. In a zoom lens barrel distortion appears in the middle of the lens's focal length range and is worst at the wide-angle end of the range.

[0036] In pincushion distortion, image magnification increases with the distance from

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 the optical axis. The visible effect is that lines that do not go through the center of the image are bowed inwards, towards the center of the image, like a pincushion.

[0037] A mixture of both types, sometimes referred to as “mustache” distortion or complex distortion, is less common but not rare. It starts out as barrel distortion close to the image center and gradually turns into pincushion distortion towards the image periphery, making horizontal lines in the top half of the frame look like a handlebar mustache.

[0038] These distortions are radial distortions. They imply a nonlinear radius mapping from the object to the image. For example, what is seemingly pincushion distortion is simply an exaggerated radius mapping for large radii in comparison with small radii.

Conversely, barrel distortion is simply a diminished radius mapping for large radii in comparison with small radii.

[0039] Radial distortion can be corrected using Brown's distortion model, also known as the Brown-Conrady model. The Brown-Conrady model corrects both for radial distortion and for tangential distortion caused by physical elements in a lens not being perfectly aligned. The latter is also known as decentering distortion.

x_d = + + K₂r⁴ H— ) + (P2(r² + 2x„) + 2P1x_uy_u)(l + P₃r² + P₄r⁴ H— ) Eq.

yd = &amp;(! + + K₂r⁴ + ···) + (P^r² + 2y²) + 2P2xuyu)(l + P3r² + P4r⁴ + - ) Eq.

where:

(x_d> ya) =distorted image point as projected on image plane using specified lens, (x_u, yu) — undistorted image point as projected by an ideal pin-hole camera, (x_c, Vc) ⁼ distortion center (assumed to be the principal point),

K_n = n^th= radial distortion coefficient,

Pn = η^Α tangential distortion coefficient, r = /(^^7+(^-^7. .

and

... =an infinite series.

[0040] In our case, barrel and pincushion distortions are quadratic, meaning they increase as the square of distance from the center. In mustache distortion the quartic (degree 4) term is significant: in the center, the degree 2 barrel distortion is dominant, while at the edge the degree 4 distortion in the pincushion direction dominates. Other distortions are in principle possible - pincushion in center and barrel at the edge, or higher order distortions (degree 6, degree 8) - but do not generally occur in practical cases, and higher order distortions are small relative to the main barrel and pincushion effects.

[0041] Barrel distortion typically will have a negative term for KI whereas pincushion distortion will have a positive value. Moustache distortion will have a nonmonotonic radial geometric series where for some r the sequence will change sign.

[0041a] In another aspect, a system for projecting interactive images on a threedimensional object comprises: a camera capable of providing distance-to-target information; a projector and the camera both positioned adjacent the object, such that the object is positioned within a field of view of the projector; and a computer, connected to the camera and the projector, programmed to carry out a computer implemented method having the steps of: scanning the object within the field of view with the camera; projecting a trigger area on a surface of the object; warping a polygon to match contours of the surface by employing a non-affine transform to map a set of points of the polygon to the surface; associating the polygon with the trigger area; associating a set of content with the polygon; warping the set of content to the polygon; determining a touch event; and, projecting the warped set of content on the surface, wherein the camera monitors defined trigger areas for interaction and □ate Reçue/Date Received 2021-01-18 utilizes depth data related to the location or touch box of the trigger area associated with the trigger area data structure.

[0041b] In another aspect, a method for projecting an interactive image on a threedimensional object comprises the steps of: providing a camera, capable of providing distanceto-target information, and a projector connected to a computer; positioning the camera and the projector adjacent the object, such that the object is positioned within a field of view of the projector; defining positions of a set of trigger areas on a surface of the object; projecting the set of trigger areas on the surface; warping a polygon to match contours of the surface by employing a non-affine transform to map a set of points of the polygon to the surface; associating the polygon with the trigger area; associating a set of content with the polygon; warping the set of content to the polygon; repeating the steps of projecting the set of trigger areas and determining the presence of the touch event until the touch event is detected; projecting the warped set of content on the surface; determining the presence of an end event; repeating the steps of projecting the set of content and determining the presence of an end event until the end event has occurred; and, terminating the projecting, wherein the camera monitors defined trigger areas for interaction and utilizes depth data related to the location or touch box of the trigger area associated with the trigger area data structure.

10a

Date Reçue/Date Received 2021-01-18

[0041c] In another aspect, a system for projecting an interactive image onto an object comprises: an infrared sensor; a projector; a controller operatively connected to the infrared sensor and the projector, the controller programmed to: transform a content image to conform to a surface of the object by employing a non-affine transform to map a set of points of a polygon encompassing the content image to the surface resulting in a mapped content image; transform a trigger image to conform to the surface of the object by employing the non-afiine transform to map a set of points of a polygon encompassing the trigger image to the surface resulting in a mapped trigger image; send a mapped trigger image signal to the projector whereby the projector projects the mapped trigger image onto the surface of the object; receive a data signal from the infrared sensor; generate a trigger signal based on the data signal; and, in response to the trigger signal, send a mapped content image signal to the projector, whereby the projector projects the mapped content image onto the surface of the object, wherein the infrared sensor monitors defined trigger areas for interaction and utilizes depth data related to the location or touch box of the trigger area associated with the trigger area data structure.

10b

Date Reçue/Date Received 2021-01-18

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

BRIEF DESCRIPTION OF DRAWINGS

[0042] Reference will now be made to the following drawings:

[0043] Figure 1 is a system architecture diagram of a preferred embodiment.

[0044] Figure 2A is a schematic of a camera/projection unit of a preferred embodiment.

[0045] Figure 2B is a schematic of a camera/projection unit of a preferred embodiment.

[0046] Figure 3 is a plan view of a display of a preferred embodiment.

[0047] Figure 4 is a plan view of a display of a preferred embodiment.

[0048] Figure 5 is a top view of a display of a preferred embodiment.

[0049] Figure 6 is a plan view of a display of a preferred embodiment.

[0050] Figure 7 is a plan view of a display of a preferred embodiment.

[0051] Figure 8 is a plan view of a display of a preferred embodiment.

[0052] Figure 9 is a plan view of a display of a preferred embodiment.

[0053] Figure 10 is a plan view of a display of a preferred embodiment.

[0054] Figure 11 is a flow chart of a calibration method of a preferred embodiment.

[0055] Figure 12 is a flow chart of a run time process of a preferred embodiment.

[0056] Figure 13 is a flow chart of a run time process for projection mapping of a preferred embodiment.

[0057] Figure 14 is a flow chart of a facial recognition process of a preferred embodiment.

[0058] Figure 15 is a system architecture diagram of a preferred embodiment.

[0059] Figure 16A is an elevation view of a preferred embodiment.

CA 02971280 2017-00-15

WO 2016/109749

PCT/US2015/068192

[0060] Figure 16B is an elevation view of a preferred embodiment.

[0061] Figure 17 is an isometric view of trigger areas of a preferred embodiment.

[0062] Figure 18A is a diagram of a touchable area application in accordance with one embodiment of the disclosure.

[0063] Figure 18B is a diagram of a trigger data structure in accordance with one embodiment of the disclosure.

[0064] Figure 18C is a diagram of a content data structure in accordance with one embodiment of the disclosure.

[0065] Figure 19A is a data flow diagram of a system using trigger areas in accordance with one embodiment of the disclosure.

[0066] Figure 19B is a flow chart for updating a trigger area in accordance with one embodiment of the disclosure.

[0067] Figure 19C is a view of an area of a trigger area in accordance with one embodiment of the disclosure.

[0068] Figure 19D is a view of a volume of a trigger area in accordance with one embodiment of the disclosure.

[0069] Figure 19E is a flow chart for updating content in accordance with one embodiment of the disclosure.

[0070] Figure 19F shows the matrix transformations applied to content in accordance with one embodiment of the disclosure.

[0071] Figure 20A is a data flow diagram of a system using trigger areas in accordance with one embodiment of the disclosure.

[0072] Figure 20B is a flow chart for determining a touch of a trigger area in

CA 02971280 2017-0S-1S

WO 2016/109749

PCT/US2015/068192 accordance with one embodiment of the disclosure.

[0073] Figure 21A is a view of the application when a first tab is selected in accordance with one embodiment of the disclosure.

[0074] Figure 2 IB is a view of the application when a second tab is selected in accordance with one embodiment of the disclosure.

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192

DETAILED DESCRIPTION

[0075] In the descriptions that follow, like parts are marked throughout the specification and drawings with the same numerals, respectively. The drawing figures are not necessarily drawn to scale and certain figures may be shown in exaggerated or generalized form in the interest of clarity and conciseness.

[0076] It will be appreciated by those skilled in the art that aspects of the present disclosure may be illustrated and described herein in any of a number of patentable classes or context including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Therefore, aspects of the present disclosure may be implemented entirely in hardware or combining software and hardware implementation that may all generally be referred to herein as a “circuit,” “module,” “component,” or “system” (including firmware, resident software, micro-code, etc.). Further, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.

[0077] Any combination of one or more computer readable media may be utilized. The computer readable media may be a computer readable signal medium or a computer readable storage medium. For example, a computer readable storage medium may be, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium would include, but are not limited to: a portable computer diskette, a hard disk, a random access memory (“RAM”), a read-only memory (“ROM”), an erasable programmable read-only memory (“EPROM” or Flash memory), an appropriate optical fiber with a repeater, a portable compact disc read-only memory (“CD-ROM”), an optical storage

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 device, a magnetic storage device, or any suitable combination of the foregoing. Thus, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0078] Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, markup languages, style sheets and JavaScript libraries, including but not limited to Windows Presentation Foundation (WPF), HTML/CSS, XAML, and JQuery.

[0079] Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, systems and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a computer interconnected with dedicated and special purpose electronic devices, or other dedicated programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create an integrated machine to implement the functions described in the flowchart and the block diagrams.

[0080] These computer program instructions may also be stored in a computer readable medium that when executed can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions when stored in the computer readable medium produce an article of manufacture including instructions which when executed, cause a computer to implement the function/act specified in the flowchart

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192 and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatuses or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0081] Referring to Figure 1, system 100 includes network 101, content server 102 connected to network 101, and camera/projector unit 104 connected to network 101. Content server 102 is further connected to content database 103 for the storage of multimedia and other relevant data.

[0082] In a preferred embodiment, network 101 is a wide area network such as the internet, but can include other wide area and intranet networks.

[0083] In a preferred embodiment, camera/projector unit 104 downloads content stored in content database 103 from content server 102 and saved to local memory for later projection. In another embodiment, camera/projector unit 104 streams content stored in content database 103 from content server 102 for projection.

[0084] Referring to Figure 2A in one embodiment, camera/projector unit 200 includes enclosure 201, computer/processor 202, projector 203 connected to computer/processor 202, and camera 204 connected to computer/processor 202.

[0085] Referring to Figure 2B in another embodiment, camera/projector unit 205 positions mirror 206 at angle a adjacent to projector 203 to redirect the field of view of projector 203

[0086] In a preferred embodiment, computer/processor 202 is a laptop computer

CA 02971260 2017-0€-15

WO 2016/109749 PCT/US2015/068192 having a suitable processor and memory for storage of content and database, and any other relevant data. In another embodiment, computer/processor 202 is a tablet computer.

[0087] A projection application is stored in the memory of computer/processor 202 for defining and projecting a set of trigger areas with projector 203, detecting a user presence and recognizing the user with camera 204, detecting a selection of the set of trigger areas with camera 204, and projecting content with projector 203, as will be further described below.

[0088] In a preferred embodiment, camera 204 is a 3-D range camera, such as the Kinect camera from Microsoft Corporation. Camera 204 provides image data in the form of streaming video at the rate of 30 frames per second in a standard 640 x 480 VGA. The focal length of the camera is 525 pixels with a field of view of approximately 63°. Camera 204 also provides distance-to-target information from an on-board infrared transmitter/receiver. The focal length of the infrared transmitter/receiver is 580 pixels with a field of view of approximately 58°. The infrared data is provided as a 16 bit number at a refresh rate of200 ps.

[0089] Referring to Figure 3, display 300 includes camera/projector unit 301 having field of view 302. Camera/ projector unit 301 is connected to arm 304. Arm 304 is attached to and extends from an overhead ceiling. In a preferred embodiment, field of view 302 can range from approximately 45° to 75°. Product 303 is positioned in field of view 302.

[0090] Referring to Figure 4 in another embodiment, display 400 includes camera/projector unit 401 having field of view 403. Camera/projector unit 401 is connected to and supported by stand 402. Stand 402 is removably connected to the floor. Product 404 is positioned in field of view 403.

[0091] Referring to Figure 5, display SOO includes camera/projector unit 501 attached to arm 502 which is attached to base 508. Camera/projector unit 501 projects trigger areas 504

CA 02971280 2017-08-15

WO 2016/109749 PCT/US2015/068192 and 505 onto the surface of product 503. When a user touches the surface of product 503 in trigger areas 504 and 505, camera/projector unit 501 detects the presence of the user in trigger areas 504 and 505 and projects contents 506 and 507, respectively, onto the surface of product 503. In a preferred embodiment, each of contents 506 and 507 is a video. Any type of graphical images may be employed. For example, if a user touches the hood of the car shown in the drawing, an image of a running engine taken from the same perspective as the user may be projected on the hood to simulate a “see through” effect to alert the user to “see” the running engine through the hood.

[0092] Referring to Figure 6 in another embodiment, display 600 includes camera/projector unit 601 having field of view 602 covering product 603. Camera/projector unit 601 projects trigger areas 604, 605, 606, and 607 onto the surface of product 603. When a user touches the surface of product in trigger areas 604, 605, 606, and 607, camera/projector unit 601 projects content 615 onto the surface of product 603.

[0093] In one embodiment, if a user touches the surface of product 603 in any of the trigger areas 604,605,606, and 607, camera/projector unit 601 projects a different color/pattem 616 onto product 603, thereby enabling the user to visualize a potential change to the color or pattern of product 603.

[0094] Stand 608 is positioned adjacent to product 603 and is covered by field of view 602. Stand 608 includes placard 609, support 610 attached to placard 609, and base 611 attached to support 610. Base 611 is removably attached to the floor. Camera/projector unit 601 projects trigger areas 612 and 613 onto the surface of placard 609. When a user touches the surface of placard 609 in trigger areas 612 and 613, camera/projector unit 601 projects content 614 onto placard 609, thereby transforming placard 609 into an interactive digital sign. Content

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

614 preferably includes information, videos, and/or pictures of product 603. Content 614 is retrieved from the content server or from the local memory of camera/projector unit 601.

[0095] Referring to Figure 7, display 700 includes camera/projector units 701 and 702. Camera/projector unit 701 has field of view 704 covering product 703 including product side 706. Camera/projector unit 702 has field of view 705 covering product 703 including product side 707. Camera/projector unit 701 projects trigger area 708 onto the surface of product side 706. When a user touches the surface of product side 706 in trigger area 708, camera/projector unit 701 projects a video or graphics content onto the surface of product side 706. Camera/projector unit 702 projects trigger area 709 onto the surface of product side 707. When the user touches the surface of product side 707 in trigger area 709, camera/projector unit 702 projects a video or graphics content onto the surface of product side 707.

[0096] Referring to Figure 8 in another embodiment, display 800 includes product 802 positioned in view of camera/projector unit 801. Camera/projector unit 801 detects the position and presence of user 803 on product 802. In one embodiment, an identity of user 803 is determined through a facial recognition process, as will be further described below. Trigger areas 805 and 806 are projected by camera/projector 801 onto the surface of product 802 adjacent to user 803 in response to the presence of user 803.

[0097] In one embodiment, content 804 is automatically projected onto the surface of product 802 in response to the presence and identity of user 803.

[0098] In another embodiment, content 804 is projected onto the surface of product 802 in response to user 803 touching any of trigger areas 805 and 806.

[0099] In another embodiment, camera/projector unit 801 projects a first video as content 804 onto the surface of product 802 based on the detected presence of user 803. When

CA 02971260 2017-06-15

WO 2016/109749 PCTÏUS2015/068192 user 803 touches any of trigger areas 805 and 806, content 804 changes to a second video and a third video, respectively. For example, when camera/projector unit 801 detects the presence of user 803 on product 802, content 804 is an introduction or welcome video. When user 803 touches trigger area 805, content 804 is switched to a graphical brochure. When user 803 touches trigger area 806, content 804 is switched to a product animation. Any type of content may be employed for content 804.

[0100] Referring to Figure 9 in another embodiment, display 900 includes camera/projector 901 and product 902. In this embodiment, projected content 903 includes and sequentially displays the components of product 902 including base layer 904, support layer 905, padding layer 906 and top layer 907. In a preferred embodiment, projected content 903 includes a set of animations which represent a “pealing away” of each layer. For example, in response to a user interacting with a trigger area, top layer 907 is “pealed away” to reveal padding layer 906. Each layer in succession can be projected individually on product 902 thus enabling the user to visualize the different layers of the product. In one embodiment, projected content 903 covers the entirety of product 902. In another embodiment, projected content 903 covers a portion of product 902.

[0101] Referring to Figure 10 in another embodiment, display 1000 includes camera/projector 1001 and product 1002. Product 1002 rests on floor 1004. Content 1003 is projected by camera/projector unit 1001 onto floor 1004 surrounding product 1002. Content 1003 includes animation which gives the appearance that product 1002 is driving through various environments, such as rain, snow, or mud to give the appearance that product 1002 is moving.

[0102] Referring to Figure 11, method 1100 for calibrating camera/projector unit 200

CA 02971260 2017-00-15

WO 2016/109749 PCT/US2015/068192 will be described. At step 1101, a baseline image of a surface is determined. In this step, the infrared transmitter/receiver of camera 204 captures a baseline image including depth and color data of the surface of a product. In one embodiment, a thin cover, such as a sheet may be employed to cover a highly reflective surface, such as a surface of a car in order to capture an accurate baseline image.

[0103] At step 1102, a position for each of a set of trigger areas for the product is selected. In a preferred embodiment, each trigger area is an ellipse. Other shapes may be employed. At step 1103, the set of trigger areas is projected on the surface. At step 1104, each trigger area is calibrated by placing a three-dimensional object on the trigger area, such as a hemispherically shaped object. Other three-dimensional objects may be employed. In this step, a position of the three-dimensional object relative to the saved baseline image is captured with camera/projector unit 200.

[0104] At step 1105, the captured position of each three-dimensional object is saved in the memory of camera/projector unit 200. At step 1106, a polygon is warped to match the surface, thereby ensuring that the polygon conforms to the surface on which it will be projected. The polygon may correspond to the entire surface or a section of the surface.

[0105] In a preferred embodiment, a non-affine transform is employed to map a set of points of the polygon to the surface. For example, if the polygon is a square in a flat plane and the surface is an angled surface with respect to the projector, then the points of the square are mapped to a quadrilateral that distorts the square image so that when projected on the angled surface, the projected quadrilateral appears as a square on the angled surface. Other transforms, mapping, and distortion techniques may be employed.

[0106] At step 1107, the warped polygon is associated with the trigger area. At step

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

1108, a set of content is determined for the polygon. In this step, a file location of the desired set of content is associated with the polygon. The set of content is downloaded and saved to local memory or linked to a stream for the set of content. At step 1109, the set of content is warped to match the warped polygon so that the selection of the trigger area projects the desired content undistorted on the surface.

[0107] Referring to Figure 12 in one embodiment, runtime process 1200 will be described. At step 1201, runtime process 1200 begins. At step 1202, a set of trigger areas is projected onto a surface. At step 1203, a touch event is determined, i.e., a determination is made whether a user is in any of the set of trigger areas. In this step, the camera is queried for a user position. The user position is compared to the coordinates of a boundary for the trigger area. If the user position is not within the boundary of the trigger area, then runtime process returns to step 1202. If the user position is within the boundary of the trigger area, then runtime process proceeds to step 1204.

[0108] At step 1204, a polygon with a set of content associated with the trigger area is projected onto the surface. At step 1205, the presence of an end event is determined. An end event is a predetermined time period or another touch event by the user in the set of trigger areas. The predetermined time period is the runtime of the set of content or any desired time period. If an end event has not occurred, then runtime process 1200 returns to step 1204 to continue projecting the polygon and the set of content. If an end event has occurred, then runtime process 1200 proceeds to step 1206.

[0109] At step 1206, whether a terminate command has been received is determined. If a terminate command has not been received, then runtime process 1200 returns to step 1202. If a terminate command has been received, then runtime process 1200 ends at step 1207.

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

[0110] Referring to Figure 13 in another embodiment, runtime process 1300 for projection mapping will be described. At step 1301, a scan of an area, such as a room is performed by a camera to detect any surfaces. At step 1302, the scan of a detected surface is mapped to a virtual area. At step 1303, a polygon and a set of content is warped to match the virtual area.

[0111] In a preferred embodiment, a non-affine transform is employed to map a set of points of the polygon and the set of content to the surface. In other embodiments, other transforms, mapping, and distortion techniques may be employed.

[0112] At step 1304, the camera is queried to determine a presence of a user in the area. At step 1305, if a presence is not detected, then runtime process 1300 returns to step 1304. If a presence is detected, then a set of trigger areas are projected onto the surface at step 1306. At step 1307, a facial recognition process begins to recognize an identity of the user, as will be further described below.

[0113] At step 1308, if the identity of the user is recognized then runtime process 1300 proceeds to step 1309. At step 1309, the polygon and a set of content associated with the user is projected onto the surface. At step 1311, the presence of an end event is determined. An end event is a predetermined time period or another touch event by the user in the set of trigger areas. The predetermined time period is the runtime of the content or any desired time period. If an end event has not occurred, then runtime process 1300 returns to step 1309 to continue projecting the polygon and the set of content. If an end event has occurred, then runtime process 1300 proceeds to step 1313.

[0114] If the identity of the user is not recognized in step 1308, then runtime process 1300 proceeds to step 1310. At step 1310, the polygon and a set of predetermined content is

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 projected onto the surface.

[0115] At step 1312, the presence of an end event is determined. If an end event has not occurred, then runtime process 1300 returns to step 1310 to continue projecting the polygon and the set of predetermined content. If an end event has occurred, then runtime process 1300 proceeds to step 1313.

[0116] At step 1313, a touch event determination is made as to whether a user is in the set of trigger areas. In this step, the camera is queried for a user position. The user position is compared to the coordinates of a boundary for any of the set of trigger areas. If the position is not within the boundary of any of the set of the trigger areas, then runtime process 1300 returns proceeds to step 1308. If the position is within the boundary of any of the set of trigger areas, then runtime process proceeds to step 1314.

[0117] At step 1314, a polygon with a set of content associated with the trigger area is projected onto the surface. At step 1315, the presence of an end event is determined. If an end event has not occurred, then runtime process 1300 returns to step 1314 to continue projecting the polygon and the set of content. If an end event has occurred, then runtime process 1300 proceeds to step 1316.

[0118] At step 1316, whether a terminate command has been received is determined. If a terminate command has not been received, then runtime process 1300 returns to step 1304. If a terminate command has been received, then runtime process 1300 ends at step 1317.

[0119] Referring to Figure 14, step 1307 will be further described as facial recognition process 1400. Facial recognition process begins at step 1401. At step 1402, a camera is queried to record the locations of facial features including eye location and color, ear location, chin location and hair color. In a preferred embodiment, the query is accommodated by

CA 02971260 2017-0€-15

WO 2016/109749 PCT/US2015/068192

Microsoft SKD Rei. 7.1 LIB file Microsoft.Kinect.Toolkit.FaceTracking. Any type of facial tracking may be employed. In a preferred embodiment, at least 121 different locations are determined. At step 1403, the distances between facial features are measured. At step 1404, a “facial map” is created. The facial map includes a table of the facial features and the distances between them. At step 1405, the facial map is compared to a set of digital images stored in memory. A result of the comparison is output at step 1406. At step 1407, the process ends.

[0120] Referring to Figure 15, in one embodiment, components of interactive projection system 1500 are local computer 1502, camera 1504, and projector 1506. Local computer 1502, camera 1504, and projector 1506 can be separate components positioned individually or all be contained in a single housing. Camera 1504 collects color and depth data within its field of view, constantly monitoring defined trigger areas for interaction. Projector 1506 displays projectable content and trigger areas on any surface within its field of view. Local computer 1502 stores and manipulates camera data, content files, locations of projected content, and locations of trigger areas and projected trigger areas. Local computer 1502 analyzes camera data to determine if interaction with any trigger areas has occurred as will be further described. Local computer 1502 includes a transform function capable of warping content images in order to project the content on three-dimensional surfaces undistorted as will be further described.

[0121] In a preferred embodiment, local computer 1502 contains a processor; a memory for storage of content, a database, and any other relevant data; and a software module including hardware drivers. Minimum requirements for local computer 1502 include an Intel Core 17 processor, at least eight (8) GB of RAM, at least five (5) GB of local memory, a 2 GB graphics processor, Windows operating system 8.1 or higher, and a USB3 connection to camera 1504. An example of a suitable computer is the Latitude E6540 laptop computer available from

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192

Dell. In a preferred embodiment, hardware drivers include an image capturing program such as Kinect for Windows available from Microsoft.

[0122] Local computer 1502 is connected to camera 1504. Camera 1504 is a 3-D range camera. Camera 1504 includes firmware for movement interpretation and image recognition. Camera 1504 provides image color data in the form of streaming video at the rate of 30 frames per second in a standard 640 X 480 VGA format at 4 bytes/pixel. The focal length of camera 1504 is 525 pixels with a field of view that can range between approximately 45° to 75°. Camera 1504 also provides distance to target information from an on-board infrared transmitter/receiver pair. The focal length of the infrared transmitter/receiver is 580 pixels with a field of view in the range of approximately 45° to 75°. The infrared data is provided as a 16 bit number at a refresh rate of 200 ps An example of a suitable camera is the Kinect camera available from Microsoft.

[0123] Local computer 1502 is connected to projector 1506 through an HDMI port and cable. Projector 1506 is a PC ready projector having a brightness of 10,000 lumens and a resolution of 1920 X 1200 WUXGA. Projector 1506 has a throw distance in the range of 4 feet to 40 feet with a standard lens. Alternate lenses are available which can dramatically increase the throw distance of projector 1506. Projector 1506 has a field of view angle in the range of 45° to 75° but can be adjusted with different lenses. An example of a suitable projector is the PLWU8100F available from ViviBright.

[0124] Local computer 1502 is connected to microphone 1512 and speaker 1514. Microphone 1512 allow for the reception of audible commands that can be used to trigger the display or playback of images, video, and audio with respect to one or more content areas. Speaker 1514 allows for the playback of the audio channels associated with a video and for the

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 playback of audio files that have been associated with a trigger area.

[0125] In an alternate embodiment, components of interactive projection system 1500 further include network 1510, content server 1522, database 1524, and control computer 1520. Content server 1522 is connected to network 1510. Network 1510 connects local computer 1502 to content server 1522. Content server 1522 is further connected to database 1524 for the storage of multimedia and other relevant data. Network 1510 is a wide area network such as the Internet but can include other wide area or intranet networks. Local computer 1502 can download content stored in database 1524 from content server 1522 and save the content to local memory for later projection. Local computer 1502 can also stream content stored in database 1524 from content server 1522 for projection with projector 1506.

[0126] Control computer 1520 is connected to content server 1522 and connected to local computer 1502 through network 1510. Through control computer 1520, a user can manipulate local computer 1502 and set up interactive projection system 1500 from a remote location. Minimum requirements for control computer 1520 include an Intel Core 17 processor, at least eight (8) GB of RAM, at least five (5) GB of local memory, a 2 GB graphics processor, and Windows operating system 8.1 or higher.

[0127] Referring to Figure 16A and coordinate system 1630, the X axis represents horizontal, the f axis represents vertical, and the Z axis is out of the page. The relative positions of the components within a viewing space 1602 are shown. Viewing space 1602 is a room having a light level sufficient for the projection of projector 1506 to be adequately viewed by a user. The dimensions of viewing space 1602 are those of a typical showroom floor but can vary greatly dependent on the size of the object of the interactive display and the lens of the projector. In general, camera 1504 is positioned to achieve field of view 1604. Field of view 1604 can

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 range from 45° · 75°. Camera 1504 is suspended from ceiling 1640 with arm 1606 but can also be attached to a floor stand with equal success. Object 1608 is positioned on floor 1610 within field of view 1604. The locations of any trigger areas must be within field of view 1604.

[0128] In general, projector 1506 is positioned to achieve field of view 1624. Field of view 1624 can range from 45° - 75°. Projector 1506 may be suspended from ceiling 1640 with arm 1616 but can also extend from a floor stand. Object 1608 is positioned on floor 1610 within field of view 1624. The location of any projected content must be within field of view 1604. Projector 1506 and camera 1504 are separated by a distance 1662 along the Xaxis and by distance 1664 along the Y axis.

[0129] Placard 1622 extends from floor 1610 via stand 1628. As shown, content may be displayed on placard 1622 because placard 1622 is positioned within field of view 1624 of projector 1506. In another embodiment placard 1622 is positioned within field of view 1604 of camera 1504. In another embodiment, placard 1622 is in field of view 1624 of projector 1506 and field of view 1604 of camera 1504 and can display both content areas and trigger areas. In a preferred embodiment, the total height 1636 of the stand and placard is about 3 to 4 feet, the length 1638 of the placard is about 16 inches, and the width 1652 of the placard is about 12 inches. Distance 1650 is the distance between ceiling 1640 and floor 1610 and is about 16 feet in one preferred embodiment.

[0130] Field of view 1624 of projector 1506 is about 72 degrees and includes field of view 1642 and field of view 1644. Field of view 1642 is the field of view for content area 1632 on the surface of object 1608 and is about 7 degrees. Field of view 1644 is the field of view for content area 1634 on placard 1622 and is about 5 degrees. Placard 1622 and object 1608 are within distance 1660 of about 8 feet of each other so that the images projected onto placard 1622

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192 and onto the object are within field of view 1624 of projector 1506.

[0131] Field of view 1604 of camera 1504 includes field of view 1646 and field of view 1648. Field of view 1646 is the field of view for trigger area 1620 on the surface of object 1608 and is about 4 degrees. Field of view 1648 is the field of view for trigger area 1626 on the surface of object 1608 and is about 3 degrees. Field of view 1604 of camera 1504 is different from field of view 1624 of projector 1506 so that camera 1504 may monitor trigger areas that are outside of field of view 1624 of projector 1506.

[0132] Referring to Figure 16B, camera 1504 is positioned generally over object 1608. The position of camera 1504 on the X axis and Z axis is dictated by the desired positions of trigger areas. In one embodiment, camera 1504 is positioned within a range of 5 to 16 feet along the Y axis above object 1608 and within a range of 10 to 20 feet of floor 1610. Trigger areas shown within the field of view of the camera. Trigger area 1620 is positioned on object 1608. Trigger area 1626 is positioned on the surface of object 1608. The system may project an indicator of trigger area 1626 on the surface of object 1608

[0133] The position of projector 1506 on the X and Z axes is dictated by the desired positions of content areas 1632 and 1634. In one embodiment, projector 1506 is positioned within a range of 4 to 40 feet along the Y axis above any surface in which content is desired to be displayed. The ideal distance being dictated by the lens. The surface on which the system displays content should be within the field of view of the projector. In one embodiment, content area 1632 is positioned on the hood portion of object 1608, which in this case is a car. Content area 1634 is positioned on placard 1622.

[0134] Flat surface 1654, shown in Figure 16C, is a hypothetical surface to be targeted at object 1608. Object 1608 includes curved surface 1656, onto which content area

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

1632 is projected. Angle 1658 is the difference between flat surface 1654 and curved surface 1656

[0135] The trigger areas are distinct from the content areas but may be linked with each other. In an alternative embodiment where a trigger area is on the headlight of a car, the associated content area may be on the hood of the car so that when the trigger area on the headlight is touched, a video is played on the content area on the hood and the audio associated with the video is played via one or more speakers. Furthennore, instead of a video being played on the hood, the trigger area on the headlight may be linked to an audio recording so that when the trigger area on the headlight is touched, the audio recording (also known as an audio response) is played.

[0136J In additional embodiments, a computer monitoring the trigger areas is also in communication with the car and can control the functions of the car in response to interaction with the trigger areas. This allows for control of the object—the car—in concert with media displayed or projected onto the object. In one embodiment, interaction with a trigger area on the hood of a car causes a video of the engine to be played on the hood and causes the system to start up the engine of the car. When the object is a car, additional features can be controlled in response to interaction with trigger areas, such as: turning on or off headlights and taillights, locking and unlocking doors, opening and closing windows, opening the trunk, and the like.

[0137] Referring to Figure 16C, surface 1654 is a hypothetical flat surface above object 1608 through which an image from the projected passes. The plane of surface 1654 is perpendicular to the direction that the projector is pointing and is tangential to object 1608.

[0138] Referring to Figure 16D, surface 1656 is the surface of object 1608, which is curved and not flat. Images projected onto curved surfaces, such as surface 1656, instead of onto

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192 flat surfaces, such as hypothetical surface 1654, are adjusted by warping and transforming the image to match the curvature of the surface.

[0139] Referring to Figure 17A, trigger area is described. Trigger areas 1702 and 1704 are predefined 3-dimensional spaces stored in the database. Trigger areas are monitored by the infrared sensors in camera 1504 for interaction with the system by a user. When the infrared sensors detect a user in a trigger area, a predefined action, such as displaying stored content, is carried out by the system.

[0140] A trigger area can be any three-dimensional shape bounded by a series of points which form lines that enclose a volume. Referring to coordinate system 1706, trigger area 1702 is bounded by a set of points including Xi, Yi, Zi; X2, Yi, Zi; Xi, Y2, Zi; X2, Y2, Zi; and function 1708. Function 1708, in this example, is a hyperbolic function defined by the equation

C » x² + y² Eq. 12

[0141] Other hyperbolic functions or linear functions may be used to define any set of points, in the X, Y, or Z directions, defining a general shape, so long as the function is closed with respect to the discrete area. Trigger area 1704 is a three-dimensional polygon. The various X, Y, and Z values for each trigger area can vary. The X and Y values range between 2 and 12 inches, preferably around 7 to 8 inches. The Z value can range between 1 and 4 inches, preferably about 2 inches.

[0142] Figure ISA is a diagram of a set of data elements stored in the computer and accessed by a trigger area software program, such as touchable area application 1802, in accordance with one embodiment. Touchable area application 1802 is an application running on a computer system that allows for the display of projected content onto objects and surfaces based user interaction with trigger areas that have been associated with and linked to the

CA 02971260 2017-08-15

WO 2016/109749 PCT/US2015/068192 projected content. Touchable area application 1802 includes tabs 1804,1806, and 1808; one or more content controls 1852; and one or more trigger controls 1854.

[0143] Tabs 1804,1806, and 1808 are user interface elements displayed in the application window of touchable area application 1802 that allow for the selection between different modes of the application, such as modes 1910,1912, and 1914 shown in Figure 19A. Tabs 1804,1806, and 1808 are displayed in display area 1816.

[0144] Display area 1816 is the display space of touchable area application 1802. Touchable Area Application includes trigger controls 1854 and content controls 1852 that are displayed in display area 1816.

[0145] Trigger controls 1854 are each associated with an instance of trigger area data structure 1818. Trigger area data structure 1818 include several fields of data to identify a trigger area that is monitored by a camera with the computer, such as camera 1504 and computer 1502 of Figure 15.

[0146] Content controls 1852 are each associated with an instance of content data structure 1820. Content data structure 1820 identifies the content and the content area in which the content is displayed by the system. Content data structure 1820 includes several data fields to identify the content and control its display.

[0147] Figure 18B is a diagram of a trigger area data structure in accordance with one embodiment of the disclosure. Trigger area data structure 1818 is associated with trigger control 1854 and includes coordinates 1822, type 1824, name 1826, identifier 1828, depth data 1830, content data structure 1820, and filename 1834. The data associated with trigger area data structure 1818 is stored to one or more files that allows trigger data structure area 1818 to be saved and reloaded by touchable area application 1802.

CA 02971260 2017-06-15

WO 2016/109749 PCTÏUS2015/068192

[0148] Coordinates 1822 identify x and y locations for each of four comers for where trigger control 1852 is displayed in display area 1816.

[0149] Type 1824 identifies a type of content associated directly with trigger area data structure 1818, such as an image or video.

[0150] Name 1826 is a user friendly name that identifies an instance of a trigger control.

[0151] Identifier 1828 is a unique identifier, such as a globally unique identifier (GUID), which is given to each trigger control 1854 of touchable area application 1802 to access trigger control 1854.

[0152] Depth data 1830 identifies the three-dimensional location or touch box associated with trigger area data structure 1818 and trigger control 1854.

[0153] Content data structure 1820 identifies the content area that is associated with trigger area data structure 1818 and trigger control 1854.

[0154] Filename 1834 identifies a file, such as an image file, that provides a default image for trigger control 1854. Filename 1834 may point to a fully transparent image so that, by default, no image is displayed by trigger control 1854.

[0155] Figure 18C is a diagram of a content data structure in accordance with one preferred embodiment. Content data structure 1820 is associated with content control 1852 and includes several data fields to identify and display content within display area 1816 of touchable area application 1802.

[0156] Coordinates 1836 identify x and y locations for each of the comers of content control 1852. The comers identified by coordinates 1836 form a polygon. When the image or video associated with content data structure 1820 is displayed, the comers of the image or video

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 are warped and transformed to fit the shape identified by coordinates 1836.

[0157] Type 1838 identifies the type of content associated with content data structure 1820 and content control 1852. Content types include image, video, sound, and the like.

[0158] Loop 1840 identifies how many times to loop the content when, for example, type 1838 of content area 1820 identifies that the content is a video.

[0159] Orientation 1842 identifies the orientation used to display images associated with content control 1852. In one embodiment, Orientation 1842 specifies that, when displayed, the image or video associated with content control 1852 is flipped about the horizontal or vertical axes or rotated by a multiple of 90 degrees.

[0160] Filename 1844 identifies the filename of the image or video associated with content data structure 1820 and content control 1852. In additional embodiments, filename 1844 is a link to static or dynamic content such as a video file, a streaming video file, or a live camera stream that is played in response to interaction with a trigger area.

[0161] Identifier 1846 is a unique identifier or GUID that is given to each content control 1852 of touchable area application 1802 to access content control 1852.

[0162] Name 1848 is a user friendly name that identifies an instance of a content control.

[0163] Figure 19A is a data flow diagram of a system calibrating trigger areas and content areas in accordance with the disclosure. In this embodiment, the system includes computer 1902, camera 1904, projector 1906, and display 1908.

[0164] Computer 1902 is connected to range camera 1904 via a USB (universal serial bus) 3.0 connection and is connected to projector 1906 and display 1908 by an HDMI (highdefinition multimedia interface) connection. Computer 1902 controls the images displayed by

CA 02671260 2017-06-15

WO 2016/109749 PCT/US2015/068192 display 1908 and projector 1906. Camera 1904 provides color data and depth data. Projector 1906 projects images sent from computer 1902 onto any surface in front of projector 1906, including one or more display stands, objects, and screens. Display 1908 displays the application to aid the calibration process.

[0165] The touchable area application uses three modes to perform the calibration process, first mode 1910, second mode 1912, and third mode 1914. When first mode 1910 is selected, user interface controls are associated with the touchable area application. For example, in the case where a static image is displayed by default and two videos are shown based on interaction with two trigger areas, the static image, the two videos, and the two trigger controls are added to the touchable area application. In First mode 1910, the application performs steps 1916,1918,1920, and 1922

[0166] In second mode 1912, the three-dimensional real world locations fortrigger areas are selected. In second mode 1912, the application performs steps 1924,1926,1928,1930, and 1932.

[0167] In third mode 1914, the projected locations of the projected content is selected, fine-tuned, and aligned. In third mode 1914 the application performs steps 1934,1936, 1938, and 1940.

[0168] At step 1916, a first tab is selected in the touchable area application to set the mode of the touchable area application to first mode 1910. One example of code for selecting a first tab is shown in the computer program listing appendix in the computer program listing of the file named MainWindow xaml cs at lines 922-926.

[0169] At step 1918, the display image of the application is sent to display 1908 so that the user interface controls—trigger controls and content controls—can be added and

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 manipulated.

[0170] At step 1920, trigger controls and content controls are added and updated. New trigger controls and content controls may be added and the associations between the trigger controls, trigger areas, content controls, and content areas may be updated. The comers of the content controls and trigger controls can be moved or dragged to reshape the images and video that are projected onto the content areas and trigger areas. One example of code that performs this function is shown in the computer program listing appendix in the computer program listing of the file named NonAffineTransformControl.xaml.es at lines 437-476. This code calls a function to determine a transformation matrix, which is shown in the computer program listing appendix in the computer program listing of the file named NonAffineTransformControl.xaml.es at lines 1220-1258. One example of code for updating trigger or content controls is shown in the computer program listing appendix in the computer program listing of the file named NonAffineTransformControl.xaml.es at lines 478-512.

[0171] At step 1922, for each change to the user interface controls from step 1920, the image sent to display 1908 is updated. This provides feedback to allow the sequences, timing, and interaction between the trigger controls and content controls.

[0172] At step 1924, a second tab is selected to shift the application to second mode 1912 to allow for the selection of the physical locations of the trigger areas that are associated with trigger controls. One example of code for selecting a second tab is shown in the computer program listing appendix in the computer program listing of the file named MainWindow.xaml.es at lines 927-971.

[0173] At step 1926, computer 1902 receives color and depth data from camera 1904.

[0174] At step 1928, the display area of the application window of the application is

CA 02671260 2017-06-15

WO 2016/109749 PCT/US2015/068192 changed to show a combination of the color and depth data received from camera 1904 with the trigger controls overlaid onto the image.

[0175] At step 1930, the trigger controls that were added in first mode 1910 are associated physical locations that are monitored by camera 1904. The associations are created by dragging and moving the trigger controls displayed on display 1908. One example of code for updating trigger controls is shown in the computer program listing appendix in the computer program listing of the file named NonAffineTransformControl.xaml.es at lines 478-512.

[0176] At step 1932, for each change to a trigger control and each change in the color and depth data received from camera 1904, computer 1902 updates the image sent to display 1908

[0177] At step 1934, a third tab, such as third tab 1808 of Figure 18, is selected to set the mode of the application to third mode 1914 to allow for selection of the location of the content areas that will be associated with the content controls. One example of code for selecting a third tab is shown in the computer program listing appendix in the computer program listing of the file named MainWindow.xaml.es at lines 916-920.

[0178] At step 1936, computer 1902 stops displaying the color and depth data from camera 1904 and displays the touchable area application as a full screen window on projector 1906 including the user interface controls. Projector 1906 projects the image from computer 1902 onto the surfaces and objects in front of projector 1906. For example, when camera 1904 projects onto the hood of a car the content displayed by the touchable area application is an image of the engine. In an alternative embodiment, the image sent by computer 1902 to projector 1906 is also sent to display 1908 so that the image is displayed by both display 1908 and by projector 1906.

CA 02971280 2017-00-15

WO 2016/109749 PCT/US2015/068192

[0179] At step 1938, the content control is updated. For example, the sizes of the content control are adjusted to match the surfaces onto which they are being projected. When the hood of the car has a trapezoidal shape, the comers of the content control are dragged to match the trapezoidal shape of the hood. One example of code for updating content controls is shown in the computer program listing appendix in the computer program listing of the file named NonAffineTransformControl.xaml.es at lines 478-512.

[0180] At step 1940, for each update to the content controls from step 1938, the image sent to the projector is updated to include the changes made in step 1938.

[0181] Figure 19B is a flow chart for updating a trigger area associated with a trigger control in accordance with one embodiment of the disclosure. The method for updating a trigger area of Figure 19B is one embodiment for step 1930 of Figure 19A.

[0182] At step 1942, color and depth data are received from a camera, such as camera 1904. The color and depth data are received as one or more byte streams that contain two frames of data, one based on the color sensor of the camera and one based on the infrared depth sensor of the camera. The resolution of the color sensor and the depth sensor differ so that the resolution of the two frames of data also differ. As one example, the color data may have a resolution of 1920 by 1080 with 32 bits of color information per pixel and the depth data may have a resolution of 640 by 480 with 16 bits of depth information per pixel.

[0183] At step 1944, a combination of the color and depth data is displayed. The color data resolution is cropped to the depth data resolution. Within the cropped resolution, color data is displayed if the depth data indicates there is a surface or object within the range of the depth sensor.

[0184] At step 1946, a trigger area associated with a trigger control is selected. The

CA 02971260 2017-0€-15

WO 2016/109749 PCT/US2015/068192 trigger area identifies the three-dimensional coordinates of a trigger area that is stored in a trigger area data structure associated with a trigger control and is with respect to the camera that provides the depth data. As shown in Figure 19C, area 1952 is an area of a surface within the range and viewing area of the depth sensor. In a preferred embodiment, the shape of the area defaults to a square or rectangle, but any other shape can be used.

[0185] At step 1948, a depth for the trigger area is selected. The depth combined with the trigger area creates a volume that is associated with a trigger area and is used to determine if the trigger area has been touched. Figure 19D show's volume 1954 created from area 1952 through the addition of depth 1956.

[0186] At step 1950, the trigger area and depth are stored. The trigger area and depth are associated with the trigger area and saved to the database. In a preferred embodiment, the trigger area and depth are stored in depth data 1830 of trigger area data structure 1818 of Figure 18B.

[0187] Figure 19E is a flow chart for updating a content area in accordance with one embodiment of the disclosure. The method for updating a content area of Figure 19E is one embodiment for step 1938 of Figure 19A.

[0188] At step 1958, an initial flat projection area is selected. The native resolution of the content that is selected and associated with a content control provides the initial flat projection area. Figure 19F shows initial flat projection 1968 of content that has been selected.

[0189] At step 1960, an area related to an affine transformation is selected. In a preferred embodiment, the area for the affine transformation is selected by dragging the comers the content area into a desired shape. As an example, when the content area is a rectangular image, one or more of the comers are dragged to form a scaled parallelogram shape, as shown by

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 transformation 1970 of Figure 19F.

[0190] At step 1962, an area related to a non-affine transformation is selected. In a preferred embodiment, the area is selected by dragging one or more of the comers and lines that from the boundary of the content area into curved shapes, as shown by transformation 1972 of Figure 19F.

[0191] At step 1964, a brush or filter is created from the affine transformation and the non-affine transformation. The brush will be applied to the content associated with the content control to transform the original size and shape of the content to the selected size and shape of the area selected via steps 1960 and 1962 that correspond to an affine transformation and a nonaffine transformation. When the content is a video, the brush is applied to each frame of the video as the video is displayed.

[0192] At step 1966, the brush created from the selected areas for the affine transformation and the non-affine transformation are saved to persistent storage

[0193] Figure 19F shows the matrix transformations applied to content in accordance with one embodiment of the disclosure. Flat projection 1968 is an initial projection that has not been warped and transformed. Projection 1970 has been warped and transformed using an affine transformation. Projection 1972 has been warped and transformed using a non-affine transformation.

[0194] The following code listing provides an example of calculating a transformation:

J! The input array of points describes a 2D rectangle il (with Z assumed to be zero) in the order // lower-left, upper-left, lower-right, upper-right.

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 // The returned transform maps the points (0,0,0),

JI (0, 1, 0), (1, 0, 0), and (1, 1, 0) to these points.

private Matrix3D CalculateNonAffineTransform(Point3D[] points)

JI Affine transform

H-----------JI This matrix maps (0, 0) —> (xO, yO)

JI (0, 1)--> (xl, yl)

JI (1, 0) -> (x2, y2) // (1, l)->(x2 + xl+x0,y2 + yl+y0)

Matrix3D A = new Matrix3DQ;

A.M11 = points[2].X - points[0].X;

A.M12 = points[2].Y - points[0].Y;

A.M21 = points[l].X - points[0].X;

A.M22 = points[l].Y - points[0].Y;

A.OffsetX = points[0].X;

A.OffsetY = points[0].Y;

J! Calculate point (a, b) that get mapped by the affine

J! transform to (x3, y3) double den = A.M1 1 * A.M22 - AM 12 * A.M21;

double a = (A.M22 * points[3].X - A.M21 * points[3].Y +

A M21 * A.OffsetY - A.M22 * A.OffsetX) I den;

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192 double b = (A.M11 * points[3].Y - A.M12 * points[3].X +

A.M12 * A.OffsetX - A MI 1 * AOffsetY) / den;

1! Non-affine transform //--------------JI This matrix maps (0, 0) —> (0, 0) // (0,1)->(0,1) // (1,0)->(1,0) // (1, 1)—>(a, b)

Matrix3D B = new Matrix3D();

B.M11 =a/(a + b -1);

B.M22 =b/(a + b -1);

B.M14 = B.M11 - 1;

B.M24 = B.M22- 1;

return B * A,

[0195] Figure 20A is a data flow diagram of a system using trigger areas in accordance with the disclosure. The system includes computer 2002, range camera 2004, and projector 2006 and has already been calibrated to align the trigger areas and content areas to the surfaces and objects in front of projector 2006.

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192

[0196] Computer 2002 is connected to range camera 2004 via a USB (universal serial bus) 3.0 connection and is connected to projector 2006 by an HDMI (high-definition multimedia interface) connection. Computer 2002 controls the image displayed by projector 2006 based on data received from range camera 2004. Range camera 2004 provides color data and depth data. Projector 2006 projects images sent from computer 2002 onto any surface in front of projector 2006, including one or more display stands, objects, and screens.

[0197] At step 2008, computer 2002 sends the display image that will be received and displayed by projector 2006. The display image has a resolution of, e.g., 1920 by 1080 pixels, which is a native resolution for projector 2006.

[0198] At step 2010, a frame of data is sent from range camera 2004 and received by computer 2002. The color data and the depth data are measured by range camera 2004 and are separated into two different byte streams with different resolutions. The color data is 1920 by 1080 by 32 bits with 8 bits for each of red, green, blue, and alpha channels. The alpha channel defaults to 1. The depth data is 640 by 480 by 16 bits with 13 bits for depth information that measures depth from range camera 2004 in 1 millimeter increments.

[0199] At step 2012, computer 2002 determines if a trigger area has been touched.

The determination is made by comparing a base line of the depth data received from range camera 2004 with the current depth data received from range camera 2004. In one preferred embodiment, the baseline depth data is an average of three frames of data captured during the calibration process.

[0200] In one embodiment, detection of a trigger area is determined by calculating whether 20% or more of a volume associated with the trigger area is occupied for at least three frames in a row. One example of code for performing this function is shown in the computer

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 program listing appendix in the computer program listing of the file named MainWindow.xaml.es at lines 494-527. The occupation of the volume is determined by the depth associated with the pixels related to the volume of the trigger area. The depth should be less than the baseline depth, but greater than a threshold depth so that a trigger area is only activated by an object that is near the real world location of the trigger area.

[0201] In another embodiment, for each pixel of a trigger area that is occupied, the surrounding eight pixels (forming a 3 by 3 or nine pixel matrix) are tested to see if they are each similarly occupied. This additional precision allows for the touch of a single finger to activate a trigger area.

[0202] At step 2014, computer 2002 updates the display image to be sent to projector 2006 based on frame data received from range camera 2004. When no trigger areas are clicked or activated, the display image is updated for content areas that includes video. When a trigger area is properly activated, the content areas associated with that trigger area are activated.

[0203] At step 2016, computer 2002 sends the updated image. The updated image is based on the content being displayed and the interaction with trigger areas.

[0204] Figure 20B is a flow chart for determining a touch of a trigger area in accordance with one embodiment of the disclosure. One example of code for performing steps analogous to the steps of the flow chart of Figure 20B is shown in the computer program listing appendix in the computer program listing of the file named MainWindow.xaml .es at lines 494527.

[0205] At step 2018, depth data is received from a camera, such as range camera 2004 of Figure 20. In a preferred embodiment, the depth data has a resolution of 640 by 480 pixels with 13 bits of depth information per pixel and aligned to 16 bits.

CA 02971260 2017-0€-15

WO 2016/109749 PCT/US2015/068192

[0206] At step 2020, a determination is made as to whether 20% or more of the trigger volume associated with a trigger area has been filled by an object. Each pixel of the depth data related to the trigger volume is compared to its corresponding baseline depth to determine of the pixel includes an object that is a threshold amount above the baseline depth. When 20% or more of the pixels associated with the trigger volume have are determined to include an object, then a frame counter is incremented and the method proceeds to step 2022. Otherwise, the frame counter is decremented to a minimum value of zero and the method returns to step 2018 to wait for the next frame of depth data.

[0207] At step 2022, the frame count for the trigger area is compared. In a preferred embodiment, an object must be detected in relation to the touchable are for three consecutive frame of data. When it has been detected for three consecutive frames of data, the method proceeds to step 2024. Otherwise, the method returns to step 2018 to wait from the next frame of depth data

[0208] At step 2024, an object of sufficient size has been detected a sufficient number of times to activate a touch event that is associated with the trigger area. In a preferred embodiment the touch event is a mouse click event that is associated with a trigger control associated with the trigger area that causes the display of content via a content control onto the content area. The content displayed in the content area may be static, dynamic, or streamed.

[0209] In Figures 21A and 21B, a system is being calibrated to display content onto an object. A touchable area application, such as touchable area application 1802 of Figure 18, includes tabs 2116, 2118, and 2120 that allow for switching between different modes—such as modes 1910,1912, and 1914 of Figure 19A—in the touchable area application for calibration. The first mode to add, associate, and manipulate trigger areas and content areas is accessed by

CA 02971280 2017-00-15

WO 2016/109749 PCT/US2015/068192 selecting tab 2116. The second mode to identify the three-dimensional real world location of the trigger areas associated with trigger controls is accessed by selecting tab 2118. The third mode to display, align, and manipulate the content areas with respect to the object is accessed by selecting tab 2120.

[0210] Figure 21A is a view of the application when a first tab is selected. When first tab 2116 is selected, the application window 2102 is seen on a first display, which is connected to and may be substantially co-located with a computer, such as computer 1902 of Figure 19A. Displaying application window 2102 on the first display allows a user to set up the trigger areas and content areas for the system along with the linking and interaction behaviors associated with the trigger controls, content controls, trigger areas, and content areas. Trigger controls 2104 and 2106 are displayed in application window 2102. Application window 2102 also displays content control 2108, content control 2110, content control 2112 and mouse pointer 2114. Application window 2102 also includes tabs 2116,2118, and 2120. Background 2128 of application window 2102 is black.

[0211] Trigger control 2104 is associated with content control 2110. When the system is running and trigger control 2104 is touched, the content of content control 2110 will be played on top of the content of content control 2108. Similarly, trigger control 2106 is associated with content control 2112 so that touching or clicking trigger control 2106 plays the media or content of content control 2112 on top of the content of content control 2108.

[0212] Content control 2108 includes the default content that is normally displayed by the system. In the embodiment of Figures 21A and 2 IB, content control 2108 is linked to a static image of an engine, such as the engine of car 2122. Content control 2110 is linked to trigger control 2104 and includes a video of the performance characteristics of the object, i.e., car

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

2122 of Figures 21B. Content control 2112 is linked to a video of the safety systems of car 2122 and is associated with trigger control 2106.

[0213] Figure 21B is a view of the application when second tab 2118 is selected. When second tab 2118 is selected, instead of showing content controls on background 2128, application window 2102 shows a combination of color and depth data from the camera. When there is no depth data for a pixel, the pixel is shown as black. Otherwise, the RGB (red green blue) color value from the camera is shown. In one embodiment when the resolution of the color data does not match the resolution of the depth data, parts of the image where there is no color data are shown only using the depth data, such as in areas 2130 and 2132. Area 2134 is shown using the combination of color and depth data and areas 2130 and 2132 are shown using only depth data since there is no color data for that portion. Alternative embodiments show any one or combination of color data and depth data on application window 2102.

[0214] Trigger controls 2104 and 2106 can be dragged to any part of the view shown application window 2102 in Figure 21B. When a trigger control is moved, it is associated with the three-dimensional physical location that corresponds with the pixels displayed in application windows 2102 in Figure 2 IB. A touch box is created for the trigger area associated with the trigger control so that when it is determined that an object sufficiently fills the touch box, the trigger control is activated by creating a virtual mouse click for the trigger control associated with the trigger area in the display area of the touchable area application.

[0215] In an additional embodiment, when the projector can display an image in view of the camera, the projector flashes an image related to the size and location of the trigger area. When the flashing image related to the trigger area is identified from the color data from the camera, the three dimensional coordinates that correspond to the trigger area are identified from

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192 the depth data and are associated with the trigger control to automatically calibrate the three dimensional location of the trigger area with the trigger control.

[0216] It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept. It is understood, therefore, that this disclosure is not limited to the particular embodiments herein, but it is intended to cover modifications within the spirit and scope of the present disclosure as defined by the appended claims.

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 1.</td><td> using</td><td> System;</td><td></td><td></td><td></td><td></td>

<td> 2.</td><td> using</td><td> System.Collections.Generic;</td><td></td><td></td><td></td><td></td>

<td> 3.</td><td> using</td><td> System.Configuration;</td><td></td><td></td><td></td><td></td>

<td> 4.</td><td> using</td><td> System.10;</td><td></td><td></td><td></td><td></td>

<td> 5.</td><td> using</td><td> System.Linq;</td><td></td><td></td><td></td><td></td>

<td> 6.</td><td> using</td><td> System.Runtime.InteropServices;</td><td></td><td></td><td></td><td></td>

<td> 7 .</td><td> using</td><td> System.Text;</td><td></td><td></td><td></td><td></td>

<td> 8 .</td><td> using</td><td> Sys tem. Threading ;</td><td></td><td></td><td></td><td></td>

<td> 9.</td><td> using</td><td> System.Threading.Tasks ;</td><td></td><td></td><td></td><td></td>

<td> 10.</td><td> using</td><td> Sys tem. Windows ;</td><td></td><td></td><td></td><td></td>

<td> 11.</td><td> using</td><td> System. Windows. Controls ;</td><td></td><td></td><td></td><td></td>

<td> 12.</td><td> using</td><td> System. Windows. Data ;</td><td></td><td></td><td></td><td></td>

<td> 13.</td><td> using</td><td> Sys tem. Windows. Documents;</td><td></td><td></td><td></td><td></td>

<td> 14.</td><td> using</td><td> System.Windows. Input;</td><td></td><td></td><td></td><td></td>

<td> 15.</td><td> using</td><td> System.Windows.Media;</td><td></td><td></td><td></td><td></td>

<td> 16.</td><td> using</td><td> System.Windows.Media.Imaging;</td><td></td><td></td><td></td><td></td>

<td> 17.</td><td> using</td><td> System.Windows.Media.Media3D;</td><td></td><td></td><td></td><td></td>

<td> 18.</td><td> using</td><td> System.Windows.Navigation;</td><td></td><td></td><td></td><td></td>

<td> 19.</td><td> using</td><td> Sys tem. Windows. Shapes ;</td><td></td><td></td><td></td><td></td>

<td> 20.</td><td> using</td><td> System.Windows.Threading;</td><td></td><td></td><td></td><td></td>

<td> 21.</td><td> using</td><td> System.Xml;</td><td></td><td></td><td></td><td></td>

<td> 22.</td><td> using</td><td> System.Xml.Serialization;</td><td></td><td></td><td></td><td></td>

<td> 23.</td><td> using</td><td> WpfAnimatedGif ;</td><td></td><td></td><td></td><td></td>

<td> 24.</td><td></td><td></td><td></td><td></td><td></td><td></td>

<td> 25.</td><td></td><td></td><td></td><td></td><td></td><td></td>

<td> 26.</td><td> using</td><td> Microsoft.Kinect;</td><td></td><td></td><td></td><td></td>

<td> 27.</td><td> using</td><td> System.Runtime.InteropServices;</td><td></td><td></td><td></td><td></td>

<td> 28.</td><td> using</td><td> System.Diagnostics;</td><td></td><td></td><td></td><td></td>

<td> 29.</td><td></td><td></td><td></td><td></td><td></td><td></td>

<td> 30.</td><td colspan="2"> namespace NonAffineTransform</td><td></td><td></td><td></td><td></td>

<td> 31.</td><td> {</td><td></td><td></td><td></td><td></td><td></td>

<td> 32.</td><td></td><td></td><td></td><td></td><td></td><td></td>

<td> 33.</td><td colspan="2"> public class WispInteractiveData</td><td></td><td></td><td></td><td></td>

<td> 34.</td><td> {</td><td></td><td></td><td></td><td></td><td></td>

<td> 35.</td><td></td><td> public string wispID { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 36.</td><td></td><td> public int wispindex { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 37.</td><td></td><td> public int wispPixDepth { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 38. 39.</td><td></td><td></td><td></td><td></td><td></td><td></td>

<td> 40.</td><td colspan="2"> public class WispCoveredFrameltem</td><td></td><td></td><td></td><td></td>

<td> 41.</td><td> {</td><td></td><td></td><td></td><td></td><td></td>

<td> 42.</td><td></td><td> //wispID we are working with</td><td></td><td></td><td></td><td></td>

<td> 43.</td><td></td><td> public string wispID { get; set; )</td><td></td><td></td><td></td><td></td>

<td> 44.</td><td></td><td> //Captures all the pixels we are worrying</td><td> about</td><td colspan="2"> (helps</td><td> in</td>

<td></td><td colspan="2"> avg of how many are covered</td><td></td><td></td><td></td><td></td>

<td> 45.</td><td></td><td> public int totalpixels { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 46.</td><td></td><td colspan="2"> //for the loop = to count how many of the pixels</td><td> for</td><td colspan="2"> this</td>

<td></td><td colspan="2"> wisp have been covered.</td><td></td><td></td><td></td><td></td>

<td> 47.</td><td></td><td> public int currentPixCount { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 48.</td><td></td><td> //CountNum - must have at least a certain</td><td colspan="3"> coverage for</td><td> a</td>

<td></td><td colspan="2"> certain number of frames to be considered a click</td><td></td><td></td><td></td><td></td>

<td> 49.</td><td></td><td> public int frameCountNum { get; set; }</td><td></td><td></td><td></td><td></td>

<td> 50.</td><td></td><td colspan="3"> //after click is achieved, then this is set to true.</td><td> a</td><td> ne’</td>

click cannot happen while this is true

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 51.</td><td> click.</td><td> //set to false after countNum hits 0 so we can have a new</td>

<td> 52. 53. 54. 55. 56.</td><td> )</td><td> public bool isCurrentlylnClickState { get; set; }</td>

<td> 57. 58. 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81. 82. 83. 84. 85. 86. 87. 88. 89. 90. 91. 92. 93. 94. 95. 96. 97. 98. 99. 100.</td><td colspan="2"> public partial· class MainWindow : Window { III &lt;surmary&gt; III Active Kinect sensor III &lt;/sumnary&gt; private KinectSensor kinectSensor = null; private const int MapDepthToByte = 8000 / 256; /11 &lt;summary&gt; III Reader for depth frames III &lt;/summary&gt; /// private CoordinateMapper coordinateMapper = null; private MultiSourceFrameReader multiFrameScurceReader = null; III &lt;summary&gt; III Description of the data contained in the depth frame III &lt;/sumnary&gt; private FrameDescription depthFrameDescription = null; III &lt;summary&gt; III Bitmap to display HI &lt;/su.tmary&gt; private WriteableBitmap depthBitmap = null; III &lt;suriL’nary&gt; III Intermediate storage for frame data converted to color III &lt;/sumnary&gt; public static byte[] depthpixels = null; //displayMe ushort[] _depthData = null; byte[] _colorData = null; byte[] _displayPixels = null; ColorSpacePoint[] _colorPoints = null; readonly int bytes_per_pixel = iPixelFormats.Bgr32.Bits?erPixel +7)/ 8; WriteableBitmap _bitmap = null; int depthWidth = 0; int depthHeight = 0; int colorwidth = 0; int colorHeight = 0;</td>

101.

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 102. 103. 104. 105. 106. 107. 108 . 109. 110.</td><td> //inverse arrays byte[] InverseDepthPixels = null; 111 &lt;summary&gt; III Current status text to display #region initSpaceeVars //READ in default settings -these can be changed in the app.config file</td>

<td> 111.</td><td> private int FRAME_NUM_REQUIRED_FCR_CLICK = int. Parse(ConfigurationManager.AppSettings[conf_FRAME_NUM_REQUIRED_ FOR_CLICK]);</td>

<td> 112.</td><td> private double WISP_PERCENT_COVERAGE_FOR_CLICK = double.Parse(ConfigurationManager.AppSettings[conf_WISP_PERCENT_COV ERAGE_FCR_CLICK]) ; II percentage</td>

<td> 113.</td><td> public int MIN_DEPTH_CLICK = int.Parse(ConfigurationManager.AppSettings[conf_MIN_DEPTE_CLICK]) ;</td>

<td> 114.</td><td> public int MAX_DEPTH_CLICK = int.Parse(ConfigurationManager.AppSettings[”conf_MAX_DEPTE_CLICK]) ;</td>

<td> 115.</td><td> public int FRAME_SKIP = int.Parse (ConfigurationManager.AppSettings[”conf_FRAME_SKIP]) ;</td>

<td> 116. 117. 118.</td><td> public int FRAME_SKIP_COUNTER = 0; List&lt;WispCoveredFrameIteiri&gt; WispViewTabList = new List&lt;WispCoveredFrameItem&gt;();</td>

<td> 119. 120. 121. 122. 123. 124. 125. 126. 127. 128. 129.</td><td> public bool PauseTouch = false; public int PauseTouchCounter = 0; public int PauseOnLongWaitCounter = 0; public int PauseOnLongWaitMax = 1500; public static bool enterlab2 = false; public static bool enterViewerMode = false; public static string popupRecieve_WispName = public static List&lt;Point&gt; recievePop_tempPoints = new List&lt;Point&gt;();</td>

<td> 130. 131. 132.</td><td> public static bool processPopup = false; private List&lt;WispInteractiveData&gt; wispDepths = new List&lt;WispInteractiveData&gt;();</td>

<td> 133. 134. 135. 136.</td><td> public static int theBitMapSourceWidth; //public bool startlnViewMode = false; private string _WispCoveredFrameIteni_XMLFile = WispCoveredFrameltem.xinl;</td>

<td> 137. 138. 139. 140. 141. 142. 143.</td><td> private string WispDepths XMLFile = WispDepths.xml; #endregion public MainWindow() { AppDomain currentDomain = AppDorcain.CurrentDomain;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 144.</td><td> currentDomain.UnhandledException += new UnhandledExceptionEventHandler(KillProgram);</td>

<td> 145. 146. 147. 148. 149. 150. 151. 152. 153. 154. 155. 156. 157. 158. 159. 160. 161. 162. 163. 164.</td><td> //start 3d Camera startKinect(); InitializeComponent(); //load in all possible touch points from the xml file LoadWispDepthsFromXML(); LoadWispsFromXML(); //check if monoter system issues this program a command var sivm = App.startlnViewMode; if (sivm == true) { //load viewer and full screen. //System.Windows.MessageBox.Show(true); startAppinFullView(); ) } static void KillProgram(object sender, UnhandledExceptionEventArgs args)</td>

<td> 165. 166. 167. 168. 169. 170.</td><td> { Environment.Exit(0); //Exception e = (Exception)args.Exceptionobject; //Console.WriteLine(MyHandler caught : + e.Message); //Console.WriteLine(Runtime terminating: {0}, args .IsTerminating);</td>

<td> 171. 172. 173. 174. 175. 176. 177. 178. 179. 180. 181.</td><td> } public void startKinect() { // get the kinectSensor object this.kinectSensor = KinectSensor.GetDefault(); // open the reader for the depth frames this.multiFrameSourceReader = this .kinectSensor.CpenMultiSourceFrameReader(FrameSourceTypes.Depth | FrameSourceTypes.Color);</td>

<td> 182. 183. 184.</td><td> this.multiFrameSourceReader.MultiSourceFrameArrived += this.Readsr_MultiSourceFrameArrived;</td>

<td> 185. 186.</td><td> this.coordinateMapper = this .kinectSensor.CoordinateMapper;</td>

<td> 187. 188 . 189.</td><td> // get FrameDescription from DepthFrameSource this.depthFrameDescription = this .kinectSensor.DepthFrameSource.FrameDescription;</td>

SUBSTITUTE SHEET (RULE 26)

GA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 190. 191. 192. 193.</td><td> depthwidth = depthFrameDescription.Width; depthHeight = depthFrameDescription.Height; // allocate space to put the pixels being received and converted</td>

<td> 194.</td><td> depthpixels = new byte[this.depthFrameDescription.Width * this.depthFrameDescription.Height; ;</td>

<td> 195. 196.</td><td> FrameDescription colorFraneDescription = this .kinectSensor.ColorFrameSource.FrameDescription;</td>

<td> 197. 198. 199. 200. 201. 202. 203. 204. 205. 206. 207.</td><td> colorwidth = colorFrameDescription.Width; colorHeight = colorFrameDescription.Height; //displayMe if ( displaypixels == null) { _depthData = new ushorr[depthwidth * depthHeight]; _colorData = new byte[colorwidth * colorHeight * BYTES_PER_PIXEL];</td>

<td> 208 .</td><td> _displayPixels = new byte[depthwidth * depthHeight * BYTES_PER_PIXEL];</td>

<td> 209.</td><td> _colorPoints = new ColorSpacePoint[depthwidth * depthHeight];</td>

<td> 210. 211.</td><td> 1 _bitmap = new WriteableBitmap(depthWidth, depthHeight, 96.0, 96.0, PixelFormats.Bgra32, null);</td>

<td> 212. 213. 214. 215. 216. 217. 218. 219.</td><td> //inverses InverseDepthPixels = new byte[depthwidth * depthHeight]; 1/ create the bitmap to display this.depthBitir.ap = new WriteableBitmap(this.depthFrameDescription.Width, this.depthFrameDescription.Height, 96.0, 96.0, PixelFormats.Gray8, null);</td>

<td> 220. 221. 222. 223. 224. 225. 226. 227.</td><td> // open the sensor this.kinectSensor.Open (); } //for each frame I can read - do this private void Reader MultiSourceFrameArrived(object sender, MultiSouroeFrameArrivedEventArgs e)</td>

<td> 228 . 229. 230. 231. 232.</td><td> {</td>

SUBSTITUTE SHEET (RULE 26)

GA 02971260 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 233. 234. 235. 236. 237. 238 .</td><td> bool depthFrameProcessed = false; DepthFrame df = null; ColorFrame cf = null; bool isBitmapLocked = false; MultiSourceFrame multiSourceFrame = e.FrameReference.AcquireFrame();</td>

<td> 239. 240. 241. 242. 243. 244. 245. 246. 247. 248 . 249. 250. 251. 252. 253. 254. 255. 256. 257.</td><td> if (PauseTouch == true) { PauseTouchCounter++ ; if (PauseTouchCounter &gt; 60) PauseTouch = false; ) else { PauseOnLongWaitCounter++; if (PauseOnLongWaitCounter &gt; PauseOnLongWaitMax) { PauseTouch = true; PauseOnLongWaitCounter = 0; } II txtDebug.Text = Entered Reader; H If the Frame has expired by the time we process this event, return.</td>

<td> 258 . 259. 260. 261. 262. 263.</td><td> if (multiSourceFram.e == null) { return; } II We use a try/finally to ensure that we clean up before we exit the function.</td>

<td> 264.</td><td> // This includes calling Dispose on any Frame objects that we may have and unlocking the bitmap back buffer.</td>

<td> 265. 266. 267.</td><td> try { df = multiSourceFrame.DepthFrameReferenoe.AcquireFrame();</td>

<td> 268 .</td><td> cf = multiSourceFrame.ColorFrameReference.AcquireFrame() ;</td>

<td> 269. 270. 271. 272. 273. 274. 275.</td><td> if (FRAME SKIP COUNTER &gt; FRAME SKIP) { FRAME_SKIP_COUNTER = 0; H II If any frame has expired by the time we process this event, return.</td>

<td> 276.</td><td> H /! The finally” statement will Dispose any that are not null.</td>

<td> 277. 278 .</td><td> if ((df == null) || (cf == null)) {</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

279. return;

280. }

281.

282. if (enterTabz == true)

283. {

284. //Inverse Streams

285. // depth stream

286.

287.

288. //displayMe

289. if (((depthwidth * depthHeight) == _depthData.Length) &amp;&amp; ((colorWidth * colorHeight * BYTES_PER_PIXEL) == _colcrData.Length))

290. {

291. df.CopyFrameDataToArray(_depthData);

292.

293. if (cf.RawColorlmageFormat ==

ColorIraageFormat.Bgra)

294.{

295.

cf.CopyRawFrameDataToArray(_colorData);

296.)

297.else

298 .{

299.

cf.CopyConvertedFrameDataToArray(_colorData, ColorIraageFormat.Bgra);

300.}

301.

302.

303.

304.

this .cocrdinateMapper.MapDepthFrameTcColorSpace(_depthData, colorPcints);

305.

306. Array.Clear(_display?ixels, 0, _displayPixels.Length);

307.

308. for (int y = 0; y &lt; depthHeight; ++y)

309. {

310. for (int x = 0; x &lt; depthwidth; ++x)

311. {

312. int depthindex = (y * depthwidth) + x;

313.

314.

315. ColorSpacePoint colorPoint = _colorPcints[depthindex];

316.

317. int colorX = (int) Math.Floor(colorPoint.X + 0.5);

318. int colorY = (int) Ma th.Floor(colorPoint.Y + 0.5);

319.

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

320.

321.

322.

323.

324.

325.

326.

327.

328 .

329.

330.

331.

332.

333.

334.

335.

336.

337.

338 .

339.

340.

341.

342.

343.

344.

345.

346.

347.

348 .

349.

350.

351.

352.

353.

354.

355.

if ((colorX >= 0) && (colorX < colorWidth) && (colorY >= 0) && (colorY < colorHeight)) {

int colorindex = ((colorY * colorwidth) + colorX) * BYTES_PER_PIXEL;

int displayindex = depthindex * BYTES_PER_PIXEL;

_displayPixels[displayindex + 0] = _colorData[colorindex];

_display?ixels[displayindex + 1] = _colorData[colorindex + 1];

_displayPixels[displayindex + 2] = _colorData[colorindex + 2];

_displayPixels[displayindex + 3] = Oxff;

} }

) using (DepthFrane depthFrame = df) { if (depthFrame != null) { // the fastest way to process the body index data is to directly access // the underlying buffer using (Microsoft.Kinect.KinectBuffer depthBuffer = depthFrame.LocklmageBuffer()) {

// verify data and write the color data to the display bitmap if (((this.depthFrameDescription.Width * this.depthFrameDescription.Height) == (depthBuffer.Size / this .depthFrameDescription.BytesPerPixel)) &amp;&amp;

(this.depthFrameDescription.Width == this.depthBitmap.Pixelwidth) &amp;&amp; (this.depthFrameDescription.Height == this.depthBitmap.PixelHeight)) {

// Note: In order to see the full range of depth (including the less reliable far field depth) // we are setting maxDepth to the extreme potential depth threshold ushort maxDepth = ushort.Maxvalue;

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 356.</td><td> II If you wish to filter by reliable depth distance/ uncomment the following line:</td>

<td> 357.</td><td> Illi maxDepth =</td>

<td></td><td> depthFrame.DepthMaxReliableDistance</td>

<td> 358 .</td><td></td>

<td> 359.</td><td></td>

<td></td><td> this .PrccassDepthFrameData(depthBuffer.UnderlyingBuffer.</td>

<td></td><td> depthBuffer.Size, depthFrame.DepthMinReliableDistance, maxDepth);</td>

<td> 360.</td><td> depthFrameProcessed = true;</td>

<td> 361.</td><td> }</td>

<td> 362.</td><td></td>

<td> 363.</td><td></td>

<td> 364.</td><td></td>

<td> 365.</td><td> )</td>

<td> 366.</td><td> }</td>

<td> 367.</td><td> }</td>

<td> 368 .</td><td></td>

<td> 369.</td><td> //Inverse Depth Array to match Picture</td>

<td> 370.</td><td> for (int y = 0; y &lt; depthHeight; ++y)</td>

<td> 371.</td><td> {</td>

<td> 372.</td><td> int counter = depthwidth - 1;</td>

<td> 373.</td><td> for (ir.t x = 0; x &lt; depthwidth; x++)</td>

<td> 374.</td><td> {</td>

<td> 375.</td><td> int depthindex = (y * depthwidth; + x;</td>

<td> 376.</td><td> int reverseindex = (y * depthwidth) +</td>

<td></td><td> counter;</td>

<td> 377.</td><td> InverseDepthPixels[depthindex] =</td>

<td></td><td> depthPixels[reverseindex];</td>

<td> 378 .</td><td> counter—;</td>

<td> 379.</td><td> }</td>

<td> 380.</td><td> }</td>

<td> 381.</td><td> depthPixels = InverseDepthPixels;</td>

<td> 382.</td><td> InverseDepthPixels = new byte[depthwidth *</td>

<td></td><td> depthHeight];</td>

<td> 383 .</td><td></td>

<td> 384.</td><td></td>

385.

<td> 386. 387. 388 .</td><td> // var r = wispDepths; if (processPopup == true) {</td>

<td> 389. 390.</td><td> //Remove the wisp if it exists before adding it back, (so when you move around, it doesn't ahve 1000 copies just one)</td>

<td> 391.</td><td> wispDepths.RemoveAll(c =&gt; c.wispID == popupRecieve_WispName);</td>

<td> 392.</td><td> WispViewTabList.RemoveAll(c =&gt; c.wispID == popupRecieve WispName);</td>

<td> 393. 394.</td><td> //add all points from the clicked wisp to the list.</td>

<td> 395. 396. 397.</td><td> foreach (Point pt in recievePop_tempPoints) {</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 398 .</td><td> int pixellndex = (int)(pt.X + ((int)pt.Y * this.depthBitmap. Pixelwidth)) ;</td>

<td> 399. 400. 401. 402. 403. 404. 405. 406. 407. 408 . 409.</td><td> int depth = 0; if (depthpixels[pixellndex] &gt; 0) depth = depthpixels[pixellndex]; else depth = 0; WispInteractiveData wid = new WispInteractiveData();</td>

<td> 410. 411. 412. 413. 414. 415. 416. 417. 418. 419.</td><td> wid.wispID = popupRecieve_WispName; wid.wispindex = pixellndex; wid.wispPixDepth = depth; wispDepths.Add(wid); } //Remove any Dups //wispDepths = wispDepths.Distinct ().ToList();</td>

<td> 420. 421. 422. 423.</td><td> //setup this wisp in the wisp list WispCoveredFrameltem itd = new WispCoveredFrameltem() ;</td>

<td> 424. 425. 426. 427. 428 .</td><td> itd.wispID = popupRecieve_WispName; itd.frameCountNum = 0; itd.IsCurrentlylnClickState = false; itd.currentPixCount = 0; itd.totalpixels = wispDepths.Where(x =&gt; x.wispID == popupRecieve_WispName).Count ();</td>

<td> 429. 430. 431. 432. 433. 434. 435. 436. 437. 438 . 439. 440. 441. 442. 443. 444. 445. 446.</td><td> wispviewTaoList.Add(itd); //clear //var t = this.bitmap.Pixelwidth; var j = Pause; popupRecieve_WispName = ; recievePop_tempPoints.Clear (); processPopup = false; //Build XML Values and save to XML files SaveWispsToXML() ; SaveWispDepthsToXML();</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 447.</td><td> }</td>

<td> 448 .</td><td> if (enterViewerMode == true)</td>

<td> 449.</td><td> {</td>

<td> 450.</td><td> //use WispDepths list to loop throught all</td>

<td></td><td> coordinates and</td>

<td> 451.</td><td> //ex: if 25% of the depth of a wisp is</td>

<td></td><td> greater than 50mm - trigger a click.</td>

<td> 452.</td><td> //you may have to add a touchcount (like</td>

<td></td><td> greenscreens on the spaceescreens) tc make sure a consistant 25% is covered over three or four frames.</td>

<td> 453.</td><td> I ! use WispCoveredFrameCountList for</td>

<td></td><td> external touchcounts.</td>

<td> 454.</td><td> //finally, you might have to</td>

<td> 455.</td><td></td>

<td> 456.</td><td> //FRAME_NUM_REQUIRED_FOR_CLICK = 5;</td>

<td> 457.</td><td> //WISP_PERCENT_CCVERAGE_FOR_CLICK = 25; //</td>

<td></td><td> percentage</td>

<td> 458 .</td><td> //WispViewTabList should be filled with</td>

<td></td><td> names, but no data.</td>

<td> 459.</td><td></td>

<td> 460.</td><td> WispViewTabList.Select(c =&gt; {</td>

<td></td><td> c.currentPixCount = 0; return c; }).ToList();</td>

<td> 461.</td><td></td>

<td> 462.</td><td></td>

<td> 463.</td><td> //string wispDepthListOutput =</td>

<td> 464.</td><td></td>

<td> 465.</td><td> foreach (var di in wispDepths)</td>

<td> 466.</td><td> {</td>

<td> 467.</td><td></td>

<td> 468 .</td><td></td>

<td> 469.</td><td> //throw out the 0's</td>

<td> 470.</td><td> if ((di.wispPixDepth &gt; 0) &amp;&amp;</td>

<td></td><td> (di.wispindex &gt; 0))</td>

<td> 471.</td><td> {</td>

<td> 472.</td><td></td>

<td> 473.</td><td> int depth =</td>

<td></td><td> depthpixels[di.wispindex];</td>

<td> 474.</td><td> int BaseLineDepth = di.wispPixDepth;</td>

<td> 475.</td><td></td>

<td> 476.</td><td> if (depth &gt; 1 &amp;&amp; BaseLineDepth &gt; 1)</td>

<td> 477.</td><td> {</td>

<td> 478 .</td><td> var pl =</td>

<td></td><td> depthpixels[di.wispindex];</td>

<td> 479.</td><td> if ((BaseLineDepth - depth &gt;</td>

<td></td><td> MIN_DEPTH_CLICK) &amp;&amp; (BaseLineDepth - depth &lt; MAX_DEPTH_CLICK])</td>

<td> 480.</td><td> {</td>

<td> 481.</td><td> //count this one point as a</td>

<td></td><td> click .</td>

<td> 482.</td><td> WispViewTabList.First(w =&gt;</td>

<td></td><td> w.wispID == di.wispID),currentPixCount++;</td>

<td> 483 .</td><td></td>

484.

// var tt = 0;

}

485.

486.

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749 PC17US2015/068192

487.

488 .

489. )

<td> 490.</td><td> // wispDepthListOutput += di.wispindex.ToString() + ” + di.wispPixDepth.ToString() + \r\n;</td>

<td> 491. 492. 493.</td><td> } //all touch as been calculated - time to render a click at the right time.</td>

<td> 494. 495. 496.</td><td> foreach (var wtli in WispViewTabList} { if (wtli.currentPixCount &gt; (wtli.tctalPixels * WISP_PERCENT_COVERAGE_FOR_CLICK))</td>

<td> 497. 498 . 499. 500. 501. 502. 503. 504. 505. 506.</td><td> { wtli.frameCountNum++; ) else { wtli.frameCountNum—; ) //click OR reset if (wtli.frameCountNum &gt;= FRAME_NCM_REQUIRED_FOR_CLICK)</td>

<td> 507. 508 . 509. 510.</td><td> { //VIRTUAL Click uc3.VirtualWispClick(wtli.wispID.ToString()) ;</td>

<td> 511. 512. 513. 514. 515. 516. 517. 518. 519. 520. 521. 522. 523.</td><td> wtli.frameCountNum = 0; wtli.currentPixCount = 0; PauseTouchCounter = 0; PauseTouch = true; PaussOnLongWaitCounter = C; //} } else if (wtli.frameCountNum &lt;= 0} { //CLICK RESET wtli.frameCountNum = 0 ; wtli.isCurrentlylnClickState = false;</td>

<td> 524. 525. 526. 527. 528 . 529. 530. 531. 532. 533.</td><td> } } }</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

534.

535.

<td> 536.</td><td></td><td> if (depthFrameProcessed &amp;&amp; enterTab2 =</td><td> :=</td><td> true)</td>

<td> 537. 538 . 539. 540. 541. 542. 543. 544. 545. 546. 547. 548 . 549. 550. 551. 552. 553. 554. 555. 556. 557. 558 . 559. 560. 561. 562. 563. 564. 565. 566. 567. 568 . 569. 570. 571. 572. 573. 574. 575. 576.</td><td> BYTES_PER_ Oxff; Oxff; Oxff; Oxff;</td><td colspan="3"> { //debug //foreach (var item in wispDepths) //{ Π depthpixels[item.wispindex] = 1; H //displayMe H int displaylndex = item.wispIndex * PIXEL; // _displayPixels[displayindex + 0] = H _displayPixels[displayindex + 1] = II _displayPixels[displayindex + 2] = II _displayPixels[displayindex + 3] = //} this.RenderDepthPixels(); } ) // END FRAME_SKIP else { FRAME_SKIP_COUNTER++; if (df != null) { df.Dispose (); } if (cf != null) { cf.Dispose (); } } H II Process Depth</td>

<td> 577. 578 . 579. 580. 581.</td><td> depthFrame</td><td colspan="2"> II FrameDescription dFrameDescription .FrameDescripticn; } catch (Exception ex) { //nothing</td><td></td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 582. 583. 584. 585. 586. 587. 588 . 589. 590. 591. 592. 593. 594. 595. 596. 597. 598 . 599. 600. 601. 602. 603. 604. 605. 606. 607. 608 . 609. 610. 611. 612. 613. 614. 615. 616. 617. 618. 619. 620. 621. 622.</td><td> Debug.WriteLine(ex); if (df != null) { df.Dispose (); } if (cf != null) { cf.Dispose (); } } finally { if (isBitmapLocked) { Π this.bitnap.Unlock(); } if (df != null) { df.Dispose (); } if (cf != null) { cf.Dispose (); } } )// end if TouchPause } private void RenderDepthPixels() { //write image to screen - if it's called this.depthBitmap.WritePixels( new Int32Rect(C, 0, this.depthBitmap.Pixelwidth this.depthBitmap.PixelHeight),</td>

<td> 623. 624. 625. 626. 627. 628 . 629. 630.</td><td> depthpixels, this.depthBitmap.Pixelwidth, 0); //displayMe bitmap.Lock (); Marshal.Copy(_displayPixels, 0, _bitmap.BackBuffer, _displayPixels.Length);</td>

<td> 631.</td><td> _bitmap.AddDirtyRect(new Int32Rect(0, 0, depthwidth depthHeight)) ;</td>

<td> 632.</td><td> _bitmap.Unlock ();</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

633.

634.

635.

636. if (_popup != null)

637.{

638. _popup.startimage.Source = this.depthBitmap;

639. _popup.colorImage.Source = this._bitmap;

640.}

641.}

642.

643.

644.

645.

646.

647.

648 .

649.

650.

651.

652.

653.

654.

655.

656.

657.

658 .

659.

660.

661.

662.

663.

/// <summary>

III Directly accesses the underlying image buffer of the DepthFrame to

III create a displayable bitmap.

Ill This function requires the /unsafe compiler option as we make use of direct

III access to the native memory pointed to by the depthFrameData pointer.

Ill </summary>

III <param name=ⁿdepthFrameData>Pointer to the DepthFrame image data</param>

III <param name=depthFrameDataSize”>Size of the DepthFrame image data</param>

/// <param name=minDepth>The minimum reliable depth value for the frame</param>

III <param name=maxDepth>The maximum reliable depth value for the frame</param>

private unsafe void ProcessDepthFrameData(IntPtr depthFrameData, uint depthFrameDataSize, ushort minDepth, ushort maxDepth) {

II depth frame data is a 16 bit value ushort* frameData = (ushort*)depthFrameData;

II convert depth to a visual representation for (int i = 0; i &lt; (int)(depthFrameDataSize I this.depthFrameDescription.BytesPerPixel); ++i) {

II Get the depth for this pixel ushort depth = frameData[i];

664. 11 To convert to a byte, we're mapping the depth value tc the byte range.

665. /! Values outside the reliable depth range are mapped to 0 (black).

666. //depthpixels[i] = (byte)(depth >= minDepth && depth <= maxDepth ? (depth / MapDepthToByte) : 0);

667. //this.depthPixels[i] = (byte)(depth);

668 .

669. depthPixels[i] = (byte)(depth >= minDepth depth <= maxDepth ? depth : 0);

670.

671. }

672.

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PC17US2015/068192

673.

<td> 674. 675. 676. 677. 678 . 679. 680. 681. 682. 683.</td><td> } //listens to key press on keyboard. - for testing, private void OnWindowKeyL'p (object sender, KeyEventArgs e) { switch (e.Key) { case Key.Fl: var ucs = LogicalTreeHelper.FindLogicalNode(mainWindow, uc + (tabControl.Selectedlndex + 1).ToString()) as NonAffineTransformControl;</td>

<td> 684. 685. 686. 687. 688 .</td><td> ucs.VirtualWispClick(aaa-lll-bbb); break; //toggle fullscreen case Key.Fil: var uc = LogicalTreeHelper.FindLogicalNode(mainWindow, uc + (tabControl.Selectedlndex + 1).ToString()) as NonAffineTransformControl;</td>

<td> 689.</td><td> var tab = LogicalTreeHelper.FindLogicalNode(mainWindow, tabGrid + (tabControl.Selectedlndex + 1).ToString()) as Grid;</td>

<td> 690. 691. 692. 693.</td><td> if (uc != null &amp;&amp; tab ! = null) { if (mainWindow.Windowstate == System.Windows.Windowstate.Maximized)</td>

<td> 694. 695.</td><td> { mainWindow.Windowstate = System.Windows.Windowstate.Normal;</td>

<td> 696.</td><td> mainWindow.Windowstyle = System.Windows.Windowstyle.SingleBorderWindcw;</td>

<td> 697. 698 . 699. 700.</td><td> rootGrid.Children.Remove(uc); tab. Children.Add(uc); tabControl.Visibility = Visibility.Visible;</td>

<td> 701. 702. 703. 704. 705.</td><td> } else { mainWindow.Windowstate = System.Windows.Windowstate.Maximized;</td>

<td> 706.</td><td> mainWindow.Windowstyle = System.Windows.WindowStyle.None;</td>

<td> 707. 708 . 709. 710.</td><td> tab.ChiIdren.Remove(uc); rootGrid.Children.Add(uc); tabControl.Visibility = Visibility.Hidden;</td>

711.

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

712. }

713. }

14. break;

715. case Key.F2: //TEMP functional to reset XML to test setup

<td> 716.</td><td colspan="2"> Wisp[] wisps = new Wisp[]</td><td> {</td>

<td> 717.</td><td></td><td> new Wisp ()</td><td></td>

<td> 718 .</td><td></td><td> (</td><td></td>

<td> 719.</td><td></td><td colspan="2"> WispID = aaa-lll-bbb,</td>

<td> 720.</td><td></td><td> Name</td><td> = ccc,</td>

<td> 721.</td><td></td><td colspan="2"> WispPolygon = new</td>

<td></td><td> WispPolygon()</td><td></td><td></td>

<td> 722.</td><td></td><td> {</td><td></td>

<td> 723.</td><td></td><td></td><td> Content = gif.gif,</td>

<td> 724.</td><td> gif.gif,</td><td></td><td> ClickContent =</td>

<td> 725.</td><td> Pointcollection(new</td><td> Point[] {</td><td> CornerPoints = new</td>

<td> 726.</td><td> //Coordinates of a</td><td> top-left corner</td><td> new Point(0, 0),</td>

<td> 727.</td><td> 100), //Coordinates</td><td> of a bottom-left corner</td><td> new Point(0,</td>

<td> 728 .</td><td></td><td></td><td> new Point(100,</td>

<td></td><td colspan="2"> 0), //Coordinates of a top-right corner</td><td></td>

<td> 729.</td><td> 100) //Coordinates</td><td> of a bottom-right corner</td><td> new Point(100,</td>

<td> 730.</td><td></td><td></td><td> }),</td>

<td> 731.</td><td> ContentType.Gif,</td><td></td><td> ContentType =</td>

<td> 732.</td><td></td><td></td><td> FrameRate = 0,</td>

733.

Loop =

LoopType. Forever,

734.

735.

736.

Contentpolygon ()

737.

738 .

ZOrder = 1 },

Contentpolygon = r.ew { Content =

PetzoldTattoo.jpg, 739.

CornerPoints = new

Pointcollection(new Point[] {

740. new Point(100,

0), //Coordinates of a top-left corner

741. new Point(100,

100), //Coordinates of a bottom-left corner

742. new Point(200,

0), //Coordinates of a top-right corner

743. new Point(200,

100) //Coordinates of a bottom-right corner

744. }),

745.

ContentType.Image, 746.

747.

ContentType =

FrameRate = 0,

Loop = (LoopType)2,

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

748 .

749.

0,

750.

751.

752.

753.

754.

755.

756.

757.

WispPolygon()

758 .

759.

760.

761.

ZOrder = 0, ContentOrientation =

Timeout = 3, } }, new Wisp () (

WispID = aaa-lll-bbb,

Name = ccc, WispPolygon = new {

Content = vidl.wmv,

ClickContent = null, CornerPoints = new

762.

763.

764.

765.

766.

767.

768 .

769.

770.

771.

Pointcollection(new Point [] {

100), //Coordinates of a “op-left corner

200), //Coordinates of a bottom-left corner

100), //Coordinates of a “op-right corner

200) //Coordinates of a bottom-right corner

ContentType.Video, }, new Point(0, new Point(0, new Point(100, new Point(100,

)),

ContentType =

FrameRare = 0,

Loop = (LoopType)2, ZOrder = 4

772. Contentpolygon = r.ew

ContentFolygon()

773. {

774. Content =

Pet20ldTatt00.jpg,

775. CornerPoints = new

<td></td><td> Pointcollection(new Point[] {</td><td></td><td></td><td></td>

<td> 776.</td><td> 0), //Coordinates of a top-left corner</td><td></td><td> new</td><td> Point(100,</td>

<td> 777.</td><td> 100), //Coordinates of a bottom-left corner</td><td></td><td> new</td><td> Point (100,</td>

<td> 778 .</td><td> 0), //Coordinates of a top-right corner</td><td></td><td> new</td><td> Point(200,</td>

<td> 779.</td><td> 100) //Coordinates of a bottom-right corner</td><td></td><td> new</td><td> Point(200,</td>

<td> 780.</td><td></td><td> }),</td><td></td><td></td>

<td> 781.</td><td></td><td colspan="3"> ContentType =</td>

ContentType.Image, 782.

783.

784.

FrameRaoe = 0, Loop = (LoopType)2, ZOrder = 0,

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

785

786.

787.

788 .

789.

790.

791.

792.

793.

WispPolygon()

794.

795.

wisp_png_sequence, 796.

797.

Contentorientation =

Timeout = 3, } }, new Wisp () {

WispID = aaa-222-bbb, Name = ccccxzczxxcz, WispPolygon = new { Content =

ClickContent = null, CornerPoints = new

Pointcollection(new Point[] { 798 .

200), //Coordinates of a top-left corner 799.

300), //Coordinates of a bottom-left corner 800 .

200), //Coordinates of a top-right corner 801.

300) //Coordinates of a bottom-right corner 802.

803 .

ContentType.PngSeq,

804.

805.

806.

807. }, new Point(0, new Point(0, new Point(100, new Point(100,

)),

ContentType =

FrameRare = 60, Loop = (LoopType)2, ZOrder = 2

808 .

Contentpolygon = r.ew

ContentFolygon()

809. {

810. Content =

PetzoldTattoo.jpg,

811. CornerPoints = new

Pointcollection(new Point[] {

812. new Point(100,

0), //Coordinates of a top-left corner

813. new Point(100,

100), //Coordinates of a bottom-left corner

814. new Point(200,

0), //Coordinates of a top-right corner

815. new Point (200,

100) //Coordinates of a bottom-right corner

816. }),

817. ContentType =

ContentType.Image, 818.

819.

820.

FrameRare = 0, Loop = (LoopType)2, ZOrder = 0,

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

821.

0,

822.

823 .

824.

825.

826.

827.

828 .

829.

WispPolygon()

830.

831.

Contentorientation =

Timeout = 3, } }, new Wisp () {

WispID = aaa-lll-bbb, Name = cccwqqweqw, WispPolygon = new { Content = wisp_png_sequence2, 832.

833.

ClickContent = null, CornerPcints = new

Pointcollection(new Point [] {

834. new Point(0,

300), //Coordinates of a top-left corner

835. new Point(0,

400), //Coordinates of a bottom-left corner

836. new Point(100,

300), //Coordinates of a top-right corner

837. new Point(100,

400) //Coordinates of a bottom-right corner

838. }),

839. ContentType =

ContentType.Text,

840.

841.

842.

843.

844.

ContentFolygon()

845.

846.

FrameRate = 60, Loop = (LoopType)2, ZOrder = 5 }, Contentpolygon = new {

Content =

Pet20ldTatt00.jpg,

847. CornerPcints = new

Pointcollection(new Point[] {

848. new Point(100,

0), //Coordinates of a top-left corner

849. new Point(100,

100), //Coordinates of a bottom-left corner

850. new Point(200,

0), //Coordinates of a top-right corner

851. new Point(200,

100) //Coordinates of a bottom-right corner

052. }),

853.

ContentType.Image,

854.

855.

856.

ContentType =

FrameRate = 0,

Loop = (LoopType)2, ZOrder = 0,

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PC17US2015/068192

Contentorientation = 0,

Timeout = 3, } }, new Wisp () {

WispID = aaa-lll-bbb, Name = cccasdas, WispPolygon = new

WispPolygon() { Content =

PetzoldTattoo.jpg,

ClickContent = rotate.png, CornerPoints = new Pointcollection(new Point[] { new Point(0, 400), //Coordinates of a top-left corner new Point(0, 500), //Coordinates of a bottom-left corner new Point(100, 400), //Coordinates of a top-right corner new Point(100, 500) //Coordinates of a bottom-right corner

}) ,

ContentType = ContentType.Image,

FrameRate = 60,

Loop = (LoopType)2, ZOrder = 0 },

Contentpolygon = new Contentpolygon () { Content =

PetzoldTattoo.jpg,

CornerPoints = new Pointcollection(new Point[] { new Point(100, 0), //Coordinates of a top-left corner new Point(100, 100), //Coordinates of a bottom-left corner new Point(200, 0), //Coordinates of a top-right corner new Point(200, 100) //Coordinates of a bottom-right corner )), ContentType =

ContentType.Image,

FrameRate = 0,

Loop = (LoopType)2, ZOrder = 0,

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 893.</td><td> Contentorientation = n</td>

<td> 894. 895. 896. 897. 898 . 899. 900. 901. 902.</td><td> U / Timeout = 3, } } }; //ucl.Wisps = wisps; using (var fs = File.Create(test.xml)) { XmlSerializer xs = new XmlSerializer(typeof(Wisp[]), new Type[] {</td>

<td> 903. 904. 905. 906. 907. 908 . 909. 910. 911. 912. 913. 914. 915. 916.</td><td> typeof(Wisp), typeof(ContentPolygon), typeof(WispPolygon), typeof(Polygon), typeof(ContentType), typeof(LoopType) }); xs.Serialize (fs, wisps); } break; ) } private void OnViewerMouseUp(object sender, MouseButtonEventArgs e)</td>

<td> 917. 918. 919. 920. 921. 922.</td><td> { enterViewerMode = true; enterTab2 = false; } private void OnSetupMouseUp(object sender, MouseButtonEventArgs e)</td>

<td> 923. 924. 925. 926. 927.</td><td> { enterViewerMode = false; enterTab2 = false; } private void OnTab2MouseUp(object sender, MouseButtonEventArgs e)</td>

<td> 928 . 929. 930. 931. 932. 933. 934. 935.</td><td> { enterTab2 = true; enterViewerMode = false; //if popup is closed if ( popup != null) { popup. startimage . Source = new Bitmapimage(new Uri(@pack://siteoforigin:,,,/Spaceeblk. jpg)) ;</td>

<td> 936. 937. 938 . 939. 940.</td><td> ) if ( popup == null) { //creating a popup _popup = new PopupWmdow() ;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 941. 942. 943. 944. 945.</td><td> _popup.Closed += OnPopupClosed; //finding out screen configuration var allScreens = System.Windows. Forms. Screen.AllScreens;</td>

<td> 946. 947.</td><td> //by default destination screen is the first Svstem.Windows.Forms.Screen destScreen = allScreens[0];</td>

<td> 948 . 949. 950. 951. 952.</td><td> //if there are more than one screen if (allScreens.Length &gt; 1) { //finding out current screen var currentscreen = System. Windows. Forms. Screen.FromPoint(new System.Drawing.Point ((int)Left, (int)Top)) ;</td>

<td> 953.</td><td> var screenindex = Array.IndexOf(allScreens, currentscreen);</td>

<td> 954. 955.</td><td> //destination screen is next destScreen = allScreens[(screenindex + 1) % allScreens.Length];</td>

<td> 956. 957. 958 .</td><td> } //placing popup in the middle of destination screen _popup.Left = destScreen.WorkingArea.Left + (destScreen.WorkingArea.Width - _popup.Width) / 2;</td>

<td> 959.</td><td> _popup.Top = destScreen.WorkingArea.Top + (destScreen.WorkingArea.Height - _popup.Height) / 2;</td>

<td> 960. 961. 962. 963. 964. 965. 966. 967. 968 . 969. 970. 971. 972. 973. 974.</td><td> //copy wisps to popup _popup.NonAffineTransformControl.Wisps = uc2.Wisps; //showing popup popup.Shew (); ) else //else focusing it { _popup.Focus (); ) } //! &lt;summary&gt; /// Handles popup closed event. Removes all handlers and links .</td>

<td> 975. 976. 977. 978 . 979. 980. 981. 982. 983. 984. 985.</td><td> /// &lt;/summary&gt; /// &lt;param name=sender&gt;&lt;/param&gt; /// &lt;param name=ⁿeX/param&gt; private void OnPopupClosed(object sender, EventArcs e) { popup.Closed -= OnPopupClosed; popup = null; } /// &lt;summary&gt; //1 Handles popup NonAffineTrans formControl loaded event.</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 986. 987. 988 . 989. 990. 991. 992. 993. 994. 995.</td><td> Ill &lt;/summary&gt; III &lt;param name=sender^B&gt;&lt;/param&gt; III &lt;param name=”e&gt;&lt;/parar&gt; private void OnMainWindowLoaded(object sender, RoutedEventArgs e)</td>

<td> 996. 997. 998 . 999. 1000.</td><td> { UpdateCornerPoints(;; } private void OnMainWindowLocationChanged(object sender, EventArgs e)</td>

<td> 1001. 1002. 1003. 1004. 1005.</td><td> { UpdateCornerPoints(); } private void OnMainWindowSizeChanged(object sender, SizeChangedEventArgs e)</td>

<td> 1006. 1007. 1008. 1009. 1010. 1011. 1012. 1013.</td><td> { UodateCornerPoints (); } private void UpdateCornerPoints!) { _topLeft = PointToScreen (new PointO); _bottomRight = new Point(_topLeft.X + mainWindow.Actualwidth, _topLeft.Y + nainWindow.ActualHeight);</td>

<td> 1014. 1015. 1016. 1017. 1018. 1019. 1020. 1021. 1022. 1023.</td><td> } private Popupwindcw _popup; private Point _topLeft; private Point _bottomRight; //loads the system in viewer mode and in full screen. - If there is a system crash, the monoter system will detect it and relaunch the program in this state.</td>

<td> 1024. 1025. 1026. 1027. 1028. 1029. 1030.</td><td> public void startAppinFullView() { //wait until the program loads int ct = 0; DispatcherTimer switchTabTimer = new DispatcherTimer(); switchTabTimer.Interval = TimeSpan.FromSeconds(1); switchTabTimer.Tick += (object timersender, EventArgs timerE) =&gt;</td>

<td> 1031. 1032.</td><td> { ct++ ;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 1033. 1034. 1035. 1036. 1037. 1038. 1039. 1040. 1041.</td><td> //switchTabTimer.Stop (); if (ct &gt; 0) { //viewer tab tabControl.Selectedlndex = 2; swirchTabTimer.Stop(); switchTabTimer = null; If System.Windows.MessageBox.Show(PREP); var uc = LogicalTreeHelper.FindLogicalNode(mainWindow, uc + (tabControl.Selectedlndex + 1).ToString()) as NonAffineTransformControl;</td>

<td> 1042.</td><td> var tab = LogicalTreeHelper.FindLogicalNode(mainWindow, tabGrid + (tabControl.Selectedlndex + 1).ToString()) as Grid;</td>

<td> 1043. 1044. 1045. 1046.</td><td> if (uc != null &amp;&amp; tab != null) { mainWindow.Windowstate = System.Windows.Windowstate.Maximized;</td>

<td> 1047.</td><td> mainWindow.Windowstyle = System.Windows.Windowstyle.None;</td>

1048.

<td> 1049. 1050. 1051. 1052. 1053. 1054. 1055.</td><td> tab.Children.Remove(uc); rcotGrid.Children.Add(uc); tabControl.Visibility = Visibility.Hidden; enterViewerMode = true; enterTabz = false; //System.Windows.MessageBox.Show(IT WORKS);</td>

<td> 1056. 1057. 1058. 1059. 1060. 1061. 1062. 1063. 1064. 1065. 1066. 1067. 1068. 1069. 1070. 1071.</td><td> //CAUSE ERROR TO TEST APPLICATION CRASH // int t = 0 / (ct - 1); } } ); switchTabTimer. Start () ; } //DEPTHS XML private void SaveWispsToXML() { using (var fs = System.IO.File.Create(_WispCoveredFrameItem_XMLFile))</td>

<td> 1072. 1073.</td><td> { XmlSerializer xs = new XmlSerializer(typeof(List&lt;WispCoveredFrameItem&gt;), new Type[] {</td>

<td> 1074. 1075. 1076.</td><td> typeof(List&lt;WispCoveredFrameItem&gt;) }); xs.Serialize(fs, WispViewTabList);</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

1077.

1078. fs.Close ();

1079. }

<td> 1080. 1081. 1082. 1083.</td><td> private void LoadKispsFromXML() { if ( ! string.IsNullOrEmpty(_WispCoveredFrameItem_XMLFile) &amp;&amp; System.IO.File.Exists ( WispCoveredFrameltem XMLFile))</td>

<td> 1084. 1085. 1086. 1087.</td><td> { try { using (var fs = System.IO.File.OpenRead( WispCoveredFrameltem XMLFile))</td>

<td> 1088. 1089.</td><td> { XmlSerializer xs = new XmlSerializer (typeof(List&lt;WispCoveredFrameItem&gt;), new Type [] {</td>

<td> 1090. 1091. 1092.</td><td> typeof(WispCoveredFrameltem) }); WispViewTabList = (List&lt;WisoCoveredFrameItem&gt;)xs.Deserialize(fs);</td>

<td> 1093. 1094. 1095. 1096. 1097. 1098. 1099. 1100. 1101.</td><td> } } catch { } ) } private void SaveKispDepthsToXML() { using (var fs = System.IO.File.Create ( WispDepths XMLFile))</td>

<td> 1102. 1103.</td><td> { XmlSerializer xs = new XmlSerializer(typeof(List&lt;WispInteractiveData&gt;), new Type[] {</td>

<td> 1104. 1105. 1106. 1107. 1108. 1109. 1110. 1111. 1112. 1113. 1114. 1115.</td><td> typeof(List&lt;WispInteractiveData&gt;) )); xs.Serialize(fs, wispDepths); fs.Close (); ) } private void LoadWispDepthsFromXML() { if ( ! string.IsNullOrEmpty(_WispDepths_XMLFile) &amp;&amp; System.IO.File.Exists ( WispDepths XMLFile))</td>

<td> 1116. 1117. 1118. 1119.</td><td> { try { using (var fs = System.IO.File.OpenRead( WispDepths XMLFile))</td>

<td> 1120. 1121.</td><td> { XmlSerializer xs = new XmlSerializer(typeof(List&lt;WispInteractiveData&gt;), new Type[] {</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

1122. typeof(WispInteractiveData)

1123.});

1124. wispDepths = (List<WispInteractiveData>)xs.Deserialize(fs);

1125. ’}

1126.}

1127. catch { }

1128.}

1129.}

1130.

1131.

1132.}

1133.

1134.

1135. }

1136.

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 1.</td><td> using System;</td>

<td> 2.</td><td> using System.Collections.Generic;</td>

<td> 3.</td><td> using System.Linq;</td>

<td> 4.</td><td> using System.Text;</td>

<td> 5.</td><td> using System.Threading.Tasks;</td>

<td> 6.</td><td> using System.Windows;</td>

<td> 7 .</td><td> using System.Windows.Controls;</td>

<td> 8 .</td><td> using System.Windows.Data;</td>

<td> 9.</td><td> using System.Windows.Documents;</td>

<td> 10.</td><td> using System.Windows.Input;</td>

<td> 11.</td><td> using System.Windows.Media;</td>

<td> 12.</td><td> using System.Windows.Media.Imaging;</td>

<td> 13.</td><td> using System.Windows.Media.Media3D;</td>

<td> 14.</td><td> using System. Windows. Navigation;</td>

<td> 15.</td><td> using System.Windows.Shapes;</td>

<td> 16.</td><td> using System.Windows.Threading;</td>

<td> 17. 18.</td><td> using System.Xml.Serialization;</td>

<td> 19.</td><td> namespace NonAffineTransform</td>

<td> 20.</td><td> {</td>

<td> 21.</td><td> /// &lt;summary&gt;</td>

<td> 22.</td><td> /// Interaction logic for NonAffineTransformControl.xaml</td>

<td> 23.</td><td> /// &lt;/summary&gt;</td>

<td> 24.</td><td> public partial class NonAffineTransformControl : Usercontrol</td>

<td> 25.</td><td> {</td>

<td> 26.</td><td> private class RenderedPolygon</td>

<td> 27.</td><td> {</td>

<td> 28.</td><td> III &lt;summary&gt;</td>

<td> 29.</td><td> III Rendering planes (meshes)</td>

<td> 30.</td><td> III &lt;/summary&gt;</td>

<td> 31. 32.</td><td> public MeshGecmetry3D Mesh;</td>

<td> 33.</td><td> III &lt;summary&gt;</td>

<td> 34.</td><td> III Transforming matrices</td>

<td> 35.</td><td> III &lt;/summary&gt;</td>

<td> 36. 37.</td><td> public MatrixTransform3D MatrixTransform;</td>

<td> 38.</td><td> III &lt;summary&gt;</td>

<td> 39.</td><td> III Corner points of meshes</td>

<td> 40.</td><td> III &lt;/summary&gt;</td>

<td> 41. 42.</td><td> public ?oint3D[] PointsTransformed;</td>

<td> 43.</td><td> /// &lt;summary&gt;</td>

<td> 44.</td><td> III Collection of WISP outlines</td>

<td> 45.</td><td> III &lt;/summary&gt;</td>

<td> 46. 47.</td><td> public PathFigure Outline;</td>

<td> 48.</td><td> III &lt;summary&gt;</td>

<td> 49.</td><td> III Collection of WISP outline paths</td>

<td> 50.</td><td> III &lt;/summary&gt;</td>

<td> 51. 52.</td><td> public Path OutlinePath;</td>

<td> 53.</td><td> III &lt;summary&gt;</td>

<td> 54.</td><td> III Connected control</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 55. 56.</td><td> Ill &lt;/summary&gt; public FrameworkElement Control;</td>

<td> 57. 58. 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74.</td><td> III &lt;summary&gt; III Collection of WISP materials III &lt;/summary&gt; public DiffuseMaterial Material; III &lt;summary&gt; III Polygon that is being rendered III &lt;/summary&gt; public Polygon Parent; } private class RenderedDefaultPolygon : RenderedPolygon { III &lt;summary&gt; III ID of default polygon III &lt;/summary&gt; public string ID;</td>

75.

<td> 76. 77. 78. 79. 80. 81. 82. 83. 84. 85. 86. 87.</td><td> Ill &lt;summary&gt; III How many times it's requested to hide /// &lt;/summary&gt; public int HideCount; } private class RenderedWisp { III &lt;summary&gt; III Rendered Wisp polygon III &lt;/summary&gt; public RenderedPolygon WispPolygon;</td>

<td> 88. 89. 90. 91. 92. 93. 94. 95. 96. 97. 98.</td><td> III &lt;summary&gt; III Rendered content polygon III &lt;/summary&gt; public RenderedPolygon ContentPclygon; III &lt;summary&gt; III Wisp that is being rendered III &lt;/summary&gt; public Wisp Parent;</td>

<td> 99. 100. 101. 102. 103. 104. 105.</td><td> III &lt;summary&gt; III Determines whether content polygons is shown III &lt;/summary&gt; public bool ContentPolygcnShown; III &lt;summary&gt; III Backup of content polygon tc restore after making unwanted changes</td>

<td> 106. 107.</td><td> III &lt;/summary&gt; public Contentpolygon ContentPolygonBackup;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td colspan="2"> 108.</td>

<td> 109.</td><td> Ill &lt;summary&gt;</td>

<td> 110.</td><td> III Timer which hides content polygon on timeout</td>

<td> 111.</td><td> III &lt;/summary&gt;</td>

<td> 112. 113.</td><td> public DispatcherTimer HideContentPolygonTimer;</td>

<td> 114.</td><td> Ill &lt;summary&gt;</td>

<td> 115.</td><td> III Media player which plays click sounds</td>

<td> 116.</td><td> III &lt;/summary&gt;</td>

<td> 117.</td><td> public MediaPlayer MediaPlayer;</td>

<td> 118. 119.</td><td> }</td>

<td> 120.</td><td> III &lt;summary&gt;</td>

<td> 121.</td><td> III Shows priority of mesh dragging over vertex dragging.</td>

<td> 122.</td><td> Ill Set to 1 to disable vertex dragging.</td>

<td> 123.</td><td> Ill Set to 0 to disable mesh dragging.</td>

<td> 124.</td><td> Ill Balanced values are G.6-0.7 .</td>

<td> 125.</td><td> III &lt;/summary&gt;</td>

<td> 126. 127.</td><td> private const double MeshDraggingWeight = 0.65;</td>

<td> 128.</td><td> III &lt;suncnary&gt;</td>

<td> 129.</td><td> III Shows how far next rendering plane would be placed</td>

<td> 130.</td><td> /// &lt;/summary&gt;</td>

<td> 131. 132.</td><td> private const double ZIncrenent = -10.0;</td>

<td> 133.</td><td> private static readonly Point3D[] DefaultCornerPoints = new Point3D[] (</td>

<td> 134.</td><td> new Point3D(-0.5, 0.5, 0),</td>

<td> 135.</td><td> new Point3D(-0.5, -0.5, 0),</td>

<td> 136.</td><td> new Point3D(0.5, 0.5, 0),</td>

<td> 137.</td><td> new Point3D(0.5, -0.5, 0)</td>

<td> 138. 139.</td><td> } ;</td>

<td> 140.</td><td> public static readonly Dependencyproperty WispsProperty;</td>

<td> 141.</td><td> public static readonly Dependencyproperty DefaultPolygonProperty;</td>

<td> 142.</td><td> public static readonly Dependencyproperty StorageFileProperty;</td>

<td> 143.</td><td> public static readonly Dependencyproperty ShowContentProperty;</td>

<td> 144.</td><td> public static readonly DependencyPrcperty ShowOutlinesProperty;</td>

<td> 145.</td><td> public static readonly Dependencyproperty ReadOnlyProperty;</td>

<td> 146. 147.</td><td> public static readonly Dependencyproperty PlaySour.dProperty</td>

<td> 148.</td><td> static NonAffineTransformControl()</td>

<td> 149.</td><td> {</td>

<td> 150.</td><td> WispsProperty = DependencyProperty.Register(Wisps, typeof(Wisp[]), typeof(NonAffineTransformControl)) ;</td>

<td> 151.</td><td> DefaultPolygonProperty = Dependencyproperty. Register (Defaultpolygons, typeof(Defaultpolygon[]), typeof(NonAffineTransformControl)) ;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 152.</td><td> StorageFilePrcperty =</td>

<td></td><td> Dependencyproperty.Register(Storage, typeof(string), typeof(NonAffineTrans formControl)) ;</td>

<td> 153.</td><td> ShowContentProperty = Dependencyproperty.Register(ShowContent, typeof(bool), typeof(NonAffineTransformControl), new PropertyMetadata(true;) ;</td>

<td> 154.</td><td> ShowOutlinesProperty = Dependencyproperty.Register(Showoutlines, typeof(bool), typeof(NonAffineTransformControl), new PropertyMetadata(true)) ;</td>

<td> 155.</td><td> ReadOnlyProperty = Dependencyproperty.Register(Readonly, typeof(bool), typeof(NonAffineTransformControl), new PropertyMetadata(false)) ;</td>

<td> 156.</td><td> PlaySoundProperty = DependencyProperty.Register(PlaySound, typeof(bool), typeof(NonAffineTransformControl), new PropertyMetadata(false)) ;</td>

<td> 157. 158. 159. 160. 161. 162. 163. 164. 165. 166. 167.</td><td> } public NonAffineTransformControl() { InitializeComponent(); Loaded += OnNonAffineTransformCcntrolLoaded; SizeChanged += OnNonAffineTransformControlSizeChanged; } private void OnHideContentPolygonTimerTick(object sender, EventArgs e)</td>

<td> 168. 169. 170. 171. 172. 173. 174. 175. 176. 177. 178. 179. 180. 181.</td><td> { var fe = (sender as DispatcherTimer); if (fe == null) return; var rw = fe.Tag as RenderedWisp; if (rw == null) return; HideContentPolygon(rw); } /// &lt;summary&gt; //I Gets or sets collection of polygons which is used to display images/videos/etc.</td>

<td> 182. 183. 184. 185. 186. 187. 188. 189. 190. 191. 192. 193.</td><td> /// &lt;/summary&gt; public Wisp[] Wisps { get { return (Wisp[])GetValue(WispsProperty); } set { SetValue(WispsProperty, value); if (! internalchange &amp;&amp; loaded) { Initialize (); UpdateCornerPointPositions(); }</td>

SUBSTITUTE SHEET (RULE 26)

GA 02971200 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 194.</td><td> )</td>

<td> 195. 196. 197. 198.</td><td> } III &lt;summary&gt; III Gets or sets collection of polygons which is used to display images/videos/etc.</td>

<td> 199. 200. 201. 202.</td><td> III &lt;/summary&gt; public Defaultpolygon[] Defaultpolygons { get { return (DefaultPolygon[])GetValue(DefaultPolygonProperty); }</td>

<td> 203. 204. 205. 206. 207. 208 . 209. 210. 211. 212. 213. 214. 215.</td><td> set { SetValue(DefaultPolygonProperty, value); //if (! internalchange &amp;&amp; loaded) //{ II Initialized; II UpdateCornerPointPositions() ; //} ) } /// &lt;summary&gt; /// Gets or sets path to xml file where control autosaves and loads Wisps</td>

<td> 216. 217. 218. 219. 220. 221. 222. 223. 224. 225. 226. 227.</td><td> /// &lt;/summary&gt; public string StorageFiie { get { return (string)GetValue(StorageFileProperty); } set { SetValue(StorageFileProperty, value); ) } III &lt;summary&gt; III Gets or sets value indication whether content of wisp polygons is shown.</td>

<td> 228 . 229. 230. 231. 232. 233. 234. 235.</td><td> /// &lt;/summary&gt; public bool ShowContent { get { return (bool)GetValue(ShowContentProperty); } set { SetValue(ShowContentProperty, value); viewpor~3d.Visibility = value ? Visibility.Visible : Visibility.Hidden;</td>

<td> 236. 237. 238 . 239. 240.</td><td> ) } III &lt;summary&gt; III Gets or sets value indicating whether outlines and controls of content polygons are shown.</td>

<td> 241.</td><td> Ill &lt;/summary&gt;</td>

SUBSTITUTE SHEET (RULE 26)

GA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 242.</td><td> public bool ShowOutlines</td>

<td> 243.</td><td> {</td>

<td> 244.</td><td> get { return (bool)GetValue(ShowOutlinesProperty); }</td>

<td> 245.</td><td> set</td>

<td> 246.</td><td> {</td>

<td> 247.</td><td> SetValue(ShowOutlinesProperty, value);</td>

<td> 248 .</td><td> overlay.Visibility = value ? Visibility.Visible :</td>

<td></td><td> Visibility.Hidden;</td>

<td> 249.</td><td> )</td>

<td> 250.</td><td> }</td>

<td> 251.</td><td></td>

<td> 252.</td><td> III &lt;summary&gt;</td>

<td> 253.</td><td> /// Gets or sets value indicating whether user able to move</td>

<td></td><td> polygons.</td>

<td> 254.</td><td> /// &lt;/simmary&gt;</td>

<td> 255.</td><td> public bool Readonly</td>

<td> 256.</td><td> {</td>

<td> 257.</td><td> get { return (bool)GetValue(ReadOnlyProperry); }</td>

<td> 258 .</td><td> set</td>

<td> 259.</td><td> {</td>

<td> 260.</td><td> SetValue(ReadOnlyProperty, value);</td>

<td> 261.</td><td> )</td>

<td> 262.</td><td> }</td>

<td> 263.</td><td></td>

<td> 264.</td><td> /// &lt;summary&gt;</td>

<td> 265.</td><td> /// Gets or sets value indicating whether control should</td>

<td></td><td> play click sounds.</td>

<td> 266.</td><td> Ill &lt;/summary&gt;</td>

<td> 267.</td><td> public bool PlaySound</td>

<td> 268 .</td><td> {</td>

<td> 269.</td><td> get { return (bool)GetValue(PlaySoundProperty); ·</td>

<td> 270.</td><td> set</td>

<td> 271.</td><td> {</td>

<td> 272.</td><td> SetValue(PlaySoundProperty, value);</td>

<td> 273.</td><td> )</td>

<td> 274.</td><td> }</td>

<td> 275.</td><td></td>

<td> 276.</td><td> public void VirtualWispClick(String wispID)</td>

<td> 277.</td><td> {</td>

<td> 278 .</td><td> for (int i = 0; i &lt; renderedWisps.Length; i++)</td>

<td> 279.</td><td> {</td>

<td> 280.</td><td> if (_renderedWisps[l].Parent.WispID == wispID)</td>

<td> 281.</td><td> {</td>

<td> 282.</td><td> ShowContentPolygon(_renderedWisps[i]) ;</td>

<td> 283.</td><td> break;</td>

<td> 284.</td><td> }</td>

<td> 285.</td><td> )</td>

<td> 286.</td><td> }</td>

<td> 287.</td><td></td>

<td> 288 .</td><td> III &lt;summary&gt;</td>

<td> 289.</td><td> III Returns a Wisp at specified location</td>

<td> 290.</td><td> /// &lt;/summary&gt;</td>

<td> 291.</td><td> III &lt;param name=point&gt;&lt;/param&gt;</td>

<td> 292.</td><td> //t &lt;returns&gt;&lt;/rerurns&gt;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-08-15

WO 2016/109749

PCT/US2015/068192

<td> 293. 294. 295. 296. 297. 298 .</td><td> public Wisp GetWispAtPoint(Point point) { var wisps = Wisps; if (wisps == null) return null;</td>

299. var resultMesh = GetMeshAtPoint(point);

300. if (resultMesh == null)

301. return null;

302.

303. var meshindex = Array.IndexOf(_renderedWisps.Select (rw => rw.WisoPolygon.Mesh).ToArrayO, resultMesh.MeshHit);

<td> 304.</td><td> if (meshindex &gt;= 0 S&amp; meshindex &lt; wisps.Length)</td>

<td> 305.</td><td> {</td>

<td> 306.</td><td> var resultwisp = wisps[meshindex];</td>

<td> 307.</td><td></td>

<td> 308 .</td><td> return resultwisp;</td>

<td> 309.</td><td> )</td>

<td> 310.</td><td></td>

<td> 311.</td><td> return null;</td>

<td> 312.</td><td> }</td>

<td> 313.</td><td></td>

<td> 314.</td><td> /// &lt;summary&gt;</td>

<td> 315.</td><td> /// Returns a Wisp at specified location</td>

<td> 316.</td><td> III &lt;/summary&gt;</td>

<td> 317.</td><td> /// &lt;param name=pointx/param&gt;</td>

<td> 318.</td><td> //1 &lt;returns&gt;&lt;/reurns&gt;</td>

<td> 319.</td><td> public Wisp GetWispAtPoint(double x, double y)</td>

<td> 320.</td><td> {</td>

<td> 321.</td><td> return GetWispAtPoint(new Point(x, y)) ;</td>

<td> 322.</td><td> }</td>

<td> 323.</td><td></td>

<td> 324.</td><td> /// &lt;summary&gt;</td>

<td> 325.</td><td> /// Returns a Wisp ID at specified location</td>

<td> 326.</td><td> /// &lt;/summary&gt;</td>

<td> 327.</td><td> /// &lt;param name=ⁿpointx/param&gt;</td>

<td> 328 .</td><td> /// &lt;returnsx/returns&gt;</td>

<td> 329.</td><td> public string GetWispIDAtPoint(Point point)</td>

<td> 330.</td><td> {</td>

<td> 331.</td><td> var wisp = GetWispAtPoint(point) ;</td>

<td> 332.</td><td> return wisp != null ? wisp.WispID : null;</td>

<td> 333.</td><td> }</td>

<td> 334.</td><td></td>

<td> 335.</td><td> III &lt;summary&gt;</td>

<td> 336.</td><td> III Returns a Wisp ID at specified location</td>

<td> 337.</td><td> III &lt;/summary&gt;</td>

<td> 338 .</td><td> III &lt;psram name=ⁿpointX/param&gt;</td>

<td> 339.</td><td> III &lt;returnsX/rerurns&gt;</td>

340. public string GetWispIDAtPoint(double x, double y)

341. {

342. return GetWispIDAtPoint(new Point(x, y)) ;

343. }

344.

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 345.</td><td> protected override void OnMouseLeftButtonDown (MouseButton.EventArgs args)</td>

<td> 346. 347. 348 . 349. 350. 351. 352.</td><td> { Point pt = args.GetPosition(viewport3d); var resultMesh = GetMeshAtPoint(pt); if (resultMesh == null) return;</td>

<td> 353.</td><td> var result = _renderedWisps.FirstOrDefault(rw =&gt; rw.WispFolygon.Mesh == resultMesh.MeshHit);</td>

<td> 354. 355. 356. 357. 358 . 359. 360. 361.</td><td> if (result != null) { _draggingWisp = result; draggingpolygon = result.WispPolygon; ) else { result = _renderedWrsps.FirstOrDefault(rw =&gt; rw.ContentPolygonShown &amp;&amp; rw.Contentpolygon.Mesh == resultMesh.MeshHit);</td>

<td> 362. 363. 364. 365. 366. 367. 368 . 369. 370.</td><td> if (result != null) { _draggingWisp = result; _draggingPolygon = result.Contentpolygon; _isDraggingContent = true; ) else { var defaultpolygon = _renderedDefaultPolygons.FirstOrDefault(rdp =&gt; rdp.Mesh == resultMesh.MeshHit);</td>

<td> 371. 372. 373. 374. 375. 376. 377. 378 . 379. 380. 381. 382.</td><td> if (defaultpolygon != null) { draggingPolygon = defaultpolygon; } else { return; } } ) if (’Readonly &amp;&amp; _renderedWisps.Any(rw =&gt; rw.ContentPolygonShown &amp;&amp; rw.Contentpolygon != _draggingPolygon &amp;&amp; rw.WispPolygon != draggingPolygon))</td>

<td> 383. 384. 385. 386. 387. 388 . 389. 390.</td><td> { draggingWisp = null; draggingPolygon = null; _isDragging = false; _isDraggingContent = false; return; )</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 391.</td><td> var diff12 = resultMesh.VertexWeightl resultMesh.VertexWeight2;</td>

<td> 392.</td><td> var diff13 = resultMesh.VertexWeightl resultMesh.VertexWeight3;</td>

<td> 393.</td><td> var diff23 = resultMesh.VertexWeight2 resultMesh.VertexWeight3;</td>

<td> 394. 395. 396.</td><td> /! Determine if the mouse within central part of mesh, if (Math.Abs(diff12) &lt; MeshDraggingWeight &amp;&amp; Math.Abs(diff13) &lt; MeshDraggingWeight &amp;&amp; Math.Abs(diff23) &lt; MeshDraggingWeight)</td>

<td> 397. 398 . 399. 400. 401. 402. 403.</td><td> { draggingVertexIr.dex = -1; ) Π Determine which vertex the mouse is closest to. else if (diffl2 &gt; G) { _draggingVertexIr.dex = diffl3 &gt; 0 ? resultMesh.Vertexlndexl : resultMesh.Vertexlndex3;</td>

<td> 404. 405. 406. 407.</td><td> ) else { -draggingVertexIr.dex = diff23 &gt; 0 ? resultMesh.Vertexlndex2 : resultMesh.Vertexlndex3;</td>

<td> 408 . 409. 410. 411. 412. 413. 414. 415. 416.</td><td> ) -StartDraggingMousePoint = pt; if (!Readonly) { // Get current transformed points, for (int i = 0; i &lt; 4; i++) { draggingPolygon.PointsTransformed[i] = _draggingPolygon.MatrixTransform.Matrix.Transform(_draggir.gPolygon.M esh.Positions[i]) ;</td>

<td> 417. 418. 419. 420.</td><td> } if (_draggingvertex.ndex &gt;= 0) { //—draggingPolygon.PointsTransformed[_draggingVertexIndex] = Simple2Dto3D(viewport3d, pt, -draggingPolygon.PointsTransformed[_draggingVertexIndex]. Z);</td>

<td> 421.</td><td> //-draggingPolygon.MatrixTransform.Matrix = CalculateNonAffineTransform( draggingPolygon.PointsTransformed);</td>

<td> 422. 423. 424. 425. 426.</td><td> } else ^{ //saving start coordinates for mesh dragging startDraggir.gPoint = Simple2Dto3D (viewport3d, pt) ;</td>

<td> 427. 428 .</td><td> _startDraggir.gMeshPoints = new Point3D[4]; Array.ConstrainedCopy(-draggingPolygon.PointsTransformed, 0, _startDraggingMeshPoints, 0, 4);</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 423. 430. 431. 432. 433. 434. 435. 436. 437. 438 . 439. 440. 441. 442.</td><td> } } //Commence dragging operation. _isDragging = true; CaptureMouse(); //args.Handled = true; } protected override void OnMouseMove(MouseEventArgs args) { base.OnMouseMove(args); //Determine if it is OK to move polygon or corner. if (!Readonly &amp;&amp; _isDragging &amp;&amp; _draggingPolygon != null &amp;&amp; _renceredWisps.All(rw =&gt; !rw.ContentPolygonShown || rw.Contentpolygon == _draggingPolygon))</td>

<td> 443. 444. 445. 446. 447. 448 . 449.</td><td> { Point ptMouse = args.GetPosition(viewport3d); // Vertex dragging if ( draggingVertexIndex &gt;= 0) { _draggingPolygon.PointsTransformed[_draggingVertex!ndex] = Simple2Dto3D(viewport3d, ptMouse, _draggingPolygon.PointsTransformed[_draggingVertexIndex].Z);</td>

<td> 450.</td><td> _draggingPolygon.Parent.CornerPoints[_draggingVertex!ndex] = Simple3Dto2D(viewport3d, draggingPolygon.PointsTransformed[ draggingVertexIndex]) ;</td>

<td> 451. 452. 453. 454. 455.</td><td> } // Mesh dragging else { Point3D curPosition = Simple2Dto3D(viewport3d, ptMouse);</td>

<td> 456.</td><td> vector3D offset = curPosition _s tar tDraggingPoint;</td>

<td> 457. 458 . 459. 460. 461.</td><td> offset.Z = 0; for (int i = 0; i &lt; 4; i++) { —draggingPolygon.PointsTransformed[i] = _startDraggingMeshPoints[i] + offset;</td>

<td> 462.</td><td> _draggingPo_ygon.Parent.CornerPoints[i] = Simple3Dto2D(viewport3d, -draggingPolygon.PointsTransformed[i] ) ;</td>

<td> 463. 464. 465. 466. 467.</td><td> } } //updating outline _draggingPolygon.Parent.UpdateOutline(-draggingPolygon.Outline);</td>

468 .

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 463.</td><td> Canvas.SetLeft(_draggingPolygon.Control, (_draggingPolygon.Parent.CornerPoints[1].X + draggingpolygon.Parent.CornerPoints[3].X _draggingPolygon.Control.Actualwidth) * 0.5);</td>

<td> 470.</td><td> Canvas. Seriop(_draggingPolygon. Control, Math.Max(_draggingPolygon. Parent.CornerPoints[1]. Y, _draggingPolygon.Parent.CornerPoints[3]. Y) ) ;</td>

<td> 471. 472. 473.</td><td> //updating transformation matrix. _draggingPolygon.MatrixTransform.Matrix = CalculateNonAffineTransform(_draggingPolygon.PointsTransformed);</td>

<td> 474. 475. 476. 477. 478 .</td><td> //args.Handled = true; } } protected override void OnMouseLeftButtonUp(MouseButtonEventArgs args)</td>

<td> 473. 480. 481. 482. 483. 484. 485. 486.</td><td> { base.OnMouseUp(args); if ( isDragging) { Point mousePoint = args.GetPosition(viewport3d); //if polygon was not moved and user clicked on a right one, showing content polygon</td>

<td> 487.</td><td> if (ShowContent S&amp; _draggingWisp != null &amp;&amp; _dragging«isp.WispPolygon == _draggingPolygcn &amp;&amp;</td>

<td> 488 .</td><td> Math.Àbs(mousePoin.X _startDraggingMousePoint.X) &lt; 2 &amp;&amp; Math.Abs(mousePoint.Y startDraggingMousePoint.Y) &lt; 2)</td>

<td> 489. 490. 491. 492. 493. 494. 495. 496.</td><td> { if (! draggingiίl·sp.ContentPolygonShowr.) { ShowContentPolygon( draggingWisp); } else //or hiding content polygon if it was shown { //reverting conent polygon params, cause Save Content button was not pressed</td>

<td> 497.</td><td> _draggingWisp.ContentPolygon.Parent = _draggingWisp. Parent.ContentPolygon = _draggingWisp.ContentPolygonBackup;</td>

<td> 498 . 499. 500. 501.</td><td> HideContentPolygon( draggingWisp); } } //else it was a drag operation and it's needed to save values to xml.</td>

<td> 502. 503. 504. 505. 506. 507.</td><td> else if (’Readonly &amp;&amp; ! isDraggingContent) { SaveData(); } _isDragging = false;</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 508 . 509. 510. 511. 512. 513. 514.</td><td> _isDraggingConter.t = false; ReleaseMouseCapture(); //args.Handled = true; ) } private RayMeshGeometry3DHitTestResult GetMeshAtPoint(Point point)</td>

<td> 515. 516. 517. 518. 519. 520. 521. 522.</td><td> { RayMeshGeometry3DHitIestResult resultMesh = null; II Obtain the Visual3D objects under the mouse pointer. VisualTreeHelper.HitTest(viewport3d, null, new HitTestResultCallback((hitTestResult) =&gt; { resultMesh = hitTestResult as RayMeshGeometry3DHitTestResult;</td>

<td> 523. 524.</td><td> //which is not an invisible content polygon or something else</td>

<td> 525.</td><td> if (resultMesh == null || (resultMesh .VisualHit as ModelVisual3D) == null ||</td>

<td> 526.</td><td> Array.IndexOf(_renderedwisps. Select(rw =&gt; rw.WispFolygon.Mesh).ToArray(), resultMesh.MeshHit) &lt; 0 &amp;&amp;</td>

<td> 527.</td><td> Array.IndexOf(_renderedWisps.Where(rw =&gt; rw.ContentPolygonShown).Select(rw =&gt; rw.Contentpolygon.Mesh).ToArray(), resultMesh.MeshHit) &lt; C &amp;&amp;</td>

<td> 528 .</td><td> Array.IndexOf(_renderedDefaultPolygons. Select(rdp =&gt; rdp.Mesh).ToArray(), resultMesh.MeshHit) &lt; 0)</td>

<td> 529. 530. 531. 532. 533. 534. 535. 536. 537. 538 . 539. 540. 541. 542. 543. 544. 545. 546. 547.</td><td> { resultMesh = null; return HitTestResultBehavior.Continue; } return HitTestResultBehavior.Stop; }), new PointHitTestParameters(point) ); return resultMesh; } private void ShowContentPolygon(RenderedWisp renderedWisp) { if (renderedWisp.ContentPolygonShown) return; var cp = (renderedWisp.CcntentPolygon.Parent as ContentFolygon);</td>

<td> 548 .</td><td> var wp = (renderedWisp.WispPolygon.Parent as WispPolygon);</td>

<td> 549. 550.</td><td> //determining clicking group and click availability</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 551. 552. 553. 554. 555. 556. 557. 558 . 559. 560. 561. 562. 563. 564. 565. 566.</td><td> if ( ! string.IsNullOrEmoty(wp.ClickGroup)) { if ( clickedGroups.Contains(wp.ClickGroup)) { return; } else { _clickedGroups.Add(wp.ClickGroup); } } //playing sound if (PlaySound &amp;&amp; ! string.IsNullOrEmpty(wp.ClickSound)) { renderedWisp.Mediaplayer.Open(new Uri(wp.ClickSound UriKind.Relative)) ;</td>

<td> 567. 568 . 569. 570. 571. 572. 573. 574.</td><td> renderedWisp.Mediaplayer.Volume = 1; renderedWisp.Mediaplayer.Play (); ) //hiding default polygons if (Readonly) { var defaultPolygonlds = renderedWisp.Parent.ShowHideDefaultPolygons;</td>

<td> 575.</td><td> var renderedDefaultPolygons = _renderedDefaultPolygons;</td>

<td> 576. 577. 578 . 579. 580.</td><td> if (defaultPolygonlds != null) { foreach (var id in defaultPolygonlds) { var rdp = renderedDefaultPolygons.FirstOrDefault(p =&gt; p.ID == id);</td>

<td> 581. 582. 583.</td><td> if (rdp != null) { //increasing its hide count (C visible, 1 and more - invisible;</td>

<td> 584.</td><td> rdp.EideCount = Math.Max(1, rdp.HideCount + 1);</td>

<td> 585.</td><td> rdp.Material.Brush = Brushes.Transparent;</td>

<td> 586. 587. 588 .</td><td> if (ShowOutlines) { rdp.OutlinePath.Stroke = Brushes.Transparent;</td>

<td> 583.</td><td> rdp.Control.Visibility = Visibility. Hidden;</td>

<td> 590. 591. 592. 593. 594. 595.</td><td> ) } } } )</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 596. 597. 598 . 599. 600. 601. 602. 603. 604. 605. 606. 607. 608 . 609. 610. 611.</td><td> //backuping content polygon renderedWisp.ContentPolvgonBackup = new Conter.tPolygon () { Content = cp.Content, Contentorientation = cp .ContentOrientatior., ContentType = cp.ContentType, CornerPoints = new Pointcollection(cp.CornerPoints), FrameRate = cp.FrameRate, Loop = cp.Loop, Timeout = cp.Timeout, ZOrder = cp.ZOrder J; //changing wisp polygon content to click content wp.ResetClickBrush(); renderedWisp.WispPolygon.Material. Brush = wp.ClickContentBrush;</td>

<td> 612. 613. 614.</td><td> //updating content polygon position UpdateCornerPointPositions(cp, renderedWisp.ContentPolygon.MatrixTransform);</td>

<td> 615. 616. 617. 618. 619.</td><td> for (int i = 0; i &lt; 4; i++) { //preserve z order renderedWisp.Contentpolygon.Mesh.Positions[i] = new Point3D(renderedWisp.Contentpolygon.Mesh.Positions[i].X, renderedWisp.Contentpolygon.Mesh.Positions[i]. Y, ZIncremer.t);</td>

<td> 620. 621. 622.</td><td> ) //Changing it's visibility (material, outline and controls)</td>

<td> 623.</td><td> renderedWisp.Contentpolygon.Material.Brush = cp.ContentBrush;</td>

<td> 624. 625. 626.</td><td> if (ShowOutlines) { cp.UpdateOutline(renderedWisp.ContentPolygon.Outline);</td>

<td> 627.</td><td> renderedWisp.ContentPolygon.OutlinePath. Stroke = (SolidCclorBrush) (new Brushconverter ().ConvertFrom(#00aff0)) ;</td>

<td> 628 . 629. 630.</td><td> var panel = renderedWisp.Contentpolygon.Control; panel.Visibility = Visibility.Visible; Canvas.SetLeft(panel, (cp.CornerPoints[1].X + cp.CornerPoints[3].X - panel.ActualWidzh) * 0.5);</td>

<td> 631.</td><td> Canvas.SetTop(panel, Math.Max(cp.CornerPoints[1].Y, cp.CornerPoints[3].Y)) ;</td>

<td> 632. 633. 634. 635.</td><td> ) else if (cp.Timeout &gt; 0) { renderedWisp.HideContentPolygonTimer.Interval = TimeSpan.FromSeconds(cp.Timeout);</td>

<td> 636. 637. 638 .</td><td> renderedWisp.HideContentPolygonTimer.Start (); )</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 633. 640. 641. 642. 643. 644. 645. 646. 647. 648 .</td><td> renderedWisp.ContentPolygonShown = true; } private void HideContentPolygon(RenderedWisp renderedWisp) { renderedWisp.HideContentPolygonTimer.Stop (); if (renderedWisp.ContentPolygonShown) { var wp = (renderedWisp.WispPolygon.Parent as WispPolygon);</td>

<td> 649. 650. 651. 652. 653. 654. 655. 656.</td><td> //determining clicking group and relesing it if ((string.IsNullOrEmpty(wp.ClickGroup)) { _clickedGroups.Remove(wp.ClickGroup); } //showing default polygons which were not hide by other polygons</td>

<td> 657. 658 . 659.</td><td> if (Readonly) { var defaultPolygonlds = renderedWisp. Parent.ShowHideDefaultPolygons;</td>

<td> 660.</td><td> var renderedDefaultPolygons = _renderedDefaultPolygons;</td>

<td> 661. 662. 663. 664. 665.</td><td> if (defaultPolygonlds ! = null) { foreach (var id in defaultPolygonlds) { var rdp = renderedDefaultPolygons.FirstOrDefault(p =&gt; p.ID == id);</td>

<td> 666. 667. 668 .</td><td> if (rdp != null) { //decreasing its hide count (C visible, 1 and more - invisible)</td>

<td> 669. 670. 671. 672. 673.</td><td> rdp.Hidecount—; if (rdp.HideCount &lt;= 0) { rdp.HideCount = 0; rdp.Material.Brush = rdp. Parent.ContentBrush;</td>

<td> 674. 675. 676.</td><td> if (ShowOutlines) { rdp.OutlinePath.Stroke = (SolidCclorBrush) (new Brushconverter ().ConvertFrom(”#00aff0)) ;</td>

<td> 677.</td><td> rdp.Control.Visibility = Visibility.Visible;</td>

<td> 678 . 679. 680. 681. 682. 683.</td><td> } ) } } } }</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 684. 685. 686. 687.</td><td> //changing wisp polygon content to regular content wp.ResetBrush(); renderedWisp.WispPolygon.Material.Brush = wp.ContentBrush;</td>

<td> 688 . 689.</td><td> //updating content polygon position to behind camera position</td>

<td> 690. 691. 692. 693.</td><td> renderedWisp.Contentpolygon. Parent.ResetBrush() ; for (int i = 0; i &lt; 4; i++) { renderedWisp.ContentPolygon.Mesh.Positions[i] = new Point3D(renderedWisp.ContentPolygon.Mesh.Positions[i].X, renderedWisp.ContentPolygon.Mesh.Positions[i]. Y, 100);</td>

<td> 694. 695. 696. 697.</td><td> } //hiding it (material, outline and control) renderedWisp.ContentPolygon.Material. Brush = Brushes.Transparent;</td>

<td> 698 . 699. 700.</td><td> if (ShowOutlines) { renderedWisp.ContentPolygon.OutlinePath. Stroke = Brushes.Transparent;</td>

<td> 701.</td><td> renderedWisp.ContentPolygon.Control.Visibility = Visibility.Hidden;</td>

<td> 702.</td><td> )</td>

<td> 703. 704. 705. 706. 707. 708 .</td><td> renderedWisp.ContentPolygonShown = false; ) } private void OnNonAffinelransformControlSizeChanged(object sender, SizeChangedEventArgs e)</td>

<td> 709. 710. 711. 712. 713.</td><td> { if (e.PreviousSize.Width &gt; 0) { var oldwidth = mainCamera.Width; mainCamera.Width = mainCamera.Width * e.NewSize.Width / e.PreviousSize.Width;</td>

<td> 714. 715.</td><td> mainCamera.Position = new Point3D( mainCamera.Position.X + 0.5 * (mainCamera.Width - oldwidth), // - newPosition.X + oldPosition.X,</td>

<td> 716.</td><td> mainCamera.Position.Y + 0.5 * (e.PreviousSize.Height - e.NewSize.Height) * mainCamera.Width / viewport3d.Actualwidth,</td>

<td> 717. 718. 719. 720.</td><td> mainCamera. Position.Z ); ) }</td>

<td> 721. 722.</td><td> private void OnNonAffineTransformControlLoaded(object sender, RoutedEventArgs e)</td>

<td> 723.</td><td> {</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 724.</td><td> overlay.Visibility = ShowOutlines ? Visibility.Visible : Visibility.Hidden;</td>

<td> 725. 726. 727. 728 . 729. 730. 731. 732. 733. 734. 735. 736. 737. 738 . 739. 740. 741. 742.</td><td> if (Visibility == System.Windows.Visibility.Visible) { LoadData() ; Initialize (); ) } private void Initialize () { viewport3d.Children.Clear (); overlay.Children.Clear (); var wisps = Wisps; var defaultpolygons = Defaultpolygons; if (wisps == null) return;</td>

<td> 743. 744. 745.</td><td> var modelVisual3D = new ModelVisual3D(); var model3DGroup = new Medel3DGroup(); //for each provided brush one plane is created, each next plane is a little further (first on is the most visible)</td>

<td> 746. 747. 748 . 749. 750.</td><td> var models = new List&lt;GeometryModel3D&gt;(); var wispsCount = wisps.Length; var count = defaultpolygons != null ? defaultFolygons.Length : 0;</td>

<td> 751. 752. 753.</td><td> #region Default polygons initialization renderedDefaultPolygons = new RenderedDefaultPolygon[count];</td>

<td> 754. 755. 756. 757.</td><td> for (int i = 0; i &lt; count; i++) { double z = i &lt; count ? ((defaultpolygons[i].ZOrder &gt; 0 ? defaultpolygons[i].ZOrder : i) + wispsCount + 1) * Zlncrement :</td>

<td> 758 . 759.</td><td> ZIncrement; //double z = (zOrder ! = null &amp;&amp; zOrder.Length &gt; i) ? zOrder[i] * Zlncrement : i * ZIncrenent;</td>

<td> 760. 761.</td><td> _renderedDefaultpolygons[i] = new RenderedDefaultPolygon()</td>

<td> 762. 763. 764. 765. 766. 767. 768 . 769. 770.</td><td> { Parent = defaultFolygons[i], PointsTransformed = new Point3D[4], ID = defaultpolygons[i].ID }; var pointsTransformed = new Point3D[] { new Point3D(0, 0, z), new Poinr3D(0, 1, z),</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192 new Pornt3D(l, 0, z) , new Point3D(l, 1, z) );

_renderedDefaultPolygons[i].Mesh = new MeshGeoir.etry3D () { //initial petitions of planes Positions = new

Point3DCollection(pointsTransformed), TextureCoordinates = new Pointcollection(new

Point [] { new Point(0, 0), new Point(0, 1), new Point(1, 0), new Point(1, 1) }),

Trianglelndices = new Int32Collection(new Int32[] {

0, 2, 1, 2, 3, 1 }) };

_renderedDefaultpolygons[i].MatrixTransform = new MatrixTransform3D();

if (ShowContenti { //creating brush _renderedDefaultPolygons[i].Material = new DiffuseMaterial(defaultpolygons[i]. ContentBrush);

} else { renderedDefaultPolygons[i].Material = new DiffuseMaterial(Brushes.Transparent);

} if (Showoutlines) { //creating outline _renderedDefaultPolygons[i].Outline = defaultpolygons[i].CreateOutline();

//adding to outlines collections overlay.Children.Add(new Path() {

Stroke = (SolidColorBrush)(new Brushconverter ().ConvertFrom(”#0Caff0)),

StrokeThickness = 2, Data = new PathGeometry() {

Figures = new PathFigureCollection(new PathFigure[] { _renderedDefaultPolygons[i].Outline

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

814. })

<td> 815.</td><td> }</td>

<td> 816. 817. 818. 819. 820. 821. 822.</td><td> }); var name = new TextBlock() { Text = defaultpolygons[i].ID, TextAligrment = TextAlignment.Center, Foreground = (SolidColorBrush)(new Brushconverter ().ConvertFrom(#00aff0)),</td>

<td> 823. 824. 825. 826. 827. 828 .</td><td> }; _renderedDefaultPolygons[i].Control = name; overlay.Children.Add(name); var polygon = defaultpolygons[i]; name.Loaded += new RoutedEventHandler((s, args)</td>

<td> 829. 830 .</td><td> { Canvas.SetLeft(name, (polygon.CornerPoints[1].X + polygon.Cornerpoints[3].X name.Actualwidth) * 0.5);</td>

<td> 831.</td><td> Canvas.SetTop(name, Math.Max(polygon.CornerPoints[1]. Y, polygon.CornerPoints[3]. Y));</td>

<td> 832. 833. 834. 835. 836. 837. 838 .</td><td> } ) ; } models.Add ( new GeometryModel3D() { Geometry = _renderedDefaultPolygor.s [i] .Mesh, Material = _renderedDefaultPolygons[i].Material,</td>

<td> 839.</td><td> Transform = renderedDefaultPolygons[i].MatrixTransform</td>

<td> 840 . 841. 842. 843. 844. 845. 846. 847. 848 . 849. 850.</td><td> } ); ) #endregion #region Wisps initialization _renderedWisps = new RenderedWisp[wispsCount]; for (int i = 0; i &lt; wispsCount; i++) { double z = i &lt; wispsCount ? (((wisps[i].WispPolygon.ZOrder &gt; 0) ? wisps[i].WispPolygon.ZOrder : i; + 1) * ZIncrement :</td>

<td> 851. 852.</td><td> Zlncrement; //double z = (zOrder != null &amp;&amp; zOrder.Length &gt; i) ? zOrder[i] * ZIncrement : i * ZIncrement;</td>

<td> 853. 854. 855. 856. 857. 858 .</td><td> var timer = new DispatcherTimer(); timer.Tick += OnKideContentPolygonTimerTick; _renderedwisps[i] = new RenderedWisp() {</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

HideContentPolygonTimer = timer, MediaPlayer = new MediaPlayer(), Parent = wisps[i], WispPolygon = new RenderedPolygon!) {

Parent = wisps[i].WispPolygon, PointsTransformed = new Point3D[4] } r

ConnentPolygon = new RenderedPolygon() {

Parent = wisps[i].ContentPolygon, PointsTransformed = new Point3D[4] } timer.Tag = _renderedWisps[i];

var pointsTransformed = new Point3D[] { new Point3D(0, 0, z) , new Point3D(0, 1, z) , new Point3D(l, 0, z) , new Point3D(l, 1, z) };

_renderedWisps[i].WispPolygon.Mesh = new MeshGeometry3D() {

//initial positions of planes Positions = new

Point3DCollection(pointsTransformed),

TextureCoordinates = new Pointcollection(new Point [] { new Point(0, 0), new Point(0, 1), new Point (1, C), new Point(1, 1) }),

Trianglelndices = new Int32Collection(new Int32[] {

0, 2, 1, 2, 3, 1 }) };

_renderedWisps[i].WispPolygon.MatrixTransform = new MatrixTransform3D();

if (ShowCcntent) { //creating brush renderedWisps[i].WispPolygon.Material = new DiffuseMaterial(wisps[i].WispPolygon.ContentBrush) ;

} else { _renderedwisps[i].WispPolygon.Material = new DiffuseMaterial(Brushes.Transparent);

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749 PC17US2015/068192

<td> 906. 907. 908 . 909. 910. 911.</td><td> } if (ShowOutlines) { //creating outline _renderedWisps[i].WispPolygon.Outline = wisps[i].WispPolygon.CreateOutline() ;</td>

<td> 912. 913. 914. 915. 916.</td><td> //adding to outlines collections overlay.Children.Add(new Path() { Stroke = (SolidColorBrush)(new Brushconverter ().ConvertFrom(#0Caff0)),</td>

<td> 917. 918. 919. 920.</td><td> StrokeThickness = 2, Data = new PathGeometry() { Figures = new PathFigureCollection(new PathFigure[] {</td>

<td> 921.</td><td> _renderedWisps[i].WispPolygon.Outline</td>

<td> 922. 923. 924. 925. 926. 927. 928 . 929. 930.</td><td> }) } }); var name = new TextBlock() { Text = wisps[i].Name, TextAlignment = TextAlignment.Center, Foreground = (SolidColorBrush)(new Brushconverter ().ConvertFrom(#0Caff0)),</td>

<td> 931. 932. 933. 934. 935. 936.</td><td> }; _renderedWisps[l].WispPolygon.Control = name; overlay.Children.Add(name); var polygon = wisps[i].WispPolygon; name.Loaded += new RoutedEventHandler((s, args)</td>

<td> 937. 938 .</td><td> { Canvas.SetLeft(name, (polygon.CornerPoints[1].X + polygon.Cornerpoints[3].X name.Actualwidth) * 0.5);</td>

<td> 939.</td><td> Canvas.SetTop(name, Math.Max(oolygon.CornerPoints[1]. Y, polygon.CornerPoints[3]. Y)) ;</td>

<td> 940. 941. 942. 943. 944. 945.</td><td> } ) ; } models.Add ( new GeometryModel3D() { Geometry = _renderedWisps[i].WispPolygon.Mesh,</td>

<td> 946.</td><td> Material = _renderedwisps[i].WispPolygon.Material,</td>

<td> 947.</td><td> Transform = _renderedWisps[i].WispPolygon.MatrixTransform</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 948 .</td><td> }</td>

<td> 949. 950. 951.</td><td> ); _renderedWisps[i].Contentpolygon.Mesh = new MeshGeorr.etry3D ()</td>

<td> 952. 953. 954.</td><td> { //initial petitions of planes Positions = new Point3DCollection(pointsTransformed),</td>

<td> 955.</td><td> TextureCoordinates = new Pointcollection(new Point [] {</td>

<td> 956. 957. 958 . 959. 960. 961.</td><td> new Point(0, 0), new Point(0, 1), new Point(1, 0), new Point(1, 1) }), Trianglelndices = new Int32Collection(new Int32[] {</td>

<td> 962. 963. 964. 965.</td><td> 0, 2, 1, 2, 3, 1 }) }; _renderedWisps[i].Contentpolygon.MatrixTransform = new MatrixTransform3D();</td>

<td> 966. 967.</td><td> _renderedWisps[i].Contentpolygon.Material = new DiffuseMaterial(Brushes.Transparent);</td>

<td> 968 . 969. 970. 971.</td><td> if (Showoutlines) { _renderedWisps[i].Contentpolygon.Outline = Polygon.CreateEmptyOutline();</td>

<td> 972.</td><td> _renderedWisps[i].Contentpolygon.OutlinePath = new Path ()</td>

<td> 973. 974. 975. 976. 977. 978 .</td><td> { Stroke = Brushes.Transparent, StrokeThickness = 2, Data = new PathGeometry() { Figures = new PathFigureCollection(new PathFigure[] {</td>

<td> 979.</td><td> _renderedWisps[i].ContentPolvgon.Outline</td>

<td> 980. 981. 982. 983.</td><td> }) } }; overlay.Children.Add(_renderedWisps;i].ContentPolygon.OutlinePath);</td>

<td> 984. 985. 986. 987. 988 . 989. 990.</td><td> //container for controls var panel = new StackPanelO { Visibility = Visibility.Hidden, Orientation = Orientation.Horizontal };</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td colspan="3"> 991.</td>

<td> 992.</td><td></td><td> //Save Content button</td>

<td> 993.</td><td></td><td> var button = new Button ()</td>

<td> 994.</td><td></td><td> {</td>

<td> 995.</td><td></td><td> Content = Save Content,</td>

<td> 996.</td><td></td><td> Margin = new Thickness(10)</td>

<td> 997.</td><td></td><td> };</td>

<td> 998 .</td><td></td><td> button.Click += OnSaveContentButtonClick;</td>

<td> 999.</td><td></td><td> button.Tag = _renderedWisps[i];</td>

<td> 1000.</td><td></td><td> panel.Children.Add(button);</td>

<td> 1001.</td><td></td><td></td>

<td> 1002.</td><td></td><td> //Rotate image</td>

<td> 1003.</td><td></td><td> var image = new Image ()</td>

<td> 1004.</td><td></td><td> {</td>

<td> 1005.</td><td></td><td> Source = new Bitmapimage(new</td>

<td></td><td> Uri (/r’</td><td> state.png, UriKind.Relative)),</td>

<td> 1006.</td><td></td><td> Margin = new Thickness(0, 10, 0, 10),</td>

<td> 1007.</td><td></td><td> Width = 20,</td>

<td> 1008.</td><td></td><td> Height = 20</td>

<td> 1009.</td><td></td><td> };</td>

<td> 1010.</td><td></td><td> image.MouseUp += OnRotateContentButtor.Click</td>

<td> 1011.</td><td></td><td> image.Tag = _renderedWisps[i];</td>

<td> 1012.</td><td></td><td> //nice antializrngs</td>

<td> 1013.</td><td></td><td> Renderoptions.SetBitmapScalingMode(image,</td>

<td></td><td> BitmapSi</td><td> calingMode. Fant);</td>

<td> 1014.</td><td></td><td> panel.Children.Add(image);</td>

<td> 1015.</td><td></td><td> overlay.Children.Add(panel);</td>

<td> 1016.</td><td></td><td> _renderedWisps[r].Contentpolygon.Control =</td>

<td></td><td> panel;</td><td></td>

<td> 1017.</td><td></td><td> }</td>

<td> 1018.</td><td></td><td></td>

<td> 1019.</td><td></td><td> models.Add (</td>

<td> 1020.</td><td></td><td> new GeometryModel3D()</td>

<td> 1021.</td><td></td><td> {</td>

<td> 1022.</td><td></td><td> Geometry =</td>

<td></td><td> _renderi</td><td> edWisps[i].ContentPolygon.Mesh,</td>

<td> 1023.</td><td></td><td> Material =</td>

<td></td><td colspan="2"> _renderedWisps[i].ContentPolygon.Material,</td>

<td> 1024.</td><td></td><td> Transform =</td>

<td></td><td colspan="2"> _renderedWisps[i].ContentPolygon.MatrixTransform</td>

<td> 1025.</td><td></td><td> }</td>

<td> 1026.</td><td></td><td> );</td>

<td> 1027.</td><td></td><td> )</td>

<td> 1028.</td><td></td><td> #endregion</td>

<td> 1029.</td><td></td><td></td>

<td> 1030.</td><td></td><td> _loaded = true;</td>

<td> 1031.</td><td></td><td></td>

<td> 1032.</td><td></td><td> UpdateCornerPointPositions();</td>

<td> 1033.</td><td></td><td> //storing models in correct z-order</td>

<td> 1034.</td><td></td><td> models.Sort(new Comparison&lt;GeometryModel3D&gt;((a, b)</td>

<td></td><td colspan="2"> a.Geometry.Bounds.Z.CompareTo(b.Geometry.Bounds.Z))) ;</td>

<td> 1035.</td><td></td><td> models.ForEach(o =&gt; mode!3DGroup.Children.Add(o); ;</td>

<td> 1036.</td><td></td><td> model3DGroup.Children.Add(new</td>

AmbientLight(Colors.White)) ;

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-08-15

WO 2016/109749 PCT/US2015/068192

<td> 1037. 1038. 1039.</td><td> modelVisual3D.Content = mode!3DGroup; this.viewport3d.Children.Add(modelVisual3D); }</td>

<td> 1040. 1041.</td><td> private void OnSaveContentButtonClick(object sender, RoutedEventArgs e)</td>

<td> 1042. 1043. 1044. 1045. 1046. 1047. 1048. 1049.</td><td> { var fe = (sender as FraraeworkElement); if (fe == null) return; var rw = fe.Tag as RenderedWisp; if (rw == null) return;</td>

<td> 1050. 1051. 1052. 1053. 1054. 1055. 1056. 1057. 1058. 1059. 1060.</td><td> SaveData() ; HideContentPolygon(rw); } l/l &lt;summary&gt; III Loads data from provided xml file. Ill &lt;/summary&gt; private void LoadData() { if ( ! string.IsNullOrEmpty(StorageFile) &amp;&amp; System.IO.File.Exists(StorageFile))</td>

<td> 1061. 1062. 1063. 1064.</td><td> { try { using (var fs = System. IO.File.OpenRead(StorageFile))</td>

<td> 1065. 1066.</td><td> { Xir.lSerializer xs = new XmlSerializer(typeof(Objectstorage), new Type[] {</td>

<td> 1067. 1068. 1069. 1070. 1071. 1072. 1073. 1074. 1075.</td><td> typeof(Wisp[]), typeof(Wisp), typeof(Contentpolygon), typeof (WispPolygor.), typeof(Polygon), typeof(Defaultpolygon[]), typeof(DefaultPolygon) }); var objectstorage = (Objectstorage)xs.Deserialize(fs);</td>

<td> 1076. 1077. 1078.</td><td> _internalChange = true; Wisps = objectstorage.Wisps; Defaultpolygons = objectstorage. DefaultPolygons ;</td>

<td> 1079. 1080. 1081. 1082. 1083. 1084.</td><td> internalchange = false; } } catch { } ) }</td>

SUBSTITUTE SHEET (RULE 26)

CA 02971260 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 1085. 1086.</td><td> ///</td><td> &lt;summary&gt;</td>

<td> 1087.</td><td> m</td><td> Saves data to provided xml file.</td>

<td> 1088.</td><td> III</td><td> &lt;/summary&gt;</td>

<td> 1083.</td><td colspan="2"> private void SaveData()</td>

<td> 1090. 1091. 1092.</td><td> {</td><td> if ( ! string.IsNullOrEmoty(StorageFile) &amp;&amp; Wisps != null) {</td>

<td> 1093.</td><td></td><td> using (var fs = System.IO.File.Create(StorageFile))</td>

<td> 1094.</td><td></td><td> {</td>

<td> 1095.</td><td></td><td> XmlSerializer xs = new</td>

<td></td><td colspan="2"> XmlSerializer (typeof(Objectstorage), new Type[] {</td>

<td> 1096.</td><td></td><td> typeof(Wisp[]),</td>

<td> 1097.</td><td></td><td> typeof(Wisp),</td>

<td> 1098.</td><td></td><td> typeof(Contentpolygon),</td>

<td> 1099.</td><td></td><td> typeof(WispPolygon),</td>

<td> 1100.</td><td></td><td> typeof(Polygon),</td>

<td> 1101.</td><td></td><td> typeof(DefaulnPolygon[] ) ,</td>

<td> 1102.</td><td></td><td> typeof(Defaultpolygon)</td>

<td> 1103.</td><td></td><td> });</td>

<td> 1104.</td><td></td><td> var objectstorage = new Objectstorage ()</td>

<td> 1105.</td><td></td><td> {</td>

<td> 1106.</td><td></td><td> Wisps = this.Wisps,</td>

<td> 1107.</td><td></td><td> Defaultpolygons = this.Defaultpolygons</td>

<td> 1108.</td><td></td><td> };</td>

<td> 1109.</td><td></td><td> xs.Serialize (fs, objectstorage);</td>

<td> 1110. 1111.</td><td></td><td> fs.Close ();</td>

<td> 1112.</td><td></td><td> }</td>

<td> 1113. 1114. 1115.</td><td> }</td><td> )</td>

<td> 1116.</td><td> ///</td><td> &lt;summary&gt;</td>

<td> 1117.</td><td> ///</td><td> Rotates content of Content polygon</td>

<td> 1118.</td><td> ///</td><td> &lt;/summary&gt;</td>

<td> 1119.</td><td> m</td><td> &lt;param name=ⁿsender”&gt;&lt;/param&gt;</td>

<td> 1120.</td><td> III</td><td> &lt;param name=ⁿex/param&gt;</td>

<td> 1121.</td><td colspan="2"> private void OnRotateContentButtonClick(object sender.</td>

<td></td><td colspan="2"> MouseButtonEventArgs e)</td>

<td> 1122. 1123.</td><td> {</td><td> var fe = (sender as FraneworkElement);</td>

<td> 1124.</td><td></td><td> if (fe == null)</td>

<td> 1125. 1126.</td><td></td><td> return;</td>

<td> 1127.</td><td></td><td> var rw = fe.Tag as RenderedWisp;</td>

<td> 1128.</td><td></td><td> if (rw == null)</td>

<td> 1123. 1130.</td><td></td><td> return;</td>

<td> 1131.</td><td></td><td> rw.Parent.ContentPolygon.ContentOrientation -= 90;</td>

<td> 1132.</td><td></td><td> if (rw.Parent.ContentPolygon.ContentOrientation &lt; 0)</td>

<td> 1133.</td><td></td><td> {</td>

<td> 1134.</td><td></td><td> rw.Parent.ContentPolygon.ContentOrientation += 360;</td>

<td> 1135. 1136.</td><td></td><td> )</td>

100

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749 PCT/US2015/068192

<td> 1137.</td><td> rw.Contentpolygon.Material.Brush = rw.ContentPolygon. Parent.ContentBrush;</td>

<td> 1138. 1139. 1140. 1141.</td><td> } III &lt;summary&gt; III Updates transformations matrices according new corner points positions</td>

<td> 1142. 1143. 1144. 1145. 1146. 1147. 1148. 1149. 1150. 1151. 1152. 1153. 1154.</td><td> III &lt;/summary&gt; private void UpdateCornerPointPositions() { if (!_loaded) return; //var count = Brushes.Length; var wisps = Wisps; var defaultpolygons = Defaultpolygons; if (wisps == null) return;</td>

<td> 1155. 1156.</td><td> var count = wisps.Length; //for each provided brush one plane is created, each next plane is a little further (first on is the most visible)</td>

<td> 1157. 1158. 1159.</td><td> for (int i = 0; i &lt; count; i++) { UpdateCornerPointPositions(wisps[i].WispPolygon, _renderedWisps[i].WispPolygon.MatrixTransform, i * Zlncrement);</td>

<td> 1160. 1161. 1162. 1163. 1164. 1165. 1166. 1167.</td><td> ) if (defaultpolygons != null) { var dpCount = defauitPolygons.Length; for (int i = 0; i &lt; dpCount; i++) { UpdateCornerPointPositions(defaultpolygons[i], _rendsrsdDefaultPolygons[i].MatrixTransform, (count + i) * Zlncrement);</td>

<td> 1168. 1169. 1170. 1171. 1172.</td><td> } ) } private void UpdateCornerPointPositions(Polygon polygon, MatrixTransform3D matrix, double defaultZ = Zlncrement)</td>

<td> 1173. 1174.</td><td> { double z = (polygon.ZOrder &gt; 0) ? polygon.ZOrder * Zlncrement : defaultZ;</td>

<td> 1175. 1176. 1177. 1178. 1179. 1180. 1181. 1182.</td><td> var pointsTransformec = new Point3D[] { new Point3D(0, 0, z) , new Point3D(0, 1, z) , new Point3D(l, 0, z) , new Point3D(l, 1, z) ); for (int j = 0; j &lt; 4; j++)</td>

101

SUBSTITUTE SHEET (RULE 26)

CA 02971280 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 1183.</td><td> {</td>

<td> 1184.</td><td> //transforming from display to 3d coordinates or</td>

<td></td><td> using default values</td>

<td> 1185.</td><td> var inputCornerPointRelative = (polygon.CornerPoints</td>

<td></td><td> != null &amp;&amp; polygon.Cornerpoints.Count &gt; j) ?</td>

<td> 1186.</td><td> Simple2Dto3D(viewport3d,</td>

<td></td><td> polygon.CornerPoints[j]) : DefaultCornerPoints[j];</td>

<td> 1187.</td><td> for (int k = 0; k &lt; 4; k++)</td>

<td> 1188.</td><td> {</td>

<td> 1189.</td><td> //it's needed to prevent NaN in matrix</td>

<td></td><td> calculations (two points cannot have the same coordinates)</td>

<td> 1190.</td><td> if (j != k Math.Abs(pointsTransformed[k].X -</td>

<td></td><td> inputCornerPointRelative.X) &lt; 0.C001</td>

<td> 1191.</td><td> &amp;&amp; Math.Abs(pointsTransformed[k].Y -</td>

<td></td><td> inputCornerPointRelative.Y) &lt; C.COOl·)</td>

<td> 1192.</td><td> {</td>

<td> 1193.</td><td> inputCornerPointRelative.X =</td>

<td></td><td> pointsTransformed[k].X + 0.0001;</td>

<td> 1194.</td><td> inputCornerPointRelative.Y =</td>

<td></td><td> pointsTransformed[k].Y + 0.0001;</td>

<td> 1195.</td><td> }</td>

<td> 1196.</td><td> }</td>

<td> 1197.</td><td></td>

<td> 1198.</td><td> pointsTransformed[j;.X = inputCornerPointRelative.X;</td>

<td> 1199.</td><td> pointsTransformed[j’.Y = inputcornerPointRelative.Y;</td>

<td> 1200.</td><td> }</td>

<td> 1201.</td><td> //updating matrix</td>

<td> 1202.</td><td> matrix.Matrix =</td>

<td></td><td> CalculateNonAffineTransform(pointsTransformed);</td>

<td> 1203.</td><td> //updating corner point positons (in case of incomplete</td>

<td></td><td> set provided)</td>

<td> 1204.</td><td> var outputCornerPoints = new Point[4];</td>

<td> 1205.</td><td> for (int j = 0; j &lt; 4; j++)</td>

<td> 1206.</td><td> {</td>

<td> 1207.</td><td> outputCornerPoints[j] = Simple3Dto2D(viewport3d,</td>

<td></td><td> pointsTransformed[j]);// new Point(_pointsTransforined[i][j].X,</td>

<td></td><td> _pointsTransformed[i][j].Y);</td>

<td> 1208.</td><td> )</td>

<td> 1209.</td><td></td>

<td> 1210.</td><td> _internalChange = true;</td>

<td> 1211.</td><td> polygon.CornerPoints = new</td>

<td></td><td> Pointcollection(outputCornerPoints);</td>

<td> 1212.</td><td> —internalchange = false;</td>

<td> 1213.</td><td> }</td>

<td> 1214.</td><td></td>

<td> 1215.</td><td> // The input array of points describes a 2D rectangle</td>

<td> 1216.</td><td> II (with Z assumed to be zero) in the order</td>

<td> 1217.</td><td> II lower-left, upper-left, lower-right, upper-right.</td>

<td> 1218.</td><td> II The returned transform maps the points (0, 0, C),</td>

<td> 1219.</td><td> II (0, 1, 0), (1, 0, C), and (1, 1, 0) to these points.</td>

<td> 1220.</td><td> private Matrix3D CalculateNonAffineTransform(Point3D[]</td>

<td></td><td> points)</td>

<td> 1221.</td><td> {</td>

<td> 1222.</td><td> /! Affine transform</td>

102

SUBSTITUTE SHEET (RULE 26)

GA 02971260 2017-06-15

WO 2016/109749

PC17US2015/068192

<td> 1223.</td><td> //----------------</td>

<td> 1224.</td><td> // This matrix maps (0, 0) —&gt; (xO, yO)</td>

<td> 1225.</td><td> // (0, 1) —&gt; (xl, yl)</td>

<td> 1226.</td><td> II (1, 0) —&gt; (x2, y2)</td>

<td> 1227.</td><td> H (1, 1) —&gt; (x2 + xl + xO, y2 + yl *</td>

<td> yO)</td><td></td>

<td> 1228.</td><td> Matrix3D A = new Matrix3D();</td>

<td> 1229.</td><td> A.Mil = points[2].X - points[0].X;</td>

<td> 1230.</td><td> A.M12 = points[2].Y - poinzs[0].Y;</td>

<td> 1231.</td><td> À.M21 = points[1].X - points[0].X;</td>

<td> 1232.</td><td> A.M22 = points[l].Y - poinrs[0].Y;</td>

<td> 1233.</td><td> A.OffsetX = points[0].X;</td>

<td> 1234.</td><td> A.OffsetY = points[0].Y;</td>

<td> 1235.</td><td></td>

<td> 1236.</td><td> /! Calculate point (a, b) that get mapped by the affine</td>

<td colspan="2"> transform to (x3, y3)</td>

<td> 1237.</td><td> double den = A.Mil * A.M22 - A.M12 * A.M21;</td>

<td> 1238.</td><td> double a = (A.M22 * points[3].X - A.M21 * poir.ts[3].Y +</td>

<td> 1239.</td><td> A.M21 * A.OffsetY - A.M22 * A.OffsetX) /</td>

<td> den;</td><td></td>

<td> 1240.</td><td></td>

<td> 1241.</td><td> double b = (A.Mil * points[3].Y - A.M12 * poir.ts[3].X +</td>

<td> 1242.</td><td> A.M12 * A.OffsetX - A.Mil * A.OffsetY) /</td>

<td> den;</td><td></td>

<td> 1243.</td><td></td>

<td> 1244.</td><td> Π Non-affine transform</td>

<td> 1245.</td><td> //--------------------</td>

<td> 1246.</td><td> // This matrix maps (0, 0) --&gt; (0, 0)</td>

<td> 1247.</td><td> // (0, 1) —&gt; (0, 1)</td>

<td> 1248.</td><td> // (1, 0) —&gt; (1, 0)</td>

<td> 1249.</td><td> // (1, 1) —&gt; (a, b)</td>

<td> 1250.</td><td></td>

<td> 1251.</td><td> Matrix3D B = new Matrix3D();</td>

<td> 1252.</td><td> B.M11 = a / (a + b - 1) ;</td>

<td> 1253.</td><td> B.M22 = b / (a + b - 1);</td>

<td> 1254.</td><td> B.M14 = B.M11 - 1;</td>

<td> 1255.</td><td> B.M24 = B.M22 - 1;</td>

<td> 1256.</td><td></td>

<td> 1257.</td><td> return B * A;</td>

<td> 1258.</td><td> }</td>

<td> 1259.</td><td></td>

<td> 1260.</td><td> // The following two methods only work with</td>

<td colspan="2"> Orthographiée arriéra.</td>

<td> 1261.</td><td> // with LookDirection of (0, 0, -1) and UpDirection of (0,</td>

<td> 1, 0) .</td><td></td>

<td> 1262.</td><td> II More advanced conversion routines can be found in the</td>

<td> 1263.</td><td> II Petzold.Media3D library.</td>

<td> 1264.</td><td></td>

<td> 1265.</td><td> II Converts a 2D point in device-independent coordinates</td>

<td colspan="2"> relative</td>

<td> 1266.</td><td> II to Viewport3D to 3D space.</td>

<td> 1267.</td><td> private Point3D Simple2Dto3D(Viewport3D vp, Point pt, double</td>

z = 0.0)

1268

103

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749

PCT/US2015/068192

<td> 1263.</td><td> OrthographicCamera cam = CheckRestrictions(vp);</td>

<td> 1270.</td><td> double scale = cam.Width / vp.Actualwidth;</td>

<td> 1271.</td><td> double x = scale * (pt.X - vp.Actualwidth /2)+ cam.Position.X;</td>

<td> 1272. 1273.</td><td> double y = scale * (vp.ActualHeight / 2 - pt.Y) + cam.Position.Y;</td>

<td> 1274.</td><td> return new Point3D(x, y, z) ;</td>

<td> 1275. 1276.</td><td> }</td>

<td> 1277.</td><td> // Converts a 3D point to 2D in device-independent coordinates</td>

<td> 1278.</td><td> // relative to Viewport3D.</td>

<td> 1273.</td><td> private Point Simple3Dto2D(Viewport3D vp, PointSD point)</td>

<td> 1280.</td><td> {</td>

<td> 1281.</td><td> OrthographicCamera cam = CheckRestrictions(vp);</td>

<td> 1282.</td><td> double scale = vp.Actualwidth / cam.Width;</td>

<td> 1283.</td><td> double x = vp.Actualwidth / 2 + scale * (point.X cam.Position.X);</td>

<td> 1284.</td><td> double y = vp.ActualHeight / 2 - scale * (point.Y cam.Position.Y);</td>

<td> 1285.</td><td> return new Point(x, y);</td>

<td> 1286. 1287.</td><td> }</td>

<td> 1288.</td><td> private OrthographicCamera CheckRestrictions(Viewport3D vp)</td>

<td> 1289.</td><td> {</td>

<td> 1290. 1231.</td><td> OrthographicCamera cam = vp.Camera as OrthographicCamera;</td>

<td> 1232.</td><td> if (cam == null)</td>

<td> 1233. 1294.</td><td> throw new Argumer.tException (Camera must be OrthographicCamera);</td>

<td> 1295.</td><td> if (cam.LookDirection != new Vector3D(0, 0, -1))</td>

<td> 1296. 1297.</td><td> throw new Argumer.tException (Camera LookDirection must be (0, 0, -1));</td>

<td> 1298.</td><td> if (cam.UpDirection != new Vector3D(0, 1, 0))</td>

<td> 1293. 1300.</td><td> throw new Argumer.tException (Camera UpDirection must be (0, 1, 0));</td>

<td> 1301.</td><td> return cam;</td>

<td> 1302. 1303.</td><td> }</td>

<td> 1304.</td><td> private RenderedWisp[] _renderedWisps;</td>

<td> 1305. 1306.</td><td> private RenderedDefaultPolygon[] _renderedDefaultPolygons;</td>

<td> 1307.</td><td> /// &lt;summary&gt;</td>

<td> 1308.</td><td> /// Indicates if user drags mouse</td>

<td> 1303.</td><td> /// &lt;/summary&gt;</td>

<td> 1310. 1311.</td><td> private bool _isDragging;</td>

<td> 1312. 1313.</td><td> private bool _isDraggingContent;</td>

104

SUBSTITUTE SHEET (RULE 26)

CA 02971200 2017-06-15

WO 2016/109749

PC17US2015/068192

1314. private RenderedWisp _draggingWisp = null;

1315. private RenderedPolygon _draggingPolygon = null;

1316.

1317. /// <summary>

1318. Ill Indicates index of dragging vertex

1313. Ill </summary>

1320. private int _draggingVertexIndex;

1321.

1322. Ill <summary>

1323. /// Start position for dragging mesh

1324. Ill </summary>

1325. private Point3D[] _startDraggingMeshPoints;

1326.

1327. Ill <summary>

1328. Ill Start position for mouse while dragging a mesh

1323. Ill </summary>

1330. private Point3D _startDraggingPoint;

1331.

1332. Ill <summary>

1333. Ill Start position of mouse while dragging

1334. /// </summary>

1335. private Point _startDraggingMousePoint;

1336.

1337. Ill <summary>

1338. Ill Indicates that Control Points assigned from within control class

1333. Ill </summary>

1340. private bool _internalChange = false;

1341.

1342. Ill <summary>

1343. Ill Indicates that control is loaded

1344. Ill </summary>

1345. private bool loaded = false;

1346.

1347. !/} <summary>

1348. Hl Stores a list of all clicked click groups

1343. Ill </summary>

1350. private List<string> _clickedGroups = new List<string>();

1351. }

1352. }

1353.

105

SUBSTITUTE SHEET (RULE 26)