Implement for optically inferring information from a planar jotting surface

RELATED APPLICATIONS

0001This application claims priority from U.S. Provisional Patent Application No. 60/450,244 filed on Feb. 24^th, 2003.

FIELD OF THE INVENTION

0002The present invention relates generally to acquisition of information written, drawn, sketched or otherwise marked on a jotting or writing surface by a user with the aid of a hand-held implement, such as a writing implement.

BACKGROUND OF THE INVENTION

0003The art of writing and drawing is ancient and rich in traditions. Over the ages various types of implements have been used for writing down words as well as drawing, sketching, marking and painting. Most of these implements have a generally elongate shape, an essentially round cross-section and they are terminated at one end by a writing nib or tip. They are typically designed to be hand-held and operated by the user's preferred hand (e.g., by the right hand for right-handed persons). More specifically, the user moves the implement across a writing or jotting surface such that the writing nib leaves a visible trace marking its motion on the surface. The marking can be produced by a material deposited from the nib, e.g., through abrasion of the marking material (such as charcoal in the case of a pencil) or by direct wetting of the surface by an ink (as in the case of the pen). The marking can also include any other physical trace left on the surface.

0004The most widely used writing and drawing implements include pens and pencils while the most convenient jotting surfaces include sheets of paper of various sizes and other generally planar objects capable of being marked. In fact, despite the tremendous advances in sciences and engineering, pen and paper remain among the simplest and most intuitive devices for writing, drawing, marking and sketching even in the electronic age.

0005The challenge of communicating with electronic devices is in the very input interface to the electronic device. For example, computers take advantage of input devices such as keyboards, buttons, pointer devices, mice and various other types of apparatus that encode motion and convert it to data that the computer can process. Unfortunately, none of these devices are as user-friendly and accepted as pen and paper.

0006This input interface problem has been recognized in the prior art and a variety of solutions have been proposed. Most of these solutions attempt to derive electronic, i.e., digital data from the motions of a pen on paper or some other writing surface, e.g., a writing tablet. Of these prior art teachings the following references are of note:

0007<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>U.S. Pat. Nos.</entry></row><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="1" colwidth="49pt" align="center" /><colspec colname="2" colwidth="35pt" align="center" /><colspec colname="3" colwidth="49pt" align="center" /><colspec colname="4" colwidth="35pt" align="center" /><colspec colname="5" colwidth="49pt" align="center" /><tbody valign="top"><row><entry>4,471,162</entry><entry>4,896,543</entry><entry>5,103,486</entry><entry>5,215,397</entry><entry>5,226,091</entry></row><row><entry>5,294,792</entry><entry>5,333,209</entry><entry>5,434,371</entry><entry>5,484,966</entry><entry>5,517,579</entry></row><row><entry>5,548,092</entry><entry>5,661,506</entry><entry>5,577,135</entry><entry>5,581,276</entry><entry>5,587,558</entry></row><row><entry>5,587,560</entry><entry>5,652,412</entry><entry>5,661,506</entry><entry>5,717,168</entry><entry>5,737,740</entry></row><row><entry>5,750,939</entry><entry>5,774,602</entry><entry>5,781,661</entry><entry>5,902,968</entry><entry>5,939,702</entry></row><row><entry>5,959,617</entry><entry>5,960,124</entry><entry>5,977,958</entry><entry>6,031,936</entry><entry>6,044,165</entry></row><row><entry>6,050,490</entry><entry>6,081,261</entry><entry>6,100,877</entry><entry>6,104,387</entry><entry>6,104,388</entry></row><row><entry>6,108,444</entry><entry>6,111,565</entry><entry>6,124,847</entry><entry>6,130,666</entry><entry>6,147,681</entry></row><row><entry>6,153,836</entry><entry>6,177,927</entry><entry>6,181,329</entry><entry>6,184,873</entry><entry>6,188,392</entry></row><row><entry>6,213,398</entry><entry>6,243,503</entry><entry>6,262,719</entry><entry>6,292,177</entry><entry>6,330,359</entry></row><row><entry>6,334,003</entry><entry>6,335,723</entry><entry>6,335,724</entry><entry>6,335,727</entry><entry>6,348,914</entry></row><row><entry>6,396,481</entry><entry>6,414,673</entry><entry>6,421,042</entry><entry>6,422,775</entry><entry>6,424,340</entry></row><row><entry>6,429,856</entry><entry>6,437,314</entry><entry>6,456,749</entry><entry>6,492,981</entry><entry>6,498,604</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0008<tables id="TABLE-US-00002" num="00002"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>U.S. Published applications:</entry></row><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="4"><colspec colname="offset" colwidth="21pt" align="left" /><colspec colname="1" colwidth="63pt" align="left" /><colspec colname="2" colwidth="63pt" align="left" /><colspec colname="3" colwidth="70pt" align="left" /><tbody valign="top"><row><entry /><entry>2002-0001029</entry><entry>2002-0028017</entry><entry>2002-0118181</entry></row><row><entry /><entry>2002-0148655</entry><entry>2002-0158848</entry><entry>2002-0163511</entry></row><row><entry /><entry namest="offset" nameend="3" align="center" rowsep="1" /></row></tbody></tgroup></table></tables><br /> European Patent Specifications: 0,649,549B1

0009<tables id="TABLE-US-00003" num="00003"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>International Patent applications:</entry></row><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="77pt" align="center" /><colspec colname="2" colwidth="63pt" align="center" /><colspec colname="3" colwidth="77pt" align="center" /><tbody valign="top"><row><entry>WO 02/017222 A2</entry><entry>WO 02/058029 A2</entry><entry>WO 02/064380 A1</entry></row><row><entry>WO 02/069247 A1</entry><entry>WO 02/084634 A1</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0010Although the above-referenced teachings provide a number of approaches they are cumbersome to the user. Many of these approaches provide the user with pens that are difficult to handle, impose special writing and/or monitoring conditions and/or they require cumbersome auxiliary systems and devices to track and digitize the information written on the writing surface. Thus, the problem of a user-friendly input interface based on a writing implement has not been solved.

SUMMARY OF THE INVENTION

0011The present invention provides a jotting implement that infers hand-jotted information from a jotting surface. For the purposes of this invention, hand-jotted information comprises any information marked on the jotting surface as a result of any of the following actions: writing, jotting, drawing, sketching or in any other manner marking or depositing marks on the jotting surface. Additionally, hand-jotted information for the purposes of this application also means information traced on the jotting surface without leaving any markings on the jotting surface. The jotting implement has a nib for jotting and an arrangement for determining when the nib is jotting on the jotting surface. Further, the implement has an optical unit for viewing the jotting surface. The optical unit is preferably mounted at a distal end of the implement with respect to the nib and indexed to it. For the purposes of this invention indexed to the nib means that the optical axis of the optical unit is referenced to the nib, e.g., the optical axis of the optical unit passes through the nib. The implement also has a processing unit for receiving optical data of said jotting surface from said optical unit and for determining from said optical data the physical coordinates of the nib with respect to at least one corner of the jotting surface and at least one edge of the jotting surface.

0012It should be noted that in contrast to the prior art the implement of the invention infers the physical coordinates of the nib indirectly, i.e., from the optical data of the jotting surface obtained from the optical unit. Therefore, any optical data about the jotting surface sufficient to make the determination of the physical coordinates of the nib can be used. For example, optical data of all corners or a number of corners, edges or portions thereof can be used. Alternatively, landmarks or any optically recognizable features on the jotting surface can be used as well.

0013The arrangement for determining when the nib is jotting on the jotting surface preferably comprises a pressure sensitive unit mounted in the jotting implement. Strain gauges, mechanical pressure sensors, piezoelectric elements and other types of arrangements recognizing contact between the nib and the jotting surface can be used for this purpose.

0014In the preferred embodiment the optical unit is an imaging unit for imaging the jotting surface or a portion thereof. It is further preferred that the imaging unit be equipped with a photodetector array for projecting an image of the jotting surface thereon. The processing unit has an edge detection unit or circuit (e.g., firmware in a microprocessor of the processing unit) for detecting edges and corners of the jotting surface in the image.

0015The jotting implement is further equipped with an image transformation unit for applying one or more transformations to the image. Specifically, the image transformation unit can include appropriate physical optics (e.g., lenses) for correcting the image as well as software routines for correcting the image and performing various operations on the image. For example, the image transformation unit has an image deformation transformer that corrects the image for a plane projection. Alternatively, the image transformation unit has an image deformation transformer that corrects the image for a spherical projection. In the same or a different embodiment, the image transformation unit has an image transformer for determining Euler angles of the jotting implement with respect to the jotting surface.

0016In the preferred embodiment the corrections and transformations are applied only to the edges and/or corners of the image that are identified by the edge detection unit. In other words, only a part of the image corresponding to the jotting surface and in particular its edges, corners, landmarks or other optically recognizable features and/or their portions are corrected and transformed.

0017A ratio computation module belonging to the processing unit determines the physical coordinates of the nib from the image. Again, in the preferred embodiment this determination is made from the relevant part of the image corresponding to the jotting surface and in particular its edges, corners, landmarks or other optically recognizable features and/or their portions.

0018The photodetector array can be any suitable array of photodetectors, including a photodiode or phototransistor array and preferably a CMOS photodetector array. The optics used by the imaging unit can include refractive and/or reflective optics and preferably include a catadioptric system. In any event, the field of view of the optics should be substantially larger than the area of the jotting surface such that the imaging unit can always detect at least one edge and one corner of the jotting surface for any possible position of the jotting implement when the nib is in contact with the jotting surface.

0019In order to determine the physical coordinates of the nib at a sufficient rate to determine what the user has written, sketched or drawn the implement has a frame control unit.

0020The frame control unit sets a certain frame rate at which the jotting surface is imaged. Preferably, this frame rate is at least 15 Hz, and more preferably it is in excess of 30 Hz.

0021Finally, the jotting implement is provided with a device for communicating the physical coordinates of the nib with an external unit. The device for communicating these coordinates can include any type of data transmission port including but not limited to infra-red (IR) ports, ultrasound ports, optical ports and the like. The external unit can be a computer, a hand-held device, a network terminal, a downloading unit, an electronic gateway into a wide area network (WAN) (e.g., the internet) or a local area network (LAN), a storage device, a printer or any other external unit which can store, print, relay and/or further process the physical coordinates of the nib. It should be noted that, depending on the application and requirements, the physical coordinates of the nib can be processed in real time or not.

0022In the preferred embodiment the implement is further equipped with an arrangement for initializing and recognizing the jotting surface. Of course, the sizes and types jotting surfaces can also be selected or input by the user. The arrangement for initializing and recognizing can include the optical unit and processing unit described above and a memory with standard sizes of likely jotting surfaces. For example, when the jotting surfaces are expected to be sheets of paper of standard sizes, the images of such sheets can be stored in the memory. Preferably, these stored images are taken at well-known positions and orientations of the jotting implement with respect to the jotting surface. In other words, they are taken at known physical coordinates of the nib on the jotting surface and known spatial orientation of the jotting implement (e.g., at known Euler angles).

0023The details of the invention will now be explained in the attached detailed description with reference to the attached drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

0024<figref idref="DRAWINGS">FIG. 1</figref> is a side view of a jotting implement in accordance with the invention where the jotting implement is shown in the plane of an inclination angle θ (Euler angle θ).

0025<figref idref="DRAWINGS">FIG. 2</figref> is a three-dimensional diagram illustrating the physical parameters of the jotting implement of <figref idref="DRAWINGS">FIG. 1</figref> when in use.

0026<figref idref="DRAWINGS">FIG. 3</figref> is a plan side view of the jotting implement of <figref idref="DRAWINGS">FIG. 1</figref> illustrating the principle of imaging.

0027<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of the processing unit of the jotting implement of <figref idref="DRAWINGS">FIG. 1</figref>.

0028<figref idref="DRAWINGS">FIG. 5</figref> is a diagram illustrating the image of the jotting surface projected onto a photodetector array belonging to the imaging unit.

0029<figref idref="DRAWINGS">FIG. 6</figref> is a diagram illustrating the process of edge and/or corner detection applied to the image of the jotting surface.

0030<figref idref="DRAWINGS">FIGS. 7A–D</figref> are diagrams illustrating the functions performed by the processing unit on the image to determine the orientation of the jotting implement with respect to the jotting surface in terms of Euler angles.

0031<figref idref="DRAWINGS">FIG. 8</figref> is a side view illustrating an alternative embodiment of a jotting implement having an orienting grip.

0032<figref idref="DRAWINGS">FIG. 9</figref> is a diagram illustrating the process of image correction and parametrization.

0033<figref idref="DRAWINGS">FIG. 10</figref> is a diagram illustrating the parameterized corrected image.

0034<figref idref="DRAWINGS">FIG. 11</figref> is a diagram illustrating the parametrized, corrected and transformed image from which the physical coordinates of the nib are determined.

0035<figref idref="DRAWINGS">FIG. 12</figref> is a diagram illustrating a correspondence between the image of the jotting surface and the physical jotting surface as can be used for initialization and cross-check purposes.

0036<figref idref="DRAWINGS">FIG. 13</figref> illustrates another embodiment of an optical unit using a catadioptric system.

0037<figref idref="DRAWINGS">FIG. 14</figref> illustrates the top portion of a writing implement employing the catadioptric system of <figref idref="DRAWINGS">FIG. 13</figref>.

0038<figref idref="DRAWINGS">FIG. 15</figref> is a three-dimensional diagram illustrating the use of alternative landmarks and features to determine the physical coordinates of the nib.

DETAILED DESCRIPTION

0039The present invention will be best understood by initially referring to the side view of <figref idref="DRAWINGS">FIG. 1</figref> illustrating a jotting implement <b>10</b> in accordance with the invention and the diagrams of <figref idref="DRAWINGS">FIGS. 2 through 4</figref>. Jotting implement <b>10</b> shown in <figref idref="DRAWINGS">FIG. 1</figref> is a pen, more specifically an ink pen, and still more precisely a ball-point pen. However, it will be appreciated that jotting implement <b>10</b> can be a marker, a pencil a brush or indeed any other writing, sketching, drawing or painting implement that can jot information on a jotting surface <b>12</b>. Alternatively, jotting implement <b>10</b> can also be stylus or any device that jots information on jotting surface <b>12</b> by tracing that information without leaving any permanent markings or deformations on the jotting surface. Such jotting surface can include a pressure-sensitive digitizing tablet or any other surface provided specifically for input into an electronic data processing device. In the present embodiment jotting implement has a shape generally resembling known writing, sketching, drawing or painting devices. Specifically, jotting implement <b>10</b> has an elongate body <b>14</b> of generally round cross-section designed to be held in a user's hand <b>16</b>.

0040In general, jotting surface <b>12</b> is a sheet of planar material on which implement <b>10</b> can perform a jotting function as defined above. For geometrical reasons, it is preferable that jotting surface <b>12</b> be rectangular. In the present embodiment jotting surface <b>12</b> is a sheet of paper of any standard or non-standard dimensions laying flat on a support surface <b>18</b>. In cases where jotting surface <b>12</b> is a digitizing tablet such as a tablet of a PDA device, a computer screen or any other sturdy surface then support surface <b>18</b> may not be required. It is important, however, that jotting surface <b>12</b> have optically recognizable features such as corners, edges, landmarks or the like. It is also important that these features not change their position with respect to the remainder of jotting surface <b>12</b> during the jotting operation.

0041Implement <b>10</b> has a nib <b>20</b> terminating in a ball-point <b>22</b>. A pressure sensor <b>24</b> is mounted proximate nib <b>20</b> for determining when nib <b>20</b> is jotting. Jotting occurs when ball-point <b>22</b> is in contact with jotting surface <b>12</b>. Conveniently, pressure sensor <b>24</b> is a strain gauge. Alternatively, pressure sensor <b>24</b> is a mechanical pressure sensor or a piezoelectric element. A person skilled in the art will recognize that other pressure sensors can also be used. Implement <b>10</b> also has an initialization switch <b>26</b>. Switch <b>26</b> is provided for the user to communicating whether jotting is occurring on the same jotting surface <b>12</b> or on a new jotting surface (not shown).

0042An optical unit <b>30</b> is mounted at a distal end <b>32</b> of implement <b>10</b>. Optical unit <b>30</b> is designed for viewing jotting surface <b>12</b> and it has a field of view <b>34</b> demarked by a delimiting line that extends beyond jotting surface, as described in more detail below. In the present embodiment optical unit <b>30</b> is mounted on three support members <b>36</b>. Members <b>36</b> can have any construction that ensures mechanical stability and obstructs a negligible portion of field of view <b>34</b>. Optical unit <b>30</b> has an optical axis <b>39</b> that is indexed to nib <b>20</b>. More specifically, optical axis <b>39</b> passes through nib <b>20</b>. Thus, field of view <b>34</b> of optical unit <b>30</b> is centered on nib <b>20</b>. Alternatively, optical axis <b>39</b> can be indexed to nib <b>20</b> at some predetermined offset. For reasons of symmetry of field of view <b>34</b>, however, it is preferred that optical unit <b>30</b> be indexed to nib <b>20</b> by passing optical axis <b>39</b> through nib <b>20</b> and through the center of ball-point <b>22</b>.

0043Implement <b>10</b> has a device <b>38</b> for communicating with an external unit <b>40</b> (see <figref idref="DRAWINGS">FIG. 2</figref>). In the present embodiment device <b>38</b> is an infra-red (IR) port for transmitting and receiving data encoded in IR radiation <b>42</b>. Of course, any type of data transmission port including but not limited to ultrasound ports or optical ports can be used as device <b>38</b>. Meanwhile, external unit <b>40</b> can be a computer, a hand-held device, a network terminal, a downloading unit, an electronic gateway into a wide area network (WAN) (e.g., the internet) or a local area network (LAN), a storage device, a printer or any other external unit which can store, print, relay and/or further process the physical coordinates of nib <b>20</b>.

0044Referring now to <figref idref="DRAWINGS">FIG. 2</figref>, the physical parameters of implement <b>10</b> are conveniently described in terms of a Cartesian coordinate system and a polar coordinate system. The origins of these coordinate systems coincide at the position of nib <b>20</b> and more specifically at the position where ball-point <b>22</b> contacts jotting surface <b>12</b>. The Cartesian system has its X-and Y-axes in the plane of jotting surface <b>12</b> and aligned with the width and length of jotting surface <b>12</b>. The Z-axis of the Cartesian system is perpendicular or normal to the plane of jotting surface <b>12</b>.

0045A number of features <b>44</b>A, <b>44</b>B, <b>44</b>C are defined by corresponding vectors v₁, v₂, v₃drawn from the origin of the Cartesian system. In the present case features <b>44</b>A, <b>44</b>B, <b>44</b>C are three corners of jotting surface <b>12</b>. Alternatively, features <b>44</b> can include any edge <b>43</b> of jotting surface <b>12</b> or any other optically recognizable landmark or feature of jotting surface <b>12</b>. It should be noted that features produced on jotting surface <b>12</b> by the user, including any marks jotted by implement <b>10</b>, are legitimate features for this purpose.

0046The polar coordinate system is used to define the orientation of implement <b>10</b> with respect to jotting surface <b>12</b>. The Z-axis of the polar system is coincident with the Z-axis of the Cartesian system. Since optical axis <b>39</b> is indexed to nib <b>20</b> it passes through the origins of the two coordinate systems.

0047Thus, in the polar system optical axis <b>39</b> defines the polar coordinate r and the length of r, i.e., |r| is the length of implement <b>10</b>. The inclination of implement <b>10</b> with respect to the Z-axis is expressed by polar angle θ, hereafter referred to as inclination angle θ. The angle of rotation of implement <b>10</b> about the Z-axis is expressed by polar angle φ.

0048It is preferred that optical unit <b>30</b> be an imaging unit, as shown in the plan view of <figref idref="DRAWINGS">FIG. 3</figref>. Specifically, optical unit <b>30</b> is preferably an imaging unit capable of imaging objects present in its field of view <b>34</b> and in particular imaging jotting surface <b>12</b> with relatively low distortion. In the present embodiment imaging unit <b>30</b> has a refractive imaging optics <b>46</b> indicated by lenses <b>48</b>A, <b>48</b>B. It will be appreciated by a person skilled in the art that suitable refractive imaging optics <b>46</b> include lenses which afford a wide field of view with good off-axis optical performance, such as fish-eye lenses or wide-field-of-view lenses. For more specifics on such types of lenses the reader is referred to U.S. Pat. Nos. 4,203,653; 4,235,520; 4,257,678 as well as the article by James “Jay” Kumler et al., “Fisheye lens designs and their relative performance”, SPIE, all of which are herein incorporated by reference.

0049Imaging optics <b>46</b> define an image plane <b>50</b> as indicated by the dashed line. Imaging unit <b>30</b> is further equipped with a photodetector array <b>52</b> positioned in image plane <b>50</b>. An image <b>12</b>′ of jotting surface <b>12</b> is projected onto array <b>52</b> by imaging optics <b>46</b>. Preferably, array <b>52</b> is a CMOS photodetector array. Of course, other types of photodetector arrays including arrays employing photodiodes or phototransitors of various types can be used as photodetector array <b>52</b>. A CMOS photodetector array, however, tends to be more efficient, responsive and it tends to consume less power. In addition CMOS arrays have a small pitch thus enabling high resolution.

0050Field of view <b>34</b> afforded by optics <b>46</b> is substantially larger than the area of jotting surface <b>12</b>. In fact, field of view <b>34</b> is large enough such that image <b>12</b>′ of entire jotting surface <b>12</b> is always projected onto array <b>52</b>. This condition holds for any jotting position that may be assumed by jotting implement <b>10</b> during a jotting operation performed by the user, such as writing near an edge or corner of jotting surface <b>12</b> at a maximum possible inclination angle θ (e.g., θ≈40°). Thus, forward and backward portions y₁, y₂of jotting surface <b>12</b> are always imaged on array <b>52</b> as portions y′₁, y′₂as long as not obstructed by user's hand <b>16</b> or by other obstacles.

0051It is noted that for purposes of clarity primed reference numbers are used herein to denote parts in image space corresponding to parts bearing the same but unprimed reference numbers in physical space. As additional transformations and operations are applied to parts in the image space, more primes are added to the reference numbers.

0052Jotting implement <b>10</b> has a processing unit <b>54</b>, which is illustrated in more detail in <figref idref="DRAWINGS">FIG. 4</figref>. Processing unit <b>54</b> is designed for receiving optical data of jotting surface <b>12</b>. In this embodiment the optical data is represented by image <b>12</b>′ of jotting surface <b>12</b>. From this optical data processing unit <b>54</b> determines the physical coordinates of nib <b>20</b> with respect to at least one corner and at least one edge of jotting surface <b>12</b>. In the present embodiment processing unit <b>54</b> is designed to determine vectors v₁, v₂, v₃in the Cartesian coordinate system defined in <figref idref="DRAWINGS">FIG. 2</figref>.

0053To achieve its function processing unit <b>54</b> is equipped with an image processor <b>56</b>, a frame control <b>58</b>, a memory <b>60</b> as well as an uplink port <b>62</b> and a downlink port <b>64</b>. Ports <b>62</b>, <b>64</b> belong to communication device <b>38</b>. Image processor <b>56</b> preferably includes an edge detection unit <b>66</b>, an origin localization unit <b>68</b>, an image transformation unit <b>70</b> and a ratio computation unit <b>72</b>, as better shown in <figref idref="DRAWINGS">FIG. 5</figref>. In addition to these elements, image processor <b>56</b> has a demultiplexer <b>74</b> for receiving and demultiplexing raw image data <b>76</b> containing image <b>12</b>′. Data <b>76</b> is delivered from the row <b>78</b>A and column <b>78</b>B multiplexing blocks of array <b>52</b>.

0054During operation, the user moves implement <b>10</b>. Once nib <b>20</b> of implement <b>10</b> is brought in contact with jotting surface <b>12</b> pressure sensor <b>24</b> activates the acquisition mode of optical unit <b>30</b>. In the acquisition mode processing unit <b>54</b> receives optical data i.e.image <b>12</b>′ of jotting surface <b>12</b> as imaged on the pixels of array <b>52</b>.

0055Now, image processor <b>56</b> captures raw image data <b>76</b> of image <b>12</b>′ at a certain frame rate. The frame rate is controlled by frame control <b>58</b>. The frame rate is fast enough to accurately track the jotting activity of the user. To achieve this the frame rate is set by frame control <b>58</b> at 15 Hz or even at 30 Hz or higher.

0056In contrast with the prior art, the information jotted by the user is not determined by inspecting or imaging the information itself. Rather, the jotted information is inferred by determining the physical coordinates of nib <b>20</b> or, more precisely of ball-point <b>22</b> with respect to optically recognizable features of jotting surface <b>12</b>. These recognizable features can include corners, edges or any other landmarks or features produced by the user on jotting surface <b>12</b>. To determine all information jotted by the user the physical coordinates of nib <b>20</b> with respect to the recognizable features are acquired at the set frame rate whenever the acquisition mode is activated by pressure sensor <b>24</b>.

0057In the present embodiment, the physical coordinates of nib <b>20</b> are determined with respect to three corners <b>44</b>A, <b>44</b>B and <b>44</b>C of jotting surface <b>12</b> parametrized with the aid of vectors v₁, v₂and v₃(see <figref idref="DRAWINGS">FIG. 2</figref>). To accomplish this goal, processing unit <b>54</b> recovers vectors v₁, v₂, and v₃from imaged vectors v′₁, v′₂and v′₃of image <b>12</b>′ (see <figref idref="DRAWINGS">FIG. 5</figref>). This process requires a number of steps.

0058In a first step image processor <b>56</b> of processing unit <b>54</b> demultiplexes raw image data <b>76</b> from row and column blocks <b>78</b>A, <b>78</b>B of array <b>52</b> with the aid of demultiplexer <b>74</b>. Next, image processor <b>56</b> sends image data <b>76</b> to edge detection unit <b>66</b>. Edge detection unit <b>66</b> identifies the edges and corners of image <b>12</b>′ of jotting surface <b>12</b>. This process is better illustrated in <figref idref="DRAWINGS">FIG. 6</figref> where unobstructed portions <b>80</b>′ of imaged edges <b>43</b>′ are used for edge detection. For more information on edge detection in images and edge detection algorithms the reader is referred to U.S. Pat. Nos. 6,023,291 and 6,408,109 and to Simon Baker and Shree K. Nayar, “Global Measures of Coherence for Edge Detector Evaluation”, Conference on Computer Vision and Pattern Recognition, June 1999, Vol. 2, pp. 373–379 and J. Canny, “A Computational Approach to Edge Detection”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 8, No. 6, November 1986 for basic edge detection all of which are herein incorporated by reference.

0059In practice, user's hand <b>16</b> is an obstruction that obscures a portion of jotting surface <b>12</b>. Hence, a corresponding shadow <b>16</b>′ is present in image <b>12</b>′. Another shadow <b>17</b>′ (or a number of shadows) will frequently be produced by other objects covering jotting surface <b>12</b> or located between jotting surface <b>12</b> and optical unit <b>30</b>. Such objects typically include the user's other hand and/or body parts such as hair (not shown). For the purposes of the present invention it is only necessary that image <b>12</b>′ have a few unobstructed portions <b>80</b>′ of imaged edges <b>43</b>′, preferably including two or more corners, e.g., <b>44</b>A′, <b>44</b>B′ and <b>44</b>C′ to enable recovery of vectors v₁, v₂and v₃and consequent determination of the physical coordinates of nib <b>20</b>.

0060Thus, despite shadows <b>16</b>′ and <b>17</b>′ several unobstructed portions <b>80</b>′ of imaged edges <b>43</b>′ are available to edge detection unit <b>66</b>. A number of pixel groups <b>82</b> whose optical data <b>76</b> can be used by edge detection unit <b>66</b> for edge detection purposes are indicated. It should be noted that in some circumstances a pixel group <b>83</b> which is obscured by a shadow, e.g., by shadow <b>16</b>′ may become visible and can then be used to detect corner <b>44</b>D′.

0061Edge detection unit <b>66</b> recognizes edges <b>43</b>′ and describes them in terms of their vector equations or other suitable mathematical expressions with reference to a center <b>84</b> of field of view <b>34</b>. In order to serve as reference, center <b>84</b> is set with the aid of origin localization unit <b>68</b>. This can be performed prior to operating jotting implement <b>10</b>, e.g., during first initialization and testing of jotting implement <b>10</b> and whenever re-calibration of origin location becomes necessary due to mechanical reasons. The initialization can be performed with the aid of any suitable algorithm for fixing the center of an imaging system. For further information the reader is referred to Carlo Tomasi and John Zhang, “How to Rotate a Camera”, Computer Science Department Publication, Stanford University and Berthold K. P. Horn, “Tsai's Camera Calibration Method Revisited”, which are herein incorporated by reference and attached as appendices hereto.

0062In accordance with the invention center <b>84</b> coincides with optical axis because optical unit <b>30</b> is indexed to nib <b>20</b>. Hence, for any orientation of jotting implement <b>10</b> in physical space, i.e., for any value of inclination angle θ and polar angle φ, center <b>84</b> of field of view <b>34</b> is always coincident with the position of nib <b>20</b> and its image <b>20</b>′. Systems having this property are commonly referred to as central systems in the art and they include various types of central panoramic systems and the like. It should be noted that image <b>20</b>′ of nib <b>20</b> is not actually visible in field of view <b>34</b>, because body <b>14</b> of jotting implement <b>10</b> obscures center <b>84</b> at all times.

0063Due to optical effects including aberration associated with imaging optics <b>46</b>, the detected portion of image <b>12</b>′ will exhibit a certain amount of rounding of edges <b>43</b>′, as indicated in dashed lines. This rounding can be compensated optically by lenses <b>48</b>A, <b>48</b>B and/or by any additional lenses (not shown) as well as electronically by processing unit <b>54</b>. Preferably, the rounding is accounted for by applying a transformation to detected portion of image <b>12</b>′ by image transformation unit <b>70</b>. For example, image transformation unit <b>70</b> has an image deformation transformer based on a plane projection to produce a perspective view. Alternatively, image transformation unit <b>70</b> has an image deformation transformer based on a spherical projection to produce a spherical projection. Advantageously, such spherical projection can be transformed to a plane projection with the aid of well-known methods, e.g., as described by Christopher Geyer and Kostas Daniilidis, “A Unifying Theory for Central Panoramic Systems and Practical Implications”, www.cis.upenn.edu, Omid Shakernia, et al., “Infinitesimal Motion Estimation from Multiple Central Panoramic Views”, Department of EECS, University of California, Berkeley, and Adnan Ansar and Kostas Daniilidis, “Linear Pose Estimation from Points or Lines”, Jet Propulsion Laboratory, California Institute of Technology and GRASP Laboratory, University of Pennsylvania which are herein incorporated by reference and attached as appendices hereto.

0064Now, once image <b>12</b>′ is recognized and transformed the orientation of jotting implement <b>10</b> is determined. This can be done in a number of ways. For example, when working with the spherical projection, i.e., with the spherical projection of unobstructed portions image <b>12</b>′, a direct three-dimensional rotation estimation can be applied to recover inclination angle θ and polar angle φ. For this purpose a normal view of jotting surface <b>12</b> is stored in memory <b>60</b>, such that it is available to transformation unit <b>70</b> for reference purposes. The transformation then yields the Euler angles of jotting implement <b>10</b> with respect to jotting surface <b>12</b> by applying the generalized shift theorem. This theorem is related to the Euler theorem stating that any motion in three-dimensional space with one point fixed (in this case the point where nib <b>20</b> is in contact with jotting surface <b>12</b> is considered fixed for the duration of each frame) can be described by a rotation about some axis. For more information about the shift theorem the reader is referred to Ameesh Makadia and Kostas Daniilidis, “Direct 3D-Rotation Estimation from Spherical Images via a Generalized Shift Theorem”, Department of Computer and Information Science, University of Pennsylvania, which is herein incorporated by reference.

0065Alternatively, when working with a plane projection producing a perspective view of unobstructed portions of image <b>12</b>′ one can use standard rules of geometry to determine inclination angle θ and polar angle φ. Several geometrical methods taking advantage of the rules of perspective views can be employed in this case.

0066One geometrical method is shown in <figref idref="DRAWINGS">FIG. 7A</figref>, where entire image <b>12</b>′ is shown for clarity (disregarding obstructed portions or filling them in with equations of edges <b>43</b>′ derived in the above step), two edges <b>43</b>′ are extended to vanishing point <b>86</b>. A connecting line Ψ from center <b>84</b> to vanishing point <b>86</b> is constructed. A line Σ in the plane of inclination angle θ is also constructed. Now, the angle between lines Ψ and Σ is equal to polar angle φ. Meanwhile, the length of line Ψ from center <b>84</b> to vanishing point <b>86</b> is inversely proportional to inclination angle θ. Preferably, a look-up table with values of Ψ corresponding to values of inclination angle θ is stored in memory <b>60</b> to facilitate rapid identification of angle θ during each frame. It should be noted that in order to keep track of the plane of inclination angle θ rotation of jotting implement <b>10</b> around optical axis <b>39</b> has to be known. This rotation can be established by providing a key e.g., in the form of a grip <b>90</b> on jotting implement <b>10</b>, as shown in <figref idref="DRAWINGS">FIG. 8</figref>. Grip <b>90</b> forces hand <b>16</b> of the user to hold jotting implement without rotating it around axis <b>39</b>.

0067Another geometrical method is shown in <figref idref="DRAWINGS">FIG. 7B</figref>, where entire image <b>12</b>′ is once again shown for clarity. Here, again, two edges <b>43</b>′ are extended to vanishing point <b>86</b>. A connecting line Ψ from center <b>84</b> to vanishing point <b>86</b> is constructed. A line Γ in the plane perpendicular to the plane of inclination angle θ is also constructed. Now, a line Π is constructed from vanishing point <b>86</b> and perpendicular to line Γ. The angle between lines Π and Ψ is equal to polar angle φ. Meanwhile, the length of line Π from intercept with line Γ to vanishing point <b>86</b> is inversely proportional to inclination angle θ. Preferably, a look-up table with values of Π corresponding to values of inclination angle θ is stored in memory <b>60</b> to facilitate rapid identification of angle θ during each frame. In this embodiment a key-mark <b>92</b> on array <b>52</b> or on some other part of jotting implement <b>10</b> is used to keep track of the plane perpendicular to the plane of inclination angle θ and it is indexed to an appropriate grip on the pen, e.g., as the one shown in <figref idref="DRAWINGS">FIG. 8</figref>.

0068Yet another geometrical method is shown in <figref idref="DRAWINGS">FIG. 7C</figref> based on entire image <b>12</b>′. Here, connecting line Ψ is constructed from center <b>84</b> to vanishing point <b>86</b> defined by two edges <b>43</b>′. A second vanishing point <b>94</b> is located by extending the other two edges <b>43</b>′. Second vanishing point <b>94</b> is then joined by line Ω with vanishing point <b>86</b>. Line Σ is now constructed from center <b>84</b> to line Ω such that it intersects line Ωat a right angle. The angle between lines Ψ and Σ is equal to polar angle φ and either the length of line Ψ or the length of line Σ (or even the length of line Ω) can be used to derive inclination angle θ. Once again, the use of corresponding look-up tables is recommended for rapid processing. It should be noted that this embodiment does not require the use of a key-mark or grip since rotation of jotting implement <b>10</b> around optical axis <b>39</b> (which is also the center axis of jotting implement <b>10</b>) does not affect this geometrical construction.

0069Still another geometrical method is shown in <figref idref="DRAWINGS">FIG. 7D</figref>. In this case corner angles α, β, γ and δ (when unobstructed) as well as the area integral of image <b>12</b>′ are used to determine θ and φ. Specifically, the values of corner angles α, β, γ and δ uniquely define angle φ. Likewise, the values of the area integral uniquely define θ. Corresponding look-up tables stored in memory <b>60</b> can be used for rapid processing and determination of angles θ, φ in this embodiment.

0070In the case where imaging optics <b>46</b> invert image <b>12</b>′ with respect to the physical orientation of jotting surface <b>12</b> image <b>12</b>′ needs to be inverted, as illustrated in <figref idref="DRAWINGS">FIG. 9</figref>. This inversion can be performed by transformation unit <b>70</b> at any point in time. For example, image <b>12</b>′ can be inverted before applying the above steps for determining θ and φ or after. If image <b>12</b>′ is not inverted, then no inversion needs to be performed.

0071A transformed and inverted (as necessary) image <b>12</b>″ is illustrated in <figref idref="DRAWINGS">FIG. 10</figref>. At this point vectors v″₁, v″₂and v″₃are re-computed. An additional vector v″_nfrom center <b>84</b> to a feature or landmark on an edge <b>43</b>″ is also shown. Such landmark on edge <b>43</b> of jotting surface <b>12</b> can be used instead of a corner for determining the physical coordinates of nib <b>20</b>. This is especially important when two corners are obstructed by the user or any object(s) located between jotting surface <b>12</b> and optical unit <b>30</b>.

0072At this point image <b>12</b>″ is corrected for rotations by angles θ and φ to obtain final transformed and corrected image <b>12</b>′″, as shown in <figref idref="DRAWINGS">FIG. 11</figref>. This is done by applying the appropriate inverse rotations to transformed (and inverted, as the case may be) image <b>12</b>″. (These inverse rotations correspond to Euler rotations in physical space of jotting implement <b>10</b> with respect to jotting surface <b>12</b>. Standard Euler transformation is described in any classical mechanics textbook such as Goldstein, <i>Classical Mechanics</i>).

0073Now the physical coordinates of nib <b>20</b> can be determined directly from vectors v′″₁, v′″₂, v′″₃and/or vector v′″_n. This function is performed by ratio computation unit <b>72</b>, which takes advantage of the fact that the proportions of image <b>12</b>′″ to jotting surface <b>12</b> are preserved.

0074Specifically, computation unit <b>72</b> employs the following ratios:

0075<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mrow><mfrac><msub><mi>x</mi><mn>1</mn></msub><msub><mi>x</mi><mn>2</mn></msub></mfrac><mo>=</mo><mfrac><msubsup><mi>x</mi><mn>1</mn><mi>′′′</mi></msubsup><msubsup><mi>x</mi><mn>2</mn><mi>′′′</mi></msubsup></mfrac></mrow><mo>,</mo><mi>and</mi></mrow></math></maths><maths id="MATH-US-00001-2" num="00001.2"><math overflow="scroll"><mrow><mfrac><msub><mi>y</mi><mn>1</mn></msub><msub><mi>y</mi><mn>2</mn></msub></mfrac><mo>=</mo><mrow><mfrac><msubsup><mi>y</mi><mn>1</mn><mi>′′′</mi></msubsup><msubsup><mi>y</mi><mn>2</mn><mi>′′′</mi></msubsup></mfrac><mo>.</mo></mrow></mrow></math></maths>

0076These values can be obtained from the vectors and the scaling factor due to the magnification M of imaging optics <b>46</b> can be used, as shown in <figref idref="DRAWINGS">FIG. 12</figref> as an additional cross-check and constraint to ensure that the values obtained are correct.

0077Jotting implements according to the invention admit of numerous other embodiments. For example, an alternative optical unit <b>100</b> employing a catadioptic system with a parabolic (or hyperbolic) mirror <b>102</b> and a lens <b>104</b> is shown in <figref idref="DRAWINGS">FIG. 13</figref>. The construction of optical unit <b>100</b> has to be altered to accommodate optical unit <b>100</b> on a jotting implement <b>108</b> (only top part shown) as in <figref idref="DRAWINGS">FIG. 14</figref>. In this embodiment a photodetector array <b>106</b> is placed at a distal end <b>109</b> of a jotting implement <b>108</b>. Support members <b>110</b> are extended with extensions <b>111</b> in this embodiment.

0078Jotting implement <b>10</b> can take advantage of features and landmarks other than corners and edges of a jotting surface <b>120</b>. For example, as shown in <figref idref="DRAWINGS">FIG. 15</figref>, jotting implement takes advantage of a feature <b>122</b> produced by the user. Feature <b>122</b> is in fact a letter “A” written by the user. In the present case a particularly easy-to-locate point on the letter (e.g., a point yielding high contrast for easy detection and tracking) is used for tracking and a vector v_ris constructed to this point from the origin of the Cartesian coordinate system. Jotting implement <b>10</b> also takes advantage of a landmark <b>124</b> located along an edge <b>126</b>. A vector v_sis constructed to landmark <b>124</b> from the origin. Finally, implement <b>10</b> uses a corner <b>128</b> of jotting surface <b>120</b> identified by corresponding vector v_q.

0079In this embodiment, during operation, edge detection algorithms described above and any other algorithms for detecting high-contrast points are applied to localize the lines and corners in the image and locate feature <b>122</b>, landmark <b>124</b> and corner <b>128</b>. Then, angles θ, φ are determined and the corresponding transformations applied to imaged vectors v′_q,v′_rand v′_sof the image of jotting surface <b>120</b>, as described above. The physical coordinates of nib <b>20</b> are determined from the transformed vectors.

0080Of course, a person skilled in the art will recognize that the number of features and landmarks tracked will generally improve the accuracy of determining physical coordinates of nib <b>20</b> on jotting surface <b>120</b>. Thus, the more landmarks and features are tracked, the more processing effort will be required. If real-time operation of jotting implement <b>10</b> is required, e.g., in cases where the jotting action is transmitted from jotting implement <b>10</b> to a receiver in real time, the number of features and landmarks should be limited. Alternatively, if the information jotted down can be downloaded by the user at a later time and/or no real-time processing is required, then more landmarks and features can be used to improve the accuracy with which the physical coordinates of nib <b>20</b> are determined. This will generally lead to an improved resolution of jotting surface <b>120</b>. It should also be kept in mind, that the features and landmarks have to provide absolute references, i.e., their positions on jotting surface <b>120</b> can not change in time. However, it should be remembered that the landmarks or features being used for determining the physical coordinates of nib <b>20</b> need not be the same from frame to frame.

0081It will be evident to a person skilled in the art that the present invention admits of various other embodiments.