Method and system for ink data generation, ink data rendering, ink data manipulation and ink data communication

The present invention relates to a method and system for reproducing a path of handwritten (freehand) stroke data and generating, rendering, manipulating, and transmitting ink data that describes the path with style.

Various handwriting input systems are known. The handwriting input system enables a user to input handwritten (freehand) data by using a pen-shaped device. For example, an electromagnetic resonance type pen tablet input system is known, which allows input of handwritten data including data related to pen pressure and pen tilt data. As a further example, an electrostatic type pen input system is known, which has a tool and a (tablet) sensor surface in the same way that a capacitance is generated between the finger and the sensor surface. Generates capacitance between. In addition, input systems that output relatively simple information, such as gesture information derived from a collection of measured positions, are also known.

Typically, handwritten or stroke (path or trace) data entered by a pen-shaped tool can be used within a single drawing application to generate raster data such as pixel data or image data. Is. Handwritten data or strokes generated by operating various types of devices and applications such as ink messaging, ink storage and recovery applications, email, photo annotations, remote video conferencing applications, etc. are shared between the various devices. There is a need for methods and systems that make this possible. Digital ink or ink data (hereinafter referred to as "ink data") is proposed for such needs. Raster data, such as direct pixel data or image data, typically generated according to the settings of the application used, is used to support the user's stroke input operation in the input device. On the other hand, the ink data is intermediate data, exists prior to the rasterization of stroke data, and has the form of vector data that can be used by various applications. Samples of ink data types are described in Non-Patent Documents D1 to D4 below.

<p><nplcit><text>W3C, Recommendation 20, September 2011, "Ink Markup Language (InkML)", URL: http://www.w3.org/TR/2011/REC-InkML-20110920/</text></nplcit><nplcit><text>Microsoft Corporation, etc., "Ink Serialized Format Specification", 2007, URL: http://download.microsoft.com/download/0/B/E/0BE8BDD7-E5E8-422A-ABFD -4342ED7AD886 / InkSerializedFormat (ISF) Specification.pdf</text></nplcit><nplcit><text>W3C Working Draft 11, February 2014, "Scalable Vector Graphics (SVG) 2", URL: http://www.w3.org/TR/SVG2/, W3C Recommendation, August 16, 2011, "Scalable Vector Graphics (SVG) 1.1 (2nd Edition)", URL: http://www.w3.org/TR/2011/REC-SVG11 -201110816 /</text></nplcit><nplcit><text>W3C, "HTML5 · A vocabulary and associated APIs for HTML and XHTML · W3C Recommendation 28 October 2014", URL: http: //www.w3.org/TR/html5/</text></nplcit><nplcit><text>Slate Corporation et al., "A Specification for an Ink Storage and Interchange Format", version 1.0, September 1996.</text></nplcit></p>

<p>Briefly, the data structures of InkML (Non-Patent Document 1) and ISF (Non-Patent Document 2) represent stroke data input by a pen-type device in a manner that can be shared between various applications. SVG (Non-Patent Document 3) makes it possible to draw a path defined by a control point input by a user as vector data regardless of what type of pen device is used as an input device. Provides web standards.</p><p>All of the ink data described in Non-Patent Documents 1 to 4 define the geometric information necessary for reproducing the trace (or path) formed by the movement of the pen or the finger. Such information is collectively referred to here as "stroke data".</p><p>Non-Patent Document 1 describes the most widely known ink data at present. Non-Patent Document 1 defines an object called "trace" as follows. That is, the "trace" is a basic element used to record the trajectory of the pen when the user writes digital ink.</p><p>For example, &lt;ink&gt; &lt;trace&gt; x1 y1, x2 y2, ..... xn yn &lt;/ trace&gt; &lt;/ ink&gt; describes the path of a stroke object that extends from point x1, y1 to point x2, y2. do.</p><p>Non-Patent Document 2 describes ink data generated by an ink function that can be used on an application of Microsoft® Windows®. Non-Patent Document 2 defines an object called "stroke" as follows. That is, "As mentioned earlier in the simple example, stroke is the most fundamental and important property in the ISF. Stroke contains the packet data that makes up the individual points in the stroke and is potentially. Includes other stroke-per-stroke properties as well. "</p><p>Non-Patent Document 3 does not assume pen input, but describes a standard of vector data supported by various browsers and drawing software. Non-Patent Document 3 defines information called "path" as follows. That is, it is defined as "a path can be filled, stroked, used as a clipping path, or represents the contour of a shape that can be any combination of these three." .. In SVG (Non-Patent Document 3), path objects are interpolated based on interpolation curves such as polybezier (cubic Bezier, quadratic Bezier) curves that are well known in the art.</p><p>For example, &lt;path stroke = "green" stroke-width = "5" d = "M100,200 C100,100 300,100 300,200" /&gt; starts at the start control point (100,200) and ends at the end control point (300,200). To describe. This path utilizes two control points (100,100) and (300,100) and has a path width of "5" and a color of "green".</p><p>Non-Patent Document 4 defines a class called "conic section" in which, for example, a quadratic curve command and a Bezier curve command can be used to generate an interpolated curve.</p><p>In this description, the term "stroke object" is used as a general term including "trace", "stroke", "path", and "canvas path" in Non-Patent Documents 1 to 4.</p><p>A stroke object is vector data information containing a set of point or control point coordinates whose data structure is collectively used to reproduce the trace (or path) formed by the movement of a pen or finger. According to various embodiments, the present invention generates and manipulates (eg, slices) ink data representing handwritten (freehand) stroke data on various applications or between different applications. ), Propose methods and systems for rendering and communicating. Each embodiment provides a technical solution that has not been available in the prior art of Non-Patent Documents 1-4. While the following description is constructed to disclose the four embodiments of the invention as a whole, various aspects of each embodiment, as will be apparent to those of skill in the art, are between the embodiments. It should be noted that can be combined, supplemented, exchanged, switched, modified in, thereby producing further embodiments. For example, the various methods and systems of each embodiment define ink data, as well as methods of generating, playing, drawing (rendering), manipulating, and transmitting ink data, as well as ink data. As for the structure (data objects and data formats), those described with respect to one or more other embodiments disclosed herein can be adopted.</p><p>Each of the following first to fourth embodiments deals with one or more aspects described below in various examples.</p>

<p>[Aspect 1] Introduction of operation objects that partially or wholly transform existing stroke objects on some computers.</p><p>According to one aspect, the present invention is directed to providing manipulation objects. The previously known ink data model described above is used only to process static stroke data in order to process one stroke object as one set. Semantics or syntax. including. Therefore, previously known ink data models do not have the ability to select or slice portions of stroke objects. Also, while the previously known ink data model allows stroke objects to be manipulated on one processor, multiple processors share the operations (eg, editing) performed on the stroke objects. Does not have the ability to make it possible.</p><p>FIG. 91 illustrates a "slice" object, which is an example of an operational object 270 according to an embodiment of the invention. A slice object 274 with the ability to manipulate (slice) part of the stroke object is created and transferred. In the illustrated example, a portion of one stroke object 9101 is sliced on one computer, and the operation data 9103 indicating the sliced portion is the same on the other computer as the stroke object 9101. Shared so that it can be manipulated. Deformation or manipulation (eg, slicing) of a portion of a stroke object will be described in detail later in the first and fourth embodiments of the invention. In order to share the edited and updated state of ink data among a plurality of computers, sharing one operation object 270 among them is a first, second, and fourth embodiment of the present invention. Will be described in detail later in the embodiment of.</p><p>[Aspect 2] To abstract the definition of pen event input information to absorb the difference of the device (and to make SVG more pen input-derived to improve the pen input expression ability of SVG).</p><p>According to a further aspect, the invention is directed towards abstracting handwritten input data in order to absorb any differences that exist between different input devices. This is achieved by abstracting the existing input attributes of the stroke, such as pressure and pen angle information, into higher level conceptual attributes defined within the new model. In general, the information that needs to be reproduced based on handwritten input data is at what angle the pen (stylus) was held, at what coordinate, what point was acquired, and how much pressure. It is not information about how the handwritten data was entered, such as whether was applied. Instead, the information that needs to be captured is vector data that can reproduce the "results" of such pen or drawing operations performed at a certain pen pressure, pen speed, etc.</p><p>Currently, it has the ability to accept finger input from a high-performance input device (for example, 9202C in Fig. 92) that can acquire pen pressure, pen angle, pen rotation angle data, etc., while acquiring pen pressure and pen tilt angle. There are a variety of handwriting input devices, such as widely used electrostatic tablets or other simpler devices that do not have the ability to do so (eg, 9202A in Figure 92). Therefore, transform any device-dependent attributes of handwritten input data into non-device-dependent abstracted vector data that can be used to reproduce the "result" of a pen event (9204 in Figure 92). Is desired. The ink data defined in such an abstracted format is finally put together in vector data to generate raster data (image data) as shown in 9208 of FIG. SVG1.1 (Non-Patent Document 3) discussed above defines vector data and is shown in 9206 in Figure 92. SVG1.1 (Non-Patent Document 3) does not allow the width, color, and transparency (opacity) of the stroke to be changed or adjusted, and as a result, reproduces the "result" of the pen event. Not particularly suitable. The SVG also contains data other than the path coordinate data of the stroke object, such as the control points used to generate the Bezier curve, and therefore in various applications 9220 other than specialized drawing applications. Not suitable for use.</p><p>In addition to generating raster image data (9208 in Figure 92), grouping ink data into a more abstract format within a vector is also desirable for use within signature authentication and annotation applications. That is. In doing so, it is preferable that the abstraction is not too image-derived, but it should be an abstraction attribute that can be used to define ink data in both raster and vector formats. It is the present invention to abstract Type 1 (including pressure) and Type 2 (not including pressure) device-dependent pen event data 9202 into generalized ink data, ie, intermediate data 9204 in FIG. Will be described in detail below in the first and third embodiments of.</p><p>[Aspect 3] To extend the life cycle of ink data by separating the language (information model) from the format.</p><p>For example, the content of raster data, such as digital photography, is often used by multiple devices and applications, not just a single device or application, and all in a chain in a particular "ecosystem." Shared or transferred between (although they can be processed in various formats like JPEG, GIF, TIFF, etc.). These various formats can be used because raster data contains a general information model that conceptually describes a set of pixel values.</p><p>According to a further aspect, the present invention is directed to facilitating the exchange and transfer of ink data between different formats, based on the adoption of a common language (stroke language (SL)). The stroke language (SL) is an information model that defines the meaning of the ink data of the present invention and is in conflict with the format of the ink data. That is, for example, ink data defined by abstracted attributes is processed into various raster image formats (PNG, JPEG, etc.), replaced by various vector graphic formats (SVG, InkML, HTML5, etc.), and has a stroke structure. Produced in various stream formats (ISF, InkML, etc.) that define. FIG. 93 conceptually illustrates this aspect of the invention. To add flexibility to the output format type as well as the input format type and adapt it to a variety of output and input format types, the common language (or the information model that defines the common language) is at the level of the device driver that produces the language. , It is preferable that the generated language is located in the middle of the output level when it is output to a file or packet. In particular, the ink data processing unit 100 according to various embodiments of the present invention includes an ink data generation unit 120 that generates ink data based on an abstracted language (stroke language) as two separate components. , Ink data shaping unit 140 that handles the input and output of ink data. Since the ink data generation function and the ink data shaping function for input / output are separated, the ink data processing unit 100 uses ink data to expand the use of ink data among various devices. Suitable for use as a fundamental element of the ecosystem. This aspect of the invention will be described in detail below in a fourth embodiment.</p><p>These three aspects of the invention depicted in FIGS. 91-93 will be discussed again after describing the first to fourth embodiments of the invention below.</p>

<figref num="1">It is a figure explaining the whole system which ink data is generated and used by various embodiments of this invention.</figref><figref num="2">It is an entity-relationship diagram of the ink data structure suitable for use in the embodiment of the present invention.</figref><figref num="3A">It is a figure explaining the stroke object defined by a plurality of point objects.</figref><figref num="3B">It is a figure explaining one of the two rendering (drawing) results of the stroke object shown in FIG. 3A by two different drawing style objects.</figref><figref num="3C">It is a figure illustrating the other one of the two rendering (drawing) results of the stroke object shown in FIG. 3A by two different drawing style objects.</figref><figref num="4A">It is a figure explaining the operation of the "selection" operation object used for selecting and transforming (for example, rotating) a stroke object.</figref><figref num="4B">It is a figure explaining the operation of the "slice" operation object used for slicing a stroke object.</figref><figref num="5">It is a functional block diagram of the ink data processing apparatus according to 1st Embodiment of this invention.</figref><figref num="6">It is a functional block diagram of the ink data processing unit (100) of the ink data processing apparatus of FIG. 5 according to the first embodiment of the present invention.</figref><figref num="7">It is a functional block diagram of the stroke object operation unit (122) of the ink data processing unit of FIG. 6 according to the first embodiment of the present invention.</figref><figref num="8">It is a figure explaining the process executed from the point "A" to "D" in the stroke object operation part of FIG.</figref><figref num="9">It is a flowchart explaining the sample routine executed by the ink data processing part of FIG.</figref><figref num="10">It is a figure explaining the sample of the stroke file format (SFF) file written in the interface definition language (IDL) which can be output to the point "E" of the ink data processing apparatus of FIG.</figref><figref num="11">It is a figure explaining the sample of the stroke object file which is written in the stroke file format (SFF) and can be output to the point "E" of the ink data processing apparatus of FIG.</figref><figref num="12">Three messages in stroke message format (SMF) that can be output to point "F" of the ink data processing device of FIG. 5 and one packet that can be output to point "G" of the ink data processing device of FIG. It is a figure explaining.</figref><figref num="13A">A stroke object that has undergone a catmal-rom curve interpolation operation that can be output to the point "D" of the ink data processing device of FIG. 5 and input to the graphic processing unit (300) or the ink data shaping unit (140) will be described. It is a figure.</figref><figref num="13B">It is a figure explaining the rendering (display) result of the stroke object of FIG. 13 which can be output from the graphic processing unit (300) to the point "H" of the ink data processing apparatus of FIG.</figref><figref num="14">It is a flowchart of the slice operation applied to the stroke object by 1st Embodiment of this invention.</figref><figref num="15A">It is a figure explaining the process of determining the single (intermediate) intersection (P_intersect_Mid) between two strokes performed in step S1409 of FIG.</figref><figref num="15B">It is a figure explaining the process of obtaining two (edge) intersections (P_intersect_L and P_intersect_R) between a wide stroke object to be sliced and an existing stroke object, which is executed in step S1413 of FIG.</figref><figref num="16A">It is a figure explaining the first of the two slices made by slicing a stroke object, which is derived in step S1415 of FIG.</figref><figref num="16B">It is a figure explaining the data structure of the parameter which defines the 1st slice shown in FIG. 16A.</figref><figref num="16C">It is a figure explaining the represented path of the newly generated first stroke object.</figref><figref num="17A">It is a figure explaining the second of the two slices made by slicing a stroke object, which is derived in step S1415 of FIG.</figref><figref num="17B">It is a figure explaining the data structure of the parameter which defines the 2nd slice shown in FIG. 17A.</figref><figref num="17C">It is a figure explaining the represented path of the newly generated second stroke object.</figref><figref num="18A">FIG. 16 is a diagram illustrating a process of obtaining a new end point for the first slice shown in FIG. 16A and a process of obtaining a new start point for the second slice shown in FIG. 17A.</figref><figref num="18B">It is a figure explaining the data structure of the parameter which defines a whole segment object by 1st Embodiment of this invention.</figref><figref num="19">It is a diagram illustrating a sample file containing two newly generated stroke objects, written in the Stroke File Format (SFF), representing two slices made by slicing one stroke object.</figref><figref num="20">It is a detailed flowchart of the ink data transmission process executed in step S1422 of FIG.</figref><figref num="21A">It is a figure illustrating type A of the various transmission message types that can be used for the transmission of ink data in connection with the slicing operation.</figref><figref num="21B">FIG. 6 illustrates Type B of the various transmission message types that can be used for the transmission of ink data in connection with the slicing operation.</figref><figref num="21C">It is a figure illustrating type C of the various transmission message types that can be used for the transmission of ink data in connection with the slicing operation.</figref><figref num="21D">FIG. 6 illustrates Type D of the various transmission message types that can be used for the transmission of ink data in connection with the slicing operation.</figref><figref num="22">FIG. 3 is a functional block diagram of an ink data receiving device configured to remotely receive ink data via a network according to the first embodiment of the present invention.</figref><figref num="23">It is a flowchart explaining the reception process of the operation (slice) object on the receiving side by 1st Embodiment of this invention.</figref><figref num="24A">It is a figure explaining the technical problem which is related to the prior art.</figref><figref num="24B">It is a figure explaining the technical problem which is related to the prior art.</figref><figref num="25">It is an entity-relationship diagram of an ink data structure suitable for use in the second embodiment of the present invention.</figref><figref num="26">It is the whole communication system diagram suitable for use in 2nd Embodiment.</figref><figref num="27">It is a figure explaining the transmission device (10-1) of the communication system shown in FIG.</figref><figref num="28">It is a figure explaining the sample of the recording format suitable for storing the updated state of the common drawing area (canvas) in the 2nd Embodiment of this invention.</figref><figref num="29">It is a figure explaining the relay server (10-2) of the communication system shown in FIG.</figref><figref num="30A">It is a figure explaining the communication parameter among the communication parameter, the drawing parameter, and the user policy parameter which jointly draws or defines the communication and graphic environment of a transmission apparatus.</figref><figref num="30B">It is a figure explaining the drawing parameter among the communication parameter, drawing parameter, and user policy parameter which jointly draws or defines the communication and graphic environment of a transmission apparatus.</figref><figref num="30C">It is a figure explaining the user policy parameter among the communication parameter, the drawing parameter, and the user policy parameter which jointly draws or defines the communication and graphic environment of a transmission apparatus.</figref><figref num="31">It is a figure explaining the receiving apparatus (10-3) of the communication system shown in FIG.</figref><figref num="32">It is a sequence diagram explaining the ink data communication performed between the transmission device (10-1), the relay server (10-2), and the receiving device (10-3) according to the second embodiment of the present invention. ..</figref><figref num="33">It is a flowchart which shows the sample of the process of finding the specified unit T for transmitting ink data.</figref><figref num="34">It is a figure explaining the sample of the transmission format of the communication packet and the message suitable for transmitting (communication) ink data between a plurality of devices according to the 2nd Embodiment of this invention.</figref><figref num="35A">It is a figure explaining the communication packet used in the communication protocol including a data retransmission scheme.</figref><figref num="35B">It is a figure explaining the communication packet used in the communication protocol which does not include a data retransmission mechanism.</figref><figref num="36A">It is a sequence diagram which shows the sample of the data retransmission processing using a serial ID suitable for use in the communication protocol which does not include a data retransmission mechanism.</figref><figref num="36B">It is a sequence diagram which shows the data transmission process which is suitable for use in the communication protocol which does not include a data retransmission mechanism, and data retransmission is not executed.</figref><figref num="36C">It is a figure explaining the method of calculating the control position of a message.</figref><figref num="36D">It is a figure explaining the method of calculating the control position of a message.</figref><figref num="36E">It is a figure explaining the method of calculating the control position of a message.</figref><figref num="36F">It is a figure explaining the interpolation (error concealment) processing using the control position calculated in FIGS. 36C to 36E for use in the sequence of FIG. 36B.</figref><figref num="36G">It is a figure explaining the interpolation (error concealment) processing using the control position calculated in FIGS. 36C to 36E for use in the sequence of FIG. 36B.</figref><figref num="37">It is a sequence diagram for demonstrating the ink data communication when the update request of the common drawing area issued by a transmission device is rejected by a relay server.</figref><figref num="38">The sequence diagram shown in FIG. 32 in which the relay server receives the fragmented data of the stroke object from the transmission device and combines all the fragmented data for each stroke object in order to relay this to the receiving device. This is the first modification of.</figref><figref num="39A">It is a format for use in the first modification shown in FIG. 38, and is a data transmission format in which all ink data for one stroke object is included in a combined state.</figref><figref num="39B">It is a figure explaining the sample of the data transmission format of DATA_INK_ALLOS_REQ which is a message requesting the stroke object data to be applied to the whole stroke when the stroke ID is known.</figref><figref num="40">This is the second modification of the sequence diagram shown in FIG. 32, in which the stroke object is transmitted "as is" (ie, unfragmented) from the transmitting device to the receiving device via the relay server.</figref><figref num="41">The third of the sequence diagram shown in FIG. 32, in which the relay server receives the stroke object from the transmission device and fragmentes the received stroke object into a plurality of fragmented data for relaying to the receiving device. This is a modified example of.</figref><figref num="42">It is a figure explaining the concept of the stroke start point peculiar to a user in relation to the origin of a common drawing area.</figref><figref num="43">It is a figure explaining the 2nd Embodiment concerning the transmission of the ink data in the semantic unit larger than the stroke unit.</figref><figref num="44">It is a figure explaining the 2nd Embodiment concerning the transmission of the ink data in the semantic unit larger than the stroke unit.</figref><figref num="45">According to the third embodiment of the present invention, on the one hand, the data input / output of the ink data processing unit and the method of generating the data will be described, and on the other hand, the data input / output of the ink data processing unit will be reproduced. It is a figure explaining the thing by the method.</figref><figref num="46A">It is a figure explaining the 1st of 3 configuration examples of the ink data generation method by the 3rd Embodiment of this invention.</figref><figref num="46B">It is a figure explaining the second of three configuration examples of the ink data generation method by the 3rd Embodiment of this invention.</figref><figref num="46C">It is a figure explaining the 3rd of 3 configuration examples of the ink data generation method by 3rd Embodiment of this invention.</figref><figref num="47A">It is a figure explaining the 1st of 2 configuration examples of the ink data reproduction method by 3rd Embodiment of this invention.</figref><figref num="47B">It is a figure explaining the second of the two configuration examples of the ink data reproduction method by the 3rd Embodiment of this invention.</figref><figref num="48A">It is an entity-relationship diagram of the ink data structure according to the ink data model (stroke language (SL)) according to the third embodiment of the present invention.</figref><figref num="48B">It is a detailed entity-relationship diagram of the ink data structure shown in FIG. 48A.</figref><figref num="48C">A graphic representation of a stroke object.</figref><figref num="48D">It is a figure explaining a canvas object.</figref><figref num="48E">It is a figure explaining a metadata object.</figref><figref num="48F">It is a figure explaining the drawing result of three different drawing style objects seen on the screen.</figref><figref num="48G">It is a figure explaining the operation of the operation (slice) object.</figref><figref num="48H">It is a figure explaining the operation of the operation (erase) object.</figref><figref num="48I">It is a figure explaining the operation of the operation (selection and transformation) object applied to an existing stroke object.</figref><figref num="48J">It is a figure explaining the syntax of the ink data structure arranged in the stroke file format (SFF) according to the 3rd Embodiment of this invention.</figref><figref num="48K">It is a figure explaining the syntax of the ink data structure arranged in the stroke file format (SFF) according to the 3rd Embodiment of this invention.</figref><figref num="48L">It is a figure explaining the syntax of the ink data structure arranged in the stroke file format (SFF) according to the 3rd Embodiment of this invention.</figref><figref num="49">It is a functional block diagram of the ink data processing part by the 3rd Embodiment of this invention.</figref><figref num="50A">FIG. 5 is a flow chart illustrating a process executed in the "ink data generation unit" of FIG. 49 and outputting radius and alpha information as attributes of a point object according to a third embodiment of the present invention.</figref><figref num="50B">It is a diagram illustrating a sample GUI of an application or operating system that can be used to define contextual information for pen event data.</figref><figref num="51">It is a figure explaining the process of deriving the speed in step S1205 of FIG. 50 according to the 3rd Embodiment of this invention.</figref><figref num="52">It is a flow diagram explaining the process of deriving the radius in step S1207 of FIG. 50 according to the 3rd Embodiment of this invention.</figref><figref num="53">It is a flow diagram illustrating the definition of the "phase" of the stroke as used in step S1207_01 of FIG. 52 according to the third embodiment of the present invention.</figref><figref num="54">It is a graph explaining three functions for deriving a radius from a parameter (velocity) as used in step S1207_05 and step S1207_07 of FIG. 52 according to the third embodiment of the present invention.</figref><figref num="55">It is a graph explaining the process of deriving the alpha which shows transparency (or opacity) in step S1209 of FIG. 50 by the 3rd Embodiment of this invention.</figref><figref num="56">A graph illustrating two functions for deriving alpha (transparency / opacity) from a parameter (velocity) as used in steps S1209_05 and S1209_07 of FIG. 55, according to a third embodiment of the invention. Is.</figref><figref num="57">It is a flow diagram explaining the process of shaping the radius and alpha values into an ink data format (data structure) together with X and Y coordinates according to the 3rd Embodiment of this invention.</figref><figref num="58">It is a figure explaining the Example of Step S1411 and Step S1413 of FIG. 57 according to the 3rd Embodiment of this invention.</figref><figref num="59">It is a figure explaining the conversion of the floating data type to the integer data type used in step S1411 and step S1413 of FIG. 57 according to the 3rd Embodiment of this invention.</figref><figref num="60">It is a figure explaining the efficiency improvement of compression as a result of the data type conversion of FIG. 59 by the 3rd Embodiment of this invention.</figref><figref num="61">FIG. 5 is a flow chart illustrating a process that can be performed by the "ink data shaping unit" shown in FIG. 49 to compress the generated ink data according to the third embodiment of the present invention.</figref><figref num="62">FIG. 5 is a flow chart illustrating a process executed (instead of FIG. 52) in the "ink data generation unit" shown in FIG. 49 in order to output radius information as an ink data attribute according to the third embodiment of the present invention. Is.</figref><figref num="63">FIG. 5 is a flow chart illustrating a process executed (instead of FIG. 55) in the "ink data generation unit" shown in FIG. 49 in order to output alpha information as an ink data attribute according to the third embodiment of the present invention. Is.</figref><figref num="64">It is a figure explaining the relationship between the ink data processing part and various applications according to the 3rd Embodiment of this invention.</figref><figref num="65">According to the third embodiment of the present invention, the ink data reproduction process for extracting (regenerating) the radius and alpha information in the ink data together with the X and Y coordinates, and the request from the drawing application for the extracted information and data. It is a flow diagram explaining that it outputs according to.</figref><figref num="66">It is a figure explaining the Example of Step S2011 and Step S2013 of FIG. 65 according to the 3rd Embodiment of this invention.</figref><figref num="67">It is a flow diagram explaining the drawing process which applies the selected drawing style object to the stroke object of the drawing target by the 3rd Embodiment of this invention.</figref><figref num="68">It is a figure explaining the drawing rendering example as a result of the input of the ink data generated based on the attenuation (damping) function shown in FIG. 54 according to the 3rd Embodiment of this invention.</figref><figref num="69">It is a figure explaining the drawing rendering example as a result of the input of the ink data generated based on the power function shown in FIG. 56 by the 3rd Embodiment of this invention.</figref><figref num="70">In the figure illustrating a drawing rendering example as a result of input of ink data generated based on both the decay function shown in FIG. 54 and the power function shown in FIG. 56 according to the third embodiment of the present invention. be.</figref><figref num="71">It is a figure explaining the drawing rendering example which shows the effect of the other functions (S-shaped and periodic functions) of FIG. 54 according to the 3rd Embodiment of this invention.</figref><figref num="72">It is a figure explaining the drawing rendering example which shows the effect of using the special value as the radius of the start point and the end point of the stroke of the drawing target by the 3rd Embodiment of this invention.</figref><figref num="73">It is a figure explaining the whole system in which ink data is used according to the 4th Embodiment of this invention.</figref><figref num="74">It is a functional block diagram of the ink data processing part by 4th Embodiment of this invention.</figref><figref num="75">It is a more detailed functional block diagram of the ink data processing unit according to the 4th Embodiment of this invention shown in FIG. 74.</figref><figref num="76">It is a functional block diagram of the stroke object operation part (122) shown in FIG. 75.</figref><figref num="77A">It is a flowchart explaining the method of generating a stroke object.</figref><figref num="77B">It is a flowchart explaining the method of generating a stroke object.</figref><figref num="78">It is a functional block diagram of the metadata object operation part (124) shown in FIG. 75.</figref><figref num="79">It is a flowchart explaining the method of generating a metadata object.</figref><figref num="80">It is a functional block diagram of the rendering (drawing style) object operation part (126) shown in FIG. 75.</figref><figref num="81">(Drawing) It is a flowchart explaining the method of deriving a style object and its cascading property.</figref><figref num="82">It is a functional block diagram of the operation object operation part (128) shown in FIG. 75.</figref><figref num="83A">It is a flowchart explaining the method of deriving the operation object such as a slice object.</figref><figref num="83B">It is a flowchart explaining the method of deriving the operation object such as a slice object.</figref><figref num="84">It is a functional block diagram of the ink data shaping part (140) shown in FIG. 75.</figref><figref num="85">It is a flowchart explaining the process executed in the ink data shaping part shown in FIG. 84.</figref><figref num="86">It is a flowchart explaining the method of outputting the data of a stroke file format (SFF).</figref><figref num="87">It is a flowchart explaining the method of outputting the data of JPEG format.</figref><figref num="88">It is a flowchart explaining the method of outputting the data of a stroke messaging format (SMF).</figref><figref num="89">It is a functional block diagram explaining the input process of the data (SFF / JPEG and SMF) output in various file formats and messaging formats.</figref><figref num="90A">It is a flowchart of a process for interpreting and playing an object placed in an SFF file.</figref><figref num="90B">It is a flowchart of the process for interpreting and playing an object based on the input in InkML.</figref><figref num="90C">It is a flowchart of the process of receiving and executing the operation (slice) object by SMF.</figref><figref num="91">It is a figure explaining the effect of using the ink data processing apparatus (101) of FIG. 75 with respect to side surface 1.</figref><figref num="92">It is a figure explaining the effect of using the ink data processing apparatus (101) of FIG. 75 with respect to side surface 2.</figref><figref num="93">It is a figure explaining the effect of using the ink data processing apparatus (101) of FIG. 75 with respect to side surface 3.</figref>

Detailed Description Through the various embodiments described herein, the following terms as used herein are as follows, unless specifically defined in a particular context that they may apply only to a particular embodiment. Has the meaning of.

"Pen event data" (input 1) means data input based on the user's handwriting drawing operation. The pen event data may be raw data input by a given input device or may be data obtained by processing the raw data. All pen event data is expected to have at least the position information (eg, XY coordinates) of each stroke drawn by the user, while being device dependent and attributes that identify each type of input device (eg, XY coordinates). , Pen pressure data, pen rotation or tilt angle data). For example, the pen event data received from the input device that can detect the pen pressure is different from the pen event data received from the input device that cannot detect the pen pressure.

"Ink data" (200) means a set of objects derived from pen event data. The ink data 200 captures the paths (strokes) formed based on the pen event data, and is a vector data format that is a kind of intermediate data that describes the properties (color, pen type, etc.) of each path. Take. The ink data 200 is device-independent and can be used by any device that supports attributes such as pen pressure and / or pen rotation / tilt angle, as well as devices that do not support these attributes. The ink data 200 according to the embodiment of the present invention includes a stroke object 210, a metadata object 250, a drawing style object 230, and an operation object 270. The ink data 200 will be described in detail below with reference to FIGS. 2, 3A-4B, 25, 48A-48I, and the like.

The "stroke object" (210) is a type of object or data contained in the ink data 200. The "stroke", "path", "trace", and "canvas path" described in the above-mentioned non-patent documents 1 to 4 are all stroke objects 210. The stroke object 210 represents the shape of a path (stroke) acquired by a user operation in the input device.

A "metadata object" (250) is a type of object contained within the ink data 200, such as author, pen ID, locally acquired date and time information, GPS location information, and so on. Contains non-drawing related information that describes 210.

A "drawing style object" (230) is a type of object contained in the ink data 200, and its shape is formed when the stroke object 210 is rendered (drawn, expressed, rasterized) on the screen. Contains information needed to control (stroke width, stroke style / pattern) and color. In a nutshell, the drawing style object controls the rendering of the stroke object 210.

The "operation object" (270) is a kind of object included in the ink data 200, and an operation / change operation (for example, a slice operation) is performed on all or a part of each of one or more existing stroke objects 210. Run. The application of the operation object 270 to a part of the stroke object will be described in detail later in the first embodiment.

The "stroke language (SL)" is an information model that defines the attributes and meanings of various objects that make up the ink data 200.

The "stroke file format (SFF)" is a type of recording format in which the ink data 200 to be output is serialized. Details of the SFF will be described later with reference to FIGS. 10, 11, 19, 19, 28, 48J, 48K, 48L, 57, 65, and 86.

"Recording format" means a format suitable for making the ink data 200 persistent, such as the SFF format and the SVG format. The ink data 200 in the recording format is recorded in a storage device (HDD, network storage, etc.) as a file or a database. And from there, its serialized data stream can be pulled and deserialized.

A "stroke message format (SMF)" is a type of message transmission format contained in a transmission packet or frame and is intended to be used when transmitting ink data 200 using a defined transmission protocol. Details of the SMF will be described later with reference to FIGS. 12, 21, 34, 35A, 35B, 39A, 39B, and 88.

"Transmission format" means a message format suitable for transmitting (sending a message) ink data 200 over a network, such as the SMF format.

"Ink data" means a rasterized image such as a GIF or JPEG image, includes pixel data, and can be generated (drawn) based on the ink data 200. It is not possible to return non-intermediate image format data to ink data 200.

The following terms are used to describe each of the main structures and components used to process the ink data 200, eg, as shown in FIG.

The "ink data processing unit" (100) means a processor that generates, stores, and processes ink data 200. In the present specification, the ink data processing unit used to generate the ink data 200 based on the pen event data and to arrange the ink data 200 in the defined format is represented by the reference numeral 100T. On the other hand, the ink data processing unit used in the computer to reproduce the ink data 200 generated and arranged in the defined format is represented by the reference numeral 100R. Details of the ink data processing unit 100 will be described later with reference to FIGS. 5 and 75, and also with reference to FIGS. 6, 22, 27, 31, 49, and 74. The ink data processing unit 100 generally receives / includes / receives the following three types of information. The first is pen event (input type) information (input 1), the second is context information (input 2), and the third is operation information (input 3).

The "ink data generation unit" (120) generates ink data 200 and develops ink data 200. In the present specification, the ink data generation unit that generates ink data 200 based on the input signal received from the input sensor is represented by the reference numeral 120T, and the ink data 200 that has already been generated is expanded and restored in the memory. The data generation unit is represented by the reference numeral 120R.

The "ink data shaping unit" (140) processes ink data 200 arranged in a recording format or transport format for input and output purposes. In the present specification, the ink data shaping unit that outputs the ink data 200 in the defined format is represented by the reference numeral 140T, and the ink data shaping unit that accepts the input of the ink data 200 in the defined format is represented by the reference numeral 140R.

First Embodiment The first embodiment of the present invention is directed to generating, rendering, manipulating (eg, slicing), and communicating stroke objects 210 constituting the ink data 200. It is a thing. In particular, the operation of a portion of the stroke object 210 will be described with the sharing (transmission) of the operation operation among multiple processors, as described above with reference to FIG.

Background of the First Embodiment The stroke object described in Non-Patent Documents 1 to 4 has a plurality of points or a plurality of control points necessary for generating an interpolation curve or a path by using a predetermined interpolation curve algorithm. include.

Non-Patent Documents 1 and 2 do not specify any special interpolation curve algorithm. In other words, any suitable interpolation curve algorithm can be used.

Non-Patent Documents 3 and 4 use a polybezier (cubic Bezier) curve. In the polyvezier In addition, at least one other control point is needed to determine the curvature of the curved segment between points Pi and point Pi + 1. The control point is different from either Pi or Pi + 1 and is not on (ie, outside the curve) the curve containing the curve segment. For example, a cubic Bezier curve requires two control points located outside the curve to define the curve segment.

For example, the XML notation &lt;stroke-width = "5" d = "M 100, 200 C100, 100 300,100 300,200" /&gt; used for cubic Bezier curves uses the starting point (100,200) as the control point. The end point (300,200) is used as another control point, and two control points (100,100) and (300,100) (between the start and end points and outside the curve segment) define the curve segment. It means that it is used for.

In recent years, the SVG Working Group, which is in charge of SVG (Non-Patent Document 3) at W3C, has been discussing the possibility of using the Catmull-Rom curve for the interpolated curve. Unlike the Polyvezier curve, the Catmull-Rom curve has no control points outside the curve (ie, not on the curve). The Catmull-Rom curve consists of a start point (Pi), an end point (Pi + 1), a point "before" the start point (Pi-1), and a point "behind" the end point (Pi + 2). Each curve segment is defined by one control point. All control points are on the curve. In other words, the Catmar-Rom curve passes through all its control points (although each curve segment requires two control points "before" and "behind" the curve segment. Since the two points at the very end of the stroke object do not define a curve segment).

Overview of the First Embodiment FIGS. 24A and 24B describe the technical problems encountered by the definition of ink data in the prior art described in Non-Patent Documents 1-5. Figure 24A shows the curve 2401S represented by the stroke object and to which the slice operation 2403 is applied. The stroke object representing the curve 2401S contains a set of point coordinates (p1 to p10) input via the input sensor.

In Figure 24A, the slice operation 2403 is applied between point coordinates p5 and p6 along the dividing line passing through the intersection 2405 to slice the curved segment of the stroke object. FIG. 24B shows two split curves 2411S1,2415S2 resulting from slice operation 2403. The curve 2411S1 contains the point coordinates p1 to p5, and the curve 2415S2 contains the point coordinates p6 to p10. As shown, the solid dividing curve 2411S1 ends at point coordinate p5 and is therefore shorter than the actual dividing curve extending to intersection 2405. Similarly, the split curve 2415S2 shown by the solid line starts at the point coordinate p6 and is shorter than the actual split curve starting at the intersection 2405. In Figure 24B, the incomplete curve segment 2413, shown by the dashed line, shows the portion of the curve lost by slice operation 2403.

Control to add new control points located at intersection 2405 and to define newly generated partial curve segments 2413 between p5 and intersection 2405 and between intersections 2405 and p6, respectively. It is possible to add points. However, calculating the location of new control points to represent the exact shape up to the end point 2405 of the partial curve segment 2413 is computationally intensive and not a trivial task. For example, if an interpolated curve such as a cubic Bezier curve is used, it is necessary to calculate two control points outside the curve (or path) in order to define a new segment. When using the Catmull-Rom curve, it is necessary to calculate two control points through the curve to define a new segment, and as a result, all previous to maintain the actual curvature. The control points will have to be recalculated in a potato-like manner. Both types of computations are highly complex and time consuming, making it impossible to support real-time slicing operations in graphics or drawing applications.

A method and system that allows the user to slice the stroke object that forms the ink data so that each of the two segments resulting from the slicing operation represents the actual split curve cut from the original stroke object. There is a need for. And it is preferable that such methods and systems do not require the calculation of the position of the new control points used for the interpolated curve. Such computations are complex and often computationally intensive, making it impossible to support real-time applications.

According to one aspect, the invention provides a method and system for generating, drawing, manipulating (eg, slicing) and transmitting ink data containing a stroke object 210. In this method and system, the stroke object 210 includes or is associated with range information that defines a particular portion of the stroke object 210 to be rendered. If the range information indicates full display, the entire stroke object 210 is displayed, and if the range information indicates partial display, one or both ends of the stroke object 210 are partially designated as not displayed. The object. When the slicing operation is applied to the original stroke object 210 and two new stroke objects 210 are generated, the first new stroke object 210 rasterizes the end of the first new stroke ( Or it is associated with range information that specifies a new "end" point that will be the point (or rendered or displayed as a result). Similarly, the second new stroke object 210 is associated with range information that specifies a new "start" point at which the display of the second new stroke begins. Both the first and second new stroke objects 210 maintain the same structure and control points (even if partial) as the original stroke objects, and therefore the first and second stroke objects. The display of the new stroke object 210 will exactly follow the shape of the original stroke object 210. Moreover, the calculation of a new control point becomes unnecessary.

According to another aspect, a method and system for outputting the stroke object 210 to form the ink data 200 is provided. A stroke object contains multiple point objects that represent multiple coordinate positions. At least some of these point objects serve as control points used to generate the interpolated curve segments, forming the path of the stroke object 210 as a whole. The stroke object 210 further includes range information that defines a start point in the start curve segment where the stroke object 210 begins to display and an end point in the end curve segment where the stroke object 210 ends display. When the original stroke object 210 is sliced to give birth to two new stroke objects 210, each of the two new stroke objects 210 has its own range information, a parameter that indicates its own start point and its own end point. It also contains a partial set of point objects that are exactly the same as those of the original stroke object 210.

According to another aspect, a method and a system for drawing (rendering or displaying) the ink data assembled as described above are provided. In this method and system, each stroke object 210 includes a plurality of point objects and range information. At least some of the point objects are control points used to generate the interpolated curve segment. This method and system generate a path for stroke object 210 by interpolating curved segments based on control points, and display a portion of stroke object 210 specified by range information on the screen. Draw each stroke object 210. In other words, this method and system start displaying the stroke object 210 at the start point specified in the range information and stop displaying the stroke object 210 at the end point specified in the range information. ..

Further aspects provide methods and systems that allow the user to slice the stroke object 210 of the ink data assembled as described above. When the user performs a slicing operation on the stroke object 210, this method and system calculates the position of the intersection of the slicing path and the stroke object 210. (See 2405 in Figure 24A). This method and system creates two new stroke objects 210, a first stroke object 210 and a second stroke object 210, as a result of the slicing operation. The first stroke object 210 includes a first set of point objects and first range information indicating a display start point and a display end point. In this case, the display end point is derived from the calculated intersection. The second stroke object 210 includes a second set of point objects and second range information indicating a display start point and a display end point. In this case, the display start point is derived from the calculated intersection. Typically, the first range information of the first stroke object 210 maintains the same display start point as that of the original stroke object 210, and the second range information of the second stroke object 210 is the original. Maintains the same display end point as that of stroke object 210 in.

The ink data assembled as described above is ink so that multiple users can share the experience of drawing and manipulating (slicing) strokes on a common drawing area (common "canvas") in real time. It can be easily communicated between various devices or applications capable of processing data.

According to the various methods and systems of the invention, the ink data assembled as described above is generated, output and displayed on the screen to allow the user to slice the stroke object 210. Used for and shared between different users. Using range information to display only part of the actual curve segment contained in the stroke object 210 is a sliced (newly generated) that traces the shape of the original stroke object 120 exactly to its end. ) Allows the stroke object 210 to be displayed. Also, since the sliced stroke object 210 maintains the same structure and control points (even if only partially) as the original stroke object 210, new control points are calculated or recalculated in connection with the slicing operation. You don't have to.

The methods and systems of the present invention are ink data of various types of curve interpolation such as the known Polyvezier curve (cubic bezier, quadratic bezier) algorithm and the Catmal-Rom curve algorithm in which the curve segment is known. It can be applied to the ink data interpolated by the algorithm.

Description of the First Embodiment FIG. 1 is a diagram illustrating an entire system in which the ink data 200 is used according to the embodiment of the present invention. In FIG. 1, the cloud unit 1 whose outline is drawn by a broken line represents an infrastructure such as the Internet, and the system using the ink data 200 of the present invention can operate on the cloud unit 1. The Internet as an exemplary infrastructure includes Internet Protocol suites (eg IP, TCP, HTTP), libraries, software that runs various web and mail data formats (HTML, MIME), and their communication methods (HTTP,). It is based on a standardized set with SMTP) and absorbs differences between the hardware configurations and operating systems that each vendor configures independently. In FIG. 1, the dashed line passing through the infrastructure section 1 describes the data exchange that occurs based on these infrastructure technologies. In FIG. 1, the cloud unit 10 outlined by a solid line represents the infrastructure for exchanging the ink data 200, and the cloud unit 10 establishes a common information model (language) for the ink data 200. It will be realized. The ink data 200 is generalized to be commonly used by various application services (or ecosystems) and various devices. For example, both application service # 1 and application service # 2 shown in FIG. 1 can use and exchange ink data 200 via the ink data exchange infrastructure 10. This can be realized as a necessary library for the ink data processor 100 distributedly supported by several types of computers, such as mobile terminals and servers. The solid line passing through the data exchange infrastructure 10 illustrates the exchange of ink data 200 between various applications provided for several applications that utilize a set of libraries for utilizing ink data 200. Ink data 200 area (domain)

In Figure 1, device 10-1-1 includes a pen tablet-type input device capable of outputting pressure data as an input sensor, provided for application service # 1 provided by the first provider / software vendor. Generate ink data using application # 1. The generated ink data 200 can then be output in the appropriate output form (eg, SMF in the packet) corresponding to the destination media (eg, network).

Devices 10-1-2 are tablet-type input devices capable of receiving handwritten input made by the user's finger. The sensor of device 10-1-2 does not have the ability to output pressure data and does not utilize pressure information using application # 2 provided for application service # 2 or at the destination. Generate ink data 200 with the appropriate output form for a medium.

Device 10-3 is yet another type of computer (eg, a desktop PC) that uses application service # 2. The device 10-3 can combine (integrate) the ink data 200 provided by each of the device 10-1-1 and the device 10-1-2. The device 10-3 can render (draw) the ink data 200 output from the device 10-1-1 and the device 10-1-2 on the screen in a state of being overlapped with each other.

FIG. 2 is an actual condition-related diagram of the ink data model. The ink data 200 according to an embodiment of the present invention is for controlling the shape and color of the stroke object set 202 and the stroke object 210 when rendering (drawing, expressing, rasterizing) on a screen or display. Manipulate a drawing style object (set) 230 that contains the required information, a metadata object 250 (eg author) that contains non-drawing related information that describes the stroke object 210, and an existing stroke object 210 (eg slice). Contains an operation object (set) 270 that contains the information needed to (rotate).

The stroke object 210 in the stroke object set 202 contains the information necessary to reproduce the stroke 210 (or trace, path) formed by the movement of a pointer (finger, pen, etc.). The stroke contains multiple ("N") point objects 212 (points 1 to N) (217). In other words, strokes are supported by the coordinates of multiple point objects obtained by sampling pen event data (pointer manipulation) generated by pointer movement. A point object can take any form, such as absolute or relative coordinate value form, or vector form, as long as it indicates the position of the point object in 2D, 3D, ... N-dimensional space. .. In various embodiments, the plurality of point objects serve as control points that can be used to insert a curved segment 210 between them, thereby forming a path for the stroke object 210.

According to embodiments of the present invention, the stroke object 210 further includes range information that defines a portion of the stroke object 210 to be displayed. In the illustrated embodiment, the range information is the "start parameter" 301, which is the first parameter that defines the start point in the stroke object 210's start curve segment, and the end point in the stroke object 210's end curve segment. Includes a second parameter to be defined, the "end parameter" 303. Range information is generated for the stroke object 210 after the point object is created. For example, when an operation operation such as a slice operation is performed on the stroke object 210 and two new stroke objects 210 are generated, two sets of point objects that make up each of the two new stroke objects 210 are obtained. And range information is added to each of the two new stroke objects 210.

Here, the start curve segment and the end curve segment mean a segment in which the drawing (display) operation starts or ends. Therefore, the truly first curve segment of the stroke object 210 specified to be completely invisible is not the "start" curve segment used here. Similarly, the truly last curve segment specified as not visible at all is not the "end" curve segment.

In general, there are two ways to generate (x, y) coordinates for multiple point objects. First, while the pen event data is being input, the coordinate points derived from the pen event data (pen operation) can be output as "raw value type" points. Second, after all the points to form a complete stroke are entered, a cubic spline function such as the Bezier curve or a higher-order function (eg, Lagrange polynomial) showing the approximate curve of the stroke is generated. And the minimum number of point objects needed to represent the approximate curve can be obtained as an "optimized point type". In the following description, the point object is created as a "raw value type" by the first method. However, the present invention can also use a "optimized point type" point object according to the second method.

The drawing style object (set) 230 contains information necessary to control the shape (stroke width, stroke style / pattern) and color of the stroke object 210. In short, the drawing style object 230 controls the rendering of the stroke object 210. The drawing style object 230 in the illustrated example includes a shape fill object 232 and a particle dispersal object 234.

FIGS. 3B and 3C show two results, each of which is the same stroke object 210 as in FIG. 3A rendered (drawn) by two different drawing style objects.

FIG. 3B shows the rendering result of the shapefill object 232, representing the stroke object 210 as a set of circles 321 with various radii and widths. The center of each circle is aligned in a row along the trace represented by the stroke object 210, and the perimeter of the set of circles is used to generate (calculate) the envelopes 323,325. Envelopes 323,325 are then used to draw the stroke object 210 of FIG. 3A on the screen or display.

FIG. 3C shows the rendering (drawing) result of the particle scattering object 234, and draws the stroke object 210 of FIG. 3A as a set of point sprites. A point sprite is a shaped particle 341 (thin section) that has a center and varies in size and angle of rotation 345θ with respect to a defined axis of the slice. Flakes of various sizes rotate by θ with respect to their respective defined axes, and their center moves from the trace in the direction perpendicular to the trace direction by an offset of 343. Offset 343 is a random number value derived from a given species.

The metadata object 250 (see Figure 2) contains non-drawing related information that describes the stroke object 210, such as author, pen ID, locally acquired date and time information, GPS-acquired location information, and so on.

The operation object (set) 270 contains information necessary for manipulating the existing stroke object 210 in whole or in part (eg, selecting, transforming / rotating, slicing, etc.). Such information is organized in the form of multiple operation objects, each of which can be performed on all or part of the stroke object 210 to bring about the desired operation on the stroke object 210. Each operation object 270 contains parameters that define and control a particular operation operation. For example, the selection object 272 contains parameters used to select and transform (eg, rotate by the transformation matrix) the stroke object 210 as shown in FIG. 4A. Slice object 274 contains parameters used to slice stroke object 210 as shown in Figure 4B.

FIG. 4A illustrates the operation of the selection object 272. The target of selection and deformation is the existing stroke object 210 "stroke i", which is selected by the other stroke j (j&gt; i) in FIG. 4A. The stroke j is newly input based on the newly input pen event data and includes the point objects P1 to Pn. The stroke j defines an area surrounding the existing stroke i, thereby inputting to select the existing stroke i. The selection object 272 transforms (rotates) the selected stroke i, as illustrated by arrow 405 in FIG. 4A, so a defined transformation matrix can be used. There are various ways to determine whether or not stroke i is selected by stroke j and how it is selected. For example, if stroke j intersects stroke i at a single position (P_intersect_Mid) between p1 and p2, then only the right part of stroke i can be selected and transformed by 405. The remaining left portion of stroke i is not selected and remains undeformed by deformation 405. This can be achieved by applying a slice operation to stroke i at the same time using stroke j (ie, because stroke j triggers the generation of both the selection object 272 and the slice object 274 for stroke i401. Used for). In this case, stroke i is split into two newly generated strokes. One of these newly generated strokes is completely surrounded by stroke j and is therefore selected.

FIG. 4B illustrates the operation of slice object 274. Slice object 274 is a partial operation of stroke i401 and is created by a new stroke object 403 (stroke j) that includes point objects P1 ~ P4. The stroke object 403 is not a normal stroke object 210, but is associated with type information indicating that it is an operation object configured to perform a defined operation operation on an existing stroke object 210. For example, the stroke object 403 (stroke j) can be classified as of type "input 3" (operation object), as described more fully below with reference to FIG. As illustrated in FIG. 4B, slice object 274 (represented within stroke j of type "input 3") is input to slice an existing stroke object 401 (stroke i). To achieve this, slice object 274 contains the parameters needed to slice the existing stroke object 401 (stroke i) into two slices, slice i1407 and slice i2409. The slice object 274 can function as a slicer, eraser, partial extractor, etc. in various applications. After the slicing operation is performed and two new slices 407,409 are generated, these slices can be "committed" (or finalized) to be two fully defined stroke objects 210. In this regard, not only the original stroke object 401 (stroke i), but also the (uncommitted) slices 407,409 and the slice object 274 (stroke j) itself used to generate the slices need to be maintained. There is no.

FIG. 5 is a functional block diagram of an ink data processing apparatus capable of outputting, manipulating, drawing, and transmitting (transmitting / receiving) ink data according to an embodiment of the present invention. This device generally corresponds to device 10-1-1 (sensor type 1) or device 10-1-2 (sensor type 2) of FIG.

The devices in this example are an input sensor 110, an input processing unit 111, an ink data processing unit 100, an application unit 300-1, a graphic processing unit 300, a display 113, and a communication unit 112 ("Tx, Rx"). ) And all of these are controlled by operating system 400-1 which is combined with the storage and executed by the CPU. The device may be a personal computer (PC) or mobile terminal device that includes or is coupled with an input sensor 110 in the form of a pen tablet sensor.

The input sensor 110 detects the user's handwriting motion (via a pointer such as a pen or a finger) and generates an input data signal expressing the detected handwriting motion. For example, electrostatic sensors, pressure sensitive sensors, electromagnetic resonance (EMR) based sensors can be used.

The input processing unit 111 receives input data of a type that depends on each input sensor from the input sensor 110, and converts this input data into "pen event data" that can be further processed to generate the ink data 200. do. The generated "pen event data" is input to the ink data processing unit 100 as "input 1" (see point "A" in FIG. 5). The pen event data ("input 1") includes at least the sensed coordinate position, and additionally includes pressure data, pen tilt data, etc., depending on whether the input sensor 110 has pressure / tilt detection capability. Can include. Therefore, the pen event data output from the input processing unit 111 also depends on the device / sensor. The input processing unit 111 is typically implemented as a software program that is a driver for an input sensor 110, such as an input subsystem running on an Android® operating system. The configurations of the input sensor 110 and the input processing unit 111 are not limited to those shown in the figure. For example, some or all of the input sensor 110 and the input processing unit 111 may be provided as a digital stationery device such as a pen-shaped device.

The ink data processing unit 100 includes an ink data generation unit 120 and an ink data shaping unit 140. The ink data processing unit 100 (more specifically, the ink data generation unit 120) includes pen event data received from the input processing unit 111, context information ("input 2") received from the application unit 300-1, and Responsible for generating ink data 200 based on operating information ("Input 3"). The ink data processing unit 100 is typically realized as a set of libraries dynamically and / or statically linked to the application unit 300-1.

Contextual information ("Input 2") is information that describes the context or environment of pen event data ("Input 1"), such as the type of pen tip used (eg, brush, crayon, pencil), or used. It can indicate the pen color (red, green, blue) to be used. The context information is typically selected by application unit 300-1 prior to inputting input data to the input sensor 110.

The operation information ("Input 3") should not treat the next input from the input sensor 110 as typical pen event data (normal stroke object 210), but some for the existing stroke object 210. Operation Specify that the command is for applying an operation (for example, slicing, erasing, expanding, deleting, copying, expanding, etc.). When input 3 is received, the ink data generator 120 sets the new stroke object #j and the operation object applied to the existing stroke objects # 0 to #i due to this new stroke object #j. Generate. The operation information ("input 3") is generated by the user selecting a switch, a button, or the like defined in the application that supports the application unit 300-1, and is input to the ink data generation unit 120.

The ink data generator 120 receives pen event data ("input 1"), context information ("input 2"), and operation information ("input 3") and strokes (at point "D" in FIG. 5). Generates "ink data" (ink data 200) including object 210, drawing style object 230, operation object 270, and metadata object 250. Further details of the ink data generator 120 will be described later with reference to FIG.

Further referring to FIG. 5, the ink data shaping unit 140 of the ink data processing unit 100 receives ink data from the ink data generation unit 120 via the point "D", and the format selection received from the application unit 300-1. The ink data is output in the format selected according to the information (Fmt-Sel).

Specifically, the ink data shaping unit 140 includes an ink data communication unit 144 and a recording format data processing unit 142. The ink data communication unit 144 is configured to transmit (via "F" in FIG. 5) and receive (via "F_in" in FIG. 5) ink data 200 in stroke message format (SMF). The stroke message format (SMF) is a format suitable for transmitting ink data 200 (for example, in real time) to other (remotely located) devices on the network. The recording format data processing unit 142 is configured to format the ink data into the stroke file format (SFF) (see "E" in FIG. 5). The stroke file format (SFF) is a format suitable for storing ink data 200 in a more permanent storage medium.

The graphic processing unit 300 receives the ink data 200 including the stroke object 210, the drawing style object 230, the operation object 270, and the metadata object 250 via "D", and the defined resolution via "H". A level that outputs a set of pixel values, including a pixel-by-pixel color (eg RGB) value. For example, the graphic processing unit 300 receives the point objects (p1 to pn) constituting the stroke object 210, inserts a curve between the point objects used as control points according to the curve interpolation algorithm, and the resulting stroke. The path of the object 210 is drawn (rendered) on the display 113 using a related GPU library such as the DirectX® or OpenGL® library.

According to various embodiments, the graphics processor 300 has been received to insert a curve that follows an appropriate curve interpolation algorithm, such as the Catmar-Rom curve algorithm or Polyvezier curve algorithm known to those of skill in the art. Use the point object contained within the stroke object 210.

Further, according to a typical embodiment of the present invention, the graphic processing unit 300 refers to the value 301 of the "start parameter" and the value 303 of the "final force parameter" included in the stroke object 210, and strokes. Display object 210. In other words, the graphic processing unit 300 renders (displays) only the part of the stroke object 210 that is depicted (bound) by the value 301 of the "start parameter" and the value 303 of the "final parameter". )do. The display used here means that it is finally displayed. Various techniques can be used to set whether or not a defined portion is a display target. For example, a method can be used in which the vertex information supplied to the GPU library does not include the vertex information for a predetermined portion that is not to be displayed so that the pixel data for the predetermined portion is not generated. As another example, a method can be used in which the stroke object 210 is 100% completely reproduced, but a predetermined portion that is not to be displayed is set to be transparent.

The application unit 300-1 includes one or more user applications that dynamically or statically link to the ink data processing unit 100, such as application # 1 in FIG. For example, the application unit 300-1 can include a real-time conference application, a document generation application, a drawing application, and the like using the ink data 200 according to the embodiment of the present invention. Each application in the application unit 300-1 provides, for example, a user interface (UI) that allows a user to input operation information ("input 3") to the ink data processing unit 100.

FIG. 6 is a functional block diagram of the ink data processing unit 100 including the ink data generation unit 120 and the ink data shaping unit 140.

The ink data generation unit 120 includes a stroke object operation unit 122, a metadata object operation unit 124, a drawing style object operation unit 126, and an operation object operation unit 128. These collectively operate and generate the stroke object 210, the metadata object 250, the drawing style object 230, and the operation object 270 that collectively constitute the ink data 200 according to the embodiment of the present invention.

The stroke object operation unit 122 receives the pen event data ("input 1") and generates the stroke object 210 (see "D" in FIG. 6). The stroke object operation unit 122 creates a raw value type point object as described above, and forms the stroke object 210. In a typical embodiment, the stroke object control unit 122 does not wait for the entire pen event data to be received before starting the generation of the point object, but continuously as the pen event data is input. Create a point object. The stroke object manipulation unit 122 is directed to the graphic processing unit 300 (see FIG. 5) or the ink data shaping unit 140 via a "D", as described in more detail below with reference to FIG. , Output the generated point object continuously. Application 300-1 is optimized for outputting raw ink data 200 and, for example, whether application 300-1 performs real-time communication or requires highly compressed vector data. The stroke object operation unit 122 can be controlled so as to switch between outputting the value type ink data 200.

The metadata object manipulation unit 124 receives pen event data ("input 1") indicating the start of a pen stroke (ie, "pen down"), or generates a new stroke object 210 (eg, in response to slicing). Processes context information ("input 2") according to, and extracts non-drawing related information such as author information, date and time information, and so on. The metadata object operation unit 124 creates a metadata object 250 including the extracted metadata in association with the corresponding stroke object 210.

The drawing style object operation unit 126 processes the context information ("input 2") in response to the reception of pen event data ("input 1") indicating pen down or the generation of a new stroke object 210, and causes the stroke object 210. Extract drawing-related information required for expression on the display. The drawing style object operation unit 126 creates the drawing style object 230 (for example, the shape fill object 232 and the particle scattering object 234) in association with the corresponding stroke object 210.

The operation object operation unit 128 is an operation or a modification operation (for example, shown in FIG. 2) to be applied to all or a part of the existing stroke object 210 in response to the reception of the operation information ("input 3"). Generate an operation object 270 that defines the "selection (transformation)" object 272 and slice object 274).

In FIG. 6, the two dashed arrows "M1 (local)" and "M2 (remote)" should be the direction of the operation operation, that is, the target stroke object 210 should be manipulated or deformed. It shows whether it is. As shown, the operation operation defined by the operation object 270 may be applied locally (M1) to the stroke object 210 existing in the stroke object operation unit 122, or may be applied to a remote location connected to the network. It may be applied to the stroke object 210 existing on the external network such as in the receiving device of the remote (M2) via the ink data shaping unit 140.

In FIG. 6, the stroke object 210, the metadata object 250, and the drawing style object 230 are input to the recording format data processing unit 142 and the ink data communication unit 144 of the ink data shaping unit 140, while the operation object 270. Is explained so as to be input only to the ink data communication unit 144 and not to 142. The first three are preferably stored permanently or semi-permanently, and are therefore formatted to suit the storage device, such as stroke file format (SFF), SVG format, and InkML format. Also, when a new stroke object 210 is generated, the stroke object 210 and associated metadata and drawing style objects are propagated to the receiver over the network and therefore the ink data communicator 144 and recording format data processing. Processed in both parts 142. On the other hand, the operation object 270 is inherently temporary because it defines some operation operations to be applied to the existing stroke object 210. Once an operational operation is applied (committed) to an existing stroke object 210, that operational object 270 is expelled from memory. Therefore, the operation object 270 is typically formatted into a stroke message format (SMF) suitable for transmission over the network and is not included in the stroke file format (SFF).

The recording format data processing unit 142 of the ink data shaping unit 140 outputs the ink data 200 in different recording formats (SFF, InkML of non-patent document 1, SVG of non-patent document 3, HTML5 of non-patent document 4, etc.). Includes multiple processing units 142-1,142-2 for. For example, processing unit 142-1 is configured to output SFF ink data 200 and is a Google protocol buffer (https://developers.google.com) for serializing file-formatted data in SFF format. / protocol-buffers /) and messages may be adopted. Processing unit 142-2, such as the difference between the definition of "trace" in non-patent document 1 and the definition of "path" in non-patent document 3 or "cambus path" in non-patent document 4. Format conversion processing may be performed to absorb the difference between the SFF file, InkML of Non-Patent Document 1, and SVG of Non-Patent Document 3.

FIG. 7 is a functional block diagram of the stroke object operation unit 122 in the ink data processing unit 100. As more and more pen event data is input, the stroke object operation unit 122 continuously outputs the point objects constituting the raw value type stroke object 210 to the graphic processing unit 300 or the ink data shaping unit 140. Has the ability to do. The stroke object operation unit 122 includes or is coupled with the storage device 770.

The stroke object operation unit 122 partially includes a start / end parameter setting unit 122A that sets the start point and end point, a path builder setting unit 122B that selects an appropriate path builder based on the type of device, and a number of point objects. It includes a pair stroke adder 122D that controls whether it should be attached to the formed stroke object 210, and a suffix adder 122E that fills the gap at the end of the stroke object 210 (the "delay" in FIG. 8).

The fragmented data generated and stored in memory 770 is used for real-time transmission as a fragment of the stroke object 210. Fragmented data is transmitted in bytes or units of time, as fully described below in the second embodiment.

The start / end parameter setting unit 122A sets the start parameter 301 and the end parameter 303 to their respective initial default values in response to the detection of the pen-down event (start of pen stroke) or pen-up event (end of pen stroke). .. For example, the start parameter 301 is set to its default value "0" in response to a pen-down event, and the end parameter 303 is set to its default value "1" in response to a pen-up event. The initial values of these parameters do not need to be stored in the storage device 770, for example, they may be set in the form of flags implying that these parameters are set to default values.

Pathbuilder unit 122B is configured to select the appropriate pathbuilder for the type of pen event data output from input processing unit 111, based on the input dynamics setting value 701 contained in the context information ("Input 2"). To. For example, if the pen event data contains a pressure value, a pressure path builder 122B1 containing a first (pressure) function f1 capable of deriving stroke width (W) and transparency (A) based on pressure. Is selected. On the other hand, if the pen event data does not include the pen pressure value, the velocity path builder 122B2 is selected. The velocity path builder 122B2 includes a second (velocity) function f2 capable of deriving stroke width (W) and transparency (A) based on the speed of movement of the pen. The movement speed of the pen is determined from the change amount of the point coordinates or the time stamp included in the pen event data. In other words, the velocity path builder 122B2 uses the velocity value instead of the pressure value used in the pressure path builder 122B1. All pen event data can be classified as either type with pressure information (type 1) or type without pressure information (type 2), so that all types of pen event data are pressure path builder 122B1 or speed. Will be processed by one of Pathbuilder 122B2. The reason for this, as well as how the pressure path builder 122B1 and the velocity path builder 122B2 work, will be described below with reference to the third embodiment.

The stroke object control unit 122 also smoothes the stroke object 210 based on the smoothing usage information contained in the context information ("input 2") before the stroke object 210 is generated and complete. It has a smoother 122C that initiates the application of the operation. Any suitable smoothing operation such as acceleration averaging, weighted averaging, exponential smoothing, double exponential smoothing, etc. can be used.

FIG. 8 illustrates a smoothing operation performed by the smoother 122C in columns A and B. Column A corresponds to point "A" (before smoothing) in FIG. 7, and column B corresponds to point "B" (after smoothing) in FIG. In column A, x1 to x10 represent the X coordinates of 10 points obtained from the pen event data. In FIG. 8, x1 to x10 are (10,20,30,45,60,80,90,100,110,115).

In column B, x1'~ x10' represent the X coordinates of the above 10 points after the smoothing operation has been applied. In the illustrated example, the following exponential smoothing function applies. However, the filter strength α = 0.5.

<math num="1"><img file="JP7094318B2_D0001.tif" /></math>

In FIG. 8, x1'~ x10' is (10,15,23,34,47,63,77,88,99,108).

The smoothing operation performed by the smoother 122C is applied to each point on a rotation basis each time the coordinates of each point are derived in order to continuously output the corrected (smoothed) position of each point. Will be done. Therefore, instead of waiting for the pen-up event as a whole to be detected, the stroke object operation unit 122 starts generating and outputting the "raw value type" stroke object 210 from the time when the pen-down event is detected. do.

The point x0'in column B is added when the Catmull-Rom curve is used to define an interpolated curve between a pair of control points. As discussed above, the Catmull-Rom curve is the start point (Pi), the endpoint (Pi + 1), the "before" point of the start point (Pi), and the "behind" the endpoint (Pi + 1). Define each curve segment using four control points, including the point. Therefore, the starting point x1'is duplicated to define the starting curve segment between the points x1'and x2', thereby defining the curve segment between x1'and x2'. A new point x0'that can be used with x2', x3' is generated (at the same position as x1'). One of the new points x0'can be adjusted so that the Catmull-Rom curve between x1'and x2' is in the position that best fits the input stroke. By simply duplicating the value of x1'(p1) to generate the value of x0'(p0), the process immediately defines the position of x0'and goes from x0'(p0) to x1'(p1). The components of the vector of can be set to zero. Such a process is suitable for real-time execution (no need to wait for p2 to generate p0), and the curvature of the curve segment between x1'(p1) and x2'(p2). It does not overly affect (eg, push to one side or the other).

Referring again to FIG. 7, the anti-stroke adder 122D shows how many point objects are established and stored in storage 770, and therefore can be added to the partial data of the output stroke object 210. decide. In the illustrated embodiment, the point objects P1 to Pn-1 are established in the storage device 770 and output to the graphic processing unit 300 or the ink data shaping unit 140 (instead of waiting for the entire stroke object to be completed). It has been determined that the stroke object 210 will be added to the partial data. The graphic processing unit 300 can display a partially formed stroke object. In other words, the graphic processing unit 300 displays the stroke object starting from the initial point as it grows. The ink data shaping unit 140 (or the ink data communication unit 144) can format the established partial data into a transmission format as fragmented data of a stroke object and transmit the data. The method of transmitting fragmented data will be described in more detail in the second embodiment.

Referring to FIGS. 7 and 8, the suffix addition portion 122E has the input end position of the stroke object 210 (x10 in column A) and the end position of the smoothed stroke object 210 (x10'in column B). Fill the gap (or "delay") between. Figure 8 illustrates an operation in which column C fills in the "delay" (adds a suffix). In the illustrated example, after the smoothing operation, a "delay" is generated between the originally entered position x10 (115) and the smoothed position x10'(108) at the end of the stroke object 210. There is. The stroke object operation unit 122 determines whether or not to perform the operation of adding the suffix based on the content of the smoothing usage information 705 included in the context information ("input 2"). The suffix addition operation can also be triggered each time a new point object is added to the stroke object 210. In this case, the suffix addition operation prepares the graphic processing unit 300 with a point object that can be used as a temporary visual preview. The newly generated points by the suffix addition operation are not yet part of the final stroke object 210 and are therefore not explicitly added to the ink data shaping unit 140 (or ink data). Ignored by the communication unit 144).

In the illustrated embodiment, when the suffix addition operation is executed, the stroke object operation unit 122 adds a new point object to x11', x12', and x13'. Point x12'is added to the same position as the latest position x10 (115) of the original input in column A. Point x11'is added to the smoothed point between points x10' and x12'. Finally, since this example uses a Catmull-Rom curve to define an interpolated curve between each pair of control points, the Catmull-Rom curve ends between x11'and x12'. The terminating point x12'is duplicated (at the same position as x12') to generate the new point x13' needed to define the curve segment. The position of the new point x13'can be adjusted so that the Catmull-Rom curve between x11' and x12' fits best into the input stroke. Also, even when the suffix addition operation is not performed, if the Catmull-Rom curve is used, the latest smoothing point x10'in column B will generate a new point x10'(new). Can be duplicated in, and it can in this case be used with x8', x9', x10' as a control point to define the latest curve segment between x9'and x10'. By simply duplicating the value of x9'to generate the value of x10', it overly affects the curvature of the curved segment between x8'(p8) and x9'(p9) (eg, one side). The process can quickly define the x10'position to a neutral position without pushing to one side or the other side).

In Figure 8, column D is smoothed (column A to column B), suffixed to the end (column B to column C), and continuously output under the control of anti-stroke adder 122D. The stroke object 210 is illustrated. In the stroke object 210 in this example, each curve segment (Pi ~ Pi + 1) is defined by four control points (Pi-1, Pi, Pi + 1, Pi + 2), and the resulting curve is all control. Defined to generate an interpolated curve by the Catmal-Rom curve algorithm that passes through a point. Therefore, the stroke object 210 contains a start curve segment (x1'~ x2') defined by four control points x0', x1', x2', x3', and four control points x10', x11', x12. Includes the end curve segment (x11'~ x2') defined by', x13'. The stroke object 210 also includes a start parameter 301 for the start curve segment (x1'~ x2'), which is set to a default value of "0.0" by the start / end parameter setter 122A. A default value of "0.0" means that the starting curve segment (x1'~ x2') is completely visible (rendered, represented) from the first point x1'. The stroke object 210 also includes an end parameter 303 for the end curve segment (x11'~ x12'), which is set to a default value of "1.0" by the start / end parameter setter 122A. The default value of "1.0" means that the end curve segment (x11'~ x12') is completely displayed up to the last point x12'.

FIG. 9 is a flowchart illustrating a sample process executed by the ink data generation unit 120 of FIG. 6 to generate the ink data 200. The process starts when the ink data generation unit 120 receives the pen event data ("input 1"). In step S901, the stroke object operation unit 122 receives the pen event data including the position coordinates (x, y) and the timing information indicating one of the following three types of timings as the input 1. The process is executed according to the determined timing.

1) Pen downtime when a pointer, such as a finger or pointing device (eg, a pen-type device), enters contact with another (sensing) object.

2) Pen movement time between pen downtime and pen uptime.

3) Pen-up time when the pointer moves away (pulled away) from another (sensing) object.

&lt;1. Processing flow in pen down time&gt; When the event type is "down operation" indicating a pen down event, the stroke object operation unit 122 starts the start curve segment of the newly generated stroke object 210 in step S910. Set parameter 301 to the default value ("0,0"). As mentioned above, the default value ("0,0") defines that the starting curve segment should be fully visible from its first value. At the same time, the stroke object control unit 122 also defines the end parameter 303 of the end curve segment of the stroke object 210, which defines that the end curve segment should also be fully displayed up to its last value ("1,0". ") May be set.

In step S912, the stroke object controls 122 are among the multiple pathbuilders used to build the stroke object 210, based on the input dynamics configuration parameter 701 contained in the context information ("Input 2" in Figure 7). Select one path builder (eg 122B1 or 122B2 in Figure 7).

In step S914, the stroke object operation unit 122 determines a set of parameters to be output from the path builder selected in step S912 above, based on the channel number parameter 703 included in the context information ("input 2"). The set of parameters to be output may be, for example, (x, y, W, A), (x, y, W), (x, y, A), or (x, y). However, (x, y) is the x, y coordinates of the point object, W is the stroke width value, and A is alpha (α) indicating transparency (or opacity). You may add a "z" value to the 2D coordinates (x, y) to generate the 3D coordinates.

In step S916, the stroke object control unit 122 applies a smoothing operation to a set of parameters output from the selected pathbuilder based on the smoothing usage parameter 705 contained in the context information ("Input 2"). Decide if it should be. The smoothing use parameter 705 may also indicate which of a plurality of parameters the smoothing operation applies to.

When the application of smoothing is indicated in step S918 (YES in step S916), the instructed smoothing process is executed. FIG. 7 illustrates the case where the smoothing process is applied to the (x, y, W) parameter but not to the "A" (alpha) parameter. The context information ("input 2") can additionally include information on the sampling rate of the input sensor 110 that the stroke object manipulation unit 122 can use to select the desired intensity level smoothing process. For example, when reducing the sampling rate of the input sensor 110 (eg, 10 samples per second for 100 samples per second), a stronger smoothing process with a larger smoothing value (effect) may be selected.

In step S919, the setting parameters used up to this point are output as attributes of the ink data 200. The parameter indicates, for example, whether the point object contained within the stroke object 210 is smoothed (whether S916 was YES or NO), or the type or strength of the smoothing filter that can be used. Based on this parameter, whether the point object contained within the stroke object 210 is smoothed, (further) should be smoothed, and the exact input data that can be used, for example, in signature verification applications. It can be determined whether it can be treated as.

In step S920, the first point (control point) x1'sets the starting curve segment between x1'and x2' as a Catmull-Rom curve, as described above with reference to column B in FIG. Duplicated to generate a new control point x0'to define.

In step S970, the ink data processor 100 may share (eg, receive, operate) the ink data 200 generated by the ink data processor 100 of the local computer. Determines if a (remote) user or computer exists.

If such other users are present, the ink data processing unit 100 will ink in step S972 based on the format selection (Fmt-Sel) parameter received from the application unit 300-1 (see Figure 5). The ink data communication unit 144 is controlled so that the data 200 is output in the stroke message format (SMF). The ink data communication unit 144 first includes a partial (fragmented) element containing the initial point coordinates and drawing style object 230 required for the remote user's receiver to draw the (partial) stroke object 210. Output the message DATA_INK_BGNOS1201 (see Figure 12). Upon receiving the message DATA_INK_BGNOS1201, the receiver receives the first part of the stroke object 210 in the shape, color, etc. specified using the received drawing style object 230 immediately before receiving the rest of the stroke object 210. You can start rendering (display) of.

&lt;2. Processing flow in pen movement time&gt; Returning to step S901 of the flowchart, the ink data generation unit 120 receives other new event data ("input 1"), and its type is pen down event, pen movement event. , Decide whether it is a pen-up event. When the event type is "move motion" indicating that the pen is moving within the stroke object 210, the ink data generator 120 is included in the particular input device dependent pen event data used. The x, y coordinates are received along with the time stamp and / or pressure information as a thing, and the process proceeds to step S930.

In step S930, the stroke object control unit 122 smoothes to, for example, a set of received parameters (x, y, W, A) based on the smoothing usage parameter 705 contained in the context information ("input 2"). Determine if the operation should be applied. The smoothing parameter 705 may additionally indicate which of the plurality of parameters the smoothing operation is applied to. The operation of step S930 is the same as that of step S916 described above.

If application of smoothing is indicated (YES in step S930), the indicated smoothing process is performed in step S932.

In step S934, the stroke object controls 122 determine how many of the point objects are established and stored in storage 770 to be added to the partial data of the stroke object 210 to be output. Use the anti-stroke addition part 122D. In this step, the anti-stroke addition unit 122D can also change the value of the point object before adding it to the partial data. For example, the anti-stroke addition unit 122D simulates a ballpoint pen running out of ink, so that the value of the alpha parameter can be changed on a random basis.

In step S974, as in step S970 described above, the ink data processing unit 100 determines whether there is another (remote) user sharing the ink data 200 generated by the ink data processing unit 100 in real time. do.

If such another user is present, the ink data processor 100 partially (in step S976) contains a point object following the first point object contained in the message DATA_INK_BGNOS1201 generated in step S972 above. The ink data communication unit 144 is used to generate and output the message DATA_INK_MVDOS1203 (see Fig. 12), which is a (fragmented) message. The number of point objects to be added to message DATA_INK_MVDOS1203 is determined by the anti-stroke addition unit 122D in step S934 above. Multiple DATA_INK_MVDOS messages can be generated and output, depending on the size of the stroke object 210. The receiver of the remote user receiving the DATA_INK_MVDOS message continues to render (display) the middle part of the stroke object 210, following the first part of the stroke object 210.

&lt;3. Processing flow in pen-up time&gt; Returning to step S901 of the flowchart, the ink data generation unit 120 receives other new event data ("input 1") and determines its type. When the event type is an "up action" indicating a pen-up event (ie, the stroke object 210 has been drawn and the pointer has been removed), the stroke object control unit 122 has received a set of parameters in step S950. For example, whether or not the smoothing operation is applied to (x, y, W, A) is determined together with which parameter the smoothing operation is applied to. The operation of step S950 is the same as that of steps S916 and S930 described above.

If application of smoothing is indicated (YES in step S950), the indicated smoothing process is performed in step S952. Also, when the smoothing use parameter 705 indicates so, the stroke object operation unit 122 additionally performs the suffix addition operation shown in column C of FIG. Depending on the content of the smoothing use parameter 705, the suffix addition operation may not always be performed. Also, when the smoothing operation is not performed, the suffix addition operation is not needed and will not be performed.

Step S953 ends to generate a new control point x13'to define the end curve segment between the Catmull-rom curves x11'and x12', as illustrated in column C of FIG. Point (control point) x12'is duplicated.

In step S954, the stroke object control unit 122 sets the end parameter 303 of the end curve segment (x11'~ x12') to the default value indicating that the end curve segment should be completely displayed up to its end point x12'. Set to "1.0"). This step may be skipped if the end parameter 303 is already set to the default value in step S910 described above.

In step S978, as in steps S970 and S970 described above, there is another (remote) user in which the ink data processing unit 100 shares the ink data 200 generated by the ink data processing unit 100 in real time. Decide whether or not.

If such other users are present, the ink data processor 100 in step S980 is partially (fragmented) containing the last set of point objects of the stroke object 210 with the suffix added. ) The ink data communication unit 144 is used to generate and output the message DATA_INK_ENDOS1205 (see FIG. 12). The remote user's device receiving DATA_INK_ENDOS1205 understands that it is the last message for stroke object 210 and completes the drawing operation of stroke object 210.

The method and system for generating and communicating ink data 200 according to the embodiment of the present invention described above continuously inputs pen event data and outputs the partial stroke object 210 as soon as it is established. .. The remote user's receiver, which receives the ink data 200 from the ink data processor 100, starts displaying each stroke object 210 without waiting for the entire stroke object 210 to be received, and strokes as it grows. Continue to display object 210.

According to various embodiments of the invention, the smoothing operation is selectively applied to the input pen event data parameters. Appropriate curve interpolation algorithms, such as the Catmal-Rom curve algorithm, are applied to build the interpolated curve using the smoothed point object as the control point.

FIG. 10 shows a stroke file format (SFF) prototype file written in Interface Definition Language (IDL) that can be output to point "E" from the recording format data processing unit 142 of the ink data processing device of FIG. It is a figure explaining the sample of. This prototype file describes how the ink data 200 are arranged as a sequence of bytes in the stroke file format. Lines 02-07 of the illustrated protofile contain data describing information contained in the drawing area (or drawing "canvas"). For example, line 06, enclosed in a dashed rectangle, defines that the stroke object 210 is repeated multiple times within the drawing area. The "decimal Precision" in line 04 defines the accuracy / resolution of the calculation of the point object of the stroke object 210.

The "decimal Precision" in line 04 is preferably a logarithmic value indicating the desired accuracy and / or resolution.

Lines 11-17 of the illustrated protofile represent the stroke object 210. For example, line 12 shows that the stroke object 210 contains multiple "sint32" type (variable byte size packets) point objects that are repeated.

In some embodiments, the parameters of the second and subsequent point objects are defined by offset (delta) values with respect to the corresponding parameter values of the first point object or the immediately preceding point object. The use of offset (related) values can help reduce the amount of data required to define the second and subsequent point objects that form the stroke object 210 compared to absolute values.

For example, the coordinates (x, y) of the point in row 12 are determined based on the following process using decimalPrecision.

<math num="2"><img file="JP7094318B2_D0002.tif" /></math>

Process 1. xfloat is a float stored in a computer device. The coordinates of the point object 212 are stored in memory 770 as a floating-point type value with a relatively large number of bits, such as a float type or a double type. xfloat is multiplied by 10 to the decimalPrecision power. As the data type of the resulting value, xfloat is converted (cast) to an int type, which produces xint.

Process 2. The offset of xint is derived. The derived offset is encoded as "sint32" type data.

Lines 13-14 surrounded by a dashed rectangle define the start parameter 301 and end parameter 303 of the stroke object 210. As shown, these parameters 301 and 303 are defined separately from the point objects that form the stroke object 210 defined in row 12. In the illustrated example, the start and end parameters 301 and 303 are represented as float type values and are set to the default values of "0" and "1" respectively. As will be described later, when the operation (slicing) operation is applied to the stroke object 210, the start and / or end parameters may be changed to new values.

The "variable Stroke Width" on line 15 stores the width values of the plurality of point objects contained in the stroke object 210. It uses the "sint32" type as well as the point in line 12, and the width of the second and subsequent point objects is defined by the offset (delta). The presence of this parameter implies that each of the multiple point objects contained in the stroke object 210 is individually associated with its own width value. In other words, if this parameter is not present, the stroke object 210 has a fixed width stored in the "strokeWidth" property on line 16.

FIG. 11 is a diagram illustrating a sample stroke object 210 written in stroke file format (SFF) and capable of being output to point "E" from the recording format data processing unit 142 of the ink data processing apparatus of FIG. The stroke object 210 in the illustrated stroke file format contains the drawing style object 230, the filter parameters (smoothing used, filter strength), and the point objects p0 to p13 at the coordinates x0'to x13' shown in column D of FIG. include. The start parameter field 1101 contains the start parameter 301 and the end parameter field 1103 contains the end parameter 303. The field "x0" (1105) contains the absolute x-coordinate value of the first point object of the stroke object 210. The field "rel_x1" (1107) contains the x-coordinate offset (delta) value of the second point object of the stroke object 210 with respect to the absolute x-coordinate value of the first point object.

FIG. 12 shows three messages stored in the stroke message format (SMF) that can be output to point "F" from the ink data communication unit 144 of the ink data processing device of FIG. 5, and the ink data processing device of FIG. It is a figure explaining one packet output from the work communication part 112 to the point "G".

The DATA_INK_BGNOS1201 message, which is output in step S972 of FIG. 9, contains information indicating that the message is the first message of the stroke object 210 (for example, BGNOS type F101) and is received by the ink data processor and the remote user. It includes a drawing area ID (F102) indicating a common drawing area shared between devices, and a stroke ID used to identify the stroke object 210 from among a plurality of stroke objects 210 in the drawing area. F101, F102, and F103 constitute a message header.

The DATA_INK_BGNOS1201 message also includes a drawing style object 230 (F104), filter parameters associated with the smoothing filter applied (not shown), start and end parameters (F105_SP_EP), and optionally (if space allows). ) Contains some of the first fragmented data (F105_begin) of the point object that forms part of the stroke object 210, with the CRC error correction value added at the end. For example, the "F105_begin" field can contain the point objects p0-p3 in Figure 8. The F104, F105_SP_EP, F105_begin, and CRC fields make up the message payload.

The reason why F104 is included in DATA_INK_BGNOS1201 will be described later in relation to the second embodiment. The parameters related to the smoothing filter are contained in the first message, DATA_INK_BGNOS1201, so that the device receiving the stroke object 210 will immediately start receiving the stroke object 210. It is possible to determine whether or not the smoothing filtering process should be applied to the point objects included in.

The DATA_INK_MVDOS1203 message is output in step S976 of FIG. 9 and includes a message header containing a message type field ("MVDOS") F101, a drawing area ID field F102, and a stroke ID field F103. The DATA_INK_MVDOS1203 message also contains a second fragmented data (F105_moved) containing a point object following the point object contained within DATA_INK_BGNOS1201. For example, the "F105_moved" field can contain the point objects p4 to p8 in Figure 8. Unlike the first data message, the DATA_INK_MVDOS1203 message does not contain the drawing style object 230 (F104) and is identified in F101 as a subsequent (non-first) type of data message. The DATA_INK_MVDOS1203 message contains the same drawing area ID (F102) and the same stroke ID (F103) as the first data message.

The DATA_INK_ENDOS1205 message is output in step S980 of FIG. 9, is the last data message for the stroke object 210, and is the last fragment of the stroke object 210, which can be, for example, the point objects p9 to p13 of FIG. Includes fragmentation data (F105_end). The DATA_INK_ENDOS1205 message is identified in F101 as the last data message (ENDOS) and contains the same drawing area ID (F102) and the same stroke ID (F103) as the first data message. The final data message contains a metadata object 250 in F109 that contains non-drawing related information such as author information.

The above three types of data messages are output from the ink data communication unit 144 of the ink data processing device of FIG. 5 to the point "F". The packet "G" in the last column of FIG. 12 is a packet containing all of these three types of data messages as a packet payload, and is a point "from the network communication unit 112 (Tx, Rx) of the ink data processing device of FIG. It is output to "G".

FIG. 13A is a diagram illustrating a stroke object that has undergone a catmal-rom curve interpolation operation, which is input to the graphic processing unit 300 of the ink data processing apparatus of FIG. 5 via the point "D" (FIG. 13A). The 13 stroke objects 210 can also be input to the ink data shaping unit 140 described above via point "D"). FIG. 13B shows how the stroke object 210 of FIG. 13A is output from the graphic processing unit 300 to the point "H" in order to be drawn (rendered) on the display 113 of the ink data processing apparatus of FIG. It is a figure explaining a rendering.

In FIG. 13A, the stroke object 210 includes point objects p0 to p13 corresponding to x0'to x13' illustrated in FIG. 8 above. The stroke object 210 is subject to the Catmull-Rom curve interpolation operation, so all of the point objects p0-p13 are used as control points to jointly generate the interpolated curve segments that form the curve shown in FIG. The object. For example, points p0 to p3 are used as control points to generate a curved segment between p1 and p2, and points p1 to p4 are used as control points to generate a curved segment between p2 and p3. Is done, and so on. As shown, the resulting curve passes through all of the control points p0-p13.

One feature of the Catmull-Rom curve is that the effect of moving the control points is local because each curve segment is completely defined by the four control points. For example, FIG. 13A illustrates that the curve segment 1301 between p6 and p7 is defined at four control points p5, p6, p7, p8. Moving one control point can have a strong effect on up to four curve segments, but not the remaining curve segments forming the curve. For example, moving p8 is up to p6 ~ p7, p7 ~ p8, p8 ~ p9, It may have a strong impact on the four curve segments p9 ~ p10. The Catmull-Rom curve's "local control" feature is that the resulting two new stroke objects 210 provide the (formal) information of the original stroke object 210 as needed with a minimum amount of data. Suitable for supporting slicing operations on stroke objects, which should be maintained as much as possible. In other words, the feature of "local control" is that each resulting slice (to maintain the perfect shape of the curve from one end to the other) has the original shape at the minimum number of control points. Allows you to maintain. For example, when the stroke object 210 in Figure 13A is sliced at the curved segment between p6 and p7, the first slice needs to maintain only control points p0 ~ p8 and the second slice needs to maintain control points p0 ~ p8. Only p5 ~ p13 need to be maintained. Polyvezier curves also have the characteristic of "local control" and are therefore suitable for supporting slicing operations. However, the Polyvezier curve, unlike the Catmull-Rom curve, requires control points that are not along the curve (ie, the curve does not pass through all of the control points). Having to calculate and store control points outside those curves is an extra calculation that requires storage space, and the point object of stroke object 210 provides all control points. -Not required by the Rom curve. This difference makes the less computationally demanding Catmull-Rom curve more suitable for supporting real-time applications in ink data generation, manipulation, drawing, and communication.

FIG. 13B illustrates an actual rendering (drawing) example of the stroke object 210 of FIG. 13A when output from the graphic processing unit 300 at the point "H" of FIG. Figure 13B shows the actual drawing range 1309 from the start position indicated by the start parameter SP ("0.0") of the start curve segment 1305 to the end position indicated by the end parameter EP ("1.0") of the end curve segment 1307. Is illustrated. It should be noted here that the first curve segment 1305 to be drawn is between p1 and p2, not between p0 and p1. This is because p0 is only used as a control point to define the curve segment between p1 and p2. Similarly, the last curve segment 1307 to be drawn is between p11 and p12, not between p12 and p13. This is because p13 is just the control point used to define the curved segment between p11 and p12.

&lt;Ink Data Manipulation (Slicing)&gt; Hereinafter, slicing operations according to various embodiments enabled by the system and method of the present invention will be described with reference to FIGS. 14 to 19.

FIG. 14 is a flowchart of the slicing operation applied to the existing stroke object 210. The slice operation is jointly executed by the stroke object operation unit 122 and the operation object operation unit 128. The illustrated example assumes the slicing operation shown in FIG. 4B, where the newly drawn stroke j403 slices the existing stroke i401.

First, the ink data generation unit 120 (stroke object operation unit 122) receives new pen event data ("input 1"). In step S1401, two processing threads starting from step S1403 and step S1405, respectively, are executed in parallel.

In the first processing thread, the ink data generation unit 120 generates a stroke object (stroke j) by the processing described in FIG. 9 in step S1403. In the second processing thread, in parallel, in step S1405, the ink data generation unit 120 (operation object operation unit 128) uses the operation information ("input 3") to generate a new stroke object 210. Determines if it is associated with the pen event data ("Input 1") used by the processing thread of. For example, the operation information ("input 3") may indicate that the associated stroke object 210 intends to perform a slicing operation.

If there is no such "input 3" (No in step S1405), the process proceeds to the end and the newly generated stroke object (stroke j) is treated as a normal stroke object 210. If there is an "input 3" associated with the stroke object 210 (Yes in step S1405), the stroke j is treated as an operation object 270 to perform a slice operation on one or more existing stroke objects 210. Is done.

&lt;Slice operation&gt; The loop starting from step S1407 and ending at step S1423 is repeated for each of the existing stroke objects (stroke_1 to stroke j-1). In this example, the slice operation represented by the newly generated stroke j is applied to each of the existing stroke objects. There are various techniques that can be applied to the operating object 270 to skip strokes isolated from the point of view of position. For example, the stroke object operation unit 122 can maintain an index including stroke segments in the area. Such an index can be used by the Operation Object Manipulator 128 to skip unnecessary intersection calculations.

Step S1407 sets up the loop.

In step S1409, the operation object operation unit 128 determines whether the new stroke j403 intersects the existing stroke i401 (i &lt;j), as shown in FIG. 15A. FIG. 15 illustrates a new stroke j that intersects the existing stroke i at the intersection P_intersect_Mid between the two control points p6 and p7 of the existing stroke i. In the illustrated example, since the new stroke j has a width, the intersection is calculated as the intersection of the centerline of the new stroke j (indicated by the solid line passing through P2, P3, P4) and the existing stroke i. Will be done.

Returning to FIG. 14, the fact that step S1409 determines that the new stroke j does not intersect the existing stroke i means that the slicing operation is not applied to the existing stroke i. The process increments i by 1 and determines if the new stroke j intersects the next existing stroke i + 1.

If it is determined in step S1409 that the new stroke j intersects the existing stroke i, the slicing operation of step S1411 is applied to the existing stroke i.

In step S1413, as shown in FIG. 15B, the operation object control unit 128 sets the edge (dashed line) of the new stroke j based on the calculated intersection P_intersect_Mid (55,100) and the "width" (3.58) of the new stroke j. Derivation of two intersections P_intersect_L and P_intersect_R with the existing stroke i, respectively. In FIG. 15B, p6 of the existing stroke i is at (47,100) and p7 is at (63,100), while P2 of the new stroke j is at (60,110) and P3 is at (50,90). As mentioned above, the intersection P_intersect_Mid is calculated as the intersection between the lines p6_p7 and the lines P2_P3. The width of the new stroke j along the p6_p7 direction is derived as "width" / sinθ 4.0 ("deriving width"). Here, sinθ = (110-90) ÷ sqrt ((60-50) ^ 2 + (110-90) ^ 2).

The derived width of 4.0 is then calculated for the two intersections P_intersect_L and P_intersect_R by adding half of the derived width to the intersection P_intersect_Mid (55,100) or subtracting half of the obtained width from the intersection P_intersect_Mid (55,100). Used to do. In the illustrated example, P_intersect_L is calculated as (53,100) and P_intersect_R is calculated as (57,100). It is possible to do additional calculations on the S1413. For example, if strokes i and j are not flat at the intersection, additional calculations should take into account the actual curvature in the calculations for P_intersect_L and P_intersect_R.

In FIG. 14, in step S1415, the operation object operation unit 128 generates two instances of the stroke object i obtained as a result of the slice operation, as shown in FIGS. 16A and 17A, respectively. These two instances are of the "slice" type. The first slice shown in FIG. 16A contains control points p0 to p8 that define the curved segment between p2 and p7, and the second slice shown in FIG. 17A contains the curved segment between p6 and p12. Includes control points p5 to p13 to be defined. The first slice of FIG. 16A contains the hole segment 1801 as the end curve segment between p6 and p7 where the stroke i was sliced. On the other hand, the second slice of FIG. 17A contains the hole segment 1801 as the starting curve segment. The first slice retains the default start parameter "0.0" for the start curve segment p1_p2, but now requires a new end parameter for the newly generated end curve segment p6_p7. Similarly, the second slice retains the default end parameter "1.0" for the end curve segment p11_p12, but now requires a new start parameter for the newly generated start curve segment p6_p7. ..

In step S1417, the operating object manipulation unit 128 derives a new termination parameter (new_EP1) value 303 for the first slice of FIG. 16A, as shown in FIG. 18A. In FIG. 18A, the new end parameter 1803 ("new_EP1") of the first slice is a value indicating the position of P_intersect_L (53,100) located in the end curve segment p6_p7 of the first slice. The new termination parameter 1803 can be an absolute value (eg (53,100)) or a relative value (eg +6 in the X direction from p6) or relative to the distance between p6 and p7. It may be the ratio of the distance between p6 and P_intersect_L (eg 6/16 = 0.375). Percentages are useful because they can be used without further processing in interpolation calculations performed by various sections, such as the graphic processing unit 300 of FIG.

Figure 16B illustrates the data structure of the parameters that define the first slice. This data structure includes a start parameter (float type) which is "0.0" (default value) and an end parameter (float type) which is "0.375" (ratio) derived in step S1417. In Figure 16B, row 11 shows that the data structure defines a "slice" object that, once finalized (committed), becomes a stroke object 210. The "slice_from Index" on line 12 is an index value (eg, an integer) that indicates the number of the slice start point object. In the example of FIG. 16, the "slice_from Index" is "0" because the first slice object starts at point p0. The "slice_to Index" in line 13 is an index value (eg an integer) that indicates the number of the end point object of the slice, which is "8" in Figure 16 because the first slice object ends at point p8 (" The final point p8 is the control point for the end curve segment p6_p7).

Returning to FIG. 14, in step S1419, the operation object operation unit 128 derives a new start parameter (new_SP2) for the second slice of FIG. 17A, as shown in FIG. 18A. In FIG. 18A, the new start parameter 1805 ("new_SP2") for the second slice is a value indicating the position of P_intersect_R (57,100) located within the start curve segment p6_p7 of the second slice. The new starting parameter can be an absolute value (eg (57,100)), a relative value (eg +10 in the X direction from p6), or p6 for the distance between p6 and p7. It may be the ratio of the distance between P_intersect_R and P_intersect_R (eg 10/16 = 0.625).

Figure 17B illustrates the data structure of the parameters that define the second slice. This data structure includes an end parameter (float type) which is "1.0" (default value) and a start parameter (float type) which is "0.625" (ratio) derived in step S1419. In Figure 17B, row 21 shows that the data structure defines a "slice" object that, once finalized, becomes a stroke object 210. The "slice_from Index" on line 22 is "5" because the second slice object starts at point p5 (the first point p5 is the control point for the starting curve segment p6_p7). The "slice_to Index" on line 23 is "13" because the second slice object ends at point p13.

Returning to FIG. 14, in step S1421, the operation object operation unit 128 uses the first slice in FIG. 16B (first slice object 274) and the second slice in FIG. 17B (second slice object 274). And are finalized (or committed) for rendering to the first newly generated stroke object 210 and the second newly generated stroke object 210, respectively, and stored in the storage device 770. As shown in FIG. 6, the finalizing step S1421 is an "M1 (local)" operation that occurs between the operation object operation unit 128 and the stroke object operation unit 122. At this point, the stroke object control unit 122 may discard the first and second "slice" objects and / or the original stroke object i.

Figure 16C illustrates the rendered (displayed) path of the finalized first stroke object 210, and Figure 17C shows the rendered (displayed) path of the finalized second stroke object 210. Is explained. As shown in Figure 16C, the curve segment between p1 and p6 is fully displayed, but for the end curve segment between p6 and p7, only the part up to the end point indicated by the end parameter 303 (0.375). Is displayed. The part from p6 to the end point indicated by the end parameter 303 accurately follows the shape of the original stroke object 210. Because the newly generated first stroke object 210 maintains the same control points p5 ~ p8 that define the end curve segment between p6 and p7, which was contained in the original stroke object 210. Because there is. Similarly, as shown in Figure 17C, the curve segment between p7 and p12 is fully displayed, but for the start curve segment between p6 and p7, at the start point indicated by start parameter 301 (0.675). Only the part that starts is displayed. The portion from the start point to p7 indicated by the start parameter 301 exactly follows the shape of the original stroke object 210. Because the newly generated second stroke object 210 maintains the same control points p5 ~ p8 that define the start curve segment between p6 and p7, which was contained in the original stroke object 210. Because there is.

In FIG. 14, in step S1422, the operating object control unit 128 points the newly generated first and second stroke objects 210 to the remote user's receiver, which is more complete with reference to FIGS. 20 and 21 below. Can be sent as described in. The transmission step S1422 is the "M2 (remote)" operation shown in FIG. 6, and is a remote coupled by an external network of the ink data processing unit 100 via the operation object operation unit 128 and the ink data shaping unit 140. Occurs with the user's receiver. Alternatively, in step S1422, the operation object operation unit 128 may transmit two slice objects 274 instead of the newly generated stroke object 210. In this case, step S1421 is performed at the receiving device. This will be fully described below with reference to Figure 20.

In step S1423, the loop processing that is repeated for each of the plurality of existing stroke objects is completed.

In step S1427, the operation object operation unit 128 flushes (destroys) all slice objects 274 generated and used in the slice operation step of S1411 above (if not yet).

The slicing operation described above is essentially replacing the original stroke object 210 to be sliced with two new stroke objects 210 resulting from the slicing operation. Therefore, the original stroke object 210 may be destroyed after the two new stroke objects 210 are created.

In the embodiment described in FIG. 14, a new stroke object j associated with the operation information ("input 3") is configured to perform a slice operation on one or more existing stroke objects 210. Treated as operation object 270. In other words, the stroke object j is used as the operation object 270.

In another embodiment, one or more computers (10-1-1,10-1-) in which the slice of object 274 generated during the slice operation step of S1411 in FIG. 14 shares an existing stroke object 210. 2,10-3, ...) Can be used as an operation object 270 to perform a slice operation on one or more existing stroke objects 210. In this case, slice object 274 is not destroyed. The use of slice object 274 as operation object 270 will be described more fully below with reference to FIGS. 20 and 21A.

In yet another embodiment, the hole segment 1801 shown in FIGS. 16A, 17A, 18A is defined and used as the operation object 270 to perform a slice operation on one or more existing stroke objects 210. A segment object can be created. Figure 18B illustrates the data structure of the parameters that define the hall segment. Line 01 shows that this data structure defines a "hole segment" object that is different from the slice object 274 and stroke object 210. The "hole_from Index" in line 02 is an index value (eg, an integer) indicating the number of the starting point object of the hole segment. In this example, the hole segment is between p6 and p7, so it is "6". The "hole_endParameter" in row 03 is the "0.375" (percentage, float type) derived in step S1417 of Figure 14 and indicates the end point of the end curve segment of the first slice generated by the hole segment 1801. ing. Line 04 "hole_to" "Index" is an index value (for example, an integer) indicating the number of the end point object of the whole segment. In this example, the hole segment is between p6 and p7, so it is "7". The "hole_startParameter" in row 05 is the "0.625" (percentage, float type) derived in step S1419 of Figure 14 and indicates the start point of the start curve segment of the second slice generated by the hole segment 1801. ing. The use of the whole segment object as the operating object 270 will be described more fully below with reference to FIGS. 20 and 21B.

In the embodiments described above, P_Intersect_L is used as the new end point for the first slice and P_Intersect_R is used as the new start point for the second slice, but in other embodiments the same point. May be used as both a new end point for the first slice and a new start point for the second slice. For example, for simplicity of calculation, P_Intersect_Mid may be used as both a new end point for the first slice and a new start point for the second slice.

According to an embodiment of the invention described above, the display of sliced curved segments (eg, "hole segment" 1801) has a new end parameter 303 that defines the end point at which the display of the first slice ends. And a new start parameter 301 that defines the starting point where the display of the second slice starts. Both the first and second slices maintain data that fully defines the structure of the whole segment, limiting the portion of the hole segment that is displayed as part of the first and second slices. I'm just doing it. Therefore, these slices that are rendered (displayed) exactly follow the shape of the original stroke object 210. In addition, since these slices maintain the same data (eg, control points) that completely define the Hall segment as being within the original stroke object 210, the position of the control points can be recalculated or There is no need to calculate new control points. The systems and methods of the invention are preferably used with curve interpolation algorithms characterized by "local control" as described above, such as Catmull-Rom curves and Polyvezier curves. The slice obtained by the slicing operation should then maintain a minimum amount of data (minimum number of control points) to define the whole segment obtained from the slicing operation.

FIG. 19 illustrates a sample file written in Stroke File Format (SFF) and containing the two newly generated stroke objects 210 displayed in FIGS. 16C and 17C. FIG. 19 can be compared to FIG. 11 which illustrates a sample of the original stroke object file.

In FIG. 19, the dashed box 1910 shows the file containing the first newly generated stroke object 210 shown in FIG. 16C. The first stroke object file contains points x0 to x8 corresponding to the point objects p0 to p8 in FIG. 16C. The start parameter field 1912 contains the default value "0.0" and the end parameter field 1912 contains the value "0.375" derived in step S1417 of FIG. The dashed box 1920 shows the file containing the second newly generated stroke object 210 shown in Figure 17C. The second stroke object file contains points x5 to x13 corresponding to the point objects p5 to p13 in FIG. 17C. The start parameter field 1922 contains the value "0.675" derived in step S1419 of FIG. 14, and the end parameter field 1912 contains the default value "1.0". The first stroke object file 1910 may be an overwritten or modified instance of the original stroke object file (Figure 11), duplicated from the original stroke object file to form an independent copy. There may be. The "newly generated" file used here may mean either of these file types. The second stroke object file 1920 may also be a duplicate copy of the original stroke object file, as shown in field 1921 in FIG.

FIG. 20 is a detailed flowchart of the ink data transmission process in step S1422 of FIG. In step S2012, the ink data processing unit 100 has a remote user who shares (i) the ink data 200 (a set of stroke objects 210) generated by the ink data processing unit 100 from the context information ("input 2"). Obtain information indicating whether or not to (ii) the message type used to send the ink data 200 to the remote user.

In step S2014, the ink data shaping unit 140 of the ink data processing unit 100 determines whether the ink data 200 of the local device is shared with another remote computer based on the above information (i). If there is currently no remote user or computer using the ink data 200, then in step S2022 the ink data shaping unit 140 optionally produces two newly generated stroke objects 210 (SFF format file in FIG. 19). It may be buffered in the type D message format shown in Fig. 21D. If the remote user later joins the ink data generation session of the ink data processing unit 100, the file formatting unit 140 can send a type D message to the remote user. At this point, it is not necessary to send the slicing operation object j. This is because the remote user does not see the slice operation in real time and only needs to receive the result of the slice operation, that is, the two newly generated stroke objects 210.

If it is determined in step S2014 that a remote user exists, the message to be used by the ink data processing unit 100 in step S2016 to send the ink data 200 to the remote user based on the above information (ii). Determine the type. The appropriate message type can be selected depending on the type of operational operation supported by the remote user's receiver (see Figure 22), the amount of time delay allowed in a particular application, the amount of transmit resources available, and so on. .. For example, if the information (ii) indicates that the remote user's receiver does not support any operation (eg slice) operation, the process proceeds to step S2022, where the ink data shaping unit 140 performs the slice operation. Buffer only the result, that is, the newly generated stroke object written in the type D message, and send it to the remote user.

In step S2016, if the above information (ii) indicates that the remote user's receiver supports the stroke object 210 as the operating object 270 to slice one or more stroke objects 210 on the receiver. If so, the process proceeds to step S2020. In step S2020, the ink data shaping unit 140 can use the ink data communication unit 144 to transmit the operation (slice) stroke object j written in the message type C shown in FIG. 21C.

The type C message shown in FIG. 21C has a header that includes a type field F101, a drawing area ID field F102, and a stroke ID field F103 that identifies the stroke object j. The type field F101 indicates that this message is of the "slicer" type that performs (executes) the slice operation. In this example, the display of the "slicer" type in F101 constitutes the operation information ("input 3") indicating that the associated stroke object j is the operation object 270. The payload part of a type C message contains an operation (slice) stroke object j that contains point objects p1 to pn. One of the advantages of using a type C message to send a stroke object j as operation object 270 is that both the sender and receiver can generate the same slice operation in real time at the same time. .. The sender can send the stroke object j with the operation information ("input 3") to the receiver substantially at the same time as the stroke object j is generated, and therefore the stroke object j on both sides at the same time. This is because the same operation can be performed by.

Returning to step S2016 in FIG. 20, if the above information (ii) causes the remote user's receiver to slice one or more stroke objects 210 on the receiver with the slice object 274 or whole segment object as the operating object 270. If it is shown that it supports execution, the process proceeds to step S2018. In step S2018, the ink data shaping unit 140 can use the ink data communication unit 144 to transmit the slice object 274 as the operation object 270 written in the message type A shown in FIG. 21A. Alternatively, in step S2018, the ink data shaping unit 140 uses the ink data communication unit 144 to transmit the hole segment object (FIG. 18B) as the operation object 270 written in the message type B shown in FIG. 21B. Can be done.

The type A message shown in FIG. 21A has a header that includes a type field F101, a drawing area ID field F102, and a stroke ID field F103 that identifies the stroke object j. The payload portion of a Type A message contains one or more pairs of slice objects 210 resulting from slicing one or more existing stroke objects 210 by stroke object j. Figure 21A illustrates two such pairs. One is the first slice and the second slice shown in fields F111_31 and F111_32, respectively, which are the result of slicing the existing stroke 3 by the stroke object j. The other is the first slice and the second slice shown in the fields F111_i1 and F111_i2, respectively, which are the result of slicing the existing stroke i by the stroke object j (description of the other pair). Is omitted). Type A message type field F101 indicates that this message is of the "slicer" type that performs (executes) the slice operation. In this example, the display of the "slicer" type in F101 constitutes the operation information ("input 3") indicating that the pair of related stroke objects 274 forms the operation object 270. Upon receiving a Type A message, the receiver retrieves each pair of slice objects 274 and creates two new stroke objects 210 that can be drawn on the display, the original (existing) strokes to be sliced. Refer to object 210 to finalize slice object 274. One of the benefits of using a type A message to send a slice object 274 as an operation object 270 is the slicing stroke that the data size of the slice object 274 is typically included in a type C message. Object 2

The type B message of FIG. 21B has a header that includes a type field F101, a drawing area ID field F102, and a stroke ID field F103 that identifies the stroke object j. The payload portion of a Type B message contains one or more hole segment objects resulting from slicing one or more existing stroke objects 210 by the stroke object j. Figure 21B illustrates two Hall segment objects. One is the hole segment stroke_3 in the field F111_3H, which is the result of slicing the existing stroke 3 with the stroke object j. The other is the hole segment stroke i in the field F111_iH, which is the result of slicing the existing stroke i with the stroke object j (the description of the other hole segment objects is omitted). Type B message type field F101 indicates that this message is of the "slicer" type that performs (executes) the slice operation. In this example, the "slicer" type display in F101 constitutes operational information ("input 3") indicating that the associated hall segment object forms the operational object 270. Upon receiving the type B message, the receiver retrieves the whole segment object and slices it to generate two new stroke objects 210 that can be drawn on the display, similar to the type A message described above. Refer to and execute the (existing) stroke object 210 of. One of the advantages of a type B message for sending a whole segment object as an operation object 270 is that the data size of the whole segment object is typically included in a type C message, the stroke object 210 to slice (stroke). Data size of object j) and type D message

FIG. 22 is a functional block diagram of an ink data receiving device configured to remotely receive ink data 200 via a network according to an embodiment of the present invention.

This receiver includes a network communication unit 310 (Tx, Rx), an ink data processing unit 100R, an application unit 300-2, a graphic processing unit 300R, a display 113R, and an operating system 400-2. In a typical embodiment, the ink data processing unit 100R is integrated into a library that realizes the ink data processing unit 100 on the transmitting side. Therefore, the receiving device generally performs a relative function corresponding to the function performed by the ink data processing unit 100.

In FIG. 22, the points "D", "E", "F_in", "G", and "H" correspond to each point of the ink data processing unit 100 on the transmitting side. In FIG. 22, M2'(remote) indicates the point at which the operation object 270 transmitted from the transmitting M2 (remote) is received. M1'(local) indicates the point corresponding to the sender's M1 (local), where the slice object 274 executed in step S1421 of FIG. 4 is finalized (committed) d). However, in FIG. 22, it is excluded that the slice object 274 (or the operation stroke object 210 that generates the slice object 274) is not internally generated and is received from the sender.

The network communication unit 310 (Tx, Rx) receives a packet and extracts various ink data messages as described in FIG.

The ink data processing unit 100R includes an ink data shaping unit 140R and an ink data generation unit 120R. The ink data shaping unit 140R corresponds to the ink data shaping unit 140 on the transmitting side, and similarly, the ink data 200 written in the stroke file format (SFF), which is the recording type, via the point "E". Ink data communication configured to receive the recording format operation unit 142R configured to receive and the ink data 200 written in stroke message format (SMF), which is the type of communication, via the point "F_in". Including part 144R. The data communication unit 144R operates (slices) the received message on the operation object 270, that is, one or more existing stroke objects, based on the value contained in the header field F101 of the received message ("input 3"). Determines if it contains a special type of stroke object 210 (eg, slicer type stroke object), slice object 274, or hole segment object configured to perform the operation.

The ink data generation unit 120R corresponds to the ink data generation unit 120 on the transmitting side. Unlike the transmitting ink data processor, the receiving device does not receive pen event data to generate the stroke object 210. Instead, the ink data generator 120R receives various objects such as stroke object 210, operation object 270, drawing style object 230, and metadata object 250. The ink data generation unit 120R processes (for example, operates) the received stroke object 210, and stores the operated (deformed) stroke object 210 in the storage device 770 in order to store the stroke object operation unit 122 and the operation object operation unit. Use 128.

The graphic processing unit 300R executes a process corresponding to the process executed by the graphic processing unit 300 on the transmitting side. As shown in FIGS. 13A and 13B, the graphic processing unit 300R uses the point object contained in the stroke object 210 as a control point and generates an interpolation curve according to a curve interpolation algorithm such as the Catmal-Rom curve algorithm. Reconstructs the stroke (path). The graphic processing unit 300R further refers to the start parameter and the end parameter defined for the stroke object 210, and controls the display range (part) of the stroke object 210. For example, if the start parameter is 0.5 and the end parameter is 0.5, the graphic processing unit 300R starts drawing (display) the stroke object 210 from the midpoint of the start curve segment, and at the midpoint of the end curve segment. Stop the display of the stroke object 210. The graphic processing unit 300R also refers to the drawing style object 230 received in association with the stroke object 210, and adds width, color, and other graphic attributes to the rendered stroke object 210.

The display 113R corresponds to the display 113 on the transmitting side. Application unit 300-2 is supported by operating system 400-2 and is dynamically or statically linked to a library that implements ink data processing unit 100R. The application unit 300-2 may include applications that are the same as, similar to, or different from the applications supported by the transmitting application unit 300-1.

FIG. 23 is a flowchart illustrating a reception process of an operation (slice) object that can be performed by the receiver of FIG. 22 according to an embodiment of the present invention.

In step S2301, the ink data shaping unit 140R receives the ink data message received via the network communication unit 310, and the message included in each message is transmitted from the type field F101 included in the message header ("input 3"). Take out the type of.

In step S2303, the ink data shaping unit 140R determines whether the message includes the standard stroke object 210 or the operation object 270. As mentioned above, the operation object 270 may be any of the stroke object 210, the slice object 274, or the whole segment object associated with the operation information ("input 3").

The stroke object 210 can be written by a stroke message format (SMF) file as shown in FIG. 12 or by a stroke file format (SFF) file as shown in FIG. The operating object 270 is preferably written in SMF, as shown in FIGS. 21A-21C.

If it is determined that the standard stroke object 210 has been received, the stroke object operation unit 122 adds the received stroke object 210 to the storage device 770R in step S2305.

If it is determined that an operation (slice) object has been received, the operation object operation unit 128 retrieves one or more operation objects contained in the received message in step S2311 and retrieves the list of operation objects in step 2313. Generate. For example, if a Type A message shown in Figure 21A is received, this list will contain two pairs of slice objects 274 contained in fields F111_31, F111_32, F111_i1 and F111_i2 (the other pairs are omitted). ) Is identified as an operation object. If a Type B message shown in Figure 21B is received, this list identifies the two hole segment objects contained in fields F111_3H and F111_iH (the description of the other hole segment objects is omitted) as operation objects. It will be a thing. If the type C message shown in Figure 21C is received, this list will identify the stroke object j as the operation object 270.

Subsequent steps S2315 to S2318 are repeated for each operation object contained in the list.

In step S2315, the operation object operation unit 128 determines the target stroke object 210 to which the operation object 270 should be applied. For example, in the case of a Type A message in Figure 21A, an operation object 270 in the form of a pair of slice objects 274 in fields F111_31 and F311_32 is applied to an existing stroke 3 and a pair of slice objects 274 in fields F111_i1 and F311_i2. It is determined that the shaped operation object 270 is applied to the existing stroke i.

In step S2317, the operation object control unit 128 applies the operation object 270 to the target stroke object 210 identified in step S2315 to perform the operation operation, and operates to generate a fully defined stroke object. Commit (finalize) the operation. The generated stroke object is then stored in the storage device 770R of the stroke object operation unit 122. The operation of step S2317 generally corresponds to the operation of step S1421 shown in FIG. In a nutshell, committing or finalizing an operation operation reflects the result of the operation operation on the resulting stroke object.

In step S2318, the operation object operation unit 128 determines whether all of the operation objects identified in the list in step S2313 have been executed for each target stroke object. If not, processing returns to step S2315 to process the next (remaining) operation object 270.

If all of the operation objects identified in the list are executed for each target stroke object and those operation operations are committed (finalized) to generate a new set of stroke objects, then the operation object operations Part 128 flushes away the slice object 274 and (if any) whole segment object used as the operation object in step S2319.

The transmitting side ink data processing unit 100 and the receiving side ink data processing unit 100R are described as software libraries operating on the operating system 400-1 or 400-2, but the ink data processing units 100 and 100R are specified. It may be implemented in a different manner, such as an application specific integrated circuit (ASIC) or IC.

Therefore, according to the method of processing ink data according to the first embodiment, as described in FIG. 91, a part of the stroke object 210 is modified or manipulated (for example, sliced) to be used for one or more other computer devices. It will be possible to send the change / operation to it.

Second Embodiment In the second embodiment of the present invention, the ink data 200 is communicated (transmitted, relayed) between a plurality of devices (transmission device, relay device, reception device) sharing a common drawing area. It is directed to methods and systems for (to, receive, process, and stream). In particular, this method and system makes it possible to superimpose multiple layers of ink data 200, each written by different devices in a common drawing area, in real time, in the correct communication order, and in a timely manner.

The second embodiment of the present invention is particularly suitable for realizing a real-time collaboration application in which a plurality of users can simultaneously and in real time perform handwriting (freehand) input to a common drawing area (or canvas).

For use in real-time collaboration applications, the invention provides a full set of attributes (color, trace) that can be shared without perceptible time delays through a novel communication / reproduction scheme that controls the transmission timing of pieces of ink data 200. Alternatively, a method and system having the ability to generate ink data having stroke width, rendering (drawing) style, etc.) are provided.

According to one aspect, the ink data 200 is a plurality of stroke objects (drawn) generated using different types of devices, and a drawing style object 230 (eg, which type of pen) that characterizes these stroke objects. The tip is used to draw the stroke object) and the ink data 200 is rendered within the common drawing area. Some applications, such as real-time collaboration applications, have strict (fast) time requirements. On the other hand, other types of applications do not have such strict time requirements. Devices and methods according to one aspect of the invention are configured to transmit / relay / receive ink data 200 in a timely, correct order and in the correct format, according to the requirements of the particular application in use.

According to one aspect, the invention is a method performed by a transmission device to communicate with a plurality of receivers, each sharing a drawing area with the transmission device, where the transmission device is detected by the input sensor of the transmission device. The ink data 200 representing the trace of the input operation is transmitted to a plurality of receiving devices. This method generally includes three steps: (a) ink data generation step, (b) message formation step, and (c) transmission step. The ink data generation step is (i) a step of continuously inputting pen event data (input 1) generated according to the movement of the pointer and generating fragmented data of the stroke data, and the stroke object is a pointer of the pointer. It contains multiple point objects to represent the motion trace, and the fragmented data is a step generated for each specified unit T and (ii) a pendown corresponding to the generation of pen event data at the start point of the trace. A step of generating a drawing style object 230 based on contextual information (input 2) in time, the drawing style object 230 includes a step of defining a drawing form of the trace of the stroke object. The message formation steps are (i) a step of generating a first message containing the drawing style object 230, and (ii) a step of generating one or more second messages following the first message, one or more. The second message of contains a step containing fragmented data. Finally, the transmission step includes the step of transmitting the first message and one or more second messages in sequence according to the defined communication protocol.

According to another aspect, the relay method receives ink data 200 representing a trace of the input operation detected by the input sensor of the transmission device, and shares the received ink data 200 with the transmission device in a drawing area. It is provided to relay to the receiving device of. This method generally involves three steps: (a) a receive step, (b) a control step, (c) a data message relay step, and (d) a send step. The receiving step includes receiving a control message containing information about the drawing area and receiving a data message containing the ink data 200 rendered in the drawing area. The control step involves updating a connection list that lists the communication addresses of multiple receivers that share a drawing area. The data message relay step (i) determines whether to allow the drawing area to be updated based on the stroke object 210 contained in the data message, and (ii) if the update is allowed, said connection. Generating a new data message to be relayed directly or via another relay device to the listed receivers, (iii) if the update is not allowed, the drawing area update request is rejected. Includes generating a deny message indicating that. Finally, the transmission step involves transmitting a new data message in a communication packet to the receiving device.

According to a further aspect, the invention provides a method of receiving ink data 200, which generally comprises four steps. The first step involves receiving the first fragment of the stroke object 210 in the message associated with the first fragment ID. The second step involves receiving a third fragment of the stroke object 210 in the message associated with the third fragment ID. The third fragment ID is not continuous with the first fragment ID. The third step involves interpolating the second fragment of the lost stroke object 210 and displaying the interpolated second fragment based on the first and third fragments of the received stroke object 210. .. The fourth step includes sending a request containing the stroke ID of the stroke object 210 in order to request the resend of the stroke object 210 as a whole after receiving the end of the stroke object 210.

Yet another aspect provides a method of streaming ink data containing a plurality of stroke objects using a server in which the ink data 200 is stored. This method generally involves two steps. The first step involves reading the stored stroke objects in sequence. The second step involves sending the stroke object from the server to one or more receivers at defined timings corresponding to each continuous portion of the stroke object.

Description of Second Embodiment In FIG. 1 described above with reference to the first embodiment, ink data 200 is generated, communicated (transmitted, relayed, and according to the second embodiment of the present invention. Describe the entire system that gets (received, processed, swept, etc.). For example, device 10-3 in FIG. 1 uses application service # 2, which may be a real-time collaboration type application, and ink data output from device 10-1-1 and device 10-1-2 in real time, respectively. 200 can be combined (synthesized). The device 10-3 renders (draws) the ink data 200 output from the device 10-1-1 and the device 10-1-2 on the screen in real time as a plurality of layers that are different from each other and are overlapped with each other. May be good.

The ink data 200 that is generated and communicated between different devices according to embodiments of the present invention is shared on communication resources by various types of devices, computers, operating systems, or applications.

FIG. 25 is an entity-relationship diagram of the information model for the ink data 200. FIG. 25 differs from FIG. 2 in that the stroke object of FIG. 25 does not need to include the start parameter 301 or the end parameter 303, but is otherwise the same as FIG. The ink data 200 according to the embodiment of the present invention controls the stroke object 210 (or stroke object set 202) (see FIG. 3) and the shape and color of the stroke object 210 during rendering (drawing, visualization, rasterization). Includes a drawing style object (set) 230 and contains the information needed for it. FIGS. 3B and 3C described above with reference to the first embodiment illustrate the rendering results of the same stroke object 210 shown in FIG. 3A by two different drawing style objects 230, respectively. The ink data 200 also includes a metadata object (set) 250 that includes non-drawing related information that describes the stroke object 210, such as author, pen ID, locally acquired date and time information, and GPS acquired location information. include. The ink data 200 further includes an operation object (set) containing information necessary for manipulating (eg, selecting, rotating, slicing) an existing stroke object 210. Such information is organized in the form of multiple operation objects that can be performed on the existing stroke object 210, each to bring about the desired operation of the stroke object 210. Other operation objects may include, for example, cutting (erasing), deleting, copying, enlarging, and reducing the operation object.

FIG. 3A illustrates a stroke object 210 containing a plurality of point objects (points 1 to n). The stroke object 210 contains information that represents a trace of pointer movement. The pointer is a pen-shaped tool or finger. Specifically, the stroke object 210 includes the coordinates of a plurality of point objects (points 1 to n) forming a trace. Each point object can be associated with attributes such as radius, color, and transparency (opacity).

The coordinates of a plurality of point objects (points 1 to n) are acquired or derived by appropriately sampling pen event data (or pointer operation), and are generated according to the movement of the pointer. Interpolate between sampled points as needed. As mentioned above with reference to the first embodiment, a suitable curve algorithm, such as the Catmull-Rom interpolation algorithm, may be employed to represent the trace as a smooth curve.

There are generally two ways to generate (x, y) coordinates for multiple point objects. In the first method, while the pen event data is input, the points derived from the pen event data every unit time are output as "raw value type" points. In the second method, after all the points that make up one stroke are entered, higher-order functions such as approximate curves for strokes (Catmal-Rom curve, Bezier curve, etc.) are generated. The minimum number of point objects required to represent the approximate curve is obtained as "optimized point type" points. According to one embodiment of the invention, the two methods of generating (x, y) coordinates are selectively switched due to the time constraints of the communication of the ink data 200 required by each application.

As described in the different rendering results in FIGS. 3B and 3C, each drawing style object 230 has a unique shape, ie, how the trace is rendered (drawn or represented) on the screen or display. Contains information about whether it appears on the screen or display. Therefore, using different drawing style objects to render the same stroke object 210 results in different drawing of the stroke object 210 in terms of how they appear. The shape of the trace is the shape of one or more point objects (eg, the circle in Figure 3B and the flakes or petals in Figure 3C), the width of the trace (eg the radius of each circle in Figure 3B and the size of each flake in Figure 3C). , Angle or offset associated with each point object (see Figure 3C), trace color (or point object color), trace (or point object) transparency / opacity, trace texture (or point object texture) ) Etc. can be defined. The information required to draw the stroke object 210 is contained in the drawing style object 230 associated with that stroke object 210.

FIG. 26 is an overall communication system diagram according to an embodiment of the present invention. This communication system includes a transmission device 10-1, a relay server 10-2, and a plurality of receiving devices 10-3, 10-4, ..., 10-n. They all share and use the set of library resources 10 shown in Figure 1. Relay server 10-2 can also be part of a peer-to-peer network (similar to a content delivery network) of globally distributed relay servers for improved performance. The system of FIG. 26 is suitable for performing the transmission, relay, and reception methods of ink data 200 in order to realize real-time collaboration according to the embodiment of the present invention.

The transmission device 10-1 of FIG. 26 corresponds to the device 10-1-1 or the device 10-1-2 of FIG.

The transmission device 10-1 includes an input sensor 110, an input processing unit (not shown), an ink data processing unit 100T, an application unit 300-1, and a communication unit 112 ("Tx Rx"). The ink data processing unit 100T corresponds to the 100T shown in FIG. 5 of the first embodiment. The application unit 300-1 contains one or more user applications linked to the ink data processing unit 100T and supported by a set of library resources 10 (FIG. 1). In the illustrated embodiment, the application unit 300-1 executes the real-time application # 1 that utilizes the real-time communication function.

The relay server 10-2 is typically a relay server, and specifically provides an ink data exchange service. In the illustrated embodiment, the relay server 10-2 exchanges and relays the ink data 200 in real time, so that the transmission device 10-1 and the reception device 10 located at a remote location from the relay server 10-2 are relayed. -3,10-4, ..., 10-n will be given convenience.

The relay server 10-2 includes a communication unit (510 Rx, 511 Tx), an ink data relay unit 540, an application control unit 520, and a repository configured to store canvas data (or drawing data). These are described more completely below. According to one aspect of the invention, the repository stores information about the latest state of the continuously updated drawing area (canvas) in the stroke file format (SFF) described in FIG. Relay server 10-2 is requested by the transmit / receive device to allow all transmit / receive devices to share the latest state of the drawing area (ie, what the "canvas" looks like now). And returns the latest state of the common drawing area for the requested device. According to one aspect, relay server 10-2 absorbs differences between multiple communication protocols used by multiple devices to allow communication and exchange of ink data 200 between those devices. ..

Receivers 10-3, 10-4, ..., 10-n correspond to device 3 in Fig. 1, respectively, and correspond to the communication unit (310 "Rx Tx"), ink data processing unit 100R, graphic processing unit 300, and Includes application part 300-2. The application unit 300-2 executes the application # 2 that uses the definition of the ink data 200 processed by the ink data processing unit 100R and the communication protocol. Application # 2 of the receiver may be the same as application # 1 used by transmission device 10-1, and both applications # 1 and # 2 share the same definition and communication protocol of ink data 200. It may be different as long as possible.

The classification between the transmission device 10-1, the relay server 10-2, and the plurality of receiving devices 10-3, 10-4, ..., 10-n shown in FIG. 26 is for ease of illustration. However, depending on the application and implementation of the embodiments of the present invention, the various functions of these devices may be partially or wholly integrated, or may be further divided and distributed. For example, the receiver 10-3 may be equipped to perform the ink data processing functions of the transmission device 10-1 ("ink data generation" 120T and "ink data shaping" 140T as described in FIG. 27). Also, even if the transmission device 10-1 is equipped to perform the ink data processing function of the receiver device 10-3 ("ink data shaping" 140R and "ink data generation" 120R described in FIG. 31). good. In some implementations there are multiple transmissions, while in some implementations there are no transmissions. In the latter case, for example, relay server 10-2 maintains ink data 200 in its repository and is directed to one or more clients, receivers 10-3, 10-4, ..., 10-n. Stream ink data 200.

In FIG. 26, the dashed arrow indicates an ink data control message, such as a control message that sets a common drawing area (or canvas) in which the ink data 200 input via the plurality of devices is displayed superimposed on the plurality of layers. Explain the flow of. The ink data control message name begins with the prefix "CTRL_".

The solid arrow describes the flow of an ink data message containing the actual ink data 200 that is input via the plurality of devices so that they are superimposed on each other in the common drawing area. The name of the ink data starts with the prefix "DATA_INK_".

The "SMF" (stroke message format) shown in FIG. 26 is a message generated and arranged in a packet in a transmission format, and will be described later in FIG. 34. Briefly, the ink data 200 in the transmission format is suitable for real-time communication with no perceived delay. For example, the "raw" point object described above can be used to form the stroke object 210 in transmission format.

"SFF" (stroke file format) shown in FIG. 26 is a message generated and arranged in the recording format, and will be described later in FIG. 28. Briefly, the ink data 200 in the recording format is compressed and suitable for storage devices. For example, the "optimized point type" point object described above can be used to form the stroke object 210 in recording format. In a further example, in the SMF transmission format the color information can be processed in RGB, whereas in the SFF recording format the color information can be processed in YCC for compression efficiency.

In various embodiments of the invention, a transmission format (eg, SMF) is used to communicate and exchange ink data 200 in real time between multiple devices. The recording format (eg, SFF), on the other hand, is used to store a common drawing area (canvas) in the repository (which can be sent to each device in response to initial access).

The dashed arrow "A.CTRL_JOIN_REQ" is a message issued when the transmission device 10-1 first participates in the editing session of the common drawing area facilitated by the collaboration service. The transmission device 10-1 may be the first device to participate in the common drawing area, or may be a late participant participating in an editing session already started by another transmission device. The dashed arrow "A.CTRL_JOIN_REQ" is a message sent by the transmission device 10-1 to set the drawing area (or canvas) shared by the receivers 10-3, 10-4, ..., 10-n. Indicates the transmission direction of. The CTRL_JOIN_REQ message contains or associates with a message containing information about the environment of the transmission device (eg, a set of parameters describing its transmit / receive environment) that is negotiated or shared with relay server 10-2. It may be done.

For example, the parameters that describe the transmission / reception environment of the transmission device are the stroke transmission / reception unit (size), message retransmission control settings, etc. (524-1 in Fig. 30A), drawing parameters that define the pen toolset, and the coordinate system. Includes communication parameters such as (524-2 in Figure 30B) and user policy parameters such as priority in resource contention and a list of blocked users (524-3 in Figure 30C).

The dashed arrow "A.CTRL_JOIN_REP" indicates the transmission direction of the response message sent by the relay server 10-2 in response to the CTRL_JOIN_REQ message. CTRL_JOIN_REP contains the environmental data required by the transmission device 10-1 to transmit its ink data 200. In particular, this environmental data can include information about the latest state of the drawing area (or canvas) of the receivers 10-3, 10-4, ..., 10-n.

Solid arrow "C.DATA_INK_BGNOS" "DATA_INK_MVDOS" ... "DATA_INK_ENDOS" (see Fig. 34) is an update message for updating the drawing area (or canvas), and the timing according to the defined communication protocol described in detail later and The transmission direction of the ink data 200 transmitted by the transmission device 10-1 to the relay server 10-2 in the transmission format is shown.

The solid arrows "D.DATA_INK_BGNOS" and "DATA_INK_MVDOS" are processed by the relay server 10-2 and transmitted to the receiving devices 10-3, 10-4, ..., 10-n on their respective communication lines. The transmission direction of the ink data 200 is shown.

FIG. 27 describes the transmission device 10-1 of FIG. 26 in the form of a functional block diagram. The transmission device 10-1 includes a (touch / pen) sensor 100, an input processing unit 111, an ink data processing unit 100T, an application unit 300-1, a network communication unit (Tx, Rx) 112T, a graphic processing unit 300, and a display 113. And includes operating system 400-1.

The sensor 110 detects a handwriting motion or operation of the user (through a pointer such as a pen or a finger) and generates an input data signal expressing the detected handwriting motion. For example, electrostatic sensors, pressure sensitive sensors, electromagnetic resonance (EMR) based sensors can be used.

The input processor 111 receives an input data signal that typically depends on the particular sensor device and its driver running on the operating system 400-1, and other such as sensed coordinate positions and pointer pressure information. It is converted into "pen event data" including information, and the pen event data is output as "input 1". The pen event data still depends on the particular sensor device used to detect the handwriting motion input. The input processing unit 111 is typically provided by a driver for the sensor 110 or a library that performs processing corresponding to the sensor driver. For example, when an electrostatic sensor is used, the input process may include a process of determining a gesture based on continuously entered inputs, such as a palm rejection process. Since the pen event data is sensor / device dependent, the pen event data may or may contain pointer pressure information or pointer tilt information, depending on whether the sensor 110 has pressure / tilt detection capabilities. May not be. The configuration of the sensor 110 and the input processing unit 111 is not limited to that shown, and even if all or part of the sensor 110 and the input processing unit 111 are provided in the form of a digital stationery device such as a pen-shaped device. good.

The ink data processing unit 100T includes an ink data generation unit 120T and an ink data shaping unit 140T. The ink data processing unit 100T corresponds to 100T in FIG. 5 of the first embodiment. The ink data processing unit 100T is responsible for converting sensor / device dependent pen event data ("input 1") into ink data 200 that can be used and shared by various applications of various devices.

The ink data generation unit 120T extracts or receives pen event data depending on the sensor / device, and converts it into non-device-dependent ink data 200. The ink data 200 is a digital representation of the ink applied (painted, smeared, deposited on) on paper using a real pen. The ink data generation unit 120T corresponds to the stroke object operation unit 122 in FIG. 7. The ink data generator 120T is used for ink data (point objects p0 to pn-1 stored in the memory 770 of FIG. 7) for each specified unit T such as a specified time unit (for example, 5 seconds) or a specified data size unit. ) Is pulled out to generate a strone object 210 or a fragment (part) of the strone object 210 that represents the trace input by the handwriting operation.

The ink data generation unit 120T receives pen event data ("input 1") from the sensor 110, and also receives context information ("input 1") from the application unit 300-1 (for example, a real-time collaboration application) or the operating system (400-1). Input 2 ") is received.

The context information ("input 2") is information about the context or environment of the pen event data at the time when the first part of the stroke is drawn (ie, at the time of "pen down"). The context information is typically set by the application unit 300-1 prior to the generation of pen event data by the sensor 110. For example, the context information is the type of nib used (eg brush, crayon, pencil), stroke / trace color (red, green, blue), pen stroke transparency (or opacity) value (alpha). , Stroke / trace width, etc. can be included. The ink data generator 120T is used to draw (render) the stroke object 210 on the display based on the context information ("Input 2") at the start timing of the pen event data (in S605 in Fig. 32, which will be described later). Generates a drawing style object that is created.

Contextual information ("input 2") also includes non-drawing related information about pen event data such as author information, pen ID, date and time information, location information, and so on. The ink data generation unit 120T generates a metadata object 250 based on such non-drawing-related context information.

The ink data generation unit 120T additionally receives operation information ("input 3") from the application unit 300-1. Input 3 does not define the normal stroke object 210 as the next input from the sensor 110, but is an operation that embodies and executes an operation operation (for example, slice, delete, copy, enlargement, etc.) on an existing stroke object 210. Identify that it defines object 270. When input 3 is received, the next stroke object #j is formed as an operation object 270 for one or more existing stroke objects # 0 to #i, and the operation operation is applied.

The ink data shaping unit 140T includes an ink data communication unit 144T and a recording format data processing unit 142T. Generally, the ink data shaping unit 140T shapes the fragmented data of the stroke generated by the ink data generating unit 120T for each specified unit T (for example, arranges it in a transmission packet). The ink data shaping unit 140T also shapes the drawing style object 230, the metadata object 250, and the operation object 270 generated by the ink data generating unit 120T. The ink data shaping unit 140T formats various objects and fragmented data of each stroke object 210 into a message and a transmission packet according to the format selection information (Fmt_Sel) received from the application unit 300-1.

The ink data communication unit 144T executes the following functions when the format selection information (Fmt_Sel) received from the application unit 300-1 specifies the use of a transmission format (for example, SMF) as an output format.

1) The stroke object 210 generated by the ink data generator 120T is accepted as fragmented (or complete) data for each specified unit T, and according to implicitly defined or explicitly negotiated parameters. Generate various types of data messages (messages beginning with the prefix "DATA_INK_", as shown in Figure 34).

2) Accepts the drawing style object 230 generated by the ink data generator 120T, and adds the drawing style object 230 ("F104" in FIG. 34) to the "first" data message ("DATA_INK_BGNOS" in FIG. 34).

3) Determine a unique stroke ID for the stroke object 210 and add that stroke ID ("F103" in Figure 34) to all data messages that contain fragmented data for the stroke object 210.

4) Depending on the receipt of the response message ("DATA_INK_NACK" in Figure 36A or "DATA_INK_REJOS" in Figure 37), it may or may not be with the negotiated parameters (eg, 524_1f, 524_1a in Figure 30A). Retransmission processing (see FIGS. 35A, 35B, 36A) or ignoring processing (FIG. 37) is adaptively performed according to (there may be only 524_1f).

The recording format data processing unit 142T processes the ink data 200 generated by the application unit 300-1 into a recording format (for example, SFF) different from the transmission format. For example, the ink data 200 in the recording format indicates the current state of the common drawing area (combus) shared (accessed) by a plurality of users of the receiving devices 10-3, 10-4, ..., 10-n. Therefore, it may be uploaded from the transmission device 10-1 to the relay server 10-2 depending on the initial access to the real-time collaboration application. On the other hand, in order to communicate the ink data 200 between a plurality of devices in real time, a transmission format may be used instead of the recording format.

FIG. 28 illustrates a sample recording format used in embodiments of the present invention. This recording format differs from the transmission format (Fig. 34) in the type of data included / omitted, the order of the data, and the redundancy.

Types of data included / omitted For example, when an operation object 270 is generated to modify an existing stroke object 210, the operation object 270 to transform the ink data 200 present on one or more computers ( For example, the slice object 274) described above with reference to the first embodiment is generated and transmitted in a transmission format. The recording format, on the other hand, is sufficient to record only the state after the operation (update) is complete, and therefore does not need to maintain the operation object 270 itself. Therefore, the recording format example of FIG. 28 does not include the operation object 270.

Data Order and Redundancy Metadata object 250 contains non-drawing related information about stroke object 210 such as author information, pen ID, and so on. Non-drawing related information is often the same for several stroke objects 1-5 entered at the same time. For example, the same author using the same pen creates multiple stroke objects 1-5 in this order.

In the recording format, redundancy can be reduced because the entire data content is known at the time of formatting the data written in the recording format. In the illustrated example, authors with the same value can be applied to all stroke objects 1-5, so the same value does not have to be repeated 5 times. The drawing-related information contained in the drawing style object 230 also provides redundancy in some cases, for example, the same type of nib shape (eg, a brush) is used to draw multiple stroke objects. Can include. Therefore, as another example, the pen tip shape of the same value is applied to the stroke objects 1,3,5, and this value does not need to be repeated three times in the recording format. Similarly, if the same value of nib shape is applied to stroke objects 2, 4, the value does not need to be repeated in the recording format. In this regard, it is not important to maintain the temporal order of the stroke objects (ie, the order in which they were entered) within the recording format. Also, since each stroke object is completely complete when injected into the recording format, the total number of point objects that form one stroke object can be included within the stroke object itself.

On the other hand, in the transmission format, it is difficult to reorganize the temporal order of stroke objects if real-time communication is important. In Liam Time communication, it is generally necessary to transmit information about stroke objects 1 to 5 in the temporal order in which stroke objects 1 to 5 are input and generated. Also, when sending fragmented data for a single stroke object, it is not possible to know in advance how many point objects will be included in that stroke object. Therefore, it is necessary to indicate that for the last fragmented data that completes the stroke object (the last fragmented data may contain information about the total number of point objects contained within the stroke object). ..

According to embodiments of the present invention, the same amount of information can be contained in a recording format with fewer bytes than in a transmission format. This is because the transmission format needs to have redundancy and strict temporal order for the purpose of achieving real-time communication without perceptible time delay. Therefore, for the purpose of saving storage space, the latest drawing area information stored in the repository of relay server 10-2 is preferably formed in a recording format, which is then connected to relay server 10-2 in various ways. Can be accessed and withdrawn by a small amount of transmit resources.

Referring again to FIG. 27, the network communication unit 112T (Tx, Rx) of the transmission device 10-1 generates a communication packet (see FIG. 34) containing the message generated by the ink data processing unit 100T as the payload. Packets are output over a network interface connected to the media (such as Ethernet®).

Various communication protocols may be used, as appropriate, based on implicit or (via negotiation) definitions in terms of the communication environment. For example, protocols such as TCP, HTTP over TCP, SSL that include a built-in retransmission mechanism may be used (Figure 35A), or protocols such as UDP (or RTP / UDP) that themselves do not provide a retransmission mechanism. May be used (Fig. 35B). In addition, if UDP is used, a message resending mechanism may be employed (Figure 36A), or the message resending mechanism may be omitted for the purpose of meeting the time requirements of real-time applications (Figure 36B). ). It is also possible to use different communication protocols for the control message and the data message. For example, it is possible to adopt a protocol having a retransmission mechanism for transmitting the control message indicated by the dashed arrow, while adopting a protocol having no retransmission mechanism for transmitting the data message indicated by the solid line arrow. ..

The application unit 300-1 provides an application that uses the ink data transmission method according to the embodiment of the present invention. The application unit 300-1 sends CTRL_JOIN_REQ to the relay server 10-2 (specifically, 520 in Fig. 29) that manages the latest state of the common drawing area (or canvas) via the network communication unit 112T (Tx). And so on. The application unit 300-1 uses the input device 110 to determine a drawing area (canvas) on which drawing is performed.

The application processing unit 300-1 processes the following information and provides it to the ink data processing unit 100T of the transmission device 10-1.

-Context information such as color information, pen tip shape, author information, generation date and time (input 2), -operation information that identifies incoming inputs that make up the operation object (input 3), and / or-transmission format and recording format Format selection information (Fmt_Sel) that specifies which of the should be selected.

The application processing unit 300-1 receives the ink data 200 generated by the ink data generation unit 120T and transmits it to a remote receiving device (10-3, 10-4, ..., 10-n). It has the ability to output to both the device's own local display (if any).

The graphic processing unit 300 generates pixel data based on the ink data 200. The graphic processing unit 300 has an ability to immediately draw (render) a stroke object on a local display, which is a common drawing area in which ink data 200 input via a plurality of devices are superimposed as different layers. Can represent a state.

The ink data transmission apparatus 10-1 according to the embodiment of the present invention determines the latest (updated) state of the drawing area (cambus) for use, and every specified unit T (for example, 5 milliseconds). In addition, the pen event data is continuously processed to generate fragmented (or complete) data for the stroke object 210. The ink data transmission device 10-1 further generates related drawing style objects 230 and metadata objects 250 by selecting each part of the context information (input 2), and generates an operation object based on the input 3. , Format the generated objects into a transmission format for transmission to relay server 10-2 and receivers 10-3,10-4, ..., 10-n. The defined unit T can be adaptively determined based on the requirements from the collaboration application.

FIG. 29 is a block diagram of the relay (or streaming) server 10-2. The relay server 10-2 collaborates in real time by relaying fragmented data received from one or more transmission client devices to one or more reception client devices 10-3, 10-4, ..., 10-n. Provide services. The functions of the relay server 10-2 may be contained in a single device or may be distributed to a plurality of servers linked by a network.

The relay server 10-2 includes a message receiving unit 510, a drawing area management unit 520, an ink data relay unit 540, and a message transmitting unit 511.

The message receiving unit 510 receives the control message and the data message separately from the transmission device 10-1. These messages are in the form of communication packets that follow a protocol (RTP / HTTP / HTTPS (SSL) / TCP / UDP / Websocket, etc.) and are either predefined or the device first joins the common drawing area. It is transmitted between transmission device 10-1 and relay server 10-2 with a message type (fragmentation, retransmission, maximum delay, etc.) that may sometimes be defined based on on-demand negotiations between devices. As shown in the figure, the message receiving unit 510 can adopt a plurality of receiving ports in order to distinguish between a control message and a data message. However, other methods may be employed to distinguish between control messages and data messages. For example, a separation device (server) that processes control messages may be prepared separately from the relay server 10-2 that processes data messages. Also, message headers that can be obtained from a common receive socket buffer may be used to distinguish between the two types of messages.

The drawing area management unit 520 manages the state of the drawing area in which the ink data 200 input from a plurality of devices and exchanged through the relay server 10-2 are overlapped with each other as different layers. The drawing area management unit 520 includes the service management unit 522 and the drawing area information management unit 524.

The service management unit 522 manages a service using the ink data relay method according to the embodiment of the present invention. Service management 522 provides, for example, accounting functions, new user authentication functions, contract-based viewing access to the common drawing area, and allows each user to enter ink data 200 into the common drawing area. Alternatively, it cooperates with an external server (not shown) to perform functions that are not permitted, that is, so-called AAA (Accounting, Authentication, Authorizing) functions.

The drawing area information management unit 524 manages drawing area information used for controlling the operation of the ink data relay method. The drawing area information generally contains three types of information (FIG. 30Ak524_1, FIG. 30B 524_2, FIG. 30C 524_3). The drawing area information management unit 524 manages these three types of information, and based on the update of these three types of information, enumerates one or more users connected to each drawing area (canvas). Maintain and purge Listing 541.

FIG. 30A shows a set of communication parameters 524_1 related to the transmission and reception of ink data 200. These communication parameters are exchanged, for example, between devices communicating at the time the application is launched. Communication parameter 524_1 may include:

(524_1a) The packet retransmission parameter defines whether the retransmission mechanism is built into the communication packet (or protocol stack) used to carry the ink data 200. For example, if TCP is used for transmission (with retransmission support) while UDP is used for reception (without retransmission support), TRUE is set for transmission and FALSE is set for reception. This provides, for example, the transmission of ink data from the robust and reliable transmission device 10-1 to the relay device 10-2, while at the same time multiple real-time, perceptible delay-free relay server 10-2. Provides streaming of ink data 200 to receivers 10-3, 10-4, ..., 10-n.

(524_1b) MTU (Maximum Transmission Unit), MSS (Maximum Segment Size) parameter defines MTU and MSS depending on the type of media (eg Ethernet®) to which the transmitter is connected (eg 146 bytes). ..

(524_1c) The stroke data fragmentation possibility parameter sets whether or not the stroke object 210 is fragmented into a plurality of fragments, each of which is smaller than the stroke object 210 as a whole.

(524_1d) The maximum Rx delay parameter sets the maximum allowable receive delay, for example, in milliseconds (msec).

(524_1e) The message encryption scheme parameter defines whether encryption is used and, if so, what encryption / decryption algorithm is used.

(524_1f) The message retransmission parameter defines whether to perform retransmissions at the message layer for protocols such as UDP that do not include a retransmission mechanism in the communication layer. For example, this parameter is used to switch between using retransmissions at the message layer (Figure 36A) and not using retransmissions at the message layer (Figure 36B).

(524_1g) The voice synchronization enable / disable parameter defines whether the voice and stroke data are played back in synchronization, and whether the ink data 200 is transmitted in the form of fragments or in the complete form (in stroke units). Can be used to determine whether or not. For example, when audio and stroke data are played back in sync, the stroke data can be fragmented into pieces, each with a type stamp, so that each piece can be played in sync with the corresponding voice data. ..

(524_1z) The other parameter setting identifier defines a predetermined set of communication parameters.

FIG. 30B shows a set of drawing parameters 524_2 related to drawing (rendering) ink data 200, which may include:

(524_2a) The drawing area ID parameter sets a common drawing area in which the user device can participate. In the illustrated example, FIG. 30B shows the common drawing area (or canvas) ID "# 123". In some embodiments, the relay server 10-2 is co-edited within each and presents the user with multiple IDs of multiple canvases on which the user device can select one for participation. You may do it. In another embodiment, when the user device starts a new drawing area, a unique drawing area ID may be assigned to the newly started drawing area.

(524_2b) User-local canvas offset, rotation, and scale parameters are based on the global coordinate system of the common drawing area and the local coordinate system of the user device used to participate in the editing session of the common drawing area. Define the relationship between. For example, this allows two different users to edit the top and bottom of the ink data being drawn from two different angles, respectively.

The pen toolset ID is an ID assigned to a set of pen parameters (pen tip shape, color, stroke width, ink transparency / opacity, etc.) that jointly define how pen strokes appear on the drawing area. be. Multiple IDs are predefined for multiple sets, and the user can choose one of them when joining a collaborative editing session on a common drawing area.

Figure 30C shows a set of user policy parameters 524_3 related to policies that manage user access to common drawing areas, including stroke data priority, filtering, and quality of service (QoS), which may include:

(524_3a) The user account priority parameter sets the priority given to stroke data within the ink data transmission service. In the illustrated example, this parameter is set to "High", for example, the resource arbitration (or QoS) control 544 of relay server 10-2 sets the "High" priority stroke data to the priority. It means that the other stroke data set to "Low" is preferentially processed and transmitted.

(524_3b) The block user list includes other users that one user wants to block, i.e. users who do not want to receive ink data 200 from that user. This list may also include users who do not want to send ink data 200. Instead of listing a user to deny a receive from or send to that user, it is also possible to list a user to accept a receive from or send to that user.

The ink data relay unit 540 refers to the connection list 541, which lists all the devices currently connected to the drawing area to which the data message is added (drawn), and the data message received from the transmission device 10-1. To one or more other receivers. The connection list 541 also lists which communication, drawing, and user policy protocols should be used to communicate with each device listed in the connection list 541. In various embodiments, the ink data relay unit 540 processes (forwards, drops) stroke object messages on a message-by-message basis rather than on a packet-by-packet basis, but as a "one flow".

The ink data relay unit 540 includes a new stroke flow detection unit 542, a resource arbitration (or QoS) control 544, a feedback transmission unit 546 (DATA_ToSelf), and an allcast transmission unit 548 (DATA_ToOthers).

The new stroke flow detector 542 checks the header of the received message, whether the message contains a new stroke ("DATA_INK_BGNOS"), and the message is the same as the one for which relay processing has already started (currently). ) Determines if the stroke object 210 contains fragmented data.

Resource arbitration (or QoS) control 544 controls transmit resources by reference to the start of each stroke object 210 and, if necessary, the end of each stroke object 210. When processing the beginning of a new stroke (BGNOS), the resource arbitration (or QoS) control 544 determines whether to accept the new stroke in the drawing area (canvas) based on various criteria. For example, if the stroke ID of the stroke object 210 newly received from the transmission device 10-1 is the same as the stroke ID already used in the drawing area, the resource arbitration (or QoS) control 544 is the same. A newly received stroke object 210 with a stroke ID can be rejected. As another example, if the network resource is found to be insufficient to transfer or process the stroke object 210 at a particular point in time, the resource arbitration (or QoS) control 544 improves the situation of the network resource. Rejects the input of newly received stroke objects until. This prevents the situation where the user starts typing a new stroke object 210 just to stop the input process before completing the stroke object 210 due to lack of sufficient network or computer resources. do. As a further example, if a particular device (user) is temporarily denied renewal rights, resource arbitration (or QoS) control 544 rejects all stroke objects generated by that particular device. .. For example, the resource has a lien on stroke data transmitted from a user with a "Low" priority in processing stroke data transmitted from a user with a "High" priority (524_3 in Figure 30C). It is assigned in a certain state. When the resource arbitration (or QoS) control 544 decides to reject any new strokes received from any device, the fee Send "REJECT" to the do-back transmitter 546. When the resource arbitration (or QoS) control 544 decides to accept a new stroke received from one device, it is an all-cast transmitter for all other devices (except the device that transmitted the new stroke). Send (transfer) the new stroke via 548.

According to various embodiments of the invention, each stroke object 210 is fragmented into fragmented data and sequentially transmitted and relayed so that the remotely located device is in real time with no perceptible time delay. Stroke objects 210 can be shared as if they were entered. At the same time, resource arbitration (or QoS) control 544 resolves any conflicts between multiple users and provides sufficient resources for all users who are granted the right to enter a new stroke object 210. Performs a traffic control function that ensures that (for example, while the first user is typing, other users are temporarily prevented from typing).

The feedback transmitter 546 (DATA_ToSelf) sends back a response only to the transmission device 10-1 that has sent a message to the relay server 10-2. For example, when the resource arbitration (or QoS) control 544 determines that the transmission device 10-1 does not allow the input of the new stroke object 210, the feedback transmitter 546 rejects the message only to the transmission device 10-1. (See Figure 37).

The allcast transmitter 548 (DATA_ToOthers) sends a message to all devices currently connected to the drawing area (included in the connection list 541). However, the transmission device 10-1 that sent the message to the relay server 10-2 is excluded. In the illustrated example, when a message is received from the transmission device 10-1 and it is determined that the message can be relayed, the all-cast transmission unit 548 determines that all the reception devices 10-3, 10-4, ..., 10 Send a message to -n.

In this way, the relay server 10-2 receives the ink data 200 from one transmission device, optimally controls the efficient use of network resources as a whole, and transfers it to one or more receiver devices. Selectively relay.

FIG. 31 is a block diagram of the ink data receiving device 10-3 of the ink data communication system of FIG. 26. The receiving device 10-3 includes a network communication unit 310 (Rx, Tx), an ink data processing unit 100R, an application unit 300-2, a graphic processing unit 300, and an operating system 400-2.

The application unit 300-2 running on the operating system 400-2 includes an application that utilizes the ink data receiving method according to the embodiment of the present invention. The type of application or operating system may be the same as the application (or operating system) of application unit 300-1 of transmission device 10-1 or both applications are inked together through a common drawing area-other. It may be different as long as it allows 200 inputs, transmissions, and receptions. This is because commonly defined (standardized) ink data 200 can be transmitted and received independently of a particular platform between different applications and operating systems. At the timing of communication initialization in which the receiving device 10-3 participates in the editing session of the common drawing area, the application unit 300-2 has a drawing area (canvas) in which other users probably overlap each ink data 200 as different layers. ) Download the latest status.

The application unit 300-2 can upload the ink data 200 generated by the application unit 300-2 to the drawing area of the relay server 10-2. The ink data 200 locally generated by receiver 10-3 can be in recording format, perhaps prior to the collaborative editing session. On the other hand, the ink data 200 generated in real time during the collaborative editing session can be in transmission format. Ink data 200 in either recording format or transmission format can be uploaded to the relay server 10-2. In this connection, the application unit 300-2 outputs the ink data 200 generated by the ink data generation unit 120R, and instructs the graphic processing unit 300 to overlap the ink data 200 in the drawing area.

The network communication unit 310 (Tx, Rx) corresponds to the network communication unit 112T (Tx, Rx) of the transmission device 10-1. The network communication unit 310 receives a communication packet (Pckt) via the network interface and extracts a message from the payload part of the packet. The receiving protocol used by receiver 10-2 may differ from the communication protocol used by transmitter 10-1. For example, transmission device 10-1 may employ a protocol that includes a retransmission mechanism such as TCP, HTTP over TCP, HTTPS over SSL to send a message to relay server 10-2, while relay server 10-2 , In order to send a message to receiver 10-2, a protocol such as UDP, RTP over UDP, which is not so reliable but suitable for streaming may be adopted. Which communication protocol to use may be decided through negotiation between the communicating devices at the start of communication.

The ink data processing unit 100R includes an ink data shaping unit 140R and an ink data generating unit 120R, and takes out ink data 200 from ink data formatted by either a transmission format or a recording format. The ink data processing unit 100R corresponds to the ink data processing unit 100T of the transmission device 10-1.

The ink data shaping unit 140R includes an ink data communication unit 144R and a recording format operation unit 142R. The ink data communication unit 144R processes the ink data 200 updated in the format of the transmission format. Specifically, the ink data communication unit 144R takes out the information (for example, stroke ID) in each data message, and outputs the fragmented data which is a fragment of the stroke data 210 divided by the transmitting side in the specified unit T. The ink data communication unit 144R also retrieves the drawing style object 230 contained in the first message, the "DATA_INK_BGNOS" message, for the stroke object 210. The drawing style object 230 contains the information necessary to render (draw) the associated stroke object.

The recording format operation unit 142R receives and processes the drawing area information in the recording format as being stored (archived) in the repository of the relay server 10-2 in order to reconstruct the latest drawing area.

The application 300-2 controls the ink data generation unit 120R so as to selectively receive the ink data 200 from the ink data communication unit 144R or the recording format operation unit 142R according to the format of the output / input ink data.

For example, the first time an application participates in a collaborative editing session in a common drawing area, a "CTRL_JOIN_REQ" request is issued to retrieve the latest drawing area information in a file written in recording format (SFF). The extracted ink data 200 of the latest drawing area is written in a recording format, and is processed by the recording format operation unit 142R as such. The ink data 200 subsequently extracted may be in transmission format (SMF), as in "DATA_INK_BGNOS" and other messages (DATA INK *) shown in FIG. 34, as such, the ink data communication unit 144R. One message (or multiple messages representing one stroke) is processed at one time.

In order to realize real-time processing of the ink data 200, the ink data processing unit 144R may wait for the fragmented data of the stroke object 210 to be extracted from all the point objects (or fragmented data) forming the stroke object 210. When they are received, they are output to the graphic processing unit 300.

The ink data generation unit 120R of the receiving device 10-3 executes the reception processing corresponding to the transmission processing of the ink data generation unit 120T of the transmission device 10-1.

The ink data 200, which consists of a plurality of point objects, a drawing style object 230, a metadata object 250, and a plurality of stroke objects, each of which is an operation object related to the stroke object, has already been generated on the transmitting side. Therefore, it does not need to be newly generated on the receiving side. Rather, on the receiving side, the ink data generator 120R stores and updates the ink data 200 in a form that can be used by the application # 2. For example, the ink data generation unit 120R uses the received ink data 200 of the first data type (for example, integer type) as the ink data 200 of the second data type (for example, float type, double type) that can be used by application # 2. The transformed data is provided to the graphic processing unit 300 used by the application unit 300-2 or the application unit 300-2.

The graphic processing unit 300 of the receiving device 10-3 is similar to the graphic processing unit 300 of the transmitting unit 10-1. The graphic processing unit 300 performs a process of generating pixel data based on the received ink data 200, and displays a stroke object on the display screen 113 of the receiving device 10-3.

FIG. 32 is a sequence diagram illustrating ink data communication performed between the transmission device 10-1, the relay server 10-2, and the reception device 10-3 according to the embodiment of the present invention.

In FIG. 32, the functions and sections of transmission device 10-1, relay server 10-2, and receiver device 10-3 are identified by the same reference numbers used in FIGS. 27, 29, and 31 above. .. This sequence describes a case where the transmission device 10-1 participates in a real-time collaboration session that has already been started, and ink data 200 input by a plurality of users can be overlapped with each other.

&lt;Starting the application unit 300-1&gt; In step S601, the transmission device 10-1 activates the application unit 300-1 using the ink data communication method according to the embodiment of the present invention. The activation of the application unit 300-1 triggers the activation of the ink data generation unit 120T and the ink data shaping unit 140T in the ink data processing unit 100.

First, the application unit 300-1 of the transmission unit 10-1 sets one of the plurality of drawing areas held in the image area management unit 520 of the relay server 10-2 as the target drawing area (524_2a). Issue a control message to identify (CTRL_JOIN_REQ, with or without parameters 524_1,524_2,524_3). The drawing area management unit 520 of the relay server 10-2 returns the latest state of the specified drawing area to the transmission device 10-1 in a recording format (SFF) which may be highly compressed (CTRL_JOIN_REP). After that, the application unit 300-1 calls the recording format operation unit 142T to reconstruct the latest state of the drawing area based on the ink data 200 written in the recording format. The transmission device 10-1 thereby may have already started drawing (input) of ink data 200 on which other devices such as receivers 10-3 and 10-4 overlap each other (or ink). Rebuild (or initially build) the latest state of the data (current state) locally. At this point, the transmission device 10-1 enters the input waiting state.

&lt;Ink data transmission process&gt; The transmission device 10-1 executes the ink data transmission process by using the detection of "pen event data input start" as a trigger. In step S605, the input processing unit 111 of the transmission device 10-1 detects the input of the start point of the stroke. In FIG. 32, the pen event data is continuously input to the ink data processing unit 100T during the hatched period "d" from step S605 to step S609. In the meantime, one stroke is drawn so as to start at the stroke start point and end at the stroke end point (hereinafter referred to as "stroke unit"). During this time (period d), the input processing unit 111 continuously outputs pen event data to the ink data generation unit 120T every unit T, for example, every 5 milliseconds.

The ink data processing unit 100T uses the start of pen event data input as a trigger, and starts the process of generating the drawing style object 230 based on the context information (input 2) received from the application unit 300-1. Specifically, the ink data processing unit 100 draws (descends) the stroke object 210 on the common drawing area (cambus) based on the context information (input 2) regarding the pen event data received from the application unit 300-1. ) Generates a drawing style object 230 that contains the information used to. The ink data processing unit 100 generates a drawing style object 230 by selectively extracting a part of the context data necessary for the receiving side in order to render the drawing. The drawing style object 230 varies the stroke color (red, green, blue), pen tip type (brush, crayon, pencil), transparency or opacity value (alpha), transparency or opacity within the stroke object 210. Is it permissible to vary the stroke (trace) width (eg 0.2-6.0 points), stroke (trace) width within the stroke object 210, how to rasterize, stroke texture Contains information such as the type of.

The generation of the drawing style object 230 is triggered by the detection of the input of the starting point of the stroke object 210 based on the input of pen event data, i.e. in S605 of FIG. Therefore, even if the user repeatedly changes the color of the pen before starting the drawing operation, the old (obsolete) color information is not included in the drawing style object 230 and is therefore sent to the relay server 10-2. For example, it does not clutter transmission resources.

The ink data shaping unit 140T (more specifically, the ink data communication unit 144T) of the ink data processing unit 100T uses the generated drawing style object 230 as the first data message (more specifically, the ink data communication unit 144T) for the stroke object 210 to be generated. Add to DATA_INK_BGNOS1001) (see Figure 34).

The DATA_INK_BGNOS1001 message is a stroke ID (F103) that the message is used to uniquely identify the stroke object 210 (eg message type BGNOS, F101), drawing area ID (F102), stroke object 210 within the drawing area. , Drawing style object 230 (F104), and optionally (if space allows) information indicating that it is the first message for some fragmented data (F105) that constitutes part of stroke object 210. , Followed by the CRC error correction value.

After generating the drawing style object 230 and including it in the first data message (DATA_INK_BGNOS1001), instead of waiting for the "ACCEPT" message to be received from the relay server 10-2, the transmission device 10-1 is a defined unit smaller than the stroke unit. Fragmented data is continuously generated for each T. Multiple fragmented data jointly form one stroke object 210 (S608). Specifically, until the end of the pen event data input operation is detected, the ink data generation unit 120T makes one stroke (or trace) for each specified unit T in order to manufacture the ink data 200. The pen event data to be formed is processed, and the ink data shaping unit 140T shapes the fragmented data in the form of a communication message ("DATA_INK_MVDOS_1, _2, ..., _n" in FIG. 34) corresponding to the specified unit T. , Assign a stroke ID to each of the fragmented data in each message.

The defined unit T used herein may be a unit of time equal to or greater than the sampling time unit of the pen event data input, for example 5 milliseconds. Therefore, the unit T can be 5 ms, 50 ms, 200 ms, and so on. Alternatively or additionally, the defined unit T may be a unit of data size, such as fixed lengths of 256 bytes and 512 bytes. The data (byte) size-based unit T can be set to be smaller than the data (byte) size of the message transfer unit (MTU) according to the given communication protocol shown in FIG. As a further alternative, the unit T may be dynamically changed or switched, for example, between the time unit T and the data size unit T.

The unit T may be adaptively or selectively set by the application unit 300-1 according to the requirements of each application or timing. For example, the application unit 300-1 is generated for the purpose of allowing the stroke object 210 to continuously and growly appear on the receiver 10-3 as if it were "drawn" in real time. The unit T may be set so as to correspond to the smallest unit in which the required interpolation processing can occur between the point objects. As another example, the purpose is to reduce the communication protocol overhead associated with having to send the same message header information (eg, the stroke ID associated with all messages for one stroke object 210). Therefore, the application unit 300-1 may set the unit T to a value as large as possible up to the unit (stroke unit) of the stroke object itself. As another example, if there is a big difference between the input rate and the display rate of the pen event data (for example, the pen event input arrives in 3-4 milliseconds, but the display refreshes every 15 milliseconds. In the case of), the application unit 300-1 may set the unit T so as to match the display rate for the purpose of reducing the entire message. In this case, each stroke fragmentation message will contain data for one or more point objects.

FIG. 33 is a flowchart illustrating a method of defining a unit T that can be executed by the transmission device 10-1 and / or the relay server 10-2. In step S701, it is determined whether the parameter T is explicitly negotiated between the transmitting device and the receiving device, or between the transmitting / receiving device and the relay server 10-2. If "YES", the explicitly negotiated value is set as the unit T (step S703). For example, the unit T is a parameter that is set when the stroke data is fragmented (S524_1c, "TRUE") and the maximum value of the reception delay is 100 milliseconds (S524_1d, "100" milliseconds). Can be negotiated and defined according to. On the other hand, if the determination result of step S701 is "NO", another request related to the time of the application unit 300 is used to set the unit T. For example, in step S705, it is determined whether the ink data 200 and other data (audio, video, etc.) are reproduced in synchronization. If "YES", "whether aggregation was requested" is determined in step S709. This will be fully explained later with reference to FIGS. 43 and 44. If "NO", the unit T shorter than the unit corresponding to one stroke is set as the unit T in step S711. In addition, each (fragmented) data for each specified unit T is given a time stamp for the purpose of synchronous reproduction with other types of data. On the other hand, if the determination result of step S705 is "NO", it is assumed that there is no particular restriction on the timing, and the unit T is set to a default value such as the unit of the entire stroke.

Returning to FIG. 32, when the input processing unit 111 of the transmission device 10-1 detects the end of the pen event data input (S609), that is, when the end of the input operation of one stroke object 250 is detected, the ink data processing unit 100T Generates the metadata object 250 in step S611. The metadata object 250 contains information about the generated stroke object 210 other than the information used to draw (render) the stroke object 210 on the display (included in the drawing style object 230). For example, the metadata object 250 contains author information, pen ID, and the like.

&lt;Ink data relay processing&gt; When the relay server 10-2 receives fragmented data for one stroke data 210 in a data message from the transmission device 10-1, the relay server 10-2 receives the fragmented data without modification. Transfer to -3.

The relay server 10-2 first receives the first message (DATA_INK_BGNOS) of the stroke object 210 from the transmission device 10-1. Then, in step S613 (acceptance) of FIG. 32, it is determined whether to allow a new entry (update) of the stroke object 210 starting with the first message DATA_INK_BGNOS for which resource arbitration (or QoS) control is received. The sequence in FIG. 32 illustrates the case where resource arbitration (or QoS) control determines that the stroke object 210 generated by transmission device 10-1 is allowed to be entered. Relay server 10-2 then stores the stroke ID so that it can identify and forward all subsequent data messages associated with the same stroke ID by simply checking the message headers, first for receiver 10-3. Forward the message DATA_INK_BGNOS.

The relay server 10-2 sends the subsequent data message (DATA_INK_MVDOS) for the same stroke object 210 to the resource arbitration (or QoS) control while checking the stroke ID contained in their headers, respectively. Continue to transfer without having to make an independent decision about. Thus, the resource arbitration decision needs to be made only once per stroke.

In step S615, when the last one of the plurality of data messages for the stroke object 210 is received, the relay server 10-2 forwards the last message to the receiver 10-3 for resource arbitration (or QoS). Control frees the processing resources of relay server 10-2 to transfer its stroke object 210, which is now fully completed.

In step S617, the stroke object 210 is added to the drawing area as a post-processing operation following the completion of the transfer of one stroke object 210, and is simultaneously stored in the drawing area repository of the relay server 10-2.

Since the relay (transfer) process is executed for each stroke object 210, the ink data 200 transferred in real time in the transmission format can be easily converted to the recording format when the relay process is completed, and is compressed. Stored in the repository in the recorded format. This means that each time the relay of the stroke object 210 generated by one of the multiple devices is completed, the relay server 10-2 changes the state of the common drawing area (or canvas) shared by the multiple devices. Allows for efficient updates.

&lt;Ink data reception process&gt; First, the receiving device 10-3 receives the first data message (DATA_INK_BGNOS) of the newly generated stroke object 210 from the relay server 10-2. Receiver 10-3 uses the information contained in the drawing style object 230, such as color, pen tip type, transparency / opacity, rendering method, etc., to fragment data of the stroke object 210 received on its display. Start a drawing (rendering) operation to draw.

The ink data processing unit 100R uses the same information contained in the drawing style object 230 to generate the ink data 200 by continuously processing the fragmented data of the same stroke object 210 contained in the subsequently received data message. , The ink data 200 is output to the graphic processing unit 300.

The graphic processing unit 300 starts and continues the drawing (rendering) processing of the fragmented ink data 200 as they are received. Steps S617-1 to S617-n in FIG. 32 represent the rendered form of the stroke object 210 on the display at different times, respectively. FIG. 32 shows the user drawing on paper by another user from the state G1, G2, G3 where the stroke object 210 is drawn only partially but growing to the state G4 where the stroke object 210 is completely drawn. It shows that the stroke object 210 is drawn continuously as the fragmented data of the stroke object 210 is increasingly received and processed, similar to the process of looking at the actual pen stroke.

FIG. 34 illustrates a sample transmission format of a data message arranged in the form of a communication packet suitable for real-time communication between a plurality of devices according to an embodiment of the present invention. The illustrated example uses a defined unit T that is smaller than the message forwarding unit (MTU) of the communication protocol defined by the media interface to which the transmission device 10-1 is connected (eg, Ethernet®).

The communication packet at the top of FIG. 34 shows the communication packet output from the network communication unit 112 of the transmission device 10-1.

"DATA_INK_BGNOS" in Figure 34 shows the first of multiple data messages for the new stroke object 210.

The F101 field identifies the type of data message. The F101 field of the first data message "DATA_INK_BGNOS" indicates "BGNOS" which means it is the first of multiple data messages for the new stroke object 210.

The F102 field contains the drawing area ID of the drawing area where the new stroke object 210 is added and overlaid.

The F103 field contains the stroke ID of the stroke object 210. A stroke ID is uniquely assigned to each stroke object 210 when used in a common drawing area. Various methods allow each stroke object 210 to be assigned a unique stroke ID for each drawing area. For example, the value of the (highest) stroke ID currently used in the drawing area can be incremented to get a new unique ID assigned to the new stroke object 210. Alternatively, the value of the pen ID of the pen used to enter the new stroke object 210 may be incremented to obtain a unique stroke ID for the new stroke object 210. In this way, transmitter 10-1 can independently assign a unique stroke ID to the new stroke object 210 it produces. Also, as discussed above, the resource arbitration (or QoS) control 544 of relay server 10-2 will have multiple stroke IDs that overlap if transmission device 10-1 fails to assign a unique stroke ID. Prevents the use of. The transmission device 10-1 can independently assign a unique stroke ID to the new stroke object 210, otherwise it can centrally assign a unique stroke ID to the stroke object 210 generated by a different device. The communication sequence required for is not required. Therefore, the transmission device 10-1 can start transmitting ink data immediately after the pen downtime, that is, the user starts drawing the stroke.

The F104 field is contained in the drawing style object 230 or the drawing style object 230, both of which contain a set of drawing parameters related to how the stroke object 210 is drawn (rendered). This set of parameters may be, for example, a shape parameter (a circle shown in FIG. 3B or a flake / piece shown in FIG. 3C), a radius or size of a shape, a minimum and maximum value of a radius or size of a shape, (its brush). It may include information such as pen pressure or pen tilt (which affects how the stroke drawn by pressure or pen tilt appears on the display).

In some embodiments, the entire set of drawing parameters is contained in the F104 field. In another embodiment, one or more sets of drawing parameters (or one or more drawing style objects) are previously shared between the transmitting device 10-1 and the relay server 10-2 (and the receiving device 10-3). The F104 field may contain only the ID of one parameter set (or the ID of the drawing style object) that specifies whether a set of drawing parameters is used to render the stroke object 210. Frequently used parameter settings, such as commonly used nib types (pencils, brushes, pens) and a limited number of commonly used color combinations, preliminarily give each setting a unique ID. These configuration IDs may be defined and may be included in the F104 field to communicate how the stroke object 210 is rendered on the display. Parameter settings may be explicitly negotiated between the devices at the time of initial negotiation or implicitly defined as part of the system's protocol.

The F105 field contains the first fragmented data of the plurality of fragmented data that jointly form the stroke object 210. The F105 field of the first data message contains, for example, the starting coordinate position of the stroke object 210. Figure 34 shows the first fragmented data (in F105) as contained in the first data message (DATA_INK_BGNOS), but if (must contain a drawing style object) in the first data message. The first fragmented data (F105) may be included in the second data message if there is not enough free space.

The data messages "DATA_INK_MVDOS" _1, _2, ... In Figure 34 all relate to the same stroke object 210, the second fragmented data (F106), the third fragmented data (F107), ... ··including. Unlike the first data message, these subsequent data messages do not contain a drawing style object (F104) and are identified in F101 as "subsequent" (not first) type data messages. All subsequent data messages contain the same drawing area ID and the same stroke ID as the first data message.

The data message "DATA_INK_ENDOS" is the last data message for the stroke object 210 and may contain the last fragmented data of the stroke object 210 (F108). The final data message also includes a metadata object 250 in F109 that contains non-drawing related information such as author information.

FIG. 35A illustrates a transmission format that can be used in a communication protocol that includes a built-in data retransmission mechanism such as TCP, specifically a communication packet that contains the data message "DATA_INK_MVDOS". On the other hand, FIG. 35B describes a communication packet that can be used in a communication protocol that does not include a built-in data retransmission mechanism such as UDP. Unlike the data message format in Figure 35A, the data message format in Figure 35B for use in protocols without a retransmission mechanism may include an additional field F110 containing a serial ID uniquely assigned to each data message. .. This through ID can be used to detect the loss of a message and initiate / request a resend.

Specifically, FIG. 36A is a sequence diagram showing a sample of data retransmission processing using a serial ID, which is suitable for execution in a communication protocol such as UDP that does not include a data retransmission mechanism. For example, FIG. 36A fits the case where the sending packet retransmission parameter (524_1a) is set to "FALSE" and the message retransmission parameter (524_1f) is set to "ENABLE". The hatched period "d" corresponds to the period d in FIG.

In Figure 36A, the first, second, and fourth data messages associated with serial IDs # 456, # 457, and # 459 are transmitted from transmission device 10-1 and successfully reach relay server 10-2. Indicates that it has been received. Figure 36A also shows that the third data message associated with serial ID # 458 is lost somewhere in the network between transmitter 10-1 and relay server 10-2 and is received by relay server 10-2. Indicates that it will not be done.

Relay server 10-2 expects a third data message associated with serial ID # 458 but is not received, and instead receives a fourth data message associated with serial ID # 459 at time T1. Detects that the third data message associated with serial ID # 458 has been lost. The relay server 10-2 then issues a negative response (NACK. DATA_INK_NACK) message containing serial ID # 458 indicating that the third data message associated with serial ID # 458 has not been received.

Transmission device 10-1 detects that the third data message associated with serial ID # 458 has been lost at time T2 when the NACK message is received, and the third data message associated with serial ID # 458 is lost. And the retransmission process of resending the subsequent data message (that is, # 459, # 460, ...) Following the serial ID # 458 is started. One of the reasons for resending all later data messages, not just lost data messages, is to continue sending fragmented data for stroke object 210 in chronological order to avoid perceptible time delays. .. Another reason is that when one data message is lost, the data message that follows the lost data message is also likely to be lost.

According to the retransmission processing method described above, all data messages required to form one stroke object 210 are successfully received, even when a protocol that does not include a retransmission mechanism is used. Can be ensured. This retransmission processing method has been described above as occurring between the transmission device 10-1 and the relay server 10-2, but this method also applies between the relay server 10-2 and the reception device 10-3. Can be executed.

Figure 36B shows another example of message retransmission. Similar to FIG. 36A, the hatched period "d" corresponds to the period d in FIG. FIG. 36B fits the case where the sending packet retransmission parameter (524_1a) is set to "FALSE" and the message retransmission parameter (524_1f) is set to "DISABLED". The message retransmission parameter, whether or not message retransmission is possible, has little tolerance for message transmission delay (ie, if there is a strict requirement to have no perceptible transmission delay). Is set to "DISABLED".

Similar to Figure 36A, also in Figure 36B, the first, second, and fourth data messages associated with serial IDs # 456, # 457, and # 459, respectively, are sent to transmitter 10-1 (or the sender). It is transmitted from the relay device 10-2) and successfully received by the relay server 10-2 (or the receiving device 10-3). The receiving relay server 10-2 expects a third data message associated with serial ID # 458 but does not receive it, but instead receives a fourth data message associated with serial ID # 459. Detects the loss of the third data message associated with serial ID # 458 at the time T1. Then, at time T2B, the receiver is associated with the lost # 458 based on the message associated with the successfully received serial IDs # 456, # 457, and # 459 without issuing a retransmission request. Interpolation processing (or error concealment processing) (S1205) is performed to replenish or interpolate the lost part of the data corresponding to the data message, thereby realizing continuous and uninterrupted display processing. At time T3B, processing of the last data message associated with serial ID # 461 is complete. At a subsequent time T4B, the receiver issues a DATA_INK_ALLOS_REQ message 1503 (see Figure 39B) using the received stroke ID. The sender 10-1 (or 10-2) then sends a DATA_INK_ALLOS message 1501 (see Figure 39A), which is a message containing the entire identified stroke. Recipient 10-2 (or 10-3) uses the received DATA_INK_ALLOS message 1501 at time T5B to perform data correction of the lost portion of the supplemented or stored data, if necessary.

FIG. 36C describes a problem dealt with in the interpolation process (error concealment process) in step S1205 of FIG. 36B. Suppose messages # 457, # 458, and # 459 form one complete stroke data. In Figure 36C, message # 456 includes control positions P4561, P4562, and P4563. The position P4561 of the black square is the start position of the stroke and contains the absolute coordinates of the position. In the illustrated example, this absolute coordinate is (100,0). In the figure, the position of the black square is associated with the absolute coordinates (as should be referenced). On the other hand, the position P4562 of the black circle is associated with the absolute coordinate or the difference (offset) with respect to the coordinate of the immediately preceding preceding position as what should be referred to. In the illustrated example, position P4562 is at Rel (+20,0), which is +20 horizontally and +0 vertically with respect to the absolute coordinates (100,0) of P4561. Means. The black circle position P4563 is at Rel (+20,0) with respect to the immediately preceding preceding black circle position P4562. Message # 457 contains three positions P4571, P4572, and P4573 at Rel (+20,0) with respect to the preceding position immediately preceding them, respectively. As shown in Figure 36B, it is assumed that message # 458 is not received by the receiver. Then, the cross-hatched start position P4591 of the last message # 459 cannot be determined. This is because position P4591 contains only relative offset information that is not useful unless there is a reference position.

The cross-hatched starting position P4591 can be determined by various methods according to embodiments of the present invention. The first method according to an embodiment of the present invention relates to linear prediction performed on the receiving side, as still described with reference to FIG. 36C. For example, if the message is sent every 50 milliseconds, the estimated orbital portion in the lost message can be predicted based on the velocity and acceleration of the already obtained position coordinates. In Figure 36C, the stroke is advanced by a total of Rel (+60,0) from P4563 (the last control position in message # 456) to P4573 (the last control position in message # 457) and is therefore missing. The last control position of (lost) message # 458 can also be predicted to be advanced by Rel (+60,0) with respect to P4573.

FIG. 36D illustrates a second method for obtaining a cross-hatched start position P4591 according to an embodiment of the invention. In this method, as indicated by the black square positions P4561, P4571, and P4591, the sender can be changed to include absolute coordinates at the start control position of each of the plurality of fragmented messages. In one embodiment, the transmission device 10-1 is required to output in a format different from that of the recording type, and b) (i) stroke fragmentation is used (in 524_1, "stroke data". "Defragmentability" is "TRUE"), (ii) Message retransmission is not executed at the message layer ("Packet retransmission" of 524_1a is "TRUE", and "Message retransmission" of 524_1f is "DISABLED". "), (Iii) You may choose to perform this method if one or more of the conditions that each fragmented message is explicitly instructed to contain absolute coordinates are met. Although Figure 36D shows that all fragmented messages contain absolute coordinates, the frequency with which they are included can vary. For example, communication protocol reports such as the IETF's RFC 3550 and RTP reports can be used to change the frequency of absolute coordinate content on the transmitting side based on the receiving quality detected on the receiving side.

A third method for obtaining the cross-hatched position P4591 is a backward confirmation that can be used with the linear (forward) prediction in Figure 36C. In FIG. 36E, unlike the case of FIG. 36C, the last control position P4593 of the last message # 459 is made square and contains the absolute coordinates (333,0) as shown in the illustrated example. When the disappearance of the message (# 458) is detected, the receiving side performs a backward confirmation of the cross-hatched round position P4591 based on the last control position P4593 of the last message # 459 including the absolute coordinates. In the illustrated example, the cross-hatched position P4591 calculates P4592 from control position P4593 (333,0) by Rel (-20,0) and reaches P4591 by another Rel (-20,0). It is calculated backwards and becomes (293,0).

FIG. 36F illustrates a replenishment or interpolation process (error concealment process) performed in step S1205 of FIG. 36B using the cross-hatched control position P4591 obtained according to any of the methods described above. The solid line 1201 shows the partial stroke data generated based on message # 457, and the white circle P4573 shows the last (end) control position of the fragmented stroke object 210 contained in message # 457. .. The solid line 1203 shows other partial stroke data generated based on message # 459, and the cross-hatched P4591 controls the first (beginning) of the fragmented stroke object 210 contained in message # 459. Indicates the position. The lost data (corresponding to the lost message # 458) between the partial stroke 1201 and the partial stroke 1203 uses interpolation, in the direction forward from the partial stroke 1201 as shown by the dashed arrow 1207, and It is derived from the partial stroke 1203 shown by the other dashed arrow 1209 based on the predictions taken in both directions in the backward direction.

FIG. 36G illustrates an example of bidirectional prediction processing. In this example, a straight line (or approximate curve) 1221 connecting P4573 and P4591 is formed, and the midpoint that divides this straight line 1221 into two equal straight line (or curved) segments 1223 and 1225, which can be represented as approximate curve functions, respectively. P_pred_midst is discovered.

As mentioned above, when the communication protocol does not include a retransmission mechanism at the communication layer (eg UDP), the system performs message transmission at the message layer as shown in FIG. 36A and the message as shown in FIG. 36B. You may adaptively choose not to perform message retransmissions even at the layer. Thus, when the application has strict timing requirements, such as when the ink data 200 is played back in sync with the audio data, it avoids perceptible transmission time delays and is continuous, real-time ink data. To achieve 200 reception and display, retransmission is disabled at both the communication layer and the message layer.

FIG. 37 is a sequence diagram illustrating ink data communication when the update request for the common drawing area issued by the transmission device 10-1 is rejected by the relay server. For example, transmission device 10-1 requests that a new stroke object 210 be added to the common drawing area, and the request is rejected by relay server 10-2. It is the same as that included in the sequence diagram of FIG. 32, up to the point that transmission device 10-1 sends the first data message (DATA_INK_BGNOS) containing the drawing style object 230.

When the relay server 10-2 receives the first data message, the ink data relay unit 540 shall perform step S613 (based on the specified criteria such as arbitrary restrictions on processing resources, arbitrary restrictions on network resources, user access policy, etc.). In (Reject), the reception of the first data message is rejected.

The relay server 10-2 then sends a rejection message (DATA_INK_REJOS) from the feedback transmitter 546 to the transmission device 10-1.

When the rejection message is received in response to the first data message for the stroke object 210, the transmission device 10-1 cancels the transmission process in step S610 and transmits a subsequent data message for the stroke object 210. Stop. In other words, unless a reject message is returned, the transmission device 10-1 explicitly indicates a transmission process for transmitting all fragmented data for one stroke object 210 in a plurality of consecutive data messages. Continue without waiting for the ACCEPT message to be returned.

According to the ink data transmission methods of various embodiments of the present invention, different devices may have long strokes (traces) drawn by one device (eg, taking several seconds to complete). You can start displaying the stroke at the same time it is drawn in the common drawing area without waiting for the stroke to complete. Several transmission methods that can be built into a given protocol system or that can be additionally used in protocols without a retransmission mechanism, even when the stroke object 210 is transmitted in multiple fragmented data. Can also be used on the receiving side to ensure that the complete stroke object 210 is successfully received. Also, when any of the data messages for the stroke object 210 transmitted by the transmission device 10-1 is rejected / rejected by the relay server 10-2, the transmission device 10-1 shall invalidate the transmission process. Avoid wasting valuable processing and network (sending) resources.

FIG. 38 shows the fragmented data for the stroke object 210 so that the relay server 10-2 receives the fragmented data of the stroke object 210 from the transmission device 10-1 and relays it to the receiving device 10-3. This is the first modification of the sequence diagram shown in FIG. 32 in which all are combined. The transmission of fragmented data for each specified unit T from the transmission device 10-1 to the relay server 10-2 is the same as that shown in FIG.

In this embodiment, the receiving device 10-3 sets reception parameters that define how the receiving device 10-3 receives the ink data 200 relayed by the relay server 10-2. You may negotiate with the relay server 10-2 in advance. For example, the receiving device 10-3 intends to receive the ink data 200 to the relay server 10-2 in stroke units instead of the specified unit T ("Stroke data fragmentation" in 524_1c. A control message (CTRL_JOIN_REQ524_1) containing communication setting information such as information indicating that the value of "Yes / No" is "FALSE") is transmitted. In the relay server 10-2, the relay server 10-2 inks the receiver 10-3 in stroke units (while the relay server 10-2 can relay the ink data 200 in the specified unit T to other receivers). The communication setting information received from the receiving device 10-3 is stored in the drawing area information management unit 524 so that the data 200 is relayed.

Then, as shown in FIG. 32, the transmission device 10-1 starts transmitting the fragmented data of the stroke object 210. Relay server 10-2 continues to store fragmented data for its stroke object 210 until the last data message (DATA_INK_ENDOS) is received in step S614 (buffering). When the last data message is received, relay server 10-2 prepares and sends a data message containing the entire stroke object 210, associated drawing style object 230, metadata object 250, etc. and renders it on the S617. Send it to receiver 10-3 as "DATA_INK_ALLOS".

In this example, since the relay server 10-2 transmits the ink data 200 related to all stroke objects 210 at one time, the number of point objects contained in the stroke object 210 is known. Therefore, the ink data 200 can be transmitted in a recording format instead of the transmission format. The recording format is less redundant and can be more compressed than the transmission format.

Figure 39A is a sample of the "DATA_INK_ALLOS" data message format in the recording format. This sample contains all of the ink data 200 for one stroke object 210, along with the associated drawing style object 230, metadata object 250, number of point objects contained in that stroke object 210, and so on. As shown, this message contains a message CRC, and if the message size exceeds a given communication packet MTU, the message will have more than one communication in the communication layer below the ink data message layer. It is divided into packets (packet 151, packet 152, packet 153, etc.).

When transmitting the ink data 200 on a stroke-by-stroke basis, it is possible to use a different communication protocol than that used to stream fragmented data in real time on a specified unit T. For example, a protocol that does not have a retransmission mechanism such as UDP is used to stream fragmented data every specified unit T, whereas a retransmission mechanism is used to transmit ink data 200 in stroke units. The including TCP or HTTP (which basically relies on TCP) can be used.

The receiving device 10-3 receives the ink data 200 for each stroke in step S617, and applies a graphic process for drawing (rendering) the stroke object 210 in one drawing.

The modified sequence example in Figure 38 is suitable for use with receiver 10-3 in a network environment where the traffic speed is slow and real-time streaming where each stroke must be received continuously is difficult to achieve. .. In such cases, receiver 10-3 may choose to employ a more robust reception method for stroke object reception in order to receive one stroke object at a time without data loss.

FIG. 39B illustrates a sample data message format for "DATA_INK_ALLOS_REQ" 1503, which is a message requesting data for the stroke object 210 over the entire stroke when the stroke ID is known. This message can be used to request the entire retransmission of stroke object data that uses the stroke ID.

FIG. 40 shows the stroke object 210 being transmitted "as is" (ie, unfragmented) from transmitter 10-1 to receiver 10-3 via relay server 10-2, as shown in FIG. This is the second modification of the sequence diagram shown.

In this embodiment, the transmission device 10-1 sets a transmission parameter that defines how the transmission device 10-1 transmits the ink data 200 to the relay server 10-2. You may negotiate with 10-2 in advance. For example, the transmission device 10-1 intends to transmit the ink data 200 to the relay server 10-2 in stroke units instead of the specified unit T ("Stroke data fragmentation" in 524_1c. Send a control message (CTRL_JOIN_REQ) containing communication setting information such as information indicating that the value of "Yes / No" is "FALSE"). In the relay server 10-2, the relay server 10-2 can receive the ink data 200 from the transmission device 10-1 in the specified unit T (while the relay server 10-2 can receive the ink data 200 from the other transmission device) for each stroke unit. The communication setting information received from the transmission device 10-1 is stored in the drawing area information management unit 524 so as to be controlled to receive.

In FIG. 40, as shown in FIG. 32, the transmission device 10-1 executes the ink data transmission process by using the detection of the "pen event data input start" (for example, the "pen down" event) as a trigger. Specifically, in step S605, the input processing unit 111 of the transmission device 10-1 detects the input of the stroke start point. In FIG. 40, the pen event data is the ink data processing unit 100T from step S605 to step S609 in the input processing unit 111 until one stroke starts drawing at the stroke start point and ends drawing at the stroke end point. (And similarly, from S607 to S611 in the ink data generator 120T. Collectively S608B).

The ink data processing unit 100T of the transmission device 10-1 uses the "pen event data input start" (for example, "pen down" event) (S605) as a trigger, and draws the drawing style object 230 based on the context information (input 2). Generate.

The ink data shaping unit 140T of the transmission device 10-1 can then shape the first data message (DATA_INK_BGNOS) to be transmitted to the relay server 10-2, and the relay server 10-2 accepts it (relay server 10-2). You can decide whether to reject (S613 (accept)). The first data message in this case is sent to confirm whether the relay server 10-2 can accept the ink data 200 from the transmission device 10-1, because it is such a thing. For example, it is not necessary to include the coordinates of the starting point of the stroke object 210.

Unless a rejection message (NACK) is received from the relay server 10-2, the transmission device 10-1 displays ink data 200 for the entire stroke object 210, the entire stroke object 210, and the associated drawing style object 230 and Format the message "DATA_INK_ALLOS" including the metadata object 250 and send it to the relay server 10-2.

When transmitting the ink data 200 in stroke units, it is possible to use a different data format than that used to transmit the ink data in the specified unit T. For example, the ink data 200 fragmented in the specified unit T can be transmitted in the transmission format, whereas the ink data 200 in the stroke unit can be transmitted in the recording format. Further, when transmitting the ink data 200 in stroke units, a communication protocol that provides a retransmission mechanism may be used in order to realize a stronger (highly reliable) transmission of the ink data 200.

The relay server 10-2 receives a data message (DATA_INK_ALLOS) containing the associated drawing style object 230, metadata object 250, etc., along with the entire stroke object 210, and accepts it "as is" (DATA_INK_ALLOS), receiver 10- Relay to 3. To achieve this, the receiving device 10-3 defines that the receiving device 10-3 receives the ink data 200 relayed from the relay server 10-2 in stroke units instead of the specified unit T. You may negotiate with the relay server 10-2 in advance to set.

The receiving device 10-3 receives the ink data 200 in stroke units, and in step S617, applies a graphic process for drawing (rendering) the stroke object 210 in one drawing.

The modified sequence example in Figure 40 is suitable for use with transmitters 10-1 and receivers 10-3 in network environments where traffic speeds are slow and therefore real-time is difficult to achieve, where each stroke can be continuously transmitted and received. Can be In some cases, transmitter 10-1 and receiver 10-3 may employ a more robust transmit and receive method for transmitting and receiving stroke objects 210 one stroke object 210 at a time without data loss. You can choose.

FIG. 41 shows a plurality of fragmented data of the received stroke object 210 in order for the relay server 10-2 to receive the stroke object 210 from the transmission device 10-1 and relay or transmit the stroke object 210 to the reception device 10-3. This is the third modification of the sequence diagram shown in FIG. 32, which is divided into.

Similar to the sequence shown in FIG. 40 above, transmission device 10-1 generates a data message containing ink data 200 over the entire stroke object 210, along with related drawing style objects 230, metadata objects 250, etc. ( DATA_INK_ALLOS), transmit to the relay server 10-2 (see steps S607 to S611 in the ink data generator 120T, collectively see S608C).

During the period "d", the ink data generator 120T of the transmission device 10-1 sets the time corresponding to each of the plurality of point objects derived from the pen event data to form the stroke object 210 in the ink data 200. Embed in. For example, when the application unit 300-1 is used in connection with voice (for example, 524_1g "voice synchronization enable / disable" parameter is "ENABLE"), the transmission device 10-1 determines the playback timing of voice or other data. For the purpose of synchronization, time stamp information is inserted for each slice in the ink data 200. The time is used as a "reproduction time" that can be used when the relay server 10-2 relays the ink data 200 and can be used when the receiving device 10-3 reproduces (renders) the ink data 200. Be embedded. For example, the first part (first point object) of the ink data 200 generated at time T1 is relayed by the relay server 10-2 at the corresponding time D_T1 and generated at time T2 (between T1 and T2). The second part (second point object) of the generated ink data 200 is relayed by the relay server 10-2 at the corresponding time D_T2, and so on. Similarly, the first to fourth parts (point objects) of the ink data 200 generated at times T1 to T4 are at the corresponding timings based on times T1 to T4 starting in the "graphic processing start" step of S617, respectively. , Played (drawn) by receiver 10-3.

The relay server 10-2 receives from the transmission device a data message DATA_INK_ALLOS containing ink data 200 for the entire stroke object 210 and the associated drawing style object 230, metadata object 250, etc., and the transmission device 10- This ink data 200 is divided into a plurality of fragmented data for each specified unit T in the same manner as in the method in which 1 generates fragmented data in the sequence example of FIG. The specified unit T is smaller than the stroke unit in this example. The relay server 10-2 outputs the generated fragmented data to the receiving device 10-3 at the timings of D_T1, D_T2, D_T3, and D_T4, respectively.

The receiving device 10-3 receives and processes a plurality of data messages including fragmented data that jointly form one stroke object 210, similarly to the receiving device 10-3 in the sequence example of FIG. Specifically, receiver 10-3 receives the first data message DATA_INK_BGNOS indicating that it is the first data message in a series of data messages constituting one stroke object 210. Receiver 10-3 retrieves the drawing style object 230 contained in this first data message and uses the information in the drawing style object 230 to start rendering the fragmented data of the stroke object 210. .. The ink data processing unit 100R continues processing the subsequent fragmented data received in the subsequent data message, and the drawing unit 300 such that the remaining rendering (drawing) of the stroke object 210 grows (increases). Continue in the way. FIG. 41 shows the fragmentation data of the stroke object 210 gradually from the state G1, G2, G3 where it is only partially present but growing to the state G4 where the stroke object 210 is fully drawn. It shows that it is drawn continuously as it is received and processed. In the above example, the relay server 10-2 controls the timing of relay transmission of stroke data, but the relay server 10-2 transmits the stroke data in units of strokes, and then the receiving device 10-3 receives the stroke data. It is also possible to control the reproduction timing of each fragment of the stroke data. Further, it is possible to have the relay server 10-2 maintain the stroke data and configure the relay server 10-2 to start streaming the stroke data to one or more receiving devices at the scheduled time. For example, the scheduled time is 0: on January 1, 2020.

The various embodiments and variations described above may be combined to provide additional embodiments. Aspects of this embodiment may be modified to provide further embodiments.

For example, in some embodiments, the receiving device 10-3 is configured to determine whether the relay server 10-2 accepts or rejects the data message transmitted from the transmitting device 10-1. Similarly, it may be configured to decide whether to accept or reject the data message relayed from the relay server 10-2. When the relay server 10-2 receives the deny message (DATA_INK_REJOS), the data message from the transmission device 10-1 is sent to the reception device 10-3 (while the same data message can be transferred to another receiving device). Updates connection list 541 to indicate that it will not be forwarded.

In some embodiments, the origin of the common drawing area is identical to the origin of the user-specific drawing area (ie, the origin of the display 113 in the transmit / receive device). In other embodiments, the origin and dimensions of the user-specific drawing area do not have to be the same as the origin and dimensions of the common drawing area. FIG. 42 is a diagram illustrating the concept of a user-specific stroke start point within a user-specific drawing area in relation to the origin of the common drawing area.

In FIG. 42, (0,0) at point 1601 indicates the origin of the common drawing area, while user 10-1 (0,0) at point 1603 is a user-specific drawing area for user 10-1. Point 1605, user 10-3 (0,0), indicates the origin of the user-specific drawing area for other users 10-3. The vector from (0,0) to user 10-1 (0,0) is shown to be (200,30) and from (0,0) to user 10-3 (0,0). The vector is shown to be (130,260). When starting communication, the transmitting device 10-1 and the receiving device 10-3 relay the origin vectors ((200,30) and (130,260) in the illustrated example) from their respective common drawing areas to the relay server 10-2. Share with. Therefore, when transmission device 10-1 sends a stroke A starting at offset 1-1 (1607) to relay server 10-2, transmission device 10-1 starts from user 10-1 (0,0). It is necessary to send only the vector "offset 1-1" which is a vector up to offset 1-1. After that, the relay server 10-2 uses the received vector "offset 1-1" as a vector (200,30) in order to determine the position of the start point "offset 1-1" in relation to the origin of the common drawing area. Can be combined.

The relay server 10-2 then defines the position of the start point "offset 1-1" with respect to the origin of the common drawing area, from point 1601 (0,0) to point 1603 user 10-1 (0,0). The coupling vector leading to offset 1-1 (1607) via can be transmitted to receiver 10-3. The receiver 10-3 then subtracts its origin vector (130,260) from the received coupling vector for its own user-specific drawing area with its origin of user 10-3 (0,0) (1605). , The position of the start point "offset 1-1" can be defined. Alternatively, the relay server 10-2 calculates the position of the start point "offset 1-1" with respect to the origin of the user 10-3 (0,0) (1605) in the user-specific drawing area of the receiving device 10-3. A subtraction operation may be performed. When the relay server 10-2 transmits the calculated position of the start point "offset 1-1" with respect to the origin of the user 10-3 (0,0) (1605) to the receiver 10-3, the receiver 10 -3 can immediately process the received position within its own user-specific drawing area.

In these embodiments, since the relay server 10-2 performs coordinate conversion with respect to the origin of the common drawing area, the transmission and receiving devices 10-1 and 10-3 do not need to perform the conversion by themselves, and their own origin. Coordinate data (vector data) is transmitted and received as coordinate data for. That is, once the transmission and reception devices 10-1 and 10-3 share their origin with respect to the origin of the common drawing area with the relay server 10-2, the coordinate conversion process is completely performed by the transmission and reception devices 10-1 and Can be transparent to 10-3.

Although the specified transmission unit T for transmitting the ink data 200 is described as being the same as or smaller than the stroke unit, it is also possible to transmit the ink data 200 in a unit larger than the stroke unit. That is, it is also possible to "collect" the ink data 200 for each unit having a meaning larger than that of the stroke unit. For example, in the process of the specified unit T shown in FIG. 33, it is determined in step S709 whether aggregation is requested. If "YES", the unit T is set to a value larger than the stroke unit, that is, a unit of meaning in step S713.

FIG. 43 is a diagram illustrating the concept of the semantic unit, and FIG. 44 is a diagram illustrating a sample of a flow for transmitting the ink data 200 in the semantic unit.

FIG. 43 illustrates the word "Network" consisting of seven handwritten characters. 7011 indicates the first stroke, 7012 indicates the second stroke (jointly constitutes the first letter "N"), 7013 indicates the third stroke, 7014 and 7015 are solidarity, respectively. Shows the 4th and 5th strokes that make up the 3rd letter "t". Here, the first and second strokes 7011 and 7012 are regarded as constituting one semantic unit because they jointly indicate one character. Similarly, the fourth and fifth strokes 7014 and 7015 are considered to constitute one semantic unit because they jointly represent one character.

FIG. 44 illustrates a sample flow in which a plurality of strokes constituting one semantic unit are "collected" and stroke data is transmitted in the semantic unit. In this example, application 300-1 is linked to the character recognition engine. In FIG. 44, the hatched period "d_for_7011" is the period during which the first stroke 7011 is detected. The ink data generator 120T detects that the first stroke 7011 does not constitute a complete semantic unit, based on the analysis results of the character recognition engine used by application 300-1. Since the semantic unit is not complete, in step S1911 the process skips the ink data transmission step. Then, the second stroke 7012 is detected during the hatched period "d_for_7012". In step S1913, the ink data generation unit 120T detects that the first and second strokes 7011 and 7012 constitute one semantic unit. In step S1915, the ink data generator 120T collects the first and second strokes 7011 and 7012, and also the first and second strokes 7011 and 7012 jointly form one semantic unit, and solidarity. To generate metadata information indicating that it means "N". The transmission device 10-1 transmits stroke (ink) data (1501 DATA_INK_ALLOS) together with metadata information to the receiving side (relay server 10-2 or receiving device 10-3). In step S1917, the receiver retrieves the metadata and determines that the first and second strokes 7011 and 7012 jointly form one semantic unit and jointly mean "N". Then, in step S1919, the receiver renders (draws) the first and second strokes 7011 and 7012 on the display.

Third Embodiment The third embodiment is a method of generating and reproducing ink data configured to represent handwritten data, and a method of outputting (rendering) a drawing using the reproduced ink data. It is aimed at. In particular, the process of FIG. 92 that abstracts (generalizes) the pen event data 9202 that depends on a particular input device sensor for generating ink data is described. For example, in order to generate a common attribute value of non-device dependent type, pen event data (type 1) having pen pressure data and pen event data (type 2) having no pen pressure data are abstracted (general). The process to be converted) is described.

Background of the Third Embodiment The above-mentioned non-patent document 5 describes a standard of ink data, and defines the purpose of the standard as follows. "Jot [Standard] allows ink to be used across a very wide range of applications and devices. With standard exchange formats, many scenarios are possible. Here, ink sharing There are some examples. Of course, as many applications can be created as Jot runs on a variety of platforms. "

This standard uses a flag called "inkForce Data Present" to indicate the existence of pressure data, which is one of the attributes of ink data.

Each of the multiple points (multiple dots) that make up the stroke is defined in the data structure in Table 1 below.

<tables><img file="JP7094318B2_D0003.tif" /></tables>

In the above, "force" is a value corresponding to the pen pressure. Therefore, for an electronic device that can acquire pen pressure information, the value of the inkForceDataPresent flag is set to a Boolean value indicating the existence of pen pressure data, and the pen pressure ("force") value is input as input information.

Non-Patent Document 1 describes other ink data standards and defines the purpose of the standards as follows. "Hardware and software vendors have typically supplied and represented digital inks that use proprietary or restricted formats. The lack of public and comprehensive digital ink formats is due to multiple vendors. The capture, transmission, processing, and presentation of digital ink across disparate devices being developed has been severely restricted. In response to this need, the Ink Markup Language (InkML) is a digital ink across software applications. Provides a simple and platform-neutral data format to facilitate the exchange. "

In this standard, strokes are called "traces" and their data structures can be defined by freely selected attributes from a plurality of predefined attribute sets.

By default, the trace is represented as a list of decimal combinations (X, Y) with the structure in Table 2-1 below.

Table 2-1 Data structure of Non-Patent Document 1 (default) &lt;traceFormat xml: id = "DefaultTraceFormat"&gt; &lt;channel name = "X" type = "decimal" /&gt; &lt;channel name = "Y" type = "decimal" /&gt; &lt;/ traceFormat&gt;

Additional attributes may be added to the default data structure with additional defined channel names such as F (force). For example, the attribute (channel name) indicating pen pressure is as follows.

<tables><img file="JP7094318B2_D0004.tif" /></tables>

By setting the &lt;channel name&gt; tag of &lt;traceFormat&gt; to, for example, "F (Force)", it is possible to custom-define the "trace format" including the force attribute.

Traces are then represented as a combination of decimal numbers (X, Y) according to any custom-defined "trace format". Therefore, the ink data represented in a custom-defined trace format (eg coordinates, pressure, etc.) is provided as a continuous list of values.

Non-Patent Document 2 describes other attempts to standardize ink data in industry, and defines its purpose as follows. "Ink Serialized Format, or ISF, is a Microsoft® format for storing written ink data. This format is mainly entered by stylus for mobile information terminals, tablet PCs, and ultra-mobile PCs. "Used to store the data Many mobile devices also provide custom information such as pressure and angle. In addition, [they] can be used to store custom information along with ink data. "

When pressure information is included, the following attributes may be included in the data format. This shows that pressure is the second most important after x, y.

TAG_NORMAL_PRESSURE

Information including TAG_NORMAL_PRESSURE information is then output in order.

Outline of Third Embodiment The above-mentioned ink data structure proposed in Non-Patent Documents 1, 2, and 5 is specific to the device obtained by the pen-type input device without abstracting or generalizing the information. It is intended to output the information of. For example, when pressure data is acquired as an attribute by a pen-type input device, the pressure data can be output to an electronic device capable of receiving and processing the pressure data, but most of the capacitance. It is not output for electronic devices that are not expected to have such capabilities, such as type tablets. Similarly, when the pen-type input device does not have the ability to acquire pen pressure data, even if the electronic device has the ability to receive and process the pen pressure data, the pen pressure data for that electronic device. Is never output. Moreover, some "finger" input devices, such as capacitive sensors, capable of receiving handwritten input by a finger generate finger pressure data when the finger is pressed against the sensor surface. Does not have the ability to do. In this way, not all pen-type or finger-type input devices have the ability to acquire pen pressure data, and all electronic devices (eg, tablets and PCs) acquire pen pressure data. , The use of pen pressure data in the currently available ink data is rather limited, as it does not have the ability to receive and process. This makes the currently available ink data unsuitable for the purpose of more realistically simulating and expressing handwritten data. Because, in reality, how the pen stroke (or pen trace) is due to the pressure applied by the user affecting the width and density of the pen stroke, or creating stains or stains that resize. This is because it has a great influence on whether it appears on paper.

The techniques proposed in Non-Patent Documents 1, 2, and 5 include pen movements (ie, "" that include information such as how much pressure is applied and what is the angle of the pen during a pen event. The purpose is to record and play back the "pen event"). However, the information that is ultimately needed to be recorded and reproduced is the "appearance" of the resulting series of pen events, how multiple strokes or traces appear on the screen. The appearance as a result of a series of pen events is typically used as graphic data. Non-Patent Document 3 proposes a standard for vector data. However, Non-Patent Document 3 is indifferent to the use of the pen as an input device, and as a result the data represents and is not suitable for expressing the strokes obtained as a result of the use of the pen. For example, the technique of Non-Patent Document 3 does not allow changing the stroke width or changing the transparency of the stroke in version 1.1.

A method of generating and reproducing ink data based on an ink data model (meaning or language) that does not require pressure data so that it can be used by devices that do not support pressure data, and ink based on that ink data model. There is a need for data formats. In some embodiments, the method of generating ink data has the ability to generate pressure substitute parameters based on information readily available on most, if not all, devices. Using the substitute parameters in the ink data model of the present invention to represent pressure information creates a rendered (drawn) pen stroke with a variety of nuances that simulate an actual pen stroke and a realistic appearance. Realize to provide.

Description of Third Embodiment The description of the various representative embodiments of the invention set forth below are generally summarized in the six sections set forth below.

[1] The whole picture of the system architecture. See Figure 1 and Figures 45-47B.

[2] Stroke Language (SL) and Stroke File Format (SFF) [2-1] Ink Data Model ("Stroke Language"). See Figures 48A to 48I.

[2-2] Stroke file format (SFF). See Figures 48J to 48L.

[3] Ink data processing unit 100 (generation unit 100T). See FIGS. 49-63.

[4] Ink data processing unit 100R (reproduction unit 100R). See Figures 64-66.

[5] Ink data rendering process. See Figure 67.

[6] Effect. Ink data drawing example. See Figures 68-72.

[1] Overall view of the system architecture (FIGS. 1 and 45 to 47B) In FIG. 1 described above with reference to the first embodiment, the ink data 200 of the present invention is generated, reproduced, and rendered on the screen. Explains the whole picture of the system according to the third embodiment of the present invention (drawn). In particular, the ink data exchange infrastructure 10, outlined in Figure 1, represents the infrastructure realized by a library that uses ink data 200 based on a common language model, where various supporting pressure data are supported. Ink data 200 is generated for common use by various application services and devices, as well as other application services and devices that do not support pressure data.

The device 10-1-1 shown in FIG. 1 includes a pen-type input device capable of sensing pen pressure data as an input sensor. The apparatus 10-1-1 generates the ink data 200 by using the application provided by the application service # 1. The generated ink data 200 can then be output in the appropriate output form for the destination media (eg, network).

The device 10-1-2 is a tablet-type input device capable of receiving handwritten input made by the user's finger. The sensor of device 10-1-2 does not have the ability to output pressure data, but can generate ink data 200 using the application provided for application service # 2, which does not require pressure data. .. The ink data 200 may then be output in the appropriate form corresponding to the destination media.

Device 10-3 is a desktop PC that subscribes to application service # 2. Device 10-3 processes the ink data 200 output from device 10-1-1 or device 10-1-2 using the application provided by application service # 2 (eg, rendered on a display screen). Or redistribute it).

Application service # 1 and application service # 2 running on equipment 10-1-1, 10-1-2 and equipment 10-3 both share a common information model ("stroke language") that represents ink data 200. Use.

The ink data 200 according to embodiments of the present invention is limited to devices and service applications that are made available by various types of devices on a variety of service applications and are themselves capable of acquiring or processing pressure data. Not done.

FIG. 45 describes the data input / output in the processing unit 100 (generation unit 100T) of the ink data 200, the ink data generation method on the left hand side, and the data input in the ink data processing unit (reproduction unit 100R). In addition to explaining the output, the ink data generation method is explained on the right hand side. The ink data processing unit 100T corresponds to 100T in FIG. 5 of the first embodiment.

The ink data processing unit 100T according to the embodiment of the present invention receives pen event data from the sensor of the device 10-1-1 and the sensor of the device 10-1-2. The pen event data represents the movement of the pen (or finger) that generated the stroke of the pen (or finger) and includes the coordinate data of the stroke (eg, (X, Y) position). Pen event data can also include device-dependent data such as pen pressure data and pen tilt data. In FIG. 45, the pen event type 1 data from devices 10-1-1 includes pen pressure data. The sensor of device 10-1-1 is a device capable of outputting pressure data, such as a pen tablet device driver or API for EMR type pen tablets used with a pressure sensitive stylus. In contrast, the pen event type 2 data from device 10-1-2 does not include any pressure data. Devices 10-1-2 are devices that do not have the ability to output pressure data, such as iPad® tablets and other capacitive touch sensor tablets. The ink data processing unit 100 receives device-dependent data (type 1 and type 2), and generates and outputs non-device-dependent ink data 200 according to the embodiment of the present invention. This ink data 200 can be shared by various devices regardless of their ability to process pressure data.

In order to generate the ink data 200 based on the pen event data, the ink data processor 100 also applies to the pen event data from the application or operating system used to input the pen event data, as shown in FIG. Receives the context information (input 2) of. The generated ink data 200 is a collection of various objects (stroke object 210, metadata object 250, drawing style object 230, operation object 270, etc., see Figure 48B), which are organized in the tree structure shown in Figure 5. Will be done. The context information (input 2) is, for example, the pen (tip) type (eg, pencil, crayon, brush, etc.) and / or pen color information received from the drawing application, pen tablet resolution and sampling rate, pen event. Includes input time and location information, pen ID, user ID provided by the OS, and any other information about the stroke provided by the (software) application used to generate (draw) the stroke.

The ink data model defines the meaning of the data used in the application domain (domain) that processes the ink data 200. When a common ink data model is shared between various applications and services, information can be reliably and freely shared within the system structure within the domain that processes the ink data 200.

In FIG. 45, the dashed arrow indicates each class object of the ink data 200 defined by the ink information model, and a collection of ink data objects constitutes the ink data 200.

The ink data processing unit 100 outputs the generated ink data 200 as a collection of data objects in the media format (for example, file, packet, etc.) requested by the application that requires the ink data 200. In general, there are two types of media formats.

(1) Stroke file format (SFF) that stores ink data 200 in a permanent (semi-permanent) form (2) Stroke messaging format (SMF) suitable for transmitting ink data 200 in messages or real-time communication.

The ink data processing unit 100R receives the ink data 200 as an SFF or SMF file, interprets the ink data 200 recorded on a specific medium, extracts the ink data 200 as a sequence of bytes or bits, and extracts the extracted ink data 200 as ink. Provided for various types of applications 300 that utilize data 200. The ink data processing unit 100R takes out an ink data object in response to a request from one of the applications 300-1, ..., 301-n, reproduces the information defined by the ink data model, and makes a request. The extracted (reproduced) ink data 200 is provided to the applications 300-1 and 300-2. The ink data processing unit 100R is typically embodied as a library dynamically or statically linked to the requesting application 300 and executed on a processor in a personal computer. In another embodiment, the ink data processing unit 100R may be a cloud server configured to reproduce the ink data 200 by interpreting the ink data model.

Each of the applications 300-1, ..., 300-n is fetched by the ink data processing unit 100R, pulls out the required amount / part of the reproduced ink data 200, and pulls out when performing various operations. Use the ink data 200. For example, application 300-1 is a drawing application in the illustrated embodiment. The drawing application can also extract a complete set of reproduced ink data 200, apply, for example, scaling, rasterization, and rendering operations to the extracted ink data 200 and output it as an image file.

In various embodiments, the ink data processing unit 100 and the ink data processing unit 100R are a memory and a CPU (central processing unit) or a GPU (central processing unit) or GPU (respectively) together with an application 300 that uses a library that realizes the functions of the generation unit / reproduction unit. It can be embodied in one or more computer devices that include a graphics processing unit). For example, the ink data processing unit 100, the ink data processing unit 100R, and the drawing application 300-1 can be embodied in one computer device. In this case, the ink data processing unit 100 and the ink data processing unit 100R may use the memory of the device to share a common ink data object. FIGS. 46A to 46C each describe three configuration examples of the ink data generation device or method according to the embodiment of the present invention.

FIG. 46A illustrates a device or method such that the ink data processing unit 100T resides in a server different from the stroke input terminal that can be coupled to device 10-1-2. The device 10-1-2 acquires the device-dependent pen event data including or not including the pen pressure data. In the illustrated embodiment, the sensor of the device 10-1-2 acquires the type 2 device-dependent event data that does not include the pen pressure data. The devices 10-1-2 then transmit the acquired device-dependent pen event data to the server embodying the ink data processing unit 100T via a network such as the Internet. The ink data processing unit 100T executed in the server receives device-dependent pen event data from devices 10-1-2 and generates ink data 200 defined by the ink data model. Contextual information about the pen event (input 2) can be provided to the ink data processor 100T by device 10-1-2, or, for example, if the application provided by the server utilizes a common pen type. It can be omitted if contextual information (input 2) is not needed.

FIG. 46B illustrates another device or method such that the ink data processing unit 100T resides in device 10-1. In this case, device 10-1-1 also includes a device driver for controlling device 10-1-1. Device 10-1 acquires device-dependent raw data that includes or does not include pressure data. In the illustrated embodiment, the sensor of apparatus 10-1-1 acquires raw data including pen pressure data. The sensor in device 10-1 sends raw data, including pressure data, to the terminal I / O via an appropriate interface such as USB. The device driver of the terminal processes the received raw data and produces type 1 device-dependent pen event data including pen pressure data. The ink data processing unit 100 of the device 10-1-1 receives the type 1 device-dependent pen event data from the device driver and generates the non-device-dependent ink data 200 defined by the ink data model. At this time, as shown in the figure, the ink data processing unit 100 may also receive context information (input 2) regarding pen event data from an application and / or an OS operating on the terminal. The generated ink data 200 is ordered in the form of a byte sequence or a bit or byte sequence and recorded on a suitable medium (network, storage device, etc.).

FIG. 46C describes yet another device or method such that the ink data processing unit 100T resides in a terminal (eg, a touch sensor) that includes an input sensor. For example, the device 10-1-2 of FIG. 46C is a capacitance type touch sensor that can function as a terminal, and includes an ink data processing unit 100 in addition to the touch sensor. This touch sensor includes a control IC capable of acquiring device-dependent position data (eg, the position touched by a finger), but with pen (or finger) pressure data or other pen-specific data (angle). , Tilt, etc.) may or may not be available. In the illustrated embodiment, the touch sensor acquires and transmits type 2 pen event data that does not include pen pressure data. The ink data processing unit 100T receives type 2 device-dependent pen event data from the sensor control IC and generates non-device-dependent ink data 200 defined by the ink data model. At this time, the ink data processing unit 100 can also receive context information (input 2) regarding pen event data (input 1) from an application or OS running on the device 10-1-2, if any. .. The ink data 200 may then be output in the form of a byte sequence or bit sequence recorded on a suitable medium (network, storage device, etc.), such as in persistent form (SFF) or messaging packet form (SMF).

47A and 47B illustrate the operation of the ink data reproduction method according to the embodiment of the present invention.

In FIG. 47A, the server has ink data 200 in a certain format (for example, SFF) generated by the ink data processing unit 100R (reception and playback side), the application 300-1, and the ink data processing unit 100T operating together with the server. An ink data reproduction method including a memory (RAM) that can be stored will be described. The ink data processing unit 100R corresponds to the 100R shown in FIG. 22 of the first embodiment. Application 300-1 activates the static / dynamic library of the ink data processing unit 100R to obtain the data required for application 300-1. In the illustrated embodiment, application 300-1 is a drawing display program. The ink data processing unit 100R can selectively take out necessary data from the ink data stored in the RAM and reproduce it, and provide the reproduced necessary data to the application 300-1. For example, the required data reproduced may include position data and RGB color data, but may or may not include variable stroke width data. Application 300-1 receives the required data reproduced from the ink data processor 100R and performs the necessary drawing operations such as interpolation, geometry generation and rasterization, thereby displaying (rendering) on the display screen. Output the image.

FIG. 47B illustrates another reproduction method such that apparatus 10-3 includes ink data processing unit 100R and application 300-2. In this case, the ink data 200 generated by the ink data processor 100 (not shown) is prepared in a message (eg, by SMF) according to the streaming protocol, or packetized and output to the network, eg. .. Application 300-2 requests the ink data processing unit 100R to receive the data required for application 300-2. In the illustrated embodiment, application 300-2 is a character image text conversion application that converts handwritten text into machine readable text according to any text coding scheme. As such, application 300-2 requires position data (XY data), but not necessarily RGB color data. Application 300-2 can also request information about the author of ink data 200 (eg, to distinguish handwriting specific to a particular author).

The ink data processing unit 100R can extract and reproduce the data necessary for the application 300-2 from the ink data 200 in the message (written in SMF) received from the network or the packet. Application 300-2 receives the required replayed data from the ink data processing unit 100R, performs the required text conversion operation, and outputs the text conversion result, probably along with the author ID.

As described above, the functions and operations of the ink data processing unit 100 and the ink data processing unit 100R are various devices (input devices, terminals) on the Internet depending on the type of media used in each application and each embodiment. , Servers, etc.) and can be integrated.

[2] Stroke Language (SL) and Stroke File Format (SFF) [2-1] Ink Data Model ("Stroke Language") (FIGS. 48A to 48I) FIGS. 48A and 48B are according to embodiments of the present invention. It is an entity-relationship diagram explaining an information model for ink data 200 (stroke language).

In the figure, one box represents one object or class used in this information model. For example, the objects "stroke", "point", and "metadata" are used. An object contains a class object defined by the language from which the object is derived, as well as a data structure, such as a structure represented as a collection of attribute data.

The ellipse in Figure 48B shows the attributes of the objects contained within the box. For example, the attributes "position", "color", "radius" are used for the data object "point". Of the attributes, those indicated by the solid ellipse are essential attributes for the corresponding data object. For example, the attribute "position" is essential for the data object "point". The ellipse shown by the dashed line is an extensible attribute that can be added. For example, the attribute "color" can only be added to the data object "point" and is not required. The diamonds in the figure show the relationships between the data objects connected by the diamonds. For example, a diamond classified as "included" means that one of the data objects connected by the diamond is contained in the other data object.

In general, an "information model" defines the meaning of data in an object domain, and represents concepts, relationships, constraints, rules, and processes used within the domain. The information model provides a systematic structure that allows reliable information requests among various applications and services in the context of a shared domain. The ink data model output by the ink data processing unit 100T and / or reproduced by the ink data processing unit 100R is a stroke trace input by a pen moved by the user and a pen type (pencil, brush, etc.) and It is configured to be able to represent the attributes of the pen used to enter the trace, such as the pen color.

FIG. 48A illustrates a subdomain of the information model for ink data 200. The information model for the ink data 200 is conceptually categorized into the following four subdomains based on the functions and purposes of various data objects.

(SM) Stroke model subdomain. Both include a stroke object 210 and a point object that define the position (geometric shape) and layout of the ink data 200.

(MD) Metadata subdomain. It mainly defines the metadata of the stroke object 210.

(R) Drawing style object 230 for image data (rasterized subdomain that mainly defines the information required to convert stroke data (stroke object)).

(M) Operation subdomain. It contains a set of data objects for dynamically manipulating a stroke object or an instance of a stroke object, such as deleting, splitting, moving, or copying a stroke object.

Each of these subdomains will be described below with additional reference to Figure 48B.

&lt;(SM) Stroke Model Subdomain&gt; The stroke model subdomain includes a stroke object 210, a point object, and a cambus object.

(SM1) The stroke object 210 is an essential component of the ink data model and constitutes the core of the ink data 200.

FIG. 48C illustrates the structure of the stroke object 210. The stroke object 210 includes a plurality of point objects from the "first point" to the "nth point" that collectively define the geometry of the stroke. The stroke object 210 was tagged as "trace" in InkML (Non-Patent Document 1), "path" in SVG1.1 (Non-Patent Document 3), and "Cambus Path" in the HTML5 (Non-Patent Document 4) specification. Corresponds to the data.

The stroke object 210 can include, as object attributes, a start parameter 301 and an end parameter 303, as described above with reference to the first embodiment. In the stroke object 210 including the start parameter 301 and the end parameter 303 as attributes, the start parameter 301 and the end parameter 303 are defined separately from the point object. The stroke object 210 may include attributes of "color" and "radius" (which can be replaced with "width" for convenience) as attributes that can be expanded for each stroke. These attributes will be described in detail later.

(SM2) A point object is such that N point objects are contained within one stroke object 210. The point object contains, as a required attribute, a "position" that indicates a position in a given (eg, 2D) coordinate system. The point object may include "color", "radius", etc. as expandable attributes. The "color" may be set for each point or for each stroke.

When the same attribute is defined for multiple data objects in a conceptual tree structure, the attributes given to lower level objects (eg objects closer to the point object) are given priority. For example, if the stroke color (R, G, B, and transparency alpha (α)) is represented by a 4D vector (R1, G1, B1, α1), and if the color of the third point of the stroke is (R2). If defined as, G2, B2, α2), the color of the third point is determined to be (R2, G2, B2, α2). As another example, if the color attribute is different between the shape filling rasterization (drawing style) object applied to the stroke object 210 and the stroke object 210 itself, then the color attribute of the stroke object 210 itself. Has priority.

The stroke object 210 or point object can be considered to own the "add" command. It commands that the new stroke object 210 or its new point object should be added to the data structure each time a new stroke object 210 or new point object is created.

(SM3) The canvas object indicates the size of the drawing area ("canvas") used when one or more strokes are input, and includes, for example, a width "W" and a height "H" as attributes. It also shows the amount of ink absorbed by each type of paper, such as Japanese paper, carbon paper, plain paper, copy paper, and photo paper. The paper type information can be combined with a rasterized (drawing style) object to determine the actual image representation of the ink data 200.

Figure 48D illustrates a canvas object. The picture on the left shows the size of the drawing area as the stroke is input. It illustrates that strokes are input to the drawing area defined by the width "W1" and the height "H1". The picture on the right illustrates how the canvas size (W1, H1) obtained from the ink data 200 can be used to reproduce the stroke. In the illustrated example, on a device such as a mobile terminal that includes a drawing area with a width "W2" and a height "H2" that is smaller than the area (W1, H1) used when the stroke was drawn. It is supposed to be played. In such cases, the associated size / position of the stroke with respect to the original drawing area (W1, H1) and the original drawing area extends the stroke to fit the given canvas area of the rendering device (W2, H2). Can be used to reduce (shrink), cut, or offset.

&lt;(MD) Metadata Subdomain&gt; A metadata subdomain contains multiple metadata objects 250, each defining metadata for the stroke object 210, and the metadata produces that stroke object 210. Contains contextual information (input 2) about the pen event data used for.

(MD1) The metadata object 250 has a one-to-one relationship with the stroke object 210 in order to "depict" the stroke object 210.

1) The timestamp attribute contains information about the time the stroke was recorded and represents, for example, a UNIX® time value written in a defined format (32-bit unsigned integer). Figure 48E illustrates the effect of using the timestamp attribute on a stroke-by-stroke basis. Suppose two strokes a and b were recorded at substantially the same time by two different users. The picture on the left shows that stroke a was drawn after stroke b, that is, the timestamp value of stroke a is greater than the timestamp value of stroke b. The picture on the right shows that stroke a was drawn before stroke b, that is, the timestamp value of stroke a is smaller than the timestamp value of stroke b. For example, accurate strokes entered by multiple users by determining which stroke should be above the other stroke at each intersection based on different timestamp values associated with different strokes. It will be possible to render to.

2) The author attribute contains information that identifies the author who recorded the stroke.

3) The pen ID attribute is information that identifies the pen used to record the stroke. Ideally, the ID is globally unique for each pen. When the pen ID is not available or needs to be firmly established, the pen ID is used in connection with the sensor side ID of the sensor used to detect the input made by the pen. obtain.

By using the attributes described above, the stroke metadata can describe, as an unlimited example, when, by whom, and with which particular pen the stroke was drawn.

&lt;(R) Drawing style object (rasterization) subdomain&gt; The drawing style object (rasterization) subdomain indicates what modification or modification process is associated with the stroke object 210 when a stroke is input. Contains a set of drawing style objects 230 to hold. The drawing style object 230 is constructed from contextual information (input 2) regarding pen event data based on which stroke object 210 was generated. For example, the drawing style object 230 records various modifications or corrections associated (applied) with different drawing tool attributes (brushes, pencils, crayons, etc.) and strokes such as different nib widths. The following objects (collectively referred to as "drawing style objects") can be part of a rasterized subdomain.

(R1) The rasterized style object is a "rendering" object that has an M: N ratio relationship to the stroke object 210. For example, M (eg 5) style objects (including extensible objects) can be applied to N (eg 20) stroke objects. Style objects are so-called superclass objects whose attributes can be inherited in extended relationships by other object classes such as shape-filling style objects. The style object contains "composite" as an attribute. "Composite" uses any type of function (eg, usually multiple, minimum or maximum of current and previous strokes, erase, etc.) when mixing strokes with previously generated strokes or backgrounds. Indicates whether it should be. FIG. 48F includes three conceptual examples of representative objects that extend a style object to modify or modify the stroke object 210: a shape filling style object, a particle scattering style object, and a region duplication style object.

(R2) The Shape Filling Style object is a stroke object that defines the outline and color of the stroke when the stroke object 210 is rendered on the screen, as illustrated at the top of Figure 48F. Applies to 210. The shape filling style object defines multiple circles, each positioned in relation to the plurality of point objects that make up the stroke object 210. Each circle can be associated with a radius and color attribute if each point object is not associated with a radius and color attribute. The use of radius and color attributes to define a point object or stroke object 210 consisting of a plurality of point objects is one of the features of the present invention and will be described in detail later. The shape filling style object also contains an antialiasing attribute that defines what algorithm should be used to visually remove the edges of the stroke outline defined by the envelope of a series of overlapping circles.

(R3) The particle scattering style object uses "particles" instead of the circles used in the shape filling style object described above, as illustrated in the middle of Figure 48F. The particle scattering style object, like the shape filling style object described above, includes the "radius" and "color" attributes.

The particle scattering style object also contains the attribute "random seed" (see Figure 48B). This is an integer value and is used to generate pseudo-random numbers for the purpose of simulating "roughness" and "scattering" for tools such as pencils and watercolor brushes. The "random seed" attribute is stored in a file format to allow the same picture to be accurately rendered each time the user opens the file or the remote user receives the ink data.

The "shape rotation" attribute indicates that each particle should be rotated at a random angle of rotation, or should be rotated along a certain orbit.

The "Spacing" attribute indicates the distance between two consecutive particles.

The attribute "range" indicates the offset value of each particle in the direction perpendicular to the orbital direction, as shown by the arrow in FIG. 48F. The position of the particles within the width defined by the arrow can be randomly offset (changed) based on the random numbers generated based on the random seed.

The attribute "fill" defines the texture that should be applied to the shape, such as hatching.

The attribute "fill offset" indicates the crop position of a hatch-like texture applied to a shape and is used to define and modify the crop position to avoid applying the same texture in succession.

The attribute "buildup" indicates whether the number of particles produced increases over time when the pen is located at a fixed point for a continuous period of time.

The attribute "mixing" defines which type of function (eg, usually multiple, none, etc.) should be used to calculate the color of the position where two consecutive particles overlap each other. For example, it can be defined whether the color should be the same as the color of one of those particles or a darker color (eg, a mixture of the two colors).

(R4) Area duplication style objects are used to extend style objects, as illustrated at the bottom of Figure 48. The area duplication style object sets the area defined by the closed curve that interpolates between the points. The attribute "transformation matrix" holds an affine transformation matrix that should be applied to the content within the area surrounded by the closed curve. The contents of that area are rendered at different coordinates based on the transformation matrix. Matrix-based transformations only affect the rendering style of the stroke object 210 and do not manipulate or modify the stroke object 210 itself.

&lt;(M) Operation subdomain&gt; The operation subdomain operates all or part of the existing stroke object 210 generated according to the ink data model (split, erase, copy and paste, modify, etc.). Define a transformation (or operation object) for.

(M1) Slice object 274 is an operation object 270 applied to retrieve one or two parts of the stroke object 210. Figure 48G illustrates the behavior of slice object 274. The picture above Figure 48G illustrates the stroke object 210 before the slice object 274 is applied. The slice object 274 has a "width" and is represented by other strokes that cross (and intersect) the stroke object 210. The slice object 274 with "width" is typically associated with the "erase" feature. The positions on the stroke object 210 where the slice object 274 with "width" intersects are between the kth point and the k + 1st point, and the k + 2nd point and the k + 3rd point, respectively. To position. The original stroke curve is generated by interpolating each point with a Catmull-Rom curve, when the stroke is sliced to cut the stroke into two strokes without modifying the shape of the original stroke. , No new end point object is created for the newly created end point. Instead, the value for the new end parameter 303n is set as the attribute for the first stroke and the value for the new start parameter 301n is the second stroke, as shown in the figure below in Figure 48G. Set as an attribute for. The new end parameter 303n and start parameter 301n are represented as one or several "float" points indicating an internal split point between the two points that define the first displayed segment or the last displayed segment. To. For example, the end parameter 303n for the first stroke can be defined as an internal split point between the original k-th point and the original k + 1-th point. Therefore, the shape of the original stroke can be used to represent the shape of the two newly generated strokes. Some embodiments

According to the method described above, the internal split point between the two points of the original stroke is retained as the attributes of the newly generated (sliced) stroke (new end parameter 303n and new start parameter 301n). To. As such, no new point object is created as a result of the new slicing operation, and the original collection of "input points" remains unchanged. Therefore, when the curve is derived from the interpolated Catmull-Rom curve, the outline of the curve does not change before and after the slicing operation.

The attributes "change style" and "change metadata" are such that the attributes of multiple strokes separated from the original stroke by the slicing action are changed (newly defined) for the separated strokes (eg,). , "Author" attribute), or whether the attribute is unchanged from the attribute associated with the original stroke (eg, pen color attribute).

The start and end parameters 301 and 303 are attributes indicating the start and end positions of the two strokes separated from the original stroke, respectively.

The attribute "point range" defines the range of points where the transverse stroke (slicing operation object) defined by the "width" in FIG. 48G intersects the stroke object to be operated. In Figure 48G, the point range includes the k + 1st point and the k + 2nd point.

Figure 48H illustrates the "erase" feature achieved by applying a slicing operation object. When such an "erase" operation object is applied to an existing stroke object 210, the stroke object 210 will have the exact part to be "erased" in the middle (eg, P_intersect_L and P_intersect_R in Hall segment 1801 in FIG. 18). Is divided into two strokes, a first stroke (stroke 1) with a new end parameter 303n and a second stroke (stroke 2) with a new end parameter 303n. .. In this case, when this operation to modify the original single stroke object 210 is finalized (committed), two new strokes (stroke 1 and stroke 2) are generated and the middle part is erased.

(M2) The selection object is an operation object that "contains" multiple (N) slice objects 274 (slicing operation objects), as shown in Figure 48I. The selection object "selects" the area (slice 2) surrounded by multiple (N) slice objects 274 (slice 1, slice 2, slice 3), and the existing stroke object within the range selected by it. Any part of the 210 can then be manipulated (moved, copied, enlarged, reduced) by applying a transformation matrix to that part. In FIG. 48I, the selection indicated by the lasso, which is formed by a dotted closed curve, includes a partially sliced stroke object 210, which can then be manipulated. In one example, it can be moved (moved) as illustrated in Figure 48I.

The attribute "transformation matrix" is an affine transformation matrix. The illustrated example shows the values in the region described by tx and ty. When a matrix object is used to define a particular operation, it is possible to represent points within a region (slice 2) as if they had moved to virtually different positions. In this case, when this operating object is finalized (committed) to modify the original single stroke object 210, three new strokes (stroke 1, stroke 2, and stroke 3) are generated.

The attribute "duplicate" indicates whether to keep the object in its original (pre-deformation) position, even after applying the affine transformation matrix to effectively move the object to a different position. For example, by holding the original position / area in addition to the new position / area that has been substantially moved, it is possible to realize a copy operation in which the original position / area is copied to the new position / area. become.

[2-2] Sample stroke file format (FIGS. 48J to 48L) FIG. 48J is a diagram similar to the stroke file format (SFF) in which the ink data structure is serialized and maintained according to the embodiment of the present invention. The structure (object tree) of the ink data generated and processed by the ink data processing unit 100 (100T or 100R) will be described according to the definition of the ink data model shown in 48B.

The upper part of FIG. 48J describes the ink data structure generated by the ink data processing unit 100 within its internal processing resources such as memory space. For example, in each of the i stroke objects, one or more points (points # 1 to # j) are given as an example, and one metadata object 250 and one drawing style object 230 are associated with each other. Defined (in the form of an instance tree). The data structure is by definition of the ink data model, as shown in the entity-relationship (ER) diagram of Figure 48B. In Figure 48J, strokes # 1 and stroke # 2 contain differently named style objects (styles # 1 and style # 2), although the considerable amount of data in each of those style objects is the same. This happens, for example, when stroke # 1 and stroke # 2 are drawn in the same drawing application using the same drawing tool (having the same nib type and pen color).

The lower part of FIG. 48J describes an example of a stroke file format (SFF) file in which the data structure shown in the upper part of FIG. 48J is arranged. That is, for each stroke object # 1 ~ # i, the point objects # 1 ~ # j that make up the stroke object 210, the metadata object 250 that depicts the stroke object 210, and how the stroke object 210 is on the screen. A drawing style object 230, which defines whether it is rendered (drawn), is defined in the SFF file. As shown, one file named "InkDataFile" contains information about multiple strokes # 1 ~ # i.

&lt;Information about stroke # 1&gt; (1) First, information about stroke # 1 itself is described. This information includes a byte sequence (or binary sequence, used herein interchangeably) in which the attributes of stroke # 1 such as start parameter 301 and end parameter 303 of stroke # 1 are encoded. .. This information also contains a byte sequence in which the set of point objects # 1 ~ # j that make up stroke # 1 are encoded.

(2) Second, it contains a byte sequence in which the metadata object 250 that "draws" stroke # 1 is encoded.

(3) Third, it contains a byte sequence in which the drawing style object 230 that "renders" stroke # 1 is encoded.

&lt;Information about stroke # 2&gt; The same shaping process is executed for stroke # 2 as in the case of stroke # 1 above. In the example of FIG. 48J, the value of the drawing style object # 1 is the same as the value of the drawing style object # 2. In this case, in persistent file format (SFF), it may be preferable not to repeat the same value in order to save file space and avoid redundancy. Therefore, the same value is not repeated for drawing style object # 2, and as shown, there is no drawing style object after metadata object # 2. On the playback side, the ink data processor 100 will continue to use the same drawing style object 230 to render each stroke object 210 until a new (different) drawing style object 230 is found. That is, it is agreed in advance on the file generation side and the playback side that if the style object for the stroke object 210 is omitted, the style object used for the previous stroke object 210 should be used.

Figure 48K describes the data type of each class of data objects (metadata object 250, canvas object, drawing style object 230) as they are ordered and stored in the SFF file. In Figure 48K, "InkDataFile" is an object located at the root of the data structure.

In Figure 48K, the first line of each block shows the names of the objects defined by the ink data model of the invention. The second and subsequent rows of each block show the attributes of the named objects in the left column and their data types (integers, unsigned strings, etc.) in the right column. The attributes are the coding method such as ASN.1, BER, DER, or the coding method shown in the schema file shown in FIG. 10 of the first embodiment such as "sint32 type" and "float". Can be encoded using the appropriate method of.

The meaning of the attributes described in Figure 48K is that, except for the information needed to sustain the ink data 200, such as "strokeCount", which indicates how many stroke objects are contained in a given SFF file. Same as the attributes described above with reference to 48B.

The data object InkDataFile at the root of the structure contains a binary sequence maintained within the file according to embodiments of the present invention.

The "Header" in the first line is the data required for the purpose of persistently storing the InkDataFile, and for use in processing subsequent binary sequences, such as information about the definition version of the ink data model. include. The header is also a binary coding method used, which data is compressed or not, which is required by the application 300 or the playback device 100R to reproduce the ink data 200 as intended by the ink data generator. , And includes information such as other information.

The second line, "Decimal Precision," indicates the level of accuracy of a value, such as a position coordinate value, and is expressed in decimal within the recording format. "Decimal Precision" corresponds to the parameter on line 4 of FIG. 10 of the first embodiment described above. A "Decimal Precision" is a set of data that is not defined in the ink data model but is needed for the purpose of maintaining the ink data file. The use of "Decimal Precision" is one of the features of the present invention and is described in detail below. The decimalPrecision number (or accuracy) corresponds to the reciprocal of the resolution. For example, when the position coordinate value is available at a resolution of 0.01 units (eg, 0.01 pixels), the resolution expressed as decimalPrecision can be set as 100 (the reciprocal of 0.01). For better storage purposes, the decimalPrecision value can be expressed in the form of an exponent. For example, the value 100 can be expressed as the square of the base 10 (in some cases the base may be omitted). The use of accuracy in the ink data generation and reproduction methods according to embodiments of the present invention will be fully described later with reference to FIGS. 58 and 66.

"MetaData" on the third line corresponds to the metadata object 250 described above with reference to FIGS. 48B and 48E.

"Canvas" on the fourth line corresponds to the canvas object described above with reference to FIGS. 48B and 48D.

"Style" on the fifth line corresponds to the (drawing) style object described above with reference to FIGS. 48B and 48F.

The "strokeCount" on the sixth line is a set of information required for the purpose of maintaining the ink data file, and is a code or an integer value indicating the number of strokes contained in the specific ink data file. As such, this attribute is not included in the ink data model itself. Typically, strokeCounts are added in the ink data code sequence when they are not dynamically modified but instead output to static storage media. .. The strokeCount is not set or contains no value (NULL), for example, when the ink data is continuously output in real-time communication for a remotely located ink data player.

The "strokes" on the 7th line are related to a group of object instances "strokes" (stroke objects) contained in the ink data. Each "stroke" (each stroke object 210) contains one or more point objects and various attributes, as described in Figure 48L.

FIG. 48L describes the information contained in each stroke which is the stroke object of No. 1 to N (strokeCount). The "Start_value" and "End_value" on the first and second lines correspond to the start parameter 301 and the end parameter 303 described above with reference to FIG. 48C. As described above with reference to the first embodiment, these two parameters are stored as attributes of the stroke object 210 separately from the point object contained in the stroke object 210.

The fourth line, "Stroke Type variable Alpha," indicates whether the transparency (alpha) associated with the stroke is variable along the length of the stroke. "Variable Alpha" is a flag that indicates whether the transparency of the stroke object 210 is allowed to be variable (ie, fixed) along its length, typically expressed as a Boolean value. Will be (TRUE or FALSE). In some embodiments, the attribute "StrokeType variableAlpha" is not present in the ink data model itself shown in Figure 48B and is used when the stroke object is maintained in the SFF file (ink data file). To. The use of "variable Alpha" in various embodiments of the invention will be described in detail later.

The fifth to eighth values "alpha", "red", "green", and "blue" jointly constitute the "color" attribute of the stroke object 210.

The "alpha" on line 5 is the fixed transparency value (or opacity / ink density value) used when the stroke "alpha" is not variable, for example as indicated by "variable Alpha = false". Is shown. When the "alpha" is fixed, the fixed "alpha" is applied along the length of the stroke object 210, i.e. to each of the plurality of point objects that make up the stroke object 210.

The "red", "green", and "blue" in the 6th to 8th lines are information for determining the color data of the stroke object 210 when the RGB color space is used.

The "points Count" on the 9th line indicates the number of points contained in the stroke object 210. Similar to the "strokes Count" mentioned above, the "points Count" is used to maintain the ink data file in the SFF file, where the ink data 200 is static (rather than a more temporary real-time communication medium). It can be determined at the timing of output to various storage type media. The "points Count" may be, for example, not present in the data structure or contain any value (NULL) when the ink data 200 is output to real-time media instead.

The "points" on the 10th line show information about each of the plurality of point objects constituting the stroke object 210.

The point object ("point" in FIG. 48L) is a data object that contains information about each of the 1st to N (pointsCount) points included in the stroke to be processed. As shown in FIG. 48B, one stroke object 210 contains N (pointsCount) point objects.

The "if" phrase on the first line of the box that defines the "point" is each "point" data object depending on the value of the variable Alpha (TRUE or FALSE) that indicates whether the stroke contains a variable alpha in the length direction. Means a syntax change for.

(i) Lines 2-6 show the data contained in the "point" data object when the value of variableAlpha is TRUE (ie, the stroke contains a variable alpha value in the length direction).

The "x" in the second line indicates the first coordinate value of the point to be processed.

The "y" in the second line indicates the second coordinate value of the point to be processed.

Therefore, "x" and "y" jointly define the 2D coordinates of the point.

The "radius" on the fourth line indicates the size of the radius of the circle containing the point as the center. "Radius" is an attribute associated with a point object and does not need to be associated with a pressure value or tip force. Rather, "radius" is a meaningful, generalized, higher-level concept that includes lower-level concepts such as the concept of pressure and force, as described more fully later.

The "alpha" on the fifth line indicates the alpha value associated with the point object.

Floating-point data typically used in drawing processing applications, as the data type of the data contained in lines 2-5 of the point object is described more fully later with reference to Figure 58. Note that it is an integer (int) or ushort (or uchar), not a type.

(ii) In the 7th to 11th lines, the value of variable Alpha is FALSE (that is, the stroke does not include the variable alpha value, that is, the value of alpha (transparency) is fixed with respect to the length of the stroke. If), indicate the data contained in the "point" data object.

The "x" on the 7th line indicates the first coordinate value of the point (same as the 2nd line above).

The "y" in line 8 indicates the second coordinate value of the point (same as line 3 above).

Therefore, "x" and "y" jointly define the 2D coordinates of the point.

The "radius" in line 9 indicates the size of the radius of the circle containing the point as the center (same as line 4 above).

In this case, the stroke object 210 has a fixed alpha value that should be applied to each of the plurality of point objects that make up the stroke object 210, so the alpha value is not defined for the point object.

As mentioned above, and as described more fully later with reference to FIG. 57, the syntax of the data object is that the stroke object 210 has a variable alpha value along the length or a fixed alpha value. It varies depending on the value of variable Alpha (TRUE or FALSE) indicating whether or not it has.

The twelfth and subsequent lines indicate that for each "point" data object, the syntax changes to selectively include additional attributes, depending on the value of the reserveRawflag. If the reserveRawflag is TRUE, it is possible to extend (expand) the data object without losing the original information, for example to include "timestamp" information for each point.

The ink data model is shown in FIGS. 48A-48L with data objects in the ink data file written in the stroke file format (SFF) defined in the syntax and semantics of the ink data model according to embodiments of the present invention. See above above. Next, an ink data processing unit 100 (100T) capable of operating to generate and output ink data 200 having a data structure according to an embodiment of the present invention is described.

[3] Ink Data Processing Unit (FIGS. 49 to 63) FIG. 49 is a functional block diagram of the ink data processing unit 100T according to the embodiment of the present invention. This ink data processing unit 100T corresponds to 100T in FIG. The ink data processing unit 100T outputs ink data according to the definition of the ink data model based on the pen event provided by the input device and the context information (input 2) of the pen event as described in FIGS. 48A to 48L. Generate. The ink data processing unit 100T outputs the generated ink data 200 in the form of, for example, an SFF file ("ink data file") described in FIGS. 48J to 48L, a binary sequence, a packet, or the like.

The ink data processing unit 100T includes an ink data generation unit 120 and an ink data shaping unit 140. The ink data generation unit 120 corresponds to the stroke data object operation unit 122 in FIG. 7. The ink data generator 120 provides various types of device-dependent input data ("input 1") such as type 1 pen event data including pen pressure data and type 2 pen event data not including pen pressure data. To receive. In FIG. 49, the type 1 pen event data includes time stamp information (eg, "double timestmap"), multiple sets of XY coordinates (eg, "float x, y"), and pressure data (eg, "float pressure"). The type 2 pen event data includes time stamp information (eg, "double timestmap") and multiple sets of XY coordinates (eg, "float x, y"). Instead of receiving the time stamp information, the ink data generation unit 120 may use, for example, the time when the pen event data is received as the time stamp information.

The ink data generator 120 also receives contextual information ("input 2") about the pen event data from the application or operating system used to input the pen event data. For example, in the case of strokes drawn using a drawing application, the context information ("input 2") can include various parameters set by the drawing application to draw the stroke. That is, the context information (input 2) can include configuration information defined for the purpose of generating (drawing) strokes. The contextual information (input 2) is the type 1 and type 2 pen events from the two different applications if the type 1 pen event data and the type 2 pen event data are generated by two different applications. Can be provided for each of the data. Alternatively, the contextual information (input 2) for the type 1 and type 2 pen event data is provided by the same application or operating system commonly used to generate the type 1 and type 2 pen event data. You may.

As described above with reference to the stroke data object operation unit 122 of FIG. 7 in the first embodiment, the ink data generation unit 120 receives received type 1 or type 2 pen event data and received context information (input). Based on 2), each outputs a series of point objects containing XY coordinates (positions) and radius and alpha data for the point. As mentioned above with reference to the data object "point" in FIG. 48L, radius is an attribute associated with a point and need not be associated with pressure or pen tip force, but ink data according to embodiments of the present invention. Rather, by definition of the model, it is a generalized, device-independent, advanced, meaningful concept that includes concepts at lower levels than pressure or pen tip force.

The ink data shaping unit 140 receives data for each point including the XY coordinates and radius of the points and alpha data, and shapes the input data into a data structure corresponding to the data structure of the point object described in FIG. 48L, for example. And output the formatted data. The data in the formatted point object is not the float type point data type ("float") typically used in drawing processing applications, but "int" (integer), "ushort", "uchar", etc. It will be of data type.

FIG. 50A is a flow chart illustrating a process of outputting radius and alpha information as attributes of a point object according to the embodiment of the present invention, which is executed by the ink data generation unit 120 of FIG. 49. Typically, the ink data generator 120 simply passes the XY coordinate data it receives to the ink data shaping unit 140, so a description of outputting the XY coordinate data (position (X, Y)). Is omitted.

In step S1201, the ink data generation unit 120 acquires the necessary context information (input 2) for each stroke including the point to be processed. A sample of contextual information (input 2) will be described later with reference to Figure 50B.

In step S1202, it is determined whether or not the input data includes the pen pressure data.

In step S1203, after it is determined in step S1202 that the input data contains pressure data ("TRUE"), the pressure data at the point derives the parameter (vn ... velocity) at that point. Can be used for. When a larger pen pressure is applied to a point, it is perceived that the speed at that point slows down. Therefore, vn can generally be inversely proportional to pressure data.

In step S1204, after it is determined in step S1202 ("FALSE") that the input data does not include the pen pressure data, the time information of that point is acquired. The time information may be received as input information for each point constituting the stroke, or may be set as the time when the ink data generation unit 120 (or the ink data processing unit 100T) receives the point information. ..

In step S1205, the velocity of a point and one or more adjacent points is derived based on the time information of that point, as described more fully later with reference to FIG.

In step S1207, point radius information is acquired based on vn (velocity), phase information, and pen type information, as described more fully later with reference to FIG.

In step S1209, point alpha (transparency or opacity) information is acquired based on vn (velocity), phase information, and pen type information, as described more fully later with reference to FIG. 55.

FIG. 50B illustrates a sample application or operating system GUI that can be used to set some of the contextual information (input 2) associated with the ink data generator 120. The context information (input 2) provided to the ink data generator 120 is, for example, the maximum and minimum radius values ("Cntx1"), the maximum and minimum radius values as described later with reference to FIG. Can be set for the associated maximum and minimum velocity values ("Cntx2"), the function used to derive the radius or alpha value ("Cntx3"), points in the start and end phases of the stroke. Exceptional values (see Figure 53) ("Cntx4") and pen type information (not shown) may be included. The context information (input 2) may be predefined for the ink data generator 120 or may be explicitly set by the user via the configuration GUI, as shown in FIG. 50B.

FIG. 51 is a diagram illustrating a process of deriving a speed based on time information in step S1205 of FIG. 50A according to an embodiment of the present invention.

Velocity is derived by dividing the distance by time. The denominator of the division can be the difference between the time when the current point coordinates were obtained and the time when the previous point coordinates were obtained. The numerator of division can be the difference (distance) between the current point coordinates and the previous point coordinates. When the sampling rate is fixed (when the denominator is fixed), the displacement between the current point coordinates and the previous point coordinates can be used as an indicator of velocity.

In an embodiment of the invention, the velocity is output as a parameter value (vn) having a minimum value of 0.0 and a maximum value of 1.0, that is, the minimum and maximum are standardized velocity values. The minimum and maximum values can be set to "Cntx2" in FIG. 50B.

FIG. 52 is a flow chart illustrating the process of deriving the radius, which is an attribute of the point object, in step S1207 of FIG. 50A. In general, the radius is derived from the parameter vn, which is the normalized velocity value calculated in step S1205 above. Alternatively, it can be derived from the pen pressure data in step S1203 according to the embodiment of the present invention.

In step S1207_01, the "phase" of the current point is determined. Further referring to FIG. 53, which illustrates the definition of a "phase" of a point within a stroke according to an embodiment of the invention, the phase is a concept indicating where (which position) the current point is located within the stroke. For example, the first point of a stroke is in the beginning phase and the last point of the stroke is in the ending phase. The point between the first point and the last point is in the middle phase. The phase of each point can be determined for each pen event type, such as pen down, pen move, pen up types. A pen-down event is an event in which the user starts drawing a stroke using a pointer (pen, finger, etc.), a pen-up event is an event in which the user finishes drawing a stroke using a pointer, and a pen move event is an event. It is an event that occurs between the pen-down event and the pen-up event.

In step S1207_1, after it is determined that the point to be processed is in the start or end phase, that is, that point is the first or last point of the stroke, one or more points are the first of the stroke. Can be added (before the first point) and at the end of the stroke (after the last point).

In step S1207_2, for each of the points added at the beginning or end of the stroke, a radius that is 0 or a radius greater than the normal radius (eg, double), as described more fully later with reference to Figure 72. The value of the exceptional radius set in "Ctnx4" in Figure 50B, such as, is set as the radius of that point.

In step S1207_1, after it is determined that the point to be processed is in the middle phase, that is, that point is neither the first point nor the last point of the stroke, in step S1207_03, to input pen event data. The pen type of the pen used is determined.

After the pen type is determined to be normal (default) in step S1207_03, in step S1207_05 a normal (default) such as the exponential decay (or damping) function of FIG. 54 (see "Attenuation" in FIG. 54). The radius is derived from the parameter vn using a function.

In step S1207_05 after the pen type is determined to be special, especially like a pen with a soft nib in step S1207_03, the radius is like the "S" and "periodic" functions in Figure 54. Derived from the parameter vn using a special function. Both normal and special functions can be explicitly defined or modified as part of the contextual information "Cntx3" in Figure 50B via the configuration GUI.

FIG. 54 is a graph illustrating three functions used in steps S1207_05 and S1207_07 to derive a radius from the parameter vn (velocity) according to an embodiment of the invention.

The horizontal axis shows the parameter vn (velocity) and the vertical axis shows the radius.

The solid line referred to as "damping" shows the normal damping (or damping) function used in step S1207_05 of FIG. This function defines the relationship that the radius decays exponentially as vn increases. Normal functions are used for normal (default) type pens. The use of this function to cause such a vn-radius transformation is based on the following observations.

[Observation A] The line width that increases due to pen pressure corresponds to the area where the ink soaks into the paper.

[Observation B] The faster the pen moves, the shorter the time the pen has to soak the ink at each point.

Based on the above observations A and B, it is theorized that the line width increases as the pen moves slowly, while the line width decreases as the pen moves faster. This theory is based on the fact that the faster the pen moves, the shorter the time it takes for the pen to infiltrate the paper from the line (stroke) at each point of contact. The line width is considered to be the radius of each of the series of points. Therefore, according to an embodiment of the invention, for a normal pen, a damping (or damping) function is used to convert the parameter vn to a radius, which causes the radius to be exponential as the velocity increases. Will be attenuated.

It should be noted that the ink data processing unit 100T of the present invention can calculate or acquire speed information using time stamp information, even for devices that do not have the ability to acquire pen pressure data. For example, the ink data processor 100T can use the local timing to receive pen stroke information from such a device to calculate or acquire the time stamp information, based on which the speed vn is at each point. Can be determined against. Therefore, the ink data processor 100T increases the radius of each point based on the velocity vn for various types of devices, including devices capable of acquiring pressure data and devices not capable of acquiring pressure data. It can be determined by reliability and output.

In FIG. 54, the dashed line called "S" represents a special function in which the decay occurs in stages rather than exponentially, and the dashed line called "periodic" is another special one that is periodic. Shows a function. Both of these special functions can be used to derive the radius from the velocity for a special type of pen in step S1207_07 of FIG. 52, as described more fully later with reference to Figure 71.

It should be noted that any application of these functions described above for converting velocity to radius of points can be in real time. Alternatively, the conversion of the parameter vn to the radius may be performed in advance, and the resulting data may be stored in a look-up table accessible by the ink data processor 100T.

FIG. 55 is a flow chart illustrating a process of deriving an alpha value indicating the transparency (or opacity) of each point in step S1209 of FIG. 50A according to the embodiment of the present invention.

In step S1209_03, the pen type of the pen used to input the pen event data is determined from the context information (input 2).

In step S1209_05 after the pen type is determined to be normal (default) in step S1209_03, alpha is derived from the parameter vn using a normal (power) function such as the "power" function shown in FIG. ..

In step S1209_07, after the pen type provided by the context information (input 2) is determined to be special, such as a pen with a particularly soft nib, in step S1209_07, "S" shown in FIG. 56. Alpha is derived from the parameter vn using a special function such as the "letter" function.

FIG. 56 is a graph illustrating two functions for deriving alpha (transparency / opacity) from the parameter vn, used in steps S1209_05 and S1209_07 of FIG. 55, according to an embodiment of the invention.

The horizontal axis shows the parameter vn (velocity) and the vertical axis shows the transparency alpha. For example, alpha 0.0 can mean completely transparent, and alpha 1.0 can mean completely opaque, i.e. opaque.

The solid line called "power" shows the normal power function used in step S1209_05 in FIG. This function defines the relationship that the radius increases exponentially as the velocity vn increases. The normal (power) function is used for normal (default) type pens. The use of this function to cause such a vn to radius transformation is based on the following observations.

[Observation C] The density of ink that increases due to pen pressure corresponds to the area where the ink soaks into the paper.

[Observation D] The faster the pen moves, the less ink will seep out of the pen at each point (because the pen is in contact with each point for less time).

Based on the above observations C and D, it is theorized that the ink density increases as the pen moves slowly (opacity increases), while the ink density decreases (transparency increases) as the pen moves faster. Is made. The theory is that the slower the pen moves, the more ink that seeps out of the pen at each point of contact, forming a darker line (stroke), and the faster the pen moves, the more it stains from the pen at each point of contact. It is based on the fact that the amount of ink emitted is reduced to form bright lines (strokes). Therefore, according to an embodiment of the invention, for a normal pen, a power function is used to convert the parameter vn to alpha, which causes the alpha (transparency) to exponentially increase in speed. It will increase.

It should be noted that the ink data processing unit 100T of the present invention can calculate or acquire speed information with high reliability by using the time stamp information even for the device which does not have the ability to acquire the pen pressure data. For example, the ink data processing unit 100T can use the timing of receiving pen stroke information from such a device in order to calculate or acquire the time stamp information, and the speed vn can be determined based on the timing. Therefore, the ink data processor 100T provides alpha with high reliability based on speed vn for various types of devices, including devices capable of acquiring pressure data and devices not capable of acquiring pressure data. It can be decided and output.

In FIG. 56, the dashed line referred to as the "S" is a special function that is an example of an increasing function that can be used to derive alpha from velocity for a special type of pen in step S1209_07 of FIG. Is shown.

It should be noted that any application of these functions described above with reference to Figure 56 can be in real time. Alternatively, the conversion of the parameter vn to alpha may be performed in advance, and the resulting data may be stored in a look-up table accessible by the ink data processor 100T.

As described above, the ink data generation unit 120 of the ink data processing unit 100T determines the radius and alpha values of each point object based on the input pen event data including or not including the pen pressure data. As shown in FIG. 49, the point data derivation unit 120 outputs radius and alpha information in its own internal memory in "float" data type. After that, the ink data shaping unit 140 receives radius and alpha information (float type) as attributes of the point object, and outputs them in the stroke file format (SFF, see Fig. 48L) or stroke message format (SMF). do.

FIG. 57 is a flow illustrating an ink data shaping process for shaping the input radius and alpha information together with the input X and Y coordinate data into the stroke file format (SFF) or stroke message format (SMF). It is a figure. Generally, the shaping process S140 is executed by the ink data shaping unit 140 of the ink data processing unit 100T as a part of the process of generating the ink data 200.

&lt;Serialization of data object InkDataFile&gt; In step S1401 to serialize the InkDataFile in the SFF, the ink data and the information needed to make the InkDataFile in the SFF file persistent are serialized. As an example of such required information, the decimalPrecision value is serialized and encoded. In the illustrated example, the value 100 is used as the decimalPrecision value of the "unsigned integer" data type, and the value 100 is the encoding method such as ASN.1, BER, or the schema file of Figure 10 of the first embodiment. It is encoded by an encoding method such as "sint32 type" and "float" shown in. In contrast, serializing an InkDataFile in an SMF suitable for real-time transmission or messaging does not require the information needed for the purpose of making the ink data in the SFF persistent and therefore in the SMF. Step S1401 may be omitted when shaping the ink data of. In step S1402, the strokesCount value for the InkDataFile is encoded in the SFF (see Figure 48L). In contrast, when formatting to SMF, the strokesCount value is not included, so step S1402 can be omitted and the process instead encodes the data indicating the last of all strokes processed. It may be changed.

&lt;Serialization of data object "stroke"&gt; The following steps, starting with step S1403 contained in the larger square drawn by the dotted line in Figure 57, are performed for each of the N (strokesCount) strokes contained in the InkDataFile. To. As a result, N strokes are shaped and output using the specified coding method.

In step S1405, the variableAlpha value is encoded in the stroke object 210 being processed. As mentioned above, this variable Alpha value (TRUE / FALSE) indicates whether the alpha value of the stroke is variable along the length of the stroke.

In step S1406, a pointsCount value indicating the number of point objects contained in the stroke object 210 is encoded. For example, in the case of a real-time application (ie, when formatting to SMF), if the pointsCount value is not available, step S1406 can be omitted and the process will instead end the stroke being processed. The data shown may be encoded.

&lt;Serialization of data object "point"&gt; The following steps, starting with step S1407 contained in the smaller square drawn with a dotted line in Figure 57, are performed for each of the pointsCount points contained within the shaped stroke. To. As a result, pointsCount point objects are formatted and output.

In step S1409, it is determined whether or not the alpha value of the stroke including the point to be processed is variable. That is, it is determined whether the variable Alpha value is TRUE or FALSE.

In step S1411 after it is determined in step S1409 that the alpha for that stroke is variable along its length ("TRUE") and therefore the alpha can vary from point to point, FIG. 58. The value of the XY coordinates for the point is encoded along with the radius and alpha values, as described more fully later with reference to the top of.

In step S1413 after it is determined in step S1409 that the alpha for that stroke is not variable, the value of the XY coordinates with respect to the point and as described more fully later with reference to the bottom of FIG. Only radius values are encoded, alphas are not.

At this point, the ink data 200 placed within the defined data structure according to the embodiments of the present invention will be in a suitable file format (eg SFF) or message format (eg SMF) for various types of media. Can be output.

FIG. 58 illustrates examples of steps S1411 and S1413 described above according to embodiments of the present invention.

Lines 01-07 at the top of Figure 58 are when the XY coordinate value for a point is encoded with both the radius and the alpha value when the alpha value is variable along the length of the stroke. It is a pseudo code corresponding to step S1411 in FIG. 57.

Lines 08 to 13 at the bottom of FIG. 58 are when the alpha value is not variable with respect to the stroke, the XY coordinate value and the radius value are encoded for the point, and the alpha value is not encoded. It is a pseudo code corresponding to step S1413 of 57.

In FIG. 58, the sections indicated by "A" and "B" show how the above-mentioned decimalPrecision values are used in the practice of embodiments of the present invention.

Preferably, the XY coordinate and radius values of the point are maintained in floating or double data types until just before output timing to maintain the highest possible accuracy for the values in the processor. On the other hand, for the purpose of making the ink data 200 widely (commonly) understandable by various data interpretation methods and for the purpose of efficiently compressing the resulting ink data 200, each value is represented. Therefore, it may be desirable to use the minimum number of bits.

Therefore, as shown in the upper part of FIG. 58, in step S1411, the input X, Y and radius values are first set to the resolution for standardizing their units, as shown in "A" in FIG. It is incremented by a decimalPrecision value that indicates the (magnification), and then converted (cast) to an int (integer) data type, as shown by "B".

Also in step S1413 shown at the bottom of FIG. 58, similar to step S1411, the input X, Y and radius values are incremented by the decimalPrecision value and then cast to the int (integer) data type.

In this regard, FIG. 59 illustrates the conversion of a floating data type to an integer data type used in steps S1411 and S1413 of FIG. 57 according to an embodiment of the present invention.

In FIG. 59, the input data 161 stored as a floating data type is an example according to the IEEE754 standard. In the input data 161, "s" is a coding bit, "exponent" is a power of a floating-point number, and "fraction" is a mantissa of a floating-point number.

The input data 161 is multiplied by the decimalPrecision value, as shown by "A" in FIG. 58 and as described above, to generate the additional input data 163 of FIG. The augmented input data 163 is also a floating point number, including exponents and mantissas.

The augmented input data 163 is shown (cast) by "B" in FIG. 58 and from "float" to "int" (or "short" or "ushort"), thereby having an absolute value of 165. Is generated. The absolute value 165 is no longer a floating point number. In the illustrated embodiment, all XY coordinate values and radius values are cast (converted) to int (integer), but may be cast to any non-floating point data type.

FIG. 60 illustrates the efficiency of compression as a result of the above-mentioned conversion (casting) of the data format from "float" to "int" according to the embodiment of the present invention.

In Figure 60, decimal numbers are used for ease of explanation and understanding. In the illustrated example, the pi is multiplied by various exponents of 10 (0, +1, +2 to the power).

The left-hand side of FIG. 60 shows the values obtained prior to the processing of steps S1411 and S1413 of FIG.

The right-hand side of FIG. 60 shows the values obtained after the processing of steps S1411 and S1413 of FIG.

The first three values and the last value contained in the square on the left hand side are the various floating point numbers (3.0, 3.1, 3.14) before the processing, but these are all the processing shown on the right hand side. After, it is converted to 3. The accuracy of each number is somewhat compromised by the conversion, and the frequency of use of the same value (eg 3 in this example) to facilitate efficient processing of the value in the processor, such as efficient compression of the value. Is increasing. For example, data represented by a data type integer may be encoded in the file or message protocol to be output using an encoding method such as ASN.1, BER, or DER.

FIG. 61 is a flow diagram illustrating a process that can be performed by the "ink data shaping unit" 140 of FIG. 49 to compress the attributes of a defined ink data object according to an embodiment of the present invention.

In step S181, as a preliminary step, the generated ink data 200 to be compressed is quantized as an integer precision value, as performed in steps S1411 and S1413 of FIG.

In step S183, the ink data shaping unit 140 determines the type of data compression. For example, this determination may depend on the type of output format. Also, this determination may be based on whether compression is for applications that require real-time data or for applications that utilize data within storage-type media. If compression is not performed ("NONE"), the process outputs the ink data value of the integer data type "as is". For example, if the first three values shown in FIG. 60 are used, the value "+3" represented by 32 bits can be output three times.

In step S185, after the first type of compression method is selected in step S183, the process categorizes data sequences such as X-coordinate values, Y-coordinate values, radius values, etc. into streams that follow those attributes. The first type of compression method is applied to the generated ink data 200. For example, the first type of compression method is run-length coding. For example, when the first three values described in FIG. 60 are used, a sign indicating that the value "+3" is repeated three times is used. The efficiency of the proposed coding technique has been improved using several coding methods, such as performing delta obtain. This will increase the number of repeated values when the changes between the values are rather constant.

In step S185, after the second type of compression method is selected in step S183, the process applies the second type of compression method to the generated ink data 200. For example, the second type of compression method is an entropy coding method using an exponential Golomb code. This is a variable length coding method that uses a bit length that is shorter than a value that has a larger absolute value for a value that has a smaller absolute value, such as +3.

&lt;Modification Example for Ink Data Processing Unit 100T and Ink Data Generation Method&gt; As described above, the ink data processing unit 100T according to the embodiment of the present invention has a radius and / or alpha (transparency) as attributes of points constituting each stroke. To derive (opacity) information, some process data received from various types of devices, some containing pressure data and others not, thereby being non-device dependent. Has the ability to generate ink data 200 of. The ink data processing unit 100T outputs the generated ink data 200 in various formats such as a file format (for example, SFF) or a message format (for example, SMF).

In the example of FIG. 50A, in the case where the input data contains pressure data (when the result of step S1202 is TRUE), the radius and transparency information is derived from the pressure data without using the time stamp information. To. However, the present invention is not limited to such examples.

A function that receives timing information and pressure data as input and outputs the radius can be used. In this case, it is possible to change the stroke width and / or the transparency of the stroke based on not only the pen pressure data but also the speed of the pen.

While the velocities are derived in the various embodiments described above, acceleration can be used, for example, when the pen includes an accelerometer or when only values corresponding to acceleration are available. For example, by integrating the acceleration to derive the velocity, a process similar to the above can be utilized.

FIG. 62 is a flow chart showing another example of the process executed in place of the method described above with reference to FIG. 52 in the ink data generation unit 120 of FIG. 49 in order to output radius information as an attribute of ink data. Is.

In step S192, it is determined whether or not the input data includes the pen pressure data, as in step S1202 of FIG. 50A.

In step S196 after it is determined that the data input in step S192 does not include the pen pressure data ("FALSE"), as described with reference to FIG. 54 above, the speed derived from the time stamp information is used. The radius is calculated using the relationship with the radius. That is, the relationship that the radius attenuates as the velocity increases is used.

In step S194 after it is determined that the data input in step S192 contains pressure data ("TRUE"), both the input pressure data and the time stamp information are used to derive the radius. In step S192, in the case where the pen pressure is fixed, (i) the radius decreases when the speed increases, and in the case where the speed is fixed, (ii) the radius increases when the pen pressure increases. Use a partial derivative function that includes the two variables. Therefore, it is possible to encode the radius of each point in the ink data 200 based on both the speed and pressure of the points.

In step S198, the radius derived in step S196 or step S194 is output.

FIG. 63 shows another example of processing executed in place of the above-described method in the ink data generation unit 120 of FIG. 49 in reference to FIG. 55 in order to output variable alpha information as an attribute of ink data. It is a flow diagram.

In step S2002, it is determined whether or not the input data includes the pen pressure data, as in step S1202 of FIG. 50A.

In step S2006 after it is determined that the data input in step S2002 does not include the pen pressure data ("FALSE"), as described with reference to FIG. 56 above, the speed derived from the time stamp information is used. Alpha (transparency / opacity) is calculated using the relationship with alpha. That is, the relationship is used that as the velocity increases, so does the alpha (becomes more transparent).

In step S2004 after it is determined that the data entered in step S2002 contains pressure data ("TRUE"), both the entered pressure data and the time stamp information are used to derive the alpha value. .. Step S2004 is (i) when the pen pressure is fixed, the alpha increases (becomes more transparent) as the speed increases, and in the case where the speed is fixed, (ii) when the pen pressure increases. Use a partial derivative that contains two variables: alpha decreases (becomes more opaque). Therefore, it is possible to encode the alpha (transparency) of each point of the ink data 200 based on both the speed and pressure of the points.

In step S2008, the alpha derived in step S2006 or step S2004 is output.

The processes of FIGS. 62 and 63 can be used together so that both radius and alpha values can be derived from the input pressure data and timestamp information. Alternatively, only the radius value or only the alpha value may be derived from the input pressure data and time stamp information.

A sample function for converting velocity (vn) to radius or alpha is described above with reference to FIGS. 54 and 56. In general, what function should be used to convert velocity (vn) to radius and / or alpha is simulated by the pen type (or pen tip type) and the sensor surface. It depends on the type of "paper" expected to be (eg, Japanese paper, carbon paper, plain paper, copy paper, photo paper, ink absorbing paper, etc.). Therefore, some of the functions chosen to convert velocity to radius and / or alpha can be adjusted by pen type and / or paper type. In other words, the radius and / or alpha derived from the velocity can vary depending on the type of pen and / or the type of paper.

According to a further aspect of the present invention, as described above, the pen pressure data used in various embodiments of the present invention includes pen angle (or pen tilt) data, pen rotation (pen rolling) data, and the like. Can be replaced by other types of attribute values that can be received from various types of input devices, such as. These attribute values, such as pen angle / tilt data, are used in the pressure data used in the above examples to derive radius and / or transparency (alpha) information according to various embodiments of the invention. Can be used instead. For example, some pen-type input devices have the ability to generate pen angle (pen tilt) data indicating the angle created by the pen axis with respect to the sensor surface or the normal of the sensor surface. A pen held perpendicular to the sensor surface tends to exert more pressure on the sensor surface than a pen extending in a direction more parallel to the sensor surface due to its tilt. Thus, just as pressure data can be correlated with the parameter vn (velocity), pen angle / tilt data can be correlated with vn.

For example, a function that defines the relationship that the faster the pen is held relative to the sensor surface (ie, closer to the normal of the sensor surface) (ie, the higher the pressure), the slower the velocity (vn). Can be used. Once vn is derived from the angle / tilt data, a function similar to the one described above used to transform vn into radius and / or alpha information can be used. That is, the pen angle / tilt data can be correlated with vn, which is later converted into radius and / or alpha information. In contrast, a tilted pen that is held perpendicular to the sensor surface tends to generate (draw) a wider stroke, probably due to the increased contact area between the tilted nib and the sensor surface. It can also be observed that it tends to produce narrower strokes than. In this case, an appropriate function may be used that defines the relationship that the more the pen is held straight against the sensor surface, the faster the velocity (vn) will be and the narrower the stroke will be. Once vn is derived from the angle / tilt data, vn can then be converted to radius and / or alpha information using the functions described above. What function is used to convert the angle / tilt data to vn is the type of pen (or type of pen tip) and the "paper" that the sensor surface is expected to simulate. Depends on the type (eg, Japanese paper, carbon paper, plain paper, copy paper, photo paper, ink absorbing paper, etc.). In addition, pens held perpendicular to the sensor surface may tend to produce wider and darker strokes than tilted pens, which tend to produce narrower, finer (more transparent) strokes. Can be observed. So, in some embodiments, instead of first correlating the angle / tilt data with vn and converting vn into point-by-point radius and / or alpha information, the pen angle / tilt data is directly Appropriate functions can be used to convert to radius and / or alpha information. Similarly, some embodiments

[4] Ink Data Processing Unit / Decoder (FIGS. 64 to 66) FIG. 64 shows the relationship between the ink data processing unit 100R and various applications 300-1,300-2,300-n according to the embodiment of the present invention. It is a figure explaining. The ink data processing unit 100R corresponds to 100R in FIG. 22 of the first embodiment.

As shown in FIG. 45, the ink data processor 100R essentially responds to requests from various applications 300 by files or files in memory ("RAM") or various types of media (eg, hard disks). It is a software library that retrieves the ink data 200 stored in the message format to a certain memory location in a data format that can be used by the application. For example, when the application 300-1 is a drawing processing application (using the graphic processing unit 300 in FIG. 5), the ink data processing unit 100R performs necessary data objects "strokes", "points", etc. (each stroke and). Outputs to the graphic processing unit 300 (where / or points are associated with radius and / or alpha information), but does not output unnecessary data objects such as "author". As another example, when the application 300-2 requests only the author information for the purpose of determining the participants of the electronic conference, the ink data processing unit 100R outputs the data object "author" in the metadata object 250. .. In the embodiment of the present invention, the ink data processing unit 100R is realized as the ink data reproduction processing S200 executed by the processor as described in FIG. 65. Hereinafter, the ink data reproduction process S200 will be described with respect to the drawing process in which the application 300 is the drawing application 300-1.

FIG. 65 shows the ink data reproduction process S200 for acquiring the radius and alpha information together with the X and Y coordinates by reproducing (extracting) the generated ink data according to the embodiment of the present invention, and the acquired information. It is a flow diagram explaining that the data is output in response to the request from the drawing application 300-1. The drawing application 300-1 can then use the radius and alpha information to give the stroke a more realistic look and subtle differences in representation when drawing / rendering on the stroke (Figure 68). ~ See Figure 72). In essence, the ink data reproduction process S200 is a reverse process of the process of generating (shaping) the ink data S140 described with reference to FIG. 57 above.

&lt;Reproduction or retrieval of data object InkDataFile&gt; In step S2001, context information (input 2) (or configuration information) for ink data 200 including strokes to be processed is fetched, such as a decimalPrecision value. This reproduction process reversely corresponds to the coding process in step S1401 of FIG. 57.

In step S2002, the strokesCount value contained in the data object InkDataFile is retrieved as shown in Figure 48L. For example, if the strokesCount value is not available, step S2002 can be skipped and when the process instead replays the data indicating the last of all the strokes contained in the InkDataFile. It may be determined whether to end the process.

&lt;Reproduction of data object "stroke"&gt; The following steps starting from step S2003 contained in the larger square drawn by the dotted line in Fig. 65 are executed for each of the N (strokesCount) strokes contained in the InkDataFile. The object. As a result, N strokes are played and output.

In step S2005, the variable Alpha value (see Figure 48L) in the data object "stroke" is retrieved. As mentioned above, this variable Alpha value (TRUE / FALSE) indicates whether the stroke to be processed contains a variable alpha value along the stroke length.

In step S2006, a "pointsCount value" indicating the number of point objects contained in the stroke object 210 is acquired. For example, in a real-time application, if the pointsCount value is not available, step S2006 can be omitted and the process will instead terminate the process by playing back data indicating the end of the stroke being processed. It may be determined.

&lt;Reproduction of data object "point"&gt; The following steps starting from step S2007 contained in the smaller square drawn by the dotted line in Fig. 65 are for each of the pointsCount points contained in the stroke object 210 to be reproduced. Will be executed. As a result, pointsCount point objects are played and output.

In step S2009, it is determined whether the alpha of the stroke to be processed is variable. That is, it is determined whether or not the variable Alpha value is TRUE.

In step S2011 after the variable Alpha value is determined to be TRUE in step S2009, the radius and alpha value are decoded and output together with the XY coordinate value for that point. This step reverses (converts) the data encoded in step S1411 of FIG. 57 to the data format required (used) by the particular application requesting the ink data 200 to be reproduced. Is.

In step S2013 after the variable Alpha value is determined to be FALSE in step S2009, the alpha value is set as a fixed value for the entire stroke, as described more fully later in Figure 66. , The XY coordinate value and radius for that point are decoded, reproduced and output. This step reverses (converts) the data encoded in step S1413 of FIG. 57 to the data format required (used) by the particular application requesting the ink data 200 to be reproduced. Is.

Therefore, the ink data reproduction process S200 extracts an XY coordinate value, a radius value, and an alpha value in some cases from the data object "point".

FIG. 66 illustrates examples of steps S2011 and S2013 of FIG. 65 described above according to an embodiment of the present invention.

Lines 01 to 07 at the top of FIG. 66 have variable alpha values along the length of the stroke, including the points (ie, the different points that make up the stroke can have different alpha values). In the pseudo code corresponding to step S2011 of FIG. 65, when the XY coordinate value for that point is retrieved along with both the radius and alpha values.

Lines 08-14 at the bottom of Figure 66 are XY for points if the alpha value is not variable along the length of the stroke, including the point (ie, all points that make up the stroke have the same alpha value). Pseudocode corresponding to step S2013 in FIG. 65 when a fixed value (eg, "1.0" in the illustrated example) is set as the alpha value while the coordinate and radius values are retrieved.

In FIG. 66, the sections indicated by "A" and "B" show how the above-mentioned decimalPrecision values are utilized in the practice of embodiments of the present invention. Specifically, as shown in "INV_B" in FIG. 66, the input X, Y and radius and alpha data in step S2011 are int (integer) using the inverse function of the function shown in FIG. Is first inversely converted (cast) from to a float data type. Then, as shown in "INV_A" at the top of FIG. 66, the X, Y and radius data are divided by the decimalPrecision value indicating the resolution.

In step S2013, shown at the bottom of FIG. 66, as in step S2011, the entered X, Y and radius values are cast to the float data type and then divided by the decimalPrecision value. On the other hand, the alpha value is set as a fixed value such as "1.0" in the illustrated example.

Thus, when application 300 is, for example, a drawing application that requires the input data to be of the "float" data type, the generated ink data 200 containing the integer type data will be of the requested floating point data type, or , Plays (decodes) back to the original data type of any other non-integer requested by application 300.

[5] Ink Data Drawing Process (FIG. 67) FIG. 67 is a drawing application 300-1 (and FIG. 5) using ink data 200 for drawing (rendering) strokes on a screen according to an embodiment of the present invention. It is a flow diagram explaining the drawing process S300-1 executed by the graphic processing part 300) of.

In step S200, in order to retrieve information about the stroke and the points included in each stroke, the process causes the ink data processing unit 100R to acquire and regenerate the InkDataFile as described above, whereby the retrieved information is obtained. It can be used as input data for drawing process S300-1.

Next, the drawing (rasterization) style object associated with the stroke object 210 to be processed is determined. As shown in Figure 48B, the ink data 200 defines the appearance of each stroke object 210 (in the stroke model subdomain) as it is drawn (rendered, rasterized, etc.) on the screen1 It is structured to be associated with the above drawing style objects (in the rasterized subdomain). While there are many types of drawing style objects, in the illustrated embodiment of Figure 67, two options are available: a scattering style object and a shapefill style object (see Figure 48).

When a scatter style object is selected, a vertex (point) array is derived for each stroke in the subprocess S300-1S. This derivation is made so that the vertex array consists of a set of discrete points located at low density. The process of deriving the vertex array uses the attribute values generated by the ink data processing unit 100, such as the values of "interval" and "range". The process of deriving the vertex array can also use the context information (input 2) received from the application or operating system. For example, contextual information about the paper types that the screen is expected to simulate (eg, Japanese paper, carbon paper, plain paper, copy paper, photo paper, ink absorbing paper, etc.) is located at low density in the vertex array. It can be used to increase or decrease the number of discrete points. The generated vertex array represents a series of particles. In the illustrated embodiment, the GPU controlling the drawing process S300-1S is the first vertex for the generated vertex array in order to give each particle a defined size based on the "radius" value of each point. Apply a shader. The GPU also applies a first fragment shader to the array of particles to give each particle a defined level of transparency (or opacity) based on the "alpha" value of each point. The drawing process S300-1S thus draws a given stroke in the style of "scattered" particles (see Figure 48F).

When a shapefill style object is selected, the subprocess S300-1F first derives a spline segment for each stroke. This derivation is such that each spline segment is a sufficiently smooth polynomial function defined for each part of the continuous stroke curve. That is, a set of spline segments connects vertices and defines a plurality of curved segments that jointly represent a stroke. Based on the "radius" value of each vertex (point), the GPU has a second vertex for a set of spline segments to give each circle around each vertex along the stroke a given size. Apply a shader. The GPU also applies a second fragment shader to a set of spline segments to give each circle a defined level of transparency (or opacity) based on the "alpha" value of each vertex (point). The drawing process S300-1F draws a given stroke in the "shape fill" style in this way (see Figure 48F).

[6] Effect. Ink Data Drawing Examples (FIGS. 68 to 72) FIGS. 68 to 72 explain various drawing rendering examples. These are used to explain an ink data generation method, an ink data reproduction method, and an ink data drawing (rendering) method according to an embodiment of the present invention. In FIGS. 68 to 72, "s" indicates the start position of the stroke, and "e" indicates the end position of the stroke. In all cases, it is assumed that the speed of movement of the pen increases (accelerates) from "s" to "e".

FIG. 68 illustrates an example of drawing rendering as a result of input of ink data generated based on the damping function shown in FIG. 54 according to an embodiment of the present invention. By using the decay function, the radius decreases as the velocity increases. Therefore, in all the drawing examples shown in FIG. 68, the stroke width decreases from "s" to "e". In these examples, alpha (transparency) is set as a fixed value.

In this rendering example, the line width that increases with pressure corresponds to the area where the ink seeps into the paper [Observation A], and the faster the pen moves, the more time the pen has to see out the ink at each point. Is shortened [Observation B], which follows the above-mentioned observation result. The ink data processing unit according to the embodiment of the present invention acquires speed information for each point even when the given pen event data input does not include pressure information, and the radius of each point based on the speed information. Has the ability to calculate information. Therefore, the generated ink data contains radius information for each of at least some points. The radius information is a realistic appearance and representation that closely simulates the appearance of a real stroke handwritten with ink on paper for the stroke drawn when the ink data 200 is rendered (drawn) on the screen. Can be used to make subtle differences.

FIG. 69 is a diagram illustrating an example of drawing rendering as a result of input of ink data 200 generated based on the power function shown in FIG. 56 according to the embodiment of the present invention. By using the power function, as the speed increases, the alpha (transparency) increases. Therefore, in all the drawing examples illustrated in FIG. 69, the strokes are brighter and more transparent (ie, reduced in density) from "s" to "e". In these examples, the radius is set as a fixed value.

In this rendering example, the density of the ink that increases with pen pressure corresponds to the area where the ink seeps into the paper [Observation C], and the faster the pen moves (the time the pen is in contact with each point). According to the above-mentioned observation result that the amount of ink exuding from the pen decreases at each point (because it becomes shorter) [Observation D]. The ink data processing unit according to the embodiment of the present invention acquires speed information for each point even when the given pen event data input does not include pressure information, and the alpha of each point is based on the speed information. Has the ability to calculate information. Therefore, the generated ink data contains alpha information for each of at least some points. The alpha information is a realistic appearance and representation that closely simulates the appearance of a real stroke handwritten with ink on paper for the stroke drawn when the ink data 200 is rendered (drawn) on the screen. Can be used to make subtle differences.

FIG. 70 illustrates a drawing rendering example as a result of input of ink data 200 generated based on both the decay function shown in FIG. 54 and the power function shown in FIG. 56 according to an embodiment of the present invention. It is a figure. By using the decay function, the radius decreases as the velocity increases, and by using the power function, the alpha (transparency) increases as the velocity increases. Therefore, in all the drawing examples illustrated in FIG. 70, the stroke width decreases from "s" to "e", while at the same time the stroke becomes brighter and brighter from "s" to "e". It is transparent (ie, the concentration is decreasing). The ink data processing unit according to the embodiment of the present invention acquires speed information for each point even when the given pen event data input does not include pressure information, and the radius of each point based on the speed information. And has the ability to calculate alpha information. Therefore, the generated ink data contains radius and alpha information for each of at least some points. The radius and alpha information is a realistic appearance and alpha information that closely simulates the appearance of a real stroke handwritten with ink on paper for the stroke drawn when the ink data 200 is rendered (drawn) on the screen. Can be used to give subtle differences in expression.

FIG. 71 is another function of FIG. 54 (S) as used in step S1207_07 of FIG. 52 for a special type of pen, such as a pen with a particularly soft nib, according to an embodiment of the invention. It is a figure explaining the drawing rendering example which shows the effect of a character and a periodic function).

The drawing example on the left hand side is a drawing example obtained as a result of the "S-shaped" function of FIG. 54 in which the attenuation occurs stepwise as opposed to the exponential in the "attenuation" function. Therefore, in each of the resulting rendering examples, the stroke radius (width) gradually decreases from wide to narrow, as opposed to gradually decreasing in the example of FIG. is doing. In the "S-shaped (increase)" function of FIG. 56, the increase occurs in stages, in contrast to the "S-shaped (decrease)" function of FIG. 54, in which the decay occurs in stages. Therefore, the drawing example obtained as a result of the "S-shaped (increasing)" function in FIG. 56 has an appearance similar to that of the drawing example on the left-hand side of FIG. 71, but the positions of "s" and "e". Will be replaced.

The drawing example on the right-hand side of FIG. 71 is a drawing example obtained as a result of the "periodic" function of FIG. 54 in which the output of the radius changes (increases or decreases) periodically. Therefore, in each of the strokes drawn (rendered) as a result, the radius (width) of the stroke changes periodically from "s" to "e".

FIG. 72 illustrates a drawing rendering example showing the effect of using the special values as the radii of the start point (s) and the end point (e) added in step S1207_04 of FIG. 52 according to the embodiment of the present invention. is doing.

Specifically, in step S1207_02 of FIG. 52, a special value is set as a radius for each of the start point (s) and the end point (e) added to the start and end of the stroke, respectively.

The left-hand side of FIG. 72 describes a drawing example when the radius of the start and end points is set to zero ("0"). This means that no matter how fast or slow the user moves the pen at the start or end of the stroke, the radius (width) of the start and end points of the stroke is essentially ignored in the resulting picture. means.

On the right-hand side of FIG. 72, the radius of the start and end points is larger than the radius normally calculated, i.e., the radius calculated by various embodiments of the invention using the various functions described above (eg, double). Explains a drawing example when it is set. As shown, this means that when the user draws a stroke with the pen on the paper (because the pen often pauses at the start and end of the pen stroke), the start and end points of each stroke are often on the paper. The result is that the start and end points of each stroke are highlighted in the same way as highlighted in.

As described above, according to the ink data processing unit, the ink data generation method, the ink data reproduction method, and the ink data drawing method of various embodiments of the present invention, strokes having a realistic appearance are rendered (drawn). Therefore, non-device dependent ink data can be generated and used. The ink data 200 is structured so that some of them support pressure data while others can be shared by various types of devices and applications that do not support pressure data. The structure of this ink data 200 defines a radius and / or alpha value for each point of each stroke, and this radius and / or alpha value is the pressure data for the stroke drawn on the screen. Alternatively, it can be used to give a subtle difference in realistic appearance and representation that closely simulates the appearance of a real stroke handwritten with ink on paper.

In the above description, the pressure is mostly described as the pen pressure applied by the pen, but for a device capable of acquiring (measuring) the pressure applied by a finger, for example, the pressure is the pressure of the finger. May mean. Therefore, in this description, the term "pressure" should be understood as a synonym for "pressure", and the term "pen" is a pen, finger, and any other user-based input device. It should be understood as a synonym for "indicator" which may include instruments, devices, and elements that can be used to indicate a position.

In the above description, alpha is used as an indicator of the degree of transparency (the larger the alpha, the greater the transparency), but there is also a parameter indicating the degree of opacity that the larger the value, the greater the degree of opacity. It can be used.

The ink data processing unit is generally described as a separate entity from various applications that request the ink data reproduced from the ink data processing unit, but these are jointly based, for example, based on a connection via a library link. It may be configured in one or the other.

Fourth Embodiment In the fourth embodiment of the present invention, the pen event data based on the handwriting motion of the user is received, and the context information provided by the application or the operating system supporting the handwriting motion (input 2). Is directed to the system and method of receiving. The context information (input 2) contains information about pen event data such as pen type and author ID. The system and method generate ink data 200, including stroke object 210, metadata object 250, and drawing style object 230, based on received pen event data and received contextual information (input 2). The system and method further receives operation information from an application or operating system that supports handwriting, and based on the received pen event data, received context information (input 2), and received operation information, the ink data. Operation objects that make up a part can be created.

Background of the Fourth Embodiment Digitized handwriting input data or ink data is shared among various operating systems, various applications, various services, various image formats, various existing standards for strokes, and the like. A framework that allows them to be done is desired. In short, unification of stroke data models is desired.

The Hypertext Markup Language (HTML) is an example of a successful unified framework. HTML contains (1) "text", which is essentially a combination of a fixed number of character codes, and (2) meta tags, which embed how this text should be characterized and portrayed when it is displayed. It has been widely adopted as a common language for embedding. For example, meta tags indicate fonts, sizes, colors, columns, rows, groups, tables, etc., which are commonly interpreted by different types of browsers to determine the appearance of text. Such a common language allows the generation of documents that can be displayed in substantially the same manner on different devices in different computing environments (although, for example, due to the implementation of each browser). There can be small variants and differences).

The same is desired for handwritten input data. That is, (1) a "stroke" (or "trace" or "pass") (input by the user's handwriting action), and (2) a "stroke" such as stroke color, texture, offset position, etc. A common language that defines the "objects" that characterize or describe is desired. In the following, such a common language (or information model) will be referred to as "stroke language (SL)". Stroke Language (SL) will enable the generation of digital documents that can be displayed in substantially the same manner (same appearance) on different devices in different computing environments. However, there may be some minor variants and differences, for example due to the implementation of each rendering engine.

As described in Non-Patent Documents 1, 2 and 5 above, some data configured to represent handwritten strokes in a manner that can be shared by various applications such as InkML, ISF, and JOT data structures. The structure is known.

Simply put, InkML (Non-Patent Document 1) is provided for the purpose of expressing ink input using an electronic pen or stylus by using a markup language that describes input data. be. For example, InkML defines a data structure for strokes, where the data structure &lt;trace&gt; contains a set of data generated by the input device, and the format of this data is a large number of &lt;channel&gt; elements. Defined in a unique data structure &lt;trace format&gt; that uses.

ISF (Non-Patent Document 2) is provided for the purpose of storing ink data in binary forms intended for use in mobile devices such as PDAs, tablet PCs, and others that use a stylus as an input mechanism. It is a thing. For example, the ISF defines a data structure for strokes, where the data structure TAG_STROKE contains a set of data generated by the input device, and the format of this data is TAG_NO_X, TAG_BUTTONS and various other tags. It is defined in the unique data structure TAG_STROKE_DESC_BLOCK to be used. ISF requires compression coding and has the ability to generate static (persistent) streams using the method of picking the most suitable compression method for each type of data. They use a combination of delta encoding and a tuned version of the Huffman algorithm for data generated by input coordinates, pressure levels, and other styli, for example, for custom properties such as custom drawing attributes. LZ algorithm is used for.

JOT (Non-Patent Document 5) is provided for the purpose of exchanging data input by an electronic pen or stylus between various machines using various operating systems and architectures. For example, JOT defines a data structure for strokes, where the data structure tag_INK_POINT describes a single pen event and features such as its position, force (pressure), rotation, and so on.

Also, various standards, not limited to processing handwritten strokes, exist for the purpose of describing vector graphics in an input-independent manner. SVG1.1 (Non-Patent Document 3) is one such example. Version 1.1 of SVG includes path elements that rely on straight lines and Bezier curves for the purpose of representing strokes.

4. Outline of Embodiment The embodiment of the present invention can be understood as dealing with one or more of the three aspects, in particular aspect 3.

To realize one or more aspects of the invention described above, a system and method for generating, converting, and otherwise processing ink data 200 defined in a novel language (or information model) is provided. Will be done.

Embodiments of the present invention are directed towards outputting ink data 200 including stroke objects that are statically depicted by metadata objects and / or dynamically controlled or manipulated by drawing style objects and manipulation objects. It was done. Stroke objects, metadata objects, drawing style objects, and operation objects as a whole are ink data that can be stored in a recording format (eg, stroke file format (SFF)) or transmission format (eg, stroke message format (SMF)). Configure 200.

The stroke object according to the embodiment of the present invention has a variable stroke width (that is, a width that changes along the length of the stroke), a variable stroke color or transparency, as shown in the third embodiment described above. It can have (alpha) / opacity (ie, a color or transparency / opacity that changes along the length of the stroke). The stroke object according to the embodiment of the present invention is defined by using an appropriate interpolation method such as the Catmull-Romspline method as shown in the first embodiment described above, and any part of the stroke is defined. Special parameters can be used to describe the start and / or end.

An embodiment of the present invention dynamically manipulates a remote (remotely located) stroke object, for example, depending on the connection / connection state with a remote host, in the same manner as dynamically manipulating a local stroke object. It is directed to a method of generating ink data 200 that can be manipulated.

Descriptive Figure 73 of the Fourth Embodiment is a diagram illustrating the whole system in which the ink data 200 is used according to the embodiment of the present invention. Compared to the system described in FIG. 1, the system of FIG. 73 additionally includes server # 2 supporting application service # 2 accessed by apparatus 10-1-2 and apparatus 10-3. Both the application service # 1 and the application service # 2 of FIG. 73 can be exchanged by using the ink data 200 via the ink data exchange infrastructure 10. In FIG. 73, device 10-1-1 is a pen-type input device configured to be able to output pen pressure data, and generates ink data 200 using application 300-1 provided by application service # 1. .. Application 300-1 links to a TCP / IP library and a library for the ink data processing unit 100 and the graphic processing unit 300 (not shown) that execute the ink data generation method of the present invention. The generated ink data 200 may then be output in an appropriate output form (eg, packet) corresponding to the destination media (eg, network).

The device 10-1-2 is a tablet-type input device configured to be able to receive handwritten input by the user's finger. The sensor of device 10-1-2 does not have the ability to output pen pressure data, but can generate ink data 200 using application 300-2 provided by application service # 2. Application 300-2 links to or utilizes a library such as the TCP / IP stack and a library for the ink data processor 100 on server # 2 that executes the ink data generation method of the present invention. .. The generated ink data 200 may then be output in an appropriate output form (eg, packet) corresponding to the destination media (eg, network).

Device 10-3 is a desktop PC that subscribes to application service # 2. Device 10-3 processes ink data 200 output from device 10-1-1 or device 10-1-2 using application 300-2 provided by application service # 2 (eg, display screen). Can be rendered on or redistributed). Application 300-2 dynamically links, or these, to a library such as the TCP / IP stack and a library for the ink data processor 100 on server # 2 that executes the ink data generation method of the present invention. To use.

FIG. 74 is a block diagram of the ink data processing unit 100 according to the embodiment of the present invention. The ink data processing unit 100 corresponds to the ink data processing unit 100 shown in FIG. The ink data processing unit 100 can be implemented as a library that is dynamically or statically linked to an application such as the drawing application 300-1 that utilizes the graphic processing unit 300 shown in FIG. The ink data processing unit 100 includes an ink data generation unit 120 and an ink data shaping unit 140. The ink data generator 120 generally has three types: 1) pen event type input information ("input 1"), 2) context information ("input 2"), and 3) operation information ("input 3"). Enter, include, and receive information.

Input 1.

"Pen event type input data" or simply input data or pen event data is input from an OS, a device driver, or an API for acquiring data from an input device such as a pen tablet sensor. The input data can be from various types of input devices, as illustrated on the left hand side of FIG. The input data is not limited to raw data from the input device and can include pen event data generated by processing the raw data, such as InkML and ISF data.

Input 2.

The context information indicates the context used to support the input of the pen event type input data described above. Contextual information can include, for example, date and time information about the stroke (eg, when the stroke was entered), pen type, pen color, pen ID, author ID, input device resolution and sampling rate, etc. Provided by the application (application 300-1) or OS used to generate.

Input 3.

The operation information is a command indicating that the next stroke to be entered is to construct an operation object used to operate the existing stroke object 210 instead of the normal stroke object 210. Is. Such commands can be entered by user activation of a switch or button associated with the input device and are provided by application 300-1 to the ink data generator 120. For example, when the user wants to "slice" an existing stroke object 210, the user issues operation information and performs a handwriting operation to slice the existing object. From an operational information point of view, the ink data generator 120 uses the user's slicing motion to slice the existing stroke object 210 instead of drawing other strokes based on the user's slicing motion.

The ink data generator 120 inputs / receives these three types of information (input 1, input 2, and input 3), as shown in FIG. 48B described above with reference to a third embodiment. Create a set of objects according to the stroke language (SL) definition.

The ink data shaping unit 140 is separated from the ink data language operation unit 120 that generates ink data including various objects. In FIG. 74, a group of objects generated by the ink data generation unit 120 is input to the ink data shaping unit 140 that outputs data in a "recording format" or a "transmission format" from bottom to top, that is, in the output direction. To. In FIG. 74, data written in a recording format or a transmission format reproduces a group of objects from top to bottom, that is, in the input direction, and provides the reproduced group of objects to the ink data generation unit 120. It is input to the part 140. In the figure below, SL stands for stroke language (see Figure 48B), SFF stands for stroke file format, which is a type of recording format, and SMF means stroke message format, which is a type of transmission format.

FIG. 75 is a more detailed functional block diagram of the ink data processing unit shown in FIG. 74 according to various embodiments of the present invention. The ink data processing unit 100 in this figure corresponds to the ink data processing unit 100 shown in FIG.

The ink data generation unit 120 includes an operation unit 122 of the stroke object 210, a metadata object operation unit 124, a rendering (drawing style) object operation unit 126, and an operation object operation unit 128.

The stroke object operation unit 122 receives the pen event type input data as the input (input 1) and also refers to the context information (input 2) to generate the stroke object 210 that constitutes the core of the stroke language.

The metadata object operation unit 124 generates a metadata object 250 that describes the stroke object 210 based on the pen event type input data (input 1) and context information (input 2). The metadata object contains non-drawing related information about the stroke object 210 that does not affect the appearance of the stroke object 210 when drawn on the screen, such as datetime information, author ID, and pen ID.

The rendering (drawing style) object operation unit 126 controls the rendering (drawing) of the stroke object 210 based on the stroke object generated by the stroke object operation unit 122 and also with reference to the context information (input 2), and is displayed on the screen. Generates a drawing style object 230 that defines what the stroke object 210 will look like when rendered to.

Operation object The operation unit 128 is an existing stroke that exists locally ("local") or remotely ("remote") across the network, depending on the reception of operation information ("input 3" in FIG. 75). Use the following "stroke" received as input 1 to generate an operation object 270 configured to manipulate the state of object 210.

Therefore, the ink data generator 120 generates a set of objects based on the stroke language, as shown in FIG. 48B, based on three types of input information (input 1, input 2, and input 3).

The ink data shaping unit 140 has a recording format data processing unit 142 configured to output a file in a recording format such as SFF, InkML, and JPEG format, and a transmission format such as SMF format to output a message. Including the configured ink data communication unit 144. The data defined by the stroke language that follows the definitions of these various formats are the SFF structure description file (schema file) (F142-1-1), the SVG structure description file (F142-2-1), and the SMF structure description file. It is output in the form of (F144-1) or the like. Therefore, it becomes possible to generate and reproduce various objects according to the stroke language, and the generated objects can be input / output in various recording formats and / or transmission formats. The illustrated file ink data generator 140 supports two recording formats SFF and SVG, but may support more than one recording format, and application 300-1 supports all possible file formats. Only one recording format may be supported when it is not necessary.

Operation object The output from the operation unit 128 (eg, the operation object 270 in Figure 75) is placed in the SMF message and networked to manipulate (eg, slice) one or more existing stroke objects 210 that reside remotely. Can be transmitted via.

&lt;Method of generating a language object&gt; FIGS. 76 to 83B show a stroke object operation unit 122, a metadata object operation unit 124, a rendering (drawing style) object operation unit 126, and an operation object operation unit according to the embodiment of the present invention. It is a functional block diagram and the flowchart explaining the structure and operation of a part 128.

&lt;SM (stroke model). Device / Method for Creating Stroke Object&gt; FIG. 76 is a functional block diagram of the stroke object operation unit 122 shown in FIG. 75. The stroke object operation unit 122 includes a stroke model processing unit (122_1) that inputs pen event data ("input 1") or pen event type data such as InkML data and outputs a point object. Each point object may include the x and y coordinates of the point and may also include radius and alpha values as attributes of the point, as described with reference to the third embodiment above. , The output point object can be expressed as (x, y, radius, alpha). The stroke object operation unit 122 receives the context information ("input 2") and prepares a configuration that will be output to the stroke model processing unit (122_1) for use in the generation of the point object. Includes model generator (122_2). The stroke object operation unit 122 further includes a stroke model builder unit (122_3) that assembles a plurality of generated point objects that jointly form one stroke into one stroke object 210. The stroke object operation unit 122 then outputs a plurality of generated stroke objects.

FIG. 77A is a flowchart illustrating the process of generating the stroke object 210. In S122_1, the model generator 122_2 processes the context information ("input 2") and retrieves the parameters used to define the configuration, such as the input rate. In S122_6, the specified configuration is loaded into the stroke model processing unit 122_1 and used to generate the point object. If this configuration instructed the stroke model processor 122_1 to apply smoothing, the stroke model processor 122_1 in S122_2 against an array of points to generate a smoothed point object. Apply smoothing. For example, a double exponential smoothing algorithm composed of appropriate window size, smoothing data factor, and directional smoothing factor can be used. If this configuration tells that smoothing is not applied, S122_2 will be skipped. If this configuration instructed the stroke model processor 122_1 to generate additional points, the stroke model processor 122_1 will generate additional points in S122_3 and make them at the beginning and / or end of the stroke. Add. Steps such as S122_2 and S122_3 are executed before S122_7 depending on the context information processed within the model generator 122_2. As another example, the value of the position of the point object constituting the stroke object 210 may depend on the interpolation method specified in the configuration loaded by S122_6 (for example, the Catmull-Romspline method). good. In step S122_7, the stroke model building unit 122_3 assembles the generated point object to form a new stroke object 210, and updates the existing stroke object 210.

FIG. 77B describes the detailed algorithm of S122_3 in FIG. 77A where additional points are generated and added at the beginning and / or end of the stroke. As shown, different algorithms are executed depending on the phase of the point being processed. The point phase indicates the position of the point with respect to the stroke to which the point belongs. For example, if the phase is the "beginning" indicating that the point is the start of the stroke (the "end" is similar and indicates the end of the stroke), then in the S122_4 ("end" phase). On the other hand, in S122_5), one or more points (for example, 3) are generated and added to form the beginning (end) of the stroke. Similar to S122_7 in Figure 77A, the position value of the point object generated and added in S122_4 (S122_5) is used in the interpolation method specified in the configuration loaded in S122_6 (eg Catmull-Romspline method). It may be dependent. If the phase is "middle" indicating that the point is in the middle of the stroke, no additional points are generated for that point.

&lt;MD (metadata). Device / Method for Creating Metadata Object&gt; FIG. 78 is a functional block diagram of the metadata object operation unit 124 shown in FIG. 75. The metadata object operation unit 124 receives context information ("input 2") and extracts metadata such as author ID and position from it. At least some of such metadata is placed in the configuration and sent to the metadata processing unit 124_2, which processes the stroke object 210 received from the stroke object operating unit 122 according to the configuration. The metadata object 250 is generated to depict the received stroke object 210. The metadata processing unit 124_2 extracts metadata such as time information from the received stroke object 210, and sends the extracted metadata to the metadata construction unit 124_3. The metadata generation unit 124_1 also sends the static configuration extracted from the context information to the metadata construction unit 124_3. Typically, the static configuration is common to the whole picture. The metadata construction unit 124_3 constructs the metadata object 250 based on the metadata received from the metadata generation unit 124_1 and the metadata processing unit 124_2.

FIG. 79 is a flowchart illustrating the process of generating the metadata object 250. In S124_1, the metadata generation unit 124_1 and the metadata processing unit 124_2 extract metadata such as a pen ID and a time stamp from their respective inputs. In S124_2, the metadata fetched as application context information is loaded into the metadata construction unit 124_3. In S124_3, the face of the point to be processed is determined. If that point was the starting point of stroke object 210, a metadata object 250 is created and associated with that point. Typically, only one metadata object 250 is required for each stroke object 210, so for the remaining points other than the starting point, the metadata object 250 does not need to be generated and associated.

&lt;R (rasterization). Device / Method for Creating Rendering (Drawing Style) Objects&gt; FIG. 80 is a functional block diagram of the rendering (drawing style) object operation unit 126 shown in FIG. 75. The rendering object operation unit 126 receives the context information ("input 2") and extracts information such as minimum / maximum radius information, minimum / maximum velocity information, minimum / maximum pressure information, and color information. At least some of such information is placed in the configuration and sent to the style processing unit 126_2, which processes the stroke object 210 received from the stroke object operation unit 122 according to the configuration. The drawing style object 230 is generated to define how the received stroke object 210 is drawn (rendered). The style processor 126_2 extracts and retrieves style-related parameters such as (variable) radius, (variable) color, (variable) alpha (transparency), and antialiasing parameters from the received stroke object 210. Send the information to the style construction department 126_3. The style generator 126_1 also sends the static configuration extracted from the context information to the style builder 126_3. Typically, the static configuration is common to the whole picture. For example, if all strokes in a picture have the same synthetic mixed mode, the value of that mode is a static configuration.

FIG. 81 is a flowchart illustrating a process of generating a style object. In step S126_1, the style processor 126_2 determines input characteristics such as pressure data, time stamp data, position data, etc., based on the stroke object 210 and context information available from the application. S126_2 loads the appropriate configuration according to the characteristics determined in step S126_1. For example, if the tool selected in the application is a ballpoint pen and the input characteristics include pressure data, then the configuration for a pressure-based ballpoint pen is loaded. As another example, if the selected tool is a ballpoint pen and the input characteristics do not include pressure data and include a time stamp, the configuration for a speed-based ballpoint pen is loaded (time stamp). Because the velocity can be derived from the information). In step S126_3, the loaded configuration is checked to determine if the width (or color) is variable. Then, in step S126_4, it is determined whether the width (or color) is variable point by point, that is, the stroke has a width or color that changes along its length. If yes, in S126_5 the cascading style property radius (and / or color) is generated for each point object. Otherwise, in step S126_6, a cascading style property is generated for the stroke object 210.

Another example for optional cascading properties is enhanced properties. The enhancement property is used to simulate a special ink outflow when the input device is stationary in the down state. A real-world analog to this property is the behavior of a watercolor brush on soft paper. When the watercolor brush is in a stationary position, the soft paper sucks in the paint and therefore the contact points are darker and larger. In step S126_7, if the input stroke object 210 meets the enhancement conditions specified by the configuration loaded in S126_2, an enhancement property is generated for each point object. For example, if the user is down and does not cause any movement, and the enhancement algorithm is activated in the loaded configuration, then in step S126_7 the current (last generated) point is , Replicated or updated to increase the point density at a particular location.

Prior to S126_8, other steps similar to those described above may be performed, depending on the configuration loaded in S126_2. In step S126_8, a style object is created. All of the cascading style properties updated in steps S126_5, S126_6, S126_7, etc. are incorporated into the style object in S126_8.

&lt;M (operation). Device / Method for Creating Operation Objects&gt; FIG. 82 is a functional block diagram of the operation object operation unit 128 shown in FIG. 75. The operation object operation unit 128 includes an operator generation unit 128_1 that prepares a configuration for use by the operation processor 128_2 by receiving and processing context information ("input 2"). The operation processor unit 128_2 processes the pen event data ("input 1") by referring to the configuration received from the operator generation unit 128_1 and the operation information ("input 3") received from the application. Generates an operation entity, such as a slice entity. Operation builder 128_3 receives an operation entity, such as a slice entity, and constructs an operation object, such as slice object 274. The operation object is configured to perform the specified action on the existing stroke object 210. For example, slice object 274 is used to slice an existing stroke object 210 into two slice pieces. A collection of operation (eg slice) objects generated by the operation builder 128_3 can be sent over the network to run on an existing stroke object 210 that resides remotely, or the stroke object controls. It can be run locally on an existing stroke object 210 created and stored by 122. As shown in the figure, the operation processor 128_2 can also receive a stroke object from the stroke object operation unit 122 based on any of the methods for generating the substance of the operation.

FIG. 83A is a flowchart illustrating the process of generating the operation object. In step S128_1, input characteristics such as pressure, position, and time stamp information are extracted from input 1 and operation information (input 3) is received. In step S128_2, the appropriate configuration determined from the retrieved input characteristics and application context information ("Input 2") is loaded. In step S128_3, new inputs are processed according to the loaded configuration to form the operation object. For example, if the new input is a polygonal shape, that polygon defines the operating area. For example, if the generated operation object is a lasso tool, its polygonal shape defines the scope of the lasso tool. In S128_4, one existing stroke object 210 is selected from among multiple existing stroke objects, and in S128_5, the intersection between the selected existing stroke object 210 and the operation area (for example, a polygonal shape). Is calculated. If no intersection is found in S128_6, steps S128_4, S128_5, until another existing stroke object 210 is selected and at least one intersection with the existing stroke object 210 is found. And S128_6 are repeated. If an intersection between the existing stroke object 210 and the operating area is found, S128_7 creates a slicing operation object configured to "slice" the existing stroke object 210 at the intersection. S128_14 determines if there are more existing stroke objects 210 that the operating areas can intersect. If "yes", the process returns to step S128_4. When all of the existing stroke objects are checked for their intersection with the operating area, the generated slice objects are assembled into a collection of slice objects in step S128_15.

FIG. 83B is a flowchart illustrating the process of generating the slice object. In S128_8, the configuration loaded in S128_2 is used to determine "operation accuracy". If the operation accuracy is "stroke as it is", S128_9 will generate a slice object 274 that slices the existing stroke object 210 to generate two slice chips, and each of the two pieces will have an existing stroke. Defined by a subset of the original point objects that make up object 210. In other words, the exact position of the intersection is the two slice chips, even when the intersection with the operating area is between two adjacent point objects of the existing stroke object 210. Never used to define.

If the operational accuracy is "accurate point", then in S128_10 an intersecting segment (curve) between two adjacent point objects is found. Here, the intersection segment is where the operation area intersects the existing stroke object 210. One or more intersecting segments are found. In S128_11, each intersecting segment is processed, for example, using an interpolation method to find the exact position of the intersection. In S128_12, for each intersection segment, two slice chips are generated with the exact position of the intersection as the end or start point, respectively. In S128_13, the generated slice chips are updated to become new stroke objects, and each of the newly generated stroke objects is associated with the metadata object 250 and the drawing style object 230.

As mentioned above, the ink data processor 100 and its related methods generate objects associated with the stroke object 210, including the metadata object 250, the rasterized drawing style object 230, and the operation object 270, along with the stroke object. do.

Hereinafter, the configuration and operation of the ink data shaping unit 140 will be described in detail with reference to FIGS. 84 to 90C.

FIG. 84 is a functional block diagram of the ink data ink data shaping unit 140 shown in FIG. 75. As described with reference to FIG. 75, the ink data shaping unit 140 includes a recording format data processing unit 142 that outputs a file in a recording format such as SFF, InkML, and JPEG formats, and various objects in a transmission format ( It includes a stroke object 210, a metadata object 250, a drawing style object 230, and an ink data communication unit 144 configured to output an operation object 270). Therefore, it becomes possible to generate and reproduce various objects according to the stroke language, and the generated objects can be input / output in various recording formats and / or transmission formats.

The recording format data processing unit 142 is configured to arrange a stroke object, a metadata object, and a drawing style object in the recording format. Subsections 142-1, 142-2, ... Are processing units configured to arrange objects according to their respective output file formats.

The ink data communication unit 144 arranges an operation object such as the slice object 274 in a stroke message format suitable for transmission to a remote device (in real time) over a network. Operation objects placed within the transmission format can be run on existing stroke objects that are locally present or remote with respect to the network.

FIG. 85 is a flowchart illustrating the processing executed by the ink data shaping unit 140. First, it is determined if the object needs to be sent. This determination is based on whether or not the currently locally assembled stroke object 210 is shared with the remote terminal. If affirmative, the object is placed in the Stroke Message Format (SMF) transmitted over the network in step S144. On the other hand, if it is determined that the object is not transmitted, the object is placed in an appropriate recording format selected from a plurality of recording formats. If the stroke file format (SFF) is used as the output format, the SFF generation process is executed in step S142-1. If other formats such as SVG and JPEG formats are used, in step S142-1 the SVG generation process will, for example, output a file in SVG format containing stroke language information based on the extension of SVG. Will be executed.

FIG. 86 is a flowchart illustrating a process of outputting data in the stroke file format (SFF). In S142-1-1, the SFF structure description file (F142-1-1) is analyzed to generate the SFF structure. The SFF structure is described using the interface description language. Each software application that uses SFF needs to understand (analyze) the description file in order to handle SFF data properly. For example, if the SFF structure is represented using Protocol Buffers IDL, the stub class generated by the Protocol Buffers compiler will be loaded in S142-1-1. In S142-1-2, the SFF structure is filled with various objects generated by the ink data generation unit 120 in FIG. 75. In S142-1-3, SFF structures are processed using various techniques for optimal memory representation of abstract data types, such as variable length integer coding techniques and exponential Golomb coding techniques. In S142-1-4, the memory representation of the generated SFF structure is packed in the memory stream. The generated memory stream may be stored in a file system or file structure.

S142-1-2 can include multiple steps to properly fill the SFF data structure. Compression operation for each stroke object 210 included in the ink data to be processed (S142-1-5) and for each point object contained in the stroke object 210 to be processed (S142-1-6). Is determined whether or not is executed. If "yes", then in S142-1-7 all floating point values are converted to fixed point precision values and expressed as integers. The loss of precision is compensated for by rounding the floating point value of the point object to the desired precision within the stroke object manipulation unit 122 or the rendering object manipulation unit 126. In S142-1-8, a compression algorithm such as data coding is applied to the generated integer value. In S142-1-9, objects, those that have been compressed if compression has been applied, are used to fill the SFF structure.

FIG. 87 is a flowchart illustrating a process of outputting JPEG format data. For each stroke object 210 (S142-2-1) contained in the ink data to be processed, in S142-2-2, all drawing style objects 230 connected to the stroke object 210 are read and drawn. All rasterization (drawing) properties defined within the style object are loaded (eg mixed and texture properties). At this time, all cascading properties such as color and radius values are analyzed. In S142-2-3, the stroke geometry is generated using the CPU or GPU. In S142-2-4, strokes are rasterized (drawn) by applying all rasterized / graphical information such as colors and textures to the generated geometry using the CPU or GPU. To. In S142-2-5, all rasterized strokes are assembled at once. S142-2-6 generates a bitmap containing all rendered (drawn, rasterized) stroke objects 210. In S142-2-7, this bitmap data is compressed using the JPEG algorithm.

FIG. 88 is a flowchart illustrating a process of outputting data in a stroke messaging format (SMF). In S144-1, an object is received from one of the object operation units 122, 124, 126, or 128 shown in FIG. 75. In S144-2, the object type is determined as a stroke object 210, a metadata object 250, a drawing style object 230, or an operation object. In S144-3, an identifier (for example, stroke ID, style ID) is assigned to the object in order to show the relationship between the object and the remaining objects in the ink data to be processed. In S144-4, the SMF structure description file (F144-4) is analyzed and the SMF structure corresponding to the determined object type is loaded. For example, if the SMF structure is represented using a protocol buffer, the stub class generated by the protocol buffer compiler will be loaded. In S144-5, it is determined whether or not the compression operation is executed for the determined object. If "yes", then in S144-6 all floating point values (eg x, y, radius, opacity, transparency) are converted to fixed point precision values and expressed as integers. In S144-7, a compression algorithm such as data coding is applied to the generated integer value. In S144-8, objects, those that have been compressed if compression has been applied, are used to fill the SMF structure. In S144-9, SMF data is stored in the memory stream.

FIG. 89 is a functional block diagram illustrating an input process of data (SFF / JPEG and SMF) output in various file formats and transmission formats.

In FIG. 89, the ink data output operation unit 140T shown on the left hand side executes the above-mentioned output process. The ink data 200 is output in a recording format such as SFF format and JPEG format, or is output in a transmission format such as SMF format.

These files and / or messages output in various formats are then input (received) by the ink data input operation unit 140R illustrated on the right hand side of FIG. 89. In various embodiments, the ink data input and ink data output processes are the same library, such as the same subsection 142-1 (both IN and OUT) and the same subsection 142-2 (both IN and OUT). Is executed in the same processing unit that shares.

The recording format data processing unit 142 in the ink data input operation unit 140R removes format-dependent data from the input data, extracts information about various types of ink data objects, and extracts information about the extracted ink data objects. It is output to the ink data generation unit 120 on the receiving side.

The ink data communication unit 144R in the ink data input operation unit 140R extracts an operation object from a received packet or message, and for each operation operation taken out, an existing stroke object in the ink data generation unit 120 on the receiving side. Instruct 210 to be executed (applied).

FIG. 90A is a flowchart of the process for interpreting and playing back the objects arranged in the SFF file. In S142-1 (IN) -1, the SFF structure description file is analyzed to generate the SFF structure. In S142-1 (IN) -2, the SFF structure is restored. One or more SFF structures are restored, and for each restored SFF structure (S142-1 (IN) -3), it is determined whether the restored SFF structure is compressed. If "yes", then in S142-1 (IN) -4 the restored SFF structure is decompressed, and in S142-1 (IN) -5 it is the decompressed fixed-point value expressed as an integer. Is converted back to a floating point representation. In S142-1 (IN) -6, the corresponding stroke language object is generated (eg stroke object 210, drawing style object 230, metadata object 250).

FIG. 90B is a flowchart of a process for interpreting and reproducing an object based on an input in InkML. In S142-2 (IN) -1, the InkML file is analyzed and loaded into memory. In S142-2 (IN) -2, the trace object is converted into a sample pointer input event. This process means extracting input data such as position, pressure, angle, tilt, and time stamp data and modeling the extracted input data into a sequence of pointer input events. In S142-2 (IN) -3, the sequence of pointer input events is passed to the stroke object operation unit 122. The stroke object control unit 122 also receives contextual information based on the data contained in the InkML file (eg, with or without pressure channels). The stroke object operation unit 122 creates a stroke object. In S142-2 (IN) -5, the metadata object operation unit 124 creates a metadata object. In S142-2 (IN) -5, the rasterized (drawing style) object 230 creates a drawing style object.

FIG. 90C is a flowchart of a process of receiving and executing an operation (slice) object by SMF. The S144-1 receives a collection of slice objects 274 written in SMF. In S144-2, slice object 274 is restored. In S144-3, existing stroke objects are considered to locate the stroke objects affected by the slice object 274 restored in S144-2. In S144-4, the affected stroke object is considered. In S144-5, all affected stroke objects 210 are modified (sliced) with the corresponding slice objects. All point objects within the point range identified as being within the corresponding slice object are deleted (erased). In S144-6, one or two new stroke objects will be created if desired. For example, if the deleted point object is in the middle of the sliced stroke object 210, the beginning of the original stroke object 210 constitutes the new stroke object 210 and the end of the original stroke object 210. The part may constitute another new stroke object 210. In S144-8, it is determined whether the style property of the affected stroke should be modified. If the style property should be modified, S144-9 sets the value of the new style property for the new stroke object. Otherwise, S144-7 simply copies the value of the style property of the original stroke object 210 to the newly generated stroke object. The same process applies to metadata. If the metadata should be modified, S144-11 sets the new metadata for the newly generated stroke object. Otherwise, S144-10 is the original stroke of the Simply copy the Ect 210 metadata to the newly generated stroke object. In S144-12, the values of the start parameter 301 and the end parameter 303 of the original stroke object 210 are copied to the newly created stroke object. The process described above is repeated for all affected strokes. On S144-13, a check is performed to determine if the current screen needs to be redrawn. If "yes", in S144-14, stroke objects sliced by one or more slice objects 274 in the modified area are drawn (rendered) on the screen.

Effect of the first to fourth embodiments FIG. 91 is a diagram illustrating the effect of using the processing device (101) of the ink data 200 of FIG. 75 with respect to the above-mentioned side surface 1. Operational objects according to various embodiments of the present invention allow transmission of the contents of an operational operation using a transmission format, thereby easily synchronizing the states of stroke objects located at multiple locations. For example, one device on the left-hand side of FIG. 91 and two devices on the right-hand side (corresponding to devices 10-1-1, 10-1-2, 10-3 in FIGS. 1 and 73, respectively). Suppose you are running a real-time collaboration application. Further assume that these devices share a stroke object 210 to be processed that has not yet been sliced. Then, according to the embodiment of the present invention, the following operations are possible.

1. First, the device on the left hand side performs a slice operation operation on the stroke object 210 having the specified stroke width WIDTH.

2. Next, the processing device 100 (operation object operation unit 128) of the ink data 200 generates an operation object based on the slice operation operation.

3. The ink data processor 101 then modifies the local stroke object 210 by performing a slice operation operation on its local stroke object 210 (see arrow "local" in Figure 75). This process may be executed prior to step 2 or may be executed in parallel with step 2.

4. Next, the ink data processing device 101 (ink data communication unit 144) formats the operation object into SMF format and sends the SMF data to the network (see the arrow "remote" in Fig. 75).

5. Devices 10-1-2 and 10-3 receive the operation object, which is SMF data, retrieve the stroke ID associated with the operation object, and each of the multiple stroke objects identified by the retrieved stroke ID. The operation operation (slice operation) is executed for the object. As a result, the (sliced) state of the stroke object identified by the retrieved stroke ID is synchronized between device # 1 on the left hand side and devices 10-1-2 and 10-3 on the right hand side. The object.

Therefore, the ink data processing method according to the embodiment of the present invention has the ability to dynamically manipulate stroke data locally and remotely via a network between two remotely located devices.

FIG. 92 is a diagram illustrating the effect of using the ink data processing apparatus (101) of FIG. 75 with respect to the above-mentioned side surface 2.

The left-hand side of FIG. 92 shows device-dependent raw data on the input side, and the right-hand side of FIG. 92 shows the data that will be included in the output file as a final product. The left-hand side shows the following four types of input data that can be used to generate strokes.

1. A set of point coordinates and a set of pressure information acquired by a Type 1 device capable of acquiring pressure information.

2. A set of point coordinates obtained by a type 2 device, a relatively simple device such as a device that incorporates a capacitive type touch sensor.

3. Pen rotation angle, pressure, X-direction pen tilt angle, Y-direction pen tilt angle, for example obtained by combining professional-grade hardware and applications used to generate computer graphics. Type N data including various details such as.

4. Standardized data like InkML that can represent orientation, height and pen orientation information.

As mentioned above with reference to Side 2, generally, the information that needs to be reproduced based on the handwritten input data is the angle at which the pen (stylus) was held and how much pressure was applied. It does not mean "how" the handwritten data was entered, such as stylus. Rather, the information that needs to be reproduced is the "result" of such a pen operation, which includes one or more strokes generated by the pen operation. Therefore, it is desirable to use a stroke model that produces handwritten input data that is as abstract and generalized as possible, that is, to process the handwritten data on the right hand side of FIG. 92 as much as possible. Such a stroke model can absorb the differences that exist between different devices that record "how" in various special (non-abstract) manners.

The rightmost side of FIG. 92 shows the data structure or file format contained in the image file as the final product of ink data processing. The middle part of FIG. 92 is an intermediate portion suitable for use in various applications such as textualization, signature authentication, annotation, and real-time collaboration applications obtained as a result of processing the ink data 200 according to the embodiment of the present invention. Shows vector data. The intermediate vector data includes existing SVG data (Non-Patent Document 3) that defines vector graphics in a device-independent manner, i.e., in a manner not derived from pen input. As such, SVG does not easily allow changes or adjustments to pen-derived data such as stroke width, stroke color, stroke transparency, and as a result, marks up (characterizes) the stroke data. ) Is not particularly suitable. In contrast, the stroke language (SL) based on intermediate vector data according to embodiments of the present invention marks up and characterizes stroke objects derived from raw data, and operates on such stroke objects, meta. It provides various objects such as data objects, rendering objects, and operation objects.

FIG. 93 is a diagram illustrating the effect of using the ink data processing apparatus (101) of FIG. 75 with respect to the above-mentioned side surface 3. The supply of a common stroke language (or a common information model that defines the meaning and grammar of a language) that is not tied to a particular format but can be used in various formats extends the life cycle of the ink data ecosystem. do. In Figure 93, 100-1,100-2, ..., 100-N represents various applications in which the ink data processing method according to the embodiment of the present invention is embedded. When the raw "input data" is input to application 100-1 ("step 1" in Figure 93), the ink data generator 120 of application 100-1 is the stroke language (or the information model that defines the stroke language). Abstract the raw data into the object of. The object is then converted to a recording or transmission format ("first format" or SVG in the illustrated example) and output ("Step 2" in FIG. 93). Application 100-2 receives and interprets data written in SVG to retrieve objects written in stroke language for rendering or manipulation. Application 100-2 can format the object into another recording or transmission format ("second format" or SFF in "step 3-2" of FIG. 93). The data written in SFF is then output and received by Application 100-N, which interprets the data written in SFF to retrieve objects written in stroke language for rendering or manipulation. Application 100-N can format and output objects in yet other recording or transmission formats ("third format" or bitmaps). Thus, for example, as compared to JOT in Non-Patent Document 5, which processes ink data structures using a single format, embodiments of the present invention have the ability to process ink data in various formats. Therefore, it extends the life cycle of the ink data 200. In the illustrated example of FIG. 93, the ink data 200 is made available by application 100-1, application 100-2, and further applications including the last application 100-N. Therefore, it extends the life cycle of the ink data 200. In the illustrated example of FIG. 93, the ink data 200 is made available by application 100-1, application 100-2, and further applications including the last application 100-N. Therefore, it extends the life cycle of the ink data 200. In the illustrated example of FIG. 93, the ink data 200 is made available by application 100-1, application 100-2, and further applications including the last application 100-N.

Each of the above-mentioned elements, devices, steps, and processes can be combined with other elements, devices, steps, and processes, depending on the implementation and application, or sub-elements, sub-devices, sub-steps, It will be appreciated by those skilled in the art that it can be further subdivided into sub-processes. Further, the steps and processes can be performed on a single processor or distributed within multiple processors, depending on the implementation and application, respectively.