User interface having clicktrough tools that can be composed with other tools.

IS 2 153 407 T3

DESCRIPTION

User interface with click tools through a transparent sheet that can be composed with other tools.

The present invention relates generally to processor controlled machines such as computers and more specifically to user interfaces to allow a user to interact with the machine.

A frequent use of a machine controlled by means of a processor such as a computer is to communicate information to a user of the machine and to accept information from the user, thereby allowing the user to perform a specified task. Depending on the type of the task, the user will often make use of a task-specific application program such as a word processor, (sometimes referred to as a text editor), a spreadsheet, a database, or a drawing program (sometimes referred to as a graphics editor). A reference to a specific type of program or editor is not intended to imply a standalone application program that has only particular functionality, as many programs have more than one type of functionality.

A typical application program consists of a set of instructions (the "application") that are executed in response to input signals to create and modify associated data (sometimes referred to as the underlying data). . In many cases, this associated data is stored on disk as a data file (sometimes referred to as "the file"), and parts of it are read into memory during program execution. For at least some applications, the data represents a document to be viewed (eg, printed or displayed on a screen), and the application allows a user to modify the document.

In many cases, a user provides at least some of the input signals through one or more input devices, often a keyboard and a pointing device such as a mouse. As a background, a mouse is a device that moves on a work surface, typically next to the keyboard, and provides position signals that make a cursor move on the screen according to the movements of the mouse. The cursor is a special symbol that is used by means of an interactive program such as a pointer or an attention-calling device. The rattan contains one or more pushbutton switches ("buttons") to provide additional input signals, which can be interpreted as part of a cursor event.

A display device, typically a display device such as a cathode ray tube (CRT) or a liquid crystal display (LCD) provides the user with information about the application and the underlying data, and that allows the user to generate the appropriate input signals and thus control the operation of the machine to produce the desired work product. The combination of input devices, display devices, and the nature of the information that the application provides to the user can be considered as the user interface for the application.

Although in principle it is possible for each application program to be completely self-sufficient, it is almost the universal case that the application program runs together with an operating system (“OS”). The OS is a program that programs and controls the resources of the machine to provide an interface between the application programs and the hardware of the machine. The OS provides the typical maintenance functions that all application programs will require in the same way, such as maintaining a file system, controlling the CPU, receiving inputs from input devices, communicating with storage devices, sending data to display devices, and provide a generic mechanism according to which a user can manage files and make various applications run. In the world of personal computers (PCs) and workstations, operating systems are often associated with a particular type of hardware configuration, but this is not necessarily the case. Unix is ​​an example of an OS that has been installed to run on many types of machines.

One type of operating system that has been in increasing use in recent years provides a graphical user interface ("GUI"). Apple Computer Macintosh OS, IBM OS / 2, and Microsoft's Windows (actually a GUI framework that runs on top of a character-based operating system known as DOS) are the best-known GUIs in the PC realm. The Macintosh OS has been available to date only on Apple-owned Macintosh PCs based on the 680x0 family of microprocessors whereas OS / 2 and Windows have only been available on so-called IBM-compatible PCs based on the 80x86 microprocessor family. This trend is in flux, with Microsoft's Windows NT having versions capable of running on more than one type of microprocessor.

A relevant aspect of a GUI is that a file open for a given application is given a window, which is a sizeable and movable screen region. The OS can have its own windows that show directory structures, with files and applications possibly being represented by means of icons (small graphic objects that represent actions or elements). There may be other windows that do not correspond to open files. One advantage of a GUI is that it facilitates a certain consistent user environment across applications. Some GUIs allow multiple applications to be opened at the same time.

Regardless of the type of OS, the application program, with varying amounts of OS help, typically provides the user with a visible representation (often referred to as the "screen image" or the "ima2

ES 2 153 407 T3 gene visualized ”) from the underlying data. The user acts on the visible representation and the program translates these actions into operations on the underlying data. As used herein, the term "visible representation" will refer to the visual representation of the underlying data not only for application programs, but also for all kinds of programs, including OS and various types of software. utilities.

For example, in a word processor, the underlying data consists of text with associated information that specifies how the document will look when printed on a printer. The associated information relates to the layout of the document such as paragraphs and columns, and to text attributes such as fonts, size, style, and color. Depending on the particular word processor and operating system, the screen image may be limited to the content of the text, or it may display the document substantially the way it will appear when printed (WYSIWYG - pronounced "wizzywig", as an acronym for “what you see is what you get”). Similarly, a program designed for a character-based OS such as DOS provides something close to the former, while one designed for a GUI will also provide something close to the latter.

A similar range of possible screen images will be found in other types of application programs. For example, in a drawing program, the underlying data will contain a description of each graphic object that will appear in the document. The description includes what is needed to give the object its desired appearance, including shape, size, color and line thickness, color and fill pattern, relative position on the document plane, and stacking order (in the case the object is in front of or below other objects). The screen image can show only the outline lines of the objects (wiring structure) or it can be a full WYSIWYG view.

Regardless of the type of application, the user manipulates the input devices with reference to the image on the screen in order to make the desired changes. This is usually done by positioning the cursor at a particular position on the screen that corresponds to the displayed position of an object to be modified, and by executing one or more user events such as keystrokes or keystrokes. ratéon actions. Mouse actions include button presses, button release, mouse movement, clicks, and mouse dragging. A mouse click refers to the user pressing and releasing one of the buttons without moving the mouse, but the term is also used to refer to the act of pressing one of the buttons. A drag (or sometimes click and drag) refers to the user positioning the cursor with the mouse, pressing one of the buttons, moving the mouse to a new position with the button still pressed, and releasing the button at the new position. The effect of mouse button presses, mouse button release, click and drag, can be modified by holding down a keyboard key or a different mouse button (if any).

For example, by placing the cursor at a particular position in a word processor image, you can operate to insert typed text at that position. By dragging the cursor over a part of the displayed text, the text (shown on the screen as highlighted) can be selected so that the user can apply an operation (such as deleting, moving, or changing the font) on the selected text. through some other mechanism. Depending on the application and the desired operation, the mechanism may be selecting the operation from a menu or entering a command from the keyboard.

Similarly, in a drawing program, the cursor can be placed in a mode by clicking on a tool icon (for example, Rectangle tool, Line tool, Polygon tool) so that subsequent clicks and subsequent dragging with the cursor results in the creation of graphic objects. Clicking on an existing object with a single cursor can result in selecting the objects so that an operation can be applied through some other mechanism. In a drag started with the cursor over an object, the result of the drag may be the cause of the object moving along with the cursor, or it may be the cause of the object changing size, depending on the position of the cursor on the object.

To be more productive for users, these tools should be provided that are relatively easy to learn, easy to use, and powerful. These goals are sometimes easy to achieve individually, but rarely together. However, considerable efforts have been made in attempts to design user interfaces that are more intuitive, efficient, and versatile. The example discussed later, taken from the field of drawing programs, shows the direction in which some of these efforts have been directed, and how improving one aspect of one user interface can downgrade another.

A common setting for drawing programs has a fixed tool palette on one side of the drawing area and a menu bar above the drawing area. To change tools, the user moves the cursor to the palette, clicks the mouse on the icon for the desired tool and moves the cursor back to the appropriate position in the drawing area. To perform a desired operation on a desired object, the user moves the cursor to the object, clicks with the mouse on the object to select the object, moves the cursor to the menu bar, clicks with the mouse button to display the menu, drag to the desired menu item and release the mouse button. The user then moves the cursor to the drawing area, to another item in the menu bar, or to the tool palette. AND^and This is one of many mouse movements

ES 2 153 407 T3 for even the simplest actions.

Drop-down menus and movable tool palettes allow the user to position the number of permanently open menus and the tool palette near the area where the drawing is being actively performed, and thus reduce the length of the mouse movement. Drop-down menus and movable palettes have made drawing more efficient in the sense of reducing the distances the user has to move the cursor, but they have made it less efficient in others. AND^orThese tend to take up a lot of the drawing area, especially near where the user is drawing. This can result in the user having to constantly interrupt the drawing task to move menus and palettes out of their way. This difficulty is made more complex by the fact that the programs are becoming more powerful (with greater functionality), the menus have grown more and occupy more space. Unfortunately, this example of tradeoffs found in trying to achieve the above goals was far from an exception.

It is an object of the invention to provide a method and a computer system that implement an improved user interface technique to allow a user to perform common tasks in a more efficient manner, that is, with fewer actions.

This objective was solved by means of the method claimed in independent claim 1 and by means of the computer system as claimed in independent claim 8. Preferred embodiments of the invention are the subject of the dependent claims.

The present invention provides a user interface technique that enables a user to perform many common tasks with fewer actions, therefore significantly improving productivity. The technique makes use of actions with which the user tends to become familiar, and therefore can learn more quickly. The invention can be made in the context of a simple program, or it can be embedded within an operating system so that it is available through different programs, including the operating system.

The invention works in the environment of a machine controlled by a processor for the execution of a program that operates on a set of underlying data and displays a visible representation of the same. The system also provides a description of a set of controllers, such as palettes, property palettes, menus, switches, dialog boxes, and cursors. Controllers will be referred to collectively as tools. The user normally specifies operations to the program by interacting with the tools and the visible representation (using a set of input devices), and visualizes the results of these operations on a display device. The user normally positions a cursor at the desired position and performs an action, such as a mouse click, with the cursor.

In short, the invention is characterized by what are known as click tools that can be combined with other tools (including other click tools) to facilitate composite tools. The visual description of a click tool includes a generally transparent defined active area that can be moved so that it can be positioned over a desired portion of the visible representation. When the user interacts with the visible representation across the active area, the action takes an attribute from the particular click tool. In one embodiment of the invention, which works in the context of a drawing program, in which the visible representation is a representation of a set of graphic objects, the click tools can include tools for creating objects, and tools for copy, modify and delete existing objects.

Once a click tool has been positioned relative to the visible representation, the result of an action through the tool will generally depend on the particular position within the active area in which the action occurs. In other words, the result of an action by means of a click tool will generally depend on the part of the visible representation that is under the cursor when the action is performed. While a conventional tool also has an active area, and the result of a tool action depends on the tool, the result does not normally depend on the position within the active area in which the action occurs or the location of the tool with relation to the visible representation. Conventional tools are not normally transparent, although there is no fundamental reason why they cannot be.

The act of combining a click tool with another tool was referred to as a composition. A click tool can be superimposed on another click tool, whereby an action taken through the two click tools on the visible representation will take on the attributes of both tools. If desired, the two tools can be grouped together to move as a single unit. You can also superimpose a push tool on a conventional tool, and take an action through the push tool on the conventional tool. Again, this action takes the attributes of both tools. In the case where the conventional tool is a modal tool, subsequent actions with the modal tool will carry the attributes of the click tool. In addition, a conventional modal tool can be called, and subsequent actions taken through the click tool, in which case the use of the conventional tool takes on the attribute of the click tool. In the case where a click tool overlaps so4

ES 2 153 407 T3 For only a part of another tool, the user has the option of clicking on the overlap area to get the composition action, or click on a non-overlap area to get the action of the tool only not composed.

In a specific embodiment, each click tool issues commands and is responsible for combining its action with the action that the event has accumulated at the time the tool receives the event. The new action can be evaluated from the existing action by one of a variety of methods, including adding the action order list to the end of the existing order list to create a composition action by adding the list. of orders of the action at the beginning of the existing list, removing one or more orders from the list, changing the names of the orders used in some or all of the actions received, modifying the values ​​specified after the order name of an order or modifying the <x, y> coordinates specified in the order.

In some embodiments, groups of tools were moved together under the control of the user, and thus could be thought of as lying on a transparent overlay that can be moved relative to the visible representation. The transparent overlay is preferably subjected to positioning by means of an input device controlled by the non-dominant hand of the user (eg, by the left hand of a user who is right-handed).

A more in-depth understanding of the nature and advantages of the present invention can be made by referring to the remaining parts of the specification and to the drawings.

Figure 1 is a block diagram of a computer system containing the present invention;

Figure 2 shows how the underlying data for the program and for the transparent overlay are converted to a graphical representation;

Figure 3 shows how the input signals for the transparent overlay are converted into procedure calls;

Figures 4-32 depict single views or sequences of views showing the operation of various tools including transparent tools activated by mouse click;

Figure 4 shows a shape creation tool;

Figure 5 shows a particular use of a shape creation tool;

Figure 6 shows a tool for erasing, moving and copying;

Figure 7 shows a color palette tool;

Figure 8 shows a style palette tool of type;

Figure 9 shows a symmetry clipboard tool;

Figure 10 shows a tool for transferring object attributes;

Figure 11 shows a tool for transferring graphic shapes;

Figure 12 shows a vortex selection tool;

Figure 13 shows an attribute detail tool;

Figure 14 shows a color and shape creation detail tool;

Figure 15 shows an alignment line tool;

Figure 16 shows a shape placement tool to be applied to objects;

Figure 17 shows a rotation tool;

Figure 18 shows a rotation, sizing and skewing tool;

Figure 19 shows a tool for activating alignment objects;

Figure 20 shows a grating tool;

Figure 21 shows a use grid; Figure 22 shows a geometrical measurement tool;

Figure 23 shows a text format development tool;

Figure 24 shows a detail interpretation tool;

Figure 25 shows a control management tool;

Figure 26 shows a debugging tool;

Figure 27 shows a keypad tool;

Figure 28 shows a text creation and text rotation tool;

Figure 29 shows a figure labeling tool;

Figure 30 shows a tool for loading documents into windows;

Figure 31 shows a tool with handles for moving, copying and deleting the tool;

Figure 32 shows how tools can be composed to create new tools;

Figure 33 is a flow chart of the user login routine for a particular implementation;

Figures 34A-34C show a hierarchy of applications, a display of these and the order of event delivery for these;

Figure 35 is a flow chart of the routine for the Event to the Application for the particular implementation;

Figure 36 shows is a flowchart of the transparent Overlay Event to Sheet routine for the particular implementation;

Figure 37 is a flow chart of the Event to Tool routine for the particular implementation;

Figures 38A and 38B show a command including a request for data;

Figure 39 shows the event delivery order for a redraw of the screen; Figure 40 shows the part of the hierarchy for redrawing.

The detailed description given below is organized as follows. Section 1 provides an overview of the system and provides a structural and project description.

ES 2 153 407 T3 high-level yielding of the transparent overlay of the present invention including the pulsing tools and two-hand operation. Section 2 describes several examples of the types of tools that are possible, with an emphasis on transparent tools activated by mouse click. Section 3 describes some strategies for organizing, modifying and creating tools on a sheet or sheets for the transparent overlay sheet and some general techniques for using the transparent overlay sheet working in the margin of different types of tools. Section 4 describes a current implementation of the transparent overlay. Section 5 describes some of the advantages of the transparent overlay over existing techniques. Section 6 ends the description. Section 7 provides a list of items mentioned in the specification.

1.0 System overview

Figure 1 is a block diagram of a computer system 10 that implements the present invention. In accordance with known practice, the computer system includes a processor 12 that communicates with various peripheral devices through a bus subsystem 15. These peripheral devices generally include a storage facility that includes a memory 17 and a file storage system 20, various input devices, and a display device 22 having an active display area 23. The file storage system stores the program and data files, and generally includes such standard devices as hard disk drives and floppy drives, and possibly other devices such as CD-ROM drives and optical drives.

In this context, the term "bus system" is used generically to include any mechanism to allow the various components of the system to communicate with each other as intended. With the exception of the input and display devices, the other components do not need to be in the same physical position. In this way, for example, the parts of the file storage system could be connected via various network media at great distances. Similarly, the input and display devices need not be in the same position as the processor, although it is anticipated that the present invention will be implemented in most cases in the context of PCs and workstations.

Input devices are for most standard parts, including a keyboard 25 and one or more pointing devices. A mouse 27 and a trackball 30 are shown, but other devices such as touch screens, graphic boards or electronic pointers could be used. While there may be cases of conventional systems where there is more than one pointer system, the normal situation is that the user uses only one of these devices at a time. The present invention derives significant advantages by providing the user with two such devices, one for each hand, for simultaneous or alternative use. For concretion purposes, the mouse 27 is shown having three buttons 32, while the trackball 30 is shown having one ball 35, three buttons 37, and a rotary knob 40.

The invention can be described from the point of view of a high-level user with reference to the illustrated contents of the display area 23. The screen shows a first application window 50 for a drawing program and a second application window 52 for a word processor. The drawing window is shown as having three graphic objects, a rectangle, an ellipse, and a pentagon; the word processor window is displayed containing text. An arrow-shaped cursor 55 whose position is controlled by mouse 27, is shown located on the contour of the rectangle in the drawing window, as could be the case when the user selects the rectangle in preparation to perform a specific operation on he. This is a representative situation, for example one that might occur in the case where a user is drafting a patent specification and is creating the patent drawings. Depending on the computer and the task at hand, there could be a single window occupying the total viewing area, or many windows, possibly with some kind of overlap between them.

The computer environment and the contents of the display area, as described above, are standardized. The present invention adds another aspect to the environment, a movable transparent overlay having several defined regions 60. The defined regions are shown as a plurality of adjoining rectangles in a multi-element grid, but as will be discussed later, the defined regions need not adjoin. with each other. Furthermore, there is no need for a plurality of them to be visible on the screen at the same time. As an aid in distinguishing the defined regions on the transparent overlay from the remaining elements in the viewing area, the elements on the transparent overlay are shown in a solid line and the application windows and graphic objects are shown in dashed lines. As described below, the transparent overlay preferably carries indications (such as icons or text) that specify the meaning of the particular defined regions. Therefore, while the transparent overlay is referred to as being transparent, it should be recognized that the need to delimit regions on the transparent overlay means that the transparent overlay may have some opaque or semi-transparent parts.

If a given defined region is positioned on a part of the screen, and an action is taken in that region, the action takes an attribute from the particular defined region. In this way, each defined region can be considered as the active region (or the active area) of a tool that can be taken to a relevant part of the visualization area and applied to that part. Given the nature of the way they are applied di6

ES 2 153 407 T3 tools, tools will sometimes be referred to as transparent tools activated by clicking the mouse. While many of the descriptions that follow treat the transparent overlay as a single transparent sheet, the transparent overlay may comprise what appears to be a plurality of relatively movable transparent sheets, each having a number of semi-transparent tools on it.

Making the tools appear on the top of the objects to which they are to be applied seems the most intuitive approach, that approach is the one that will generally be assumed. However, there may be certain special circumstances that warrant the reverse stacking order. For example, there may be certain applications where it is critical that none of the application objects are obscured, even by markings on the transparent overlay. This can be accommodated by making the application appear transparent and having the transparent overlay appearing behind the application. As described in a later section, the transparent overlay structure operation of the transparent overlay program will tend to be the same in any case. Therefore, the term "transparent overlay" will be used to refer to the collection of sheets containing tools, whether they appear above or below other items in the display area. In some cases, it may be desirable to allow the user to switch from one stacking order to the other.

Although there are many ways for the user to position the transparent overlay relative to the viewing area, it is preferred that this be done with the user's non-dominant hand using a trackball 30. Rectilinear positioning can be accomplished by rotation. of the ball 35 while other operations can be carried out with the buttons 37. Resizing of the transparent overlay sheet and its contents can be accomplished by rotating the rotary knob 40.

The mouse click-activated transparent tools and the transparent overlay sheet represent elements of a new user interface, but can also be used in conjunction with standard interface elements. By way of example, a stylized palette tool 62 of the type used in many prior art programs is shown. Depending on the program and the OS, the tools and attribute palettes for a given program can be fixed quickly in that program window, or they can appear as a separate window that can be moved relative to other windows of the program. While the detailed description of the transparent overlay tool examples in the next section is largely consistent with the rattan pulse activated transparent tools, conventional tools such as those on the paddle 62 can be incorporated onto the overlay sheet. transparent and moved together with other tools on the transparent overlay sheet. Paddle 62 is shown solid, implying that it is on top of the transparent overlay. Conventional tools can share one or more transparent overlay sheets with mouse-activated transparent tools, or they can be separated on separate transparent overlay sheets.

Figure 2 is a flow chart showing how the various data items stored in memory 17 or in a file storage system 20 are processed so that they appear in the display area 23. The underlying data of the designated program 70, they are usually stored in each native program format, which is a feature of the program, and is presumably optimized for that program operation. The data is subjected to processing by an interpreter 72, which converts the data to an image data structure 73 that specifies what is to appear on the screen. There are several possible formats; for example, the image data structure 73 may be a bitmap or a set of commands in a language such as Display Postscript or Quickdraw. Regardless of the details, the image data structure must contain sufficient information that can be converted to pixels (if not already a bitmap) at the screen resolution or otherwise processed for display on the screen. .

The transparent overlay is characterized by a similar hierarchy in which the underlying data of the transparent overlay, designated 75, is processed by means of an interpreter 77, which converts the data to a transparent overlay image data structure 80 The two image data structures are combined into what is shown schematically as a summation node 82, and are converted to the final display image, designated 83. The particular technique for combining data image data structures should be to ensure that the transparent overlay sheet appears as a transparent sheet with opaque or partially transparent indications. As will be discussed later, the transparent overlay can include what are known as visual filters, or it can include tools that incorporate visual filters. In such embodiments, the summing node may also act to distort or filter parts of the displayed image. The specifics of the configuration and appearance of the transparent overlay will be described later along with a description of the various tool embodiments.

The display interface of the present invention requires relative positioning of three conceptual user interface transparent overlay sheets: a cursor transparent overlay sheet, a transparent overlay sheet of transparent overlay, and a transparent overlay sheet of

ES 2 153 407 T3 application. The transparent cursor overlay sheet, at a minimum, is defined by a distinguished pointer in the plane (a cursor position), along with one or more visible objects that move in a rigid manner with that distinguished pointer. The Transparent Overlay Sheet Clear Overlay Sheet, as a monym, includes a set of tools that move together in a coordinated manner. The transparent application overlay includes one or more programs with visible renderings. Each of these transparent overlapping sheets may in turn consist of sub-sheets. For example, the cursor can carry a drag and drop object with oil; The transparent overlay tools can be made by layering more simple tools on top of each other, and the applications can overlap as in an overlap window system.

Figure 3 is a flow chart showing the relationship between these three transparent overlay sheets and the communication between the transparent overlay, designated 85, and application programs 87 and 88 (also referred to as applications # 1 and 2). The communication between the transparent overlay and the applications is the same, regardless of whether the tools on the transparent overlay appear above or below the visible representation.

When activated, the tools in the transparent overlay deliver commands, which can include arbitrary data structures, to applications. An application can respond to the command by changing its own data structures, and it can also respond by returning data to the transparent overlay. In addition, if whenever it is requested to paint itself, the application responds to the transparent overlay sheet by providing information about its visual representation (current appearance on the screen). The transparent overlay can modify this visual representation (eg, using visual filters) before presenting it to the user.

The figure also shows a reverse path from the application to the transparent overlay, as the applications return data to the transparent overlay in response to certain commands. Although the specific example discussed later deals with the operation of the mouse-activated transparent tool, the basic description applies to conventional tools that are on the transparent overlay.

The transparent overlay software works in conjunction with a window manager 92. The window manager, which can be part of the operating system, draws window frames and supervises the creation, movement, resizing and destruction of windows on the screen. The window manager takes raw input signals from input devices, routes the signals to the correct application (usually the one whose window is furthest forward under the cursor), and translates the position information into coordinates expressed in the coordinate system of the application. The window manager also provides information to the application as to where to draw the contents of the window.

The input signals to the transparent overlay can be raw input signals from the OS (for example, the mouse event coordinates) or they can be provided by means of a drag and drop object or by another application. Additionally, for those embodiments that allow overlapping of transparent overlay tools, the input signals may come from what can be seen as another transparent overlay. An additional set of input cues (not explicitly shown) to the transparent overlay includes the cues for positioning the transparent overlay relative to the visible representation.

In the embodiment shown, the input signals are translated into universal language commands, which are directed to a translator for the appropriate application. In the event that the input signal had position information that caused the orders to be routed to application # 1, the orders are found with a translator 93 that converts any of them to orders in the input language of the application. # 1, and some of them directly to calls about the procedures of application # 1. The commands in the input language of the application are directed to a parser 95, which converts them into procedure calls. The drawing also shows the case in which the input signal belongs to application # 2, in which case the universal language commands are directed to a translator 97, and possibly later to a parser 98 to generate calls to the procedures of the application. application # 2.

The following is an example of the conversion of a command found in universal language to commands in either of the two application languages. Consider the example of a painting program that works on bitmaps and a drawing program that works on vector objects. Consider also the case where the transparent overlay tool under the cursor specifies a change from a feature color to red. Topically, an order included an operator, a set of position information, and possibly one or more parameters. Examples of commands that are in the universal language might include the following:

SetColor <x, y> red,

SelectCorner <x, y>, y

Scale <x, y> <x ', y'> 20.

Consider the SetColor command. For the painting program, which works on pixels, the position of the cursor provides all the position information necessary to determine the required action, and all that is needed is a single command. This unique command relevant to the language of the paint program could be the following:

SetColorPixel <x, y> red.

For the drawing program, it would first be necessary to determine, based on the position of the cursor, which object is the one to be used.

ES 2 153 407 T3 lessons, and then apply the color to that object. The relevant sequence of order in the language of the drawing program could be as follows:

SelectObjetc <x, y>

SetColorSelectedShape red.

If the operation has been carried out with a conventional tool on the transparent overlay to set the selected object to red, the order sequence would be the same, but it had to be carried out in two stages, first when the user had selected the selected object. the conventional way, and then when the user clicks on the red button in the conventional color palette.

A variation of this configuration would be to have the translators of the transparent overlay sheet and the application tightly coupled to each other to avoid the conversion of input signals into commands of a universal language. Besides this, the transparent overlay would have to hold information about what applications it supports, and would translate input signals directly into the appropriate input language of the application.

2.0 Summary of Transparent Overlay Tool Examples

For the transparent overlay to be useful, it must contain a set of tools that help the user to use the application. Several of these tools are described below. Some of them are novel for reasons of their capacity. Others are novel only in the context of the transparent overlay. Most of the tasks performed by the non-dominant hand can also be performed by the dominant hand, at the cost of having the dominant hand interrupting its own tasks. There are, however, certain tasks that require the use of both hands. Many of the tools I describe below are clickable tools. As mentioned before, the term refers to the fact that the tool is applied by clicking on the tool on a visible part of the representation.

These tools have several interesting properties including the following. They often allow several interaction steps to be merged into one. The user's eyes never need to leave the work area. The interface is straightforward, visual and, with carefully chosen tools, easy to learn. The user's non-dominant hand is responsible only for approximate positioning; precise positioning is done with the hand that handles the mouse. The examples are directed in the first place to an environment of a drawing program (graphic editor) with tools to create, modify and delete graphic objects in a scene.

The operation of most tools would be described together with a figure that includes a series of views, indicating the appearance of the drawing scene, and in some cases the appearance of the tool, at different stages of the operation. For some of the examples, a given operation using a tool of the present invention will be compared to the same operation using conventional drawing program tools and techniques. With the exception of figures 8, 13 and 23, the objects in the scene are drawn by dashed lines and the tools on the transparent overlay are drawn by means of continuous lines, complying with the convention adopted in figure 1.

No references to a specific type of program or publisher are intended to implicate self-contained application programs. In fact, many of the so-called drawing programs have very sophisticated text handling capabilities, and many of the so-called word processors have powerful drawing modules. The demarcation is further blurred by the built-in program packages (called “job” programs) that provide the functionality of many types of programs in a single program. Accordingly, a reference to a given type of program should be taken as a reference to a program that has the declared functionality, if it is marketed as a drawing program, as a word processor, as a database program. or like a caílculo leaf.

Several of the tools are described in conjunction with a graphical editor that supports a feature referred to as "quick drag". This refers to the gravity technique described in Bier and Stone's paper on fast trailing [* Bier 86]. Along with this technique, a special point that will be referred to as the "caret", pulls to active gravity positions, such as corners of objects, and other objects that are drawn, can touch the caret.

The terms "button", "menu" and "palette" are used in conjunction with several of the tools that will be described later. The terms are generally used in accordance with their known meaning, but some indication is sometimes necessary in view of the fact that the transparent overlay of the present invention imbues these other separate familiar devices with new properties. In general, a button refers to a defined area of ​​the display area, which when clicked, causes an operation to occur. Some of the buttons used on the transparent overlay allow the user to control the particular result by clicking a particular position on the button. In general, a menu (often preceded by the adjective "drop-down" or "sudden appearance") is a list of elements or properties that the user can select by clicking on the menu bar or on the menu icon and dragging to the desired item. The term palette refers to a visible collection of buttons in which one or more of them can be selected with a click.

A drop-down menu actually replaces a palette for a drop-down or pop-up menu. In this way, the menu selection involves the simple step of selecting the menu item, rather than the compound step of selecting the menu item from the menu bar, and then selecting the item from the menu item. The

ES 2 153 407 T3 term “pallet menu” is used later to denote a pallet or a drop-down menu that can be moved with the non-dominant hand, and in this way it can be brought into the work area and moved later without distract the user from the main task at hand.

Some of the specific tools described below make use of what is known as a visual filter, filter, or lens. Each filter is a region of the screen, called a region of view, along with an operator, such as operations that increase, interpret in a wiring structure style or reveal the equation hidden in a cell of a spreadsheet, performed on the forms seen in that region. These filters generalize many different representations to pixels and many operations other than magnification. To produce their visible output, these filters can make use of the original application data structures from which the current visual representation is produced. In this way, these filters can represent application data structures in a substantially different format, highlighting information that was previously more difficult to see, suppressing information that is not of importance at the current time, or even revealing information about it. parts of the data structures that were not previously displayed. Such visual filters work in concert with the transparent overlay tools, in particular with the tools that perform operations relevant to the parts of the data structures that are to be revealed or highlighted by means of the visual filters.

Visual filters can produce not only modified views of application data structures, but also temporary transparent overlay tools, positioned relative to particular application shapes. The user can use these temporary tools in the same way as other tools on the transparent overlay; for example, these tools may include buttons that the user can click on, click through, or drag to produce commands to be delivered to the underlying application.

When multiple filters are composed, the effect is as if the model were sequentially passing through the filter stack from bottom to top, with each filter acting on the model as it passes through it. Also, when a filter has other filters below it, you can modify how the limits of these other filters are assigned on the screen within your own limit.

2.01 Place and shape objects within the scene

Figure 4 shows how the action of adding a new shape to a graphic scene is performed by using a shape palette that is on the transparent overlay sheet. The user has roughly positioned a circle on the tool near a rectangle that is in the scene. When the user presses and holds down the mouse button, a new circle of that size is created in the scene, the transparent overlay disappears, and the circle associates its center (for example) with the cursor arrow through fine positioning. Using a gravity technique such as snap drag [* Bier86] the new circle can be positioned so that its center falls exactly on the corner of the rectangle. When the user releases the button, the new shape is in its final position, and the tool reappears. If the user had placed a shape with several corners, such as a triangle, the corner closest to the cursor when the mouse button was pressed, would have been the point that would have been associated with the cursor.

In the example above, the size of the object found in the menu determined its size when it was applied to the application. In many situations, such as selecting from lines, rectangles, circles and other shapes, one wants to select the generic shape and then specify its size and position. The transparent overlay sheet enables a novel technique that has the advantage of the ability to use both hands to make selection, positioning and scaling tasks in a fluid and natural way.

Figure 5 shows a variation in the creation of the rectangle that allows the four edges of the rectangle to be positioned at the same time. Initially, the non-dominant hand has positioned a rectangle on the tool over a rectangle in the scene. The user clicks on the rectangle tool with the mouse cursor and presses a mouse button to create a rectangle of that initial size and at that position on the scene. The tool disappears. The corner of the rectangle closest to the mouse cursor is quickly pasted to that cursor. A new cursor appears at the opposite corner of the rectangle; the position of this new cursor is controlled by the non-dominant hand. Both corners of the rectangle can be positioned simultaneously and quickly locked in place using quick drag. When the mouse button is released, the rectangle is placed and the tool reappears. This two-handed creation technique can be used to position other shapes, including both ends of straight line segments, the center point and the point of the circumference of a circle (allowing the circle to be translated and resized simultaneously ), two corners of a triangle (allowing the triangle to be translated, rotated and resized simultaneously).

2.02 Mouse-activated transparent tool buttons

In most interfaces, the text that describes the operation that a button performs is placed within the active region of the same button. However, on the transparent overlay, it is often preferable to have a transparent active region, with text, an icon or other visible indication indicating the operation near the active region. This allows the user to apply an operation to the objects that are visible within the button. Each active region is known as a transparent active tool button.

ES 2 153 407 T3 clicks on the mouse. The transparent tool buttons activated by clicking the mouse can also be used to grab properties of objects.

Figure 6 shows transparent tool buttons activated by clicking the mouse for delete, move and copy operations, and the sequence of operations for deleting an object (the ellipse) from the scene. The user positions the transparent overlay so that the delete button is over a group of objects, while pointing to one of those objects with the cursor. In certain implementations, while the mouse button is pressed, the system highlights the object that will be acted upon if the button is released at that moment. When the user releases the mouse button, the selected object is deleted. When several objects intersect with the delete button, only the object that the user indicates with the mouse cursor will be the one that is actually deleted. This allows a precise specification of the operands. Also, mouse-activated transparent tool buttons allow the user to select an operation and operand with a single movement of the hands. If the user had wanted to perform a different operation, it would have been possible to use a transparent tool button activated by a different mouse click.

Figure 7 shows an array of mouse-activated transparent tool buttons used as a color palette, and the sequence of operations to change the color of an object in the scene (the pentagon). In this case, each button is a rectangle with a triangular region in the upper right corner that designates the color (different colors are denoted by different creation patterns). The user positions the part of the color palette that has the desired color on the pentagon and clicks on it with the mouse. Although the ellipse is also on the chosen button, only the pentagon, which the user indicates with the mouse cursor, has changed its color. (If the user makes a transparent click on a region devoid of objects, the program may ignore the action, or it may interpret the action as wanting to set a default value). In a conventional drawing program, the user will move the cursor to the object whose color he wishes to change (possibly after having first performed an action such as moving the cursor to the tool palette to obtain the selection of the tool), he would have clicked on the object to select it, and will move the cursor to a color palette or menu to select the desired color.

The buttons of the color palette are shown adjacent to each other, but these can be separated from each other as in the case of the Delete, Move and Copy buttons. Also, the colored regions above the buttons are shown as opaque, but may also be transparent. In case the colored regions are transparent, they can cover the entire button area. A similar array of mouse-activated transparent tool buttons may be provided to change the contour line of an object. The color for a given button may be denoted as before, but with the color only applied to the perimeter of a triangular region, or by applying the color to the entire perimeter of the button.

Figure 8 shows an array of mouse-activated transparent tool buttons used as a property palette to set the style of text in a document. Each style (normal, bold, etc.) has an active area on the tool. In this particular example, the text describing the function on the button is located in the active area. Selecting the text displayed in this area changes its style. In the example shown, the user has selected text that is inside the "bold" button, with the result that the selected text is converted to bold text. The particular event that the program recognizes that the text selection is not important until the event begins on the desired button. If the mechanism for selecting text is to drag the cursor over the text to be selected, the user will position the starting point for the selection in the active region, press the mouse button, drag to complete the selection and release the button from the rattle. The fact that the cursor will also be out of the active region when the mouse button is released is not important.

2.03 Clipboard

Clipboard tools pick up shapes and properties from underlying objects, acting as visible instances of the copy and paste keys common in many applications. Clipboards can take entire objects or specific properties such as color, line pattern, or font. They can retain one or more copies of an object. The objects or the properties captured on the clipboard can be copied by clicking on them, as in the tools of the palette. In a sense, the object or attribute that is grabbed by a clipboard tool becomes part of the tool. This is however an example of the general feature, which will be discussed further in a later section, of allowing the user to customize the transparent overlay.

Figure 9 shows the operation of a symmetry clipboard that collects the shape on which the user clicks and produces all the turns of the shape by rotations in multiples of 90 °. By moving the clipboard and clicking on it again, the user places a translated copy of the resulting mirror shape. Clicking on the small box in the upper left corner of the clipboard clears the clipboard so that new shapes can be set.

Figure 10 shows a couple of tools that can both collect the graphical properties of one object and apply these properties to other objects. The particular sequence illustrated is transferring the color from the ellipse to the right angle. These tools can be thought of11

ES 2 153 407 T3 are used as rubbing tools since their use is reminiscent of sheets of paper and charcoal used to erase words and text from monuments. The user clicks on the ellipse through a rubbing sheet that is sensitive to the color of the objects' area. The color of the aereal is "lifted" from the drawing (or actually copied while the color of the circle is not changed) to become part of the rubbing sheet. Even part of the shape of the corcle is retained as a remainder of the part from which this color comes.

The user then picks up the rub and positions his circular projection, which acted as a proprietary applicator, on a rectangle. When the user clicks on the mouse, the rectangle takes on the color of the rubbing sheet. The second wipe sheet could be used to lift a second fill color from a different object, making it possible to store multiple colors for later application.

Figure 11 shows the operation of a tool that allows the user to copy a shape from an underlying object, and then transfer the selected parts of that shape back to the application. The user clicks on the object (in this case a curve) through the tool, at which point the curve or a copy of the curve becomes part of the tool. Later, when the user is drawing a new shape, the tool can be used in the manner of a French curve. Specifically, the user positions the tool near a part of the new shape, clicks two points on the curve to specify which part of the curve is to be added to the new shape (the specified part can become highlighted or change color), and the selected part is added to the nearest end of the new shape.

2.04 Mouse-activated transparent tool buttons with visual filters

In the mouse-activated transparent tool buttons shown above, the active area for each button was completely transparent, showing the objects under the button just as if the button was not there. However, for many applications it would be advantageous to display a view of the objects under the button that highlights the information needed to successfully perform the operation. This can be done using a visual filter, as described above.

For example, in Figure 12, the user is faced with several stacked rectangles and wishes to select the upper left corner of the central rectangle. This corner was hidden by the uppermost rectangle, so it is difficult to target the corner of the desired rectangle. However, a tool that has a vortex selection button displays a wireframe view (line drawing) of the drawing showing its corner, making selection easy.

The combination of visual filters with transparent overlay tools can be especially advantageous, in a conventional drawing program, to produce a wiring view of the drawing, the user would have to explicitly call a separate command. Once this command has been given, all objects will be drawn as wireframe drawings, not as objects to be acted upon. This could lose context, which is crucial to help the user identify the correct objects to act on. On the other hand, it calls a view operation, an edit order and an operand, all with a simple movement of the hands.

2.05 Combine clear overlay with details

The transparent overlay technique of the present invention can be combined with what is related to any existing user interface technique, and the combination can produce a tool with interesting properties.

One such interface technique is the use of what are known as details, which are generally one or more strokes with a pointing stick. A detail is topically characterized by one or more characteristic points (for example, the starting and ending points of the line path, the point of intersection of two lines).

For example, Figure 13 shows operations that use a tool that combines single stroke details and slices with the transparent overlay. The tool provides an active central area (scene region) surrounded by various attribute menu segments and a region to hold a prototype object.

Initially, the prototype object, which is part of the tool, is a rectangle with a dotted outline and a first fill color. The user positions the central area of ​​the tool on a triangle that was in the scene that has a continuous line outline and a second fill color. By tracing (dragging the cursor) from the triangle to the "fill color" menu region, the user indicates that the fill color of the triangle should be applied to the prototype object. At this point, the prototype object has been colored again. However, the sector menu can also be used in reverse to apply the properties of the prototype object to the objects in the scene. For example, when the user draws from the menu of the "dotted" region to the triangle, the dotted pattern of the prototype object is applied to the triangle.

Different menus of sectors could be built that allow not only individual properties, but also arbitrary groups of properties, or even complete shapes, to be applied on the scene. For example, by tracing from the prototype object region to the scene region, you could either apply all the properties of the prototype object to the indicated object, or you could copy the same prototype object into the scene.

A menu of sectors that can be used

ES 2 153 407 T3 from the center to the outside or from the outside to the inside, it appears as a novel invention by itself. However, it wouldn't make much sense outside of the context of the transparent overlay. One-handed sector menus appear suddenly, centered on the beginning of a trace, once the trace has started [* Hopkins 91]. In this way, there is no easy way to trace from outside to inside. However, since the slice menu is on the transparent overlay, the menu appears before the stroke begins, and an inward stroke is possible. The idea of ​​tracing inside a button and outside a button is not limited to circular arrangements such as sector menus. Any style of button could potentially allow this capability.

Figure 14 shows one way that the transparent overlay can be combined with the Kurtenbach and Buxton [* Kurtenbach91] single stroke details to form a shape creation tool. While this figure shows a tool consisting of a color palette, a design like the one in figure 7 could be used as well. The user positions the cursor at the desired position in the drawing, and moves the button that has the desired color under the cursor. By starting a stroke on a particular color button, the user tells the system to create an object of that color. The direction of the stroke determines the shape that will be created; the length of the stroke, the size. Once the stroke has started, the transparent overlay disappears and a menu of sectors of shapes appears, reminding the user which directions correspond to each of the shapes. Once the stroke is complete, a new shape is added to the scene. As Kurtenbach and Buxton describe, there are many ways to accelerate this interaction. For example, if the user traces quickly, the sector menu does not need to appear.

Note that this tool allows a user to specify the position, color, size and shape of a new object with a single short movement of the hands. In a conventional drawing program, the user will first move the cursor to the tool palette to select the rectangle tool, move the cursor back to the desired starting position, draw the rectangle at the desired position and size, and I would move the cursor to a color palette or menu to select the desired color. The conventional color palette may resemble the mouse-activated transparent tool button matrix of Figure 7 or Figure 14, although the conventional palette will not normally be transparent.

2.06 Quickly pasting the tools from the transparent overlay to the scene

In the previous examples, the movement of the transparent overlay on the scene was independent of the content of the scene. However, the present invention also provides useful tools that automatically reposition themselves (or the entire sheet of the transparent overlay sheet) to align with one or more objects in the scene. The examples show that the non-dominant hand can be used to quickly paste objects into points in the scene while the dominant hand is free to perform (or be ready to perform) other interactive operations.

For example, figure 15 shows a tool used to create alignment lines for ruler and compass style construction, such as the one used in quick drag [* Bier86]. The tool has several alignment lines that pass through a common point (the center of a circle) with different angles. The tool also has a small region to display an active angle. When one of the alignment lines (for example, line a45^°) passes near a software cursor (for example, the quick-drag caret shown in the figure), that line is quickly pasted to the sign and lengthened, and the tool displays the slope of the selected alignment line. The user can then freeze this line in position (for example, by clicking on a trackball button, or by clicking on the circle in the middle of the tool with the mouse). Finally, the user can perform new operations that quickly paste the caret with the alignment line. The figure shows how to draw a new line segment using the alignment line as a guide.

The quick paste technique can be used for other alignment objects that are behind lines. Figure 16 shows a palette of alignment circles positioned on an illustration in which the caret has been placed in the lower left corner of a rectangle. When the center of one of these circles (for example, the small circle inside the large circle on the right) passes near the sign, the circle would stand out by changing to a different color and the center of the circle would quickly stick to the sign. the tip of the sign. In this example, the entire palette is quickly pasted, but it is also possible to have only the circle only temporarily away from the other circles in the palette.

The user could also quickly paste a tool to an arbitrary scene point rather than a special point like the tip of the caret. For example, the tool of Figure 17 is an anchor object, used in rapid dragging as a center of dimensioning or a center of rotation. Here, the user moves the transparent overlay sheet until the anchor is near one corner of the rectangle. Without having to move the rest of the transparent overlay, the anchor moves to quickly stick to the corner of the rectangle. The user freezes the transparent overlay sheet, and with the dominant hand, rotates the rectangle around the anchor.

Figure 18 shows a tool that generalizes the rotation tool of figure 17 to perform any rotation, dimensioning and

ES 2 153 407 T3 skewed. This tool allows the placement of an anchor position, the selection of an interactive operation and the performance of that operation with a single movement of two hands. Specifically, a menu of sectors of operations is located around the same anchor. Once more, it is assumed that the user has made the anchor quickly stick to the corner of the rectangle. The user starts an operation by clicking on the mouse button with the cursor located on the name of an operation, say the rotation operation. Once the mouse button is pressed, the system rotates the object as described in Bier and Stone's paper on fast dragging [* Bier86]. In particular, the angle that the object is rotated from its original position remains the same as the angle through which the caret has been moved, using the anchor as a center of rotation. During interactive operation (rotation in this case), the transparent overlay preferably disappears.

The above tools that quickly stick to objects in the scene are also examples of using the transparent overlay to provide the user with virtual drawing tools, such as a virtual ruler, virtual compass, or virtual protractor. As with the corresponding physical tools, the non-dominant hand can control the position and orientation of the tool, while the dominant hand is used to create objects or lines. Like physical tools, objects or edges that are created in the proximity of the tool are affected by this constraints. One benefit of using the transparent overlay for this application is that the constraints can get in and out of the effect quickly and easily, much the same as in the case of palette knives. 2.07 Combination of on / off buttons with palette menus

Many systems have modes that can be turned on or off by pressing a button that toggles their status. Said buttons may be located on a palette menu as well, making it possible to reduce cursor movement for these buttons by positioning them close to the cursor with the non-dominant hand, and making it unnecessary for the user's gaze to move from the work area. Also, as the palette menus can be large, (they do not need to be adjusted on the screen all at once, being scrolling up and down), larger and more expressive representations of the on and off buttons can be used. .

For example, Figure 19 shows a set of on and off buttons that are displayed as lines with different angles, in which the user can activate a class of snap-drag alignment objects. A rectangle in the scene has its four corners and its center point identified as hot spots. When the user clicks on a given alignment line (for example, the vertical line), the line is highlighted, and all objects with hot spots trigger the appearance of full-length alignment lines of this class, nominally line lines. vertical alignment. The tool is shown also providing a numerical indication (in a box at the bottom right) of the angle of the selected alignment line. If the user had selected the alignment lines with me from an angle, the numerical indication would not be displayed. 2.08 Guide lines and grids as secondary image

Section 2.04 has described the combination of the transparent overlay with visual filters. In those examples, the visual filter presented objects in the scene in different ways to facilitate operations on those objects in the scene. However, visual filters can also be used to display objects that are not normally displayed and to make it possible to interact with these objects. For example, a visual filter can be used to display guide lines or grids locally.

Figure 20 shows three tools, each one displaying different kinds of grating. The first two grids on the left are rectangular with different spacing. The last grid is a hexagonal grid. Although each grid only appears when the visual filter is in place, the grid coordinates are bounded to the scene, so the grid points do not move when the transparent overlay is moved. In this way, by clicking on the grid points and moving the transparent overlay, the user can edit the scene using these grids. The user has started a line on a point on the grid, moved the transparent overlay up and finished the line using the grid. In conventional programs, the effect of activating a grid becomes apparent only once the grid is activated. With these tools, however, the user can see what kind of grid would facilitate any given tool before use. All visual filter tools have this property to some extent.

Figure 21 shows how a user can make and use a custom grid tool. The custom grid tool collects a set of selected shapes and allows the user to use them as a custom grid. As with other grids, this grid can be used precisely to place objects. This tool makes use of the transparent overlay properties discussed earlier in section 2.03 on clipboard. The lines in the scene are a grid that the user has created (for example, using an editor) to specify a template for the two-column layout. The user lifts this grid over the custom grid tool, upon which the grid lines become part of the tool. However, as with the grid tools discussed above, the grid lines maintain their position even if the tool is moved, so that they remain reliable reference points. As a result, the grid lines only appear14

ES 2 153 407 T3 cen when the grating tool is present. The figure shows how the user is beginning to stretch a rectangle, and finally gluing the rectangle in place at the bottom of the left column.

One possible extension is to allow any object in the scene to be lifted onto the transparent overlay from the application. This object will then become gravity active so that objects in the scene will stick to it, allowing users to create their own custom guide lines and curves. 2.09 Measurement and definition tools

Certain of the tools described above extract graphical properties from objects. Figure 22 shows the use of a click-activated button tool that measures geometrical properties, namely coordinates, lengths, slopes, and angles. When the user clicks on a corner of an object using this tool, the coordinates of this corner are reported. If the user clicks again, the system reports the length and slope from the first point to the second point. If the user clicks for the third time, the system reports the angle formed by the last three points that were clicked. The tools that display information based on what the mouse is pointing at each moment are also useful for word processors; for example, a tool can display the definition of a word that has been selected by means of the tool.

2.10 Aiming by means of the non-dominant hand

While most tools use the non-dominant hand to position the clear overlay and use the dominant hand to aim, it also makes sense to use the non-dominant hand to aim if the objects to be pointed are large. For example, in a text document, paragraphs are generally large enough to be pointed with the non-dominant hand. Figure 23 shows a tool that reveals the hidden structure of the paragraph under the arrowhead, moving with the transparent overlay. This tool always displays the name of the print format that will be used to format this paragraph for a printer. In this example, the arrow points to a paragraph whose format is called “Body”.

2.11 Detail interpretation tools

One particularly exciting application of the transparent overlay is to provide the user with full local input interpreters. For example, a drawing program might normally interpret mouse input as a set of commands, such as select, move, create line, and so on (for example, see Rubine's work on editing graphics with details Rubine91] or Goldberg and Goodisman's work on detailed text editing [* Goldberg91]). By positioning a tool for interpreting details on the scene, the user will be able to interact with the same editor in a different way. For example, in Figure 24, the user draws an "X" on a detail interpretation tool. If "X" means "delete" for this tool, the object below the X will be deleted.

Said details interpreter can be used in various applications. For example, it provides a way to place a details interface on top of a word processor that is based on a mouse. In a painting program, the movements of the mouse or the pencil will be able to put the painting when the detail interpretation tool is absent, but they will carry out the editing orders when it is present. In addition, the interface provided by a detail interpretation tool may be common between applications. For example, if the transparent overlay is provided by the windowing system, the same detail interpreter could be moved from a drawing program to a word processor, allowing the user to use a given detail in multiple contexts. For example, the "X" in figure 24 could be used to erase shapes in a drawing program, paragraphs in a word processor, or a region in a painting program.

2.12 Combination of local command interpreters and visual filters

The idea of ​​having a local command interpreter can be combined with visual filters. For example, many drawing programs display small user interface objects, called grips, on top of objects in the scene. By pointing at these grips with the cursor and pressing the mouse button, users can perform translation, sizing, and stretching operations on objects. This idea can be combined with the transparent overlay blade to provide a variety of different kinds of grips. For example, in Figure 25, the user has selected two objects out of three found in a drawing; selected objects are highlighted with small black squares. By positioning a transform grip tool, the user can view and point to any of the grips found in a set (small white squares). Clicking and dragging the center handle will translate the selected objects. Clicking and dragging any of the other grips will stretch the selected objects.

The usefulness of visual filters that add temporary tools, positioned relative to the application objects, is particularly apparent when several of these filters are available on a single transparent overlay sheet. In this case, the user can alternately use one of the temporary tool sets and then another. For example, one of the tool sets may provide editing grips that allow for the translation and stretching described above. Another set of tools may facilitate rotation around the center of a shape or around any of the shapes.

ES 2 153 407 T3 its corners. If all of these tools are made available right away, the temporary tools will result in unacceptable clutter. Presented alternatively, however, these make it possible for the user to take advantage of a wide variety of tools whose attributes (including position, type, and number) are dependent on attributes (including position, type, and number). application objects and are thus particularly suitable for operating on them.

2.13 Reference and debugging tools

The tools can be used not only for the normal operation of an application, but also to allow a user to get help about that application or to allow a programmer to debug the application. An example is a tool such as style details or mouse actions performed within the limits in which that tool causes the tool to display information about the command that the user is trying to perform. For example, Figure 26 shows a tool that displays an image of the rat. When the user presses a mouse button while the cursor is in this tool, the mouse icon displayed on the tool shows which of the mouse buttons is being pressed. Such a tool will be useful, for example, when making video tapes of an interactive tool. A more sophisticated version of this tool will also display the name of the order that is being carried out and will offer to open the source code that implements that order.

2.14 Operations that use an intersection point

While the mouse-activated transparent tool buttons are a particularly interesting type of button made possible by the transparent overlay, even normal buttons are handy. For example, Figure 27 shows an array of buttons that act like a numeric keypad. This numeric keypad can be positioned near the area in which a user is working and be activated by means of a pencil or a cursor, making a keyboard unnecessary for some of the operations. Every time the mouse clicks on a digit, the transparent overlay moves to the right by the width of one character. This numeric keyboard can also be used as a calculator allowing the user to insert calculated numbers into a document in an easy way.

2.15 Rotary tools

Some of the above tools, such as the alignment line tools, may transfer slightly relative to the transparent overlay sheet. It is also possible to allow the tools to rotate and resize relative to the sheet of the transparent overlay sheet. For example, Figure 28 shows a tool for selecting a font and / or typing text. To produce text with an arbitrary angle, the user can rotate the tool. In this example, if the user clicks on two points of the scene through the measurement region (corners) of the tool, the tool orients itself with the slope of the line that is between the two points. measured. When the user subsequently selects a font from this tool, the new text is appended to this angle measured with the selected font. While the example shown here stays reoriented until another measurement is made, it is also possible to have the tool reoriented temporarily.

2.16 Combining graphical search, guide lines and object creation

As described above, clear overlay tools can combine multiple task steps in one motion with both hands. An extreme example of step reduction is the figure labeling tool shown in Figure 29. This tool combines the constraint-based graph search [* Kurlander92], the snap-drag alignment lines, the visual filters, and the push object through. This tool is used to put a label of a figure at a certain position in the boundary rectangle for all drawings. When this tool is moved over a region of the scene, the graphical search based on the constraint is used to find all large rectangles in that region. For each of these rectangles, the tool draws alignment lines at a fixed distance from each edge of the rectangle. Using the mouse, the user can select one of the text labels on the surface of the tool, and paste this label to the alignment lines using quick drag.

two. 17 Tool for loading documents into windows

In addition to adding one or a few objects to a drawing, the transparent overlay tools can be used to load the entire file from disk into an editor window, document frame, or another region. Figure 30 shows such a tool. The first part of the figure shows the tool, which has several document icons, positioned over a set of document windows. When the user clicks on an icon, the corresponding document (which, in the example, contains text and graphics) opens in the window behind the cursor. In the example illustrated, the user selects a file, called "home", from a set of available files that are placed in the selected window, with which the content of the file "home" is displayed in the selected window. An alternative approach would be for the user to position the tool near the desired window and drag the icon into the window.

3.0 Customization and use of the transparent overlay

No matter how the system designer is aware of user needs, there is no such thing as a prototype user. Accordingly, it is anticipated that the system will allow the user to customize the transparent overlay tools and layout.

ES 2 153 407 T3 transparent overlay to suit personal preferences and to adapt the transparent overlay to a particular task at hand. This section describes several possible ways that a user can do this. In addition, the clear overlay can be used more effectively if there is a simple and consistent set of conventions for using the clear overlay tools. This section describes several ways that users can make the use of the transparent overlay more effective.

3.01 Move, copy and delete tools

At the very least, the user will still want to create more of one tool class and less of another, and place within groups the tools that are commonly used together. If the user is to have the ability to participate and organize the transparent overlay, the user should at least have the ability to move, copy, and delete tools from the transparent overlay sheet. Figure 31 shows a technique to facilitate this ability, namely, providing handles that perform these operations on the tools. The specific example is the rubbing tool described in conjunction with Figure 9. As shown in Figure 31, the handles are icons next to the tool to facilitate desired operations. It is shown to the user clicking on one of the handles to move the tool. In practice, tool grips should probably be smaller and less in detail than shown.

Alternatively, the grips could be made so that they are invisible (and ineffective) during normal use, and are only subjected to these operations in a tool edit mode. 3.02 Organization of tools

A typical application is also having a large number of tools in its interface. To avoid clumping, it is necessary to organize these tools and sheets so that the user can quickly find any desired tool. One approach is to put all the tools on a single sheet that can be navigated by scrolling and focusing. The tools can be organized in blocks, and the resulting blocks can be glued together adjoining each other to form the single large sheet. To find any of the tools, then, the user will scroll to the desired block on the screen and scroll to the desired tool within the block. By using the scroll and focus functions together the user will be able to navigate in a rather large space of blocks. In addition, the mapping between the movements of the trackball and the movements of the transparent overlay sheet may allow for greater movements when the trackball is moved rapidly so that a small movement of the trackball is needed.

For a large number of blocks, a hierarchical organization could be used in addition to this block organization. For example, the user could create a virtual "box" containing several sheets of the transparent overlay sheet, each one tailored to a particular task. Alternatively, each block in a matrix could in fact be a stack of blocks. The user will be able to select which block is currently visible by repeatedly clicking on a button, or by using pop-up menus. The user could also use a single block (rather than an array of blocks) that cyclically passed through a set of block types.

The technique used in a current prototype allows a single block to display different sets of tools at different times. The set to be displayed can be selected in various ways. The user can click on a special tool in the set, which jumps to another set (somewhat like the arrows on Apple Computers HyperCard (TM) computers). In addition, a master view provides a table of contents of the available sets, thus allowing the user to jump to any other. To use different sets simultaneously, the user creates additional sheets. 3.03 Tool composition - Visual macros

The transparent tools activated by the mouse click and the visual filters can be composed by overlapping them, thus providing the user with the ability to interactively create new operations by combining the previous ones. This provides an intuitive and powerful macro capability. Figure 32 shows, at a user level, how to compose a tool that changes the fill colors (turn red) with a tool that changes line colors (turn blue) to form a tool that changes both the color of the fill as the line color. In the example, the two tools are held together so that their active areas are partially overlapping. If the user clicks on a part of one of the tools that does not overlap with the other tool, the resulting operation is simply the operation that is facilitated by the simple tool that is clicked on. On the other hand, if the user clicks on the overlap region, the resulting operation is a combination of the operations of both tools.

The composition of tools can be carried out in two ways. First, the two tools can be overlapped on the same transparent overlay so that they move together as a single unit. This is convenient when the combination is also going to be used frequently. It is also possible to compose tools that sit on separate transparent overlay sheets so that they can be moved independently, and be held together on the rare occasions when a particular combination is needed.

When a stack of overlapping tools (or filters) receives input (for example, a click

ES 2 153 407 T3 for a while), the entry event is passed from the top to the bottom through the tools. Each tool in turn modifies the order chain that has been built up to there. For example, a tool could concatenate an additional command over a current command string. Consider the example, discussed above, from the user's point of view, of the tool that changes the fill colors to red that is composed with the tool that changes the line colors to form a tool that changes both the fill color and color. the color of the line. If the line color tool is on top, then the command string would be “SetLineColor blue” after passing through this tool, and “SetLineColor blue; SetFillColor red ”after going through both tools.

3.04 Complicated macros

The composition of the tools described above is a powerful technique for implementing macros. However, if the user wishes to compose his own from a few tools, there is a risk that the intuitive visual advantage would be lost in a grouping of tools. In view of this possibility, the transparent overlay provides an alternative technique for creating composite tools. The technique is similar to the macro that records the facility found in some programs. This would generally require the user to put the system into macro creation mode, and then perform a desired sequence of operations on this object.

This composite tool has several advantages. For example, if you want to apply the set of operations to several objects, some of which may not yet have been drawn, applying to additional objects requires a simple step rather than reapplying the entire sequence of individual operations. Additionally, the user can easily apply the set of operations in an exploratory manner, either to simple objects or to a set of selected objects simultaneously.

3.05 Remembering selected objects

A problem that sometimes arises is that a user selects a set of objects and performs a set of operations on them, or all at once, or individually. Some time later, the user may wish to modify or undo the operations. The clear overlay sheet provides such a technique by facilitating each tool with the ability to remember which objects it has been applied to before. The user can then apply an operation to each group of objects, perform that other operation that is needed, and retrieve the set of objects, without having to remember the objects or explicitly perform a selection operation. This mechanism is useful even if the tools only remember the most recent set of shapes that was selected.

3.06 Creating and modifying tools

Some techniques for creating and modifying tools have already been described. AND^í These include the provision of move, copy, erase, and overlap tools to organize the transparent overlay sheet, discussed earlier, and techniques for copying object shapes and object attributes to create specialized clipboard tools as discussed in the previous section. section 2.03. The present invention is contemplated allowing the user additional flexibility to create and modify transparent overlay tools. For example, it would also be possible to use a drawing program or word processor to produce the geometry of a new transparent overlay tool, and then apply the behavior to the geometry using the embedded button architecture [* Bier90, Bier9la, Bier92 ].

Additionally, it would also be possible to use one sheet of the transparent overlay sheet to edit another. In such an environment a transparent overlay sheet would become editable in the same way as a drawing in a graphics editor, the different tools and techniques described above could be brought in for support. In addition, on the other hand, if the non-dominant hand can resize a sheet of the transparent overlay sheet as well as reposition it, then the user can make larger or smaller tools for special applications.

Two toolkits for creating the transparent overlay tools are currently under development. The first is a traditional toolkit in which tools are created through object-oriented programming. The second set of tools is based on embedded button technology in which users draw new tools and tool collections using a graphical editor and then apply a behavior (for example, the button functionality of transparent tools activated by clicking ratios) to these graphic forms, the behavior being in a client customization language.

3.07 Focus and shift

There are two equivalent ways for the user to position a sheet of the transparent overlay on an application. The scene may remain stationary while the user moves the transparent overlay over the scene, or the transparent overlay may remain stationary while the user moves the scene under the transparent overlay. This makes sense, to provide both of these kinds of functionality to the user. Scrolling the application allows the user to bring off-screen parts of an application to a tool on the screen, making it possible to manipulate larger applications. Moving the transparent overlay over the application allows the user to bring tools from off the screen application objects onto the screen, making it possible to use large sheets on the transparent overlay. References should be taken to position the transparent overlay sheet with

ES 2 related to visible representation to include rotation as well as translation.

While it is possible to move a transparent overlay sheet by pressing a mouse button and dragging the edge of the sheet, using the dominant hand, the use of the non-dominant hand and the second input device (for example, the trackball). One way of assigning the trackball buttons to these functions is as follows. Clicking on the right button causes the trackball and the rotary knob to move and focus the scene (application) respectively. Clicking the right button causes the trackball and rotary knob to move and focus the transparent overlay sheet respectively. Clicking on the middle button allows the scene and the transparent overlay to be moved and focused as units. The user may be given the opportunity to customize this assignment. For example, a user who wants to be able to make the transparent overlay disappear easily when it is not necessary, and make it reappear when what is preferred is to have what is preferred is to have the visibility of the transparent overlay of one lever. of the trackball buttons on and off. In addition to embodiments where there are multiple sheets of transparent overlay, it may be desirable to provide additional assignments to allow the sheets to be independently shifted and focused. This could be done by having the user click different combinations of trackball buttons for different sheets.

With these movement and sizing controls, the user can center a tool, or any application object and tool size to cover any region of the screen. Larger tools can be used to minimize blade movement when applying one tool to multiple objects. A tool that has been stretched to cover the entire work area effectively creates a command mode over the entire application. For several of the above tools, it is necessary to be able to "freeze" in some sense. For example, the “paste” tools in section 2.06 require alignment lines to remain frozen once positioned so that objects in the scene can later be pasted to them. A convention for the use of the clear overlay will fix the position of these tools relative to the clear overlay when the clear overlay is fixed to the scene (for example, when the button in the center of the trackball has been depressed. ).

Another possibility that refers to the same issues as the focus and the displacement of the transparent overlay sheet, is a tool design in which a set of tools is drawn larger and others smaller, the drawing being drawn larger.

407 T3 36 large under user control. This is somewhat like a tool highlight view, allowing the user to see many tools and use a few tools, without taking up a lot of screen space.

3.08 Keyboard to move the transparent overlay sheet

While the above-described embodiment makes use of an input device, such as a trackball, that can trigger both small and large movements in the transparent overlay as the set of positioning signals, it is also feasible to provide the set of signals. from an input device that only reports on and off transitions, such as a keyboard. For example, a set of keys on a computer keyboard could be designed to move the transparent overlay somewhat in some direction. By pressing these keys one or more times, the user can position different defined regions on a given workpiece. The design of tools on a transparent overlay could be captured to work well with this approach. For example, the tools could be arranged in a regular array whose spacing is a multiple of the distance that the transparent overlay is moved after each click of the keys for the movement of the transparent overlay.

Alternatively, a key on the numeric keypad could be used as a "reverse grip" that allows the transparent overlay to be easily attached and detached from mouse movement. When this key is held down, the transparent overlay is moved with the mouse cursor. When this key is released, the transparent overlay remains where it was placed and the mouse cursor moves independently.

The use of the keyboard in conjunction with scrolling the clear overlay has the advantage of providing relatively easy use of the clear overlay for users whose computers are not configured with a trackball and mouse.

3.09 Modal tools

While ambidextrous users can repeatedly perform an operation on a variety of objects, moving both the mouse cursor and the tool around the screen, this requires a lot of coordination and is equally user-friendly for users who only use one hand to position. again in a sequential way the transparent overlay sheet and then the mouse cursor. It is possible to overcome this problem by allowing a smooth transition between the transparent overlay metaphor and the mode interfaces. For example, the tool grips described in section 3.01 might include a button to place the cursor in a tool mode corresponding to that tool. While in this mode, users can repeatedly perform operations as if they were 19

ES 2 153 407 T3 by clicking through that tool. The cursor may take a shape similar to the shape of the tool as a way to remember the persistence of that mode. This is analogous to when a user selects a conventional modal tool.

By allowing a mouse-activated transparent tool to become a temporary modal tool, the question arises as to whether the temporary modal tool can be allowed to cooperate with other tools. One approach is to allow the temporary modal tool to be composed with other tools in the same way that you can compose a mouse-activated transparent tool. The alternative approach is to require that a temporary modal tool be the only tool that can operate on the underlying data. That is, putting a tool in mode will make it impossible to use the other tools. In such a case, the transparent overlay can be replaced during the mode, by means of a button that allows the user to exit the mode. Entering and exiting these modes can be enhanced using a detail. For example, by double clicking on a tool you can enter a mode. Doing a double click it will come out.

3.10 Dragging a tool

Typical use of a tool will include moving the cursor within the tool, pressing one of the mouse buttons, and, with possible intervention operations, releasing the mouse button. There are several possibilities between pressing and releasing the mouse button. AND^to These include the possibilities that the mouse does not move at all, that the mouse moves and stays in the tool, that the mouse leaves the tool, and that the mouse leaves the tool and comes back. One possible convention is that if an operation starts on a tool, that operation should continue until the mouse button is released. This convention will allow the tools to be relatively small, while allowing users to start an operation on the tool and then continue it outside.

3.11 Combining click-activated transparent tools with the mouse and the transparent overlay sheet containing conventional tools

As mentioned above, conventional tools can be used in conjunction with transparent overlay tools or they can be used as transparent overlay tools alone. A possible scenario here where the transparent overlay is provided with the transparent tools activated by clicking with the mouse as an enhancement to an existing program. Users who are familiar with a more modern version of the program, using a conventional user interface, may wish to be cautious when faced with a powerful new user interface. In this way, the user may decide to keep most of the conventional tools as a series of palettes that can be moved with the mouse cursor, but will add a few of these tools to the transparent overlay. As the user becomes more familiar with using the clear overlay and using the non-dominant hand to quickly and conveniently position tools, the user will be able to add more tools to the clear overlay. The user will first be able to create one more transparent overlay sheet sheets with conventional tools only, begin by experimenting with a few mouse-activated transparent tools, and later mix the mouse-activated transparent tools with conventional tools on the same sheet. .

It is also possible to compose mouse-activated transparent tools with conventional tools. For example, a mouse-activated transparent color palette such as the one shown in figure 7 could be combined with a set of tools for creating conventional modal shapes of the type on which the user clicks on a tool. and then move the cursor to the drawing area and click or drag with the cursor to form the shape. Said combination will allow the user to select the color and shape in one step by superimposing the transparent tool button activated by clicking with the mouse for the desired color on the desired conventional shape tool. The user will still have to move the cursor and form the shape in the conventional way. While this does not provide the total economy of movement that characterizes the tool shown in Figure 14, the user can still benefit from being able to select the color and shape with a single click. The modal tool can also be used with mouse-activated transparent tool buttons that specify other object properties such as line thickness, line stroke style, and appearance, and with button layouts. Mouse-activated transparent tools to specify more than one object property.

Another example of such a composition is one in which the user first clicks on the tool in a conventional way, and then proceeds to draw shapes through one or more transparent tool buttons activated by clicking the mouse for the desired set of properties. In this way, the user is able to select the properties and the position of the object to be drawn in one step, having pre-selected the shape of the object by clicking on the tool in a conventional way.

4.0 Specific implementation

This section describes a current implementation of the view-through interface that includes the transparent overlay tools and visual filters. The software is currently implemented in the mul20 structure

EN 2 153 407 T3 tiMulti-editor multi-user device (MMM) [* Bier91b] in Cedar [* Swinheart86] programming language and environment that runs on SunOS UNIX (TM) operating system compatible with Sun Microsystems SPARC workstations and other computers. The Gargoyle graphics editor [* Pier88], built into MMM, serves as a complex application for testing the interface. User input devices include a standard mouse for the dominant hand and a Microspeed FastTRAP (TM) trackball for the non-dominant hand. The trackball includes three buttons and a rotary knob that can be used to supply additional parameters to the interface.

This section describes three transparent overlay subsystems: one that handles simultaneous input from two pointing devices and updates the screen after multiple simultaneous changes, another that modifies pointing events when it passes through tools and visual filters, and another that modifies the graphical output as it passes through each tool or visual filter.

4.01 Multi-device input

The cross-over viewing interface has the following MMM features. MMM takes multiple events from multiple input devices, such as mouse and trackball, keeps track of which device produced a certain event, and puts all events in a single queue. The interface takes each of the events from the queue in order and determines to which application each element should be delivered. MMM applications are configured in a hierarchy that indicates the way in which they are nested on the screen. Each event is passed to the root application, which can pass the event onto one of its child applications, which can in turn shift the event down the structure in the tree. Mouse events are generally delivered to the most deeply nested application whose screen region contains the mouse coordinates. However, when the user is dragging or resizing an object in a particular application, all mouse coordinates go to that application until the drag or resize action is complete. The keyboard events go to the application that is currently selected. To support the transparent overlay sheet, the MMM rules for handling the trackball input were modified. When the application is movable, the trackball events and the rotary knob events go to the top-level application, which interprets them as commands to move or resize the sheet, respectively. In the case where the blade is not movable, the trackball and rotary knob events are delivered to the selected application, which interprets them as commands to move or to focus on that application.

Figure 33 is a flow chart of the user login routine, which determines the appropriate action to be taken in response to a login event. When a system containing a transparent overlay sheet receives an input, it must determine whether the event is intended to move the transparent overlay, to move the cursor, to trigger the transparent overlay, or to be delivered to a traditional application in the traditional way. usual, and then you must act on this determination. The routine first checks if the input is coming from a transparent overlay moving device, and if so, it moves the transparent overlay as a function of device movement. If not, then it checks if the input is coming from a transparent overlay sizing device, and if it is, it resizes the transparent overlay as a function of device movement. If not, then it checked if the input comes from a cursor movement device, and if it is, it moves the cursor as a function of device movement. The next thing to do, if applicable, is to pass the event to the root application (which can be, for example, a window manager) that determines where the event will be delivered next.

Note that the phrase "device for the movement of the transparent overlay" refers to a device that is currently designed for the movement of the transparent overlay. A particular phosphor device is a transparent overlay moving device at a particular time, if a software data structure called the "device table" currently designates that device as a transparent overlay moving device. This designation can be changed by the user at any time. The same is true for the phrase "transparent overlay sizing device". 4.02 Filtering the input through tools and visual filters

Commonly, MMM input events are strictly moved from the root application to the leaf applications. However, a system implementing the present invention may contain a number of transparent overlay sheets sprinkled with the applications. In order to support the transparent overlay, the input events must be passed backward in the structure of this tree. For example, Figure 34A shows an application hierarchy between the applications denoted by # 1, # 2, # 3, # 3A, # 3B, and # 4, and the transparent overlay sheets denoted by # 1 and # 2. Figure 34B shows how they might appear. the application windows and the transparent overlapping sheets on the screen, with a given hierarchy.

Application # 1 is the root application and its window contains all other relevant application windows. In the present implementation, this would be the editor of the top-level MMM rectangle, which acted as a window system. The order from left to right in each of the two lower levels of this tree structure indicates the order from top to bottom for the

ES 2 153 407 T3 applications on the screen, application # 4 being the application that is higher. Clear Overlay Sheet # 1 is located between Applications # 3 and # 4, while Sheet # 2 is located above Application # 4. Applications # 3A and # 3B are shown as contained in application window # 3, but they could be associated with it in another case.

The input events are first delivered to the transparent overlay to determine if the user is interacting with a tool or a visual filter. In case of being like this, the event is modified by the transparent overlay sheet. In any case, the event is returned to the root application, which either accepts the event itself, or passes it on to the child applications that appear further to the right in the tree structure. Consider the example in which the cursor is positioned as shown in FIG. 34B, namely, within the active area of ​​each of the transparent overlay sheets and over application # 3B. If the user gave a trigger event, such as pressing a mouse button, the event will go through sheet # 2 first, and then through sheet # 1, and then to the application containing the cursor coordinates. , namely application # 3B.

The data structure representing an MMM event is modified in three ways to support the transparent overlay. First, an event is annotated with a representation (event fields that will be referred to as "below child" and "below tool") of the parts of the application tree structure that have already been visited. . This prevents the root application from delivering the event to the sheet more at once. Second, an event is tagged with a command string that will be interpreted when it reaches its final application. For example, a color palette mouse click-activated transparent tool button annotates each mouse click event with the name of the SetFillColor command followed by a color. Finally, if the tool contains a visual filter, the mouse coordinates of an event can be modified so that the event will be correctly directed to the object that appears under the cursor through that visual filter.

Figure 34C shows how the event described above is delivered to the correct application. In particular, in order for the event to be delivered to the sheets and transparent overlays of the application in the correct order and with the correct device coordinates, the event travels up and down the application hierarchy.

4.03 Event delivery routines

Section 4.02 above describes how input events are routed through possible overlays transparent to the application you are targeting. This section and the following sections describe the event handling in additional detail with reference to the flowcharts and the pseudocode for certain of the important routines, namely the user input routine, the application event routine, the event routine to the transparent overlay, the event routine to the tool, the translation and execution routine, and the compositing routine.

The delivery of events through the transparent overlay sheets to applications, as described above and as shown in Figures 34A - 34C, uses three routines: the application event routine, which is executed when an event is delivered to a normal application, the transparent overlay event routine, which is used when the event is delivered to a transparent overlay, and the event routine to tool, used when an event is delivered to a particular tool on a transparent overlay. Each of these routines can call one or more of the others (for example, when an event is passed from a normal application to a transparent overlay sheet or from a transparent overlay sheet to a normal application, as occurs several times in the example described above).

A given event E has a "below child" field that contains an array (or other multivalued data structure), with an entry for each application and for each leaf in the tree structure. The value of each entry, below child [A], is a pointer to the application or sheet that is a child of A and is the last child of A that the event visited. A similar field "below tool" has for each entry, below tool [O], a pointer to an OR tool that is the last tool that the event visited.

Figure 35 is a flow chart from the event routine to the application. This routine assumes that some variables are initialized each time a new event is generated. For example, below child [O] is initialized for an imaginary application program that is defined to be both a child of application A and to be in front of all other children of A.

Fig. 36 is a flow chart of the event routine to the transparent overlay. When an input event is received by the transparent overlay, the transparent overlay determines which of its defined regions, called tools, should process the event. In some cases, multiple tools will process the event in turn. If a tool is a “transparent tool activated by the mouse click”, it calculates a data structure called an “command list”, which is passed down to the tools that are behind it (in case exist) or to the applications that are behind the entire transparent overlay sheet (through the parent program of the transparent overlay). This routine assumes that some variables are initialized each time a new event is generated. For example, below tool [O] is initialized to an imaginary tool that is defined to be in front of all other tools. When a transparent overlay sheet is first created, it is not in detail mode and none of its

ES 2 tools is designated as the “detail management” tool.

Figure 37 is a flowchart from the event routine to the tool. When an event E is received by a particular transparent overlay tool T, T determines from the type of E what action A is to take. If T is a conventional tool (a non-transparent tool activated by the mouse click), the action is carried out immediately. Otherwise, it delivers the event to an application behind its transparent overlay, either directly or through other tools on this or another transparent overlay. If E has already been processed by another tool, T may need to compose A with other actions that have already been associated with event E. 4.04 Overview of the translation and execution routine

A given action A can generally be an arbitrary body of code written in a suitable programming language. This language could be any general programming language such as the C language or the Pascal language, but user customization of the transparent overlay tools will be better supported if it is an easily interpreted language, such as Apple's HyperTalk, John Ousterhout's Tcl [* Ousterhout90], or a small version of LISPtal like the one used in Interleaf [* English90] or recessed button architecture [* Bier90, Bier91a, Bier92b]. To support the use of transparent overlay tools in an environment, such as the UNIX environment, in which each application runs its own important process. A can be represented as a cross-process message, such as those provided by the X windows system. The current implementation of the transparent overlay tools is done in the Cedar [* Swinehart86] programming environment, using a small LISP-like language to represent actions, and on Unix X windows using X windows messages to represent actions. the actions.

When a given software program P receives an action A, that action must be interpreted in its own context. In the case where A describes an action that could be performed by several programs, P must translate A into its own terms and then execute A on its own data structures (P). Interpreting a code body in a particular context (also called a domain) is a well-known technique in computer science. In order for A to be interpreted satisfactorily, a particular program P must provide ligatures (values) for all variables used in A. The following examples illustrate the particular use of the current implementation of this technique as it applies to use of independent application tools. 4.05 Actions using a general program

For example, an action A intended to increase the size of any rectangle behind the mouse cursor by a factor of 2 could contain code similar to

407 T3 44 this pseudocode:

FOREACHSHAPE, s, in PICTURE do if SRECTANGLE [s] and INCLUDES [s, cursorPoint] then SCALE [s, 2.0] endloop

In order to interpret this action, P must provide his own definitions for each of the words shown in the top case. You must provide your own procedure, FOREACHSHAPE, by means of which you repeat the shapes in your drawing, your own value for the PICTURE variable, your own ISRECTANGLE routine that checks if a shape is a right angle, your own INCLUDES procedure that checks if a signature contains a point, and its own SCALE procedure that resizes the shapes. In addition, you must be sure that the variable "cursorPoint" contains the current position of the cursor when it was passed through the transparent overlay routines described above.

In order to simplify the interpretation of transparent overlay actions, the current implementation of transparent overlay sheets in Cedar does not support general programs, such as the one shown in the previous example, which includes iteration, conditional, and iteration operations. calculation. Instead, the current implementation restricts actions to a list of commands, with each command being a command name followed by a set of values. These values ​​are calculated by the transparent overlay, not by the applications.

4.06 Actions with one or more orders

For example, a simple command specifying that P should select the closest object near the cursor point <23, 37> as long as it is not further than a quarter of an inch from where the cursor might be:

(SelectObject <23, 37> 0.25)

As in the LISP language, a complete command is enclosed in parentheses to make it easy to say where commands start and end.

Actions can contain several orders as in the case of the following action that selects an object, sets its interior in red and its contour line in blue:

((SelectObject <23, 37> 0.25) (SetFillColor red) (SetLineColor blue)) delimiting an additional pair of parentheses at the beginning and end of the entire action. This is an example of the list of orders referred to as L in the flowcharts above.

4.07 Translation of orders

This representation of actions is particularly easy to translate in the current prototype, since all the software programs on which the transparent overlays are being implemented already interpret commands that are expressed in this format. In this way, to translate the list of independent program commands above into a list of commands that can be interpreted by a

ES 2 153 407 T3 particular program P, it is necessary to translate each generic order into a number of orders that P understands.

For example, P may not have a command named "SelectObject" but may have a command named "Select" with the required functionality. Perhaps Select expects its second argument to be in pixels rather than centimeters (at 72 pixels by 25.4mm, 5.3mm = 18 pixels). Similarly, P may not have a "SetFillColor" command but may have a "SetAreaColor" command. If P does not have a SetLineColor command, then a correct translation of the list of commands is:

((Select <23, 37> 18) (SetAreaColor red) (SetLineColor blue))

The length of the command string is not always maintained. For example, the action could contain a command: (SetFillColorAtPoint cursorPoint red), which changes the color of the object closest to the cursor to red. A particular program P might not provide this operation as a simple command. In this case, the simple command (SetFillColorAtPoint cursorPoint red) could be translated into two commands: (Select cursorPoint) (SetAreaColor red).

If P does not already implement an interpreter for said list of commands but provides a set of procedures that facilitate the required functionality, the actions can be directly translated into procedure calls. For example, the action could be translated into three procedure calls in the Cedar programming language as:

SelectShape [picture, [23, 37], 18]; shape GetSelectedShape [picture]; SetAreaColor [shape, red];

SetLineColor [shape, blue];

4.08 Reverse the data flow

The actions described so far communicate the commands and data coming from the tool to an application program. It is also possible for a command to return data from the application program to the tool. This is done by including some empty storage in the order list. For example, a copy and paste tool might issue the action:

((Select <23, 37> 18) (GetSelected fillInValue))

Here, “fillInValue” is a pointer to a record, labeled fv for “future value”.

Figures 38A and 38B show this record before and after the application has returned the data required by the command. This record includes a "value" field in which a pointer to the data returned by the application would be placed. A second field, "ready?" has the value of false until the "value" field has been filled out. The last field "VarCond" contains a synchronization object provided by the Cedar programming language, called a condition variable. A condition variable includes a list of these programs that are waiting for this value. When the value is available, the application that provides the value communicates with each of the programs that are waiting.

In this example, the application P would select the shape found at coordinates <23, 37> and then store a pointer to it in the "value" field. If P is running in a process other than T, T must wait (hang) until P has stored the pointer. Because T holds its pointer to fv, it can reach the "value" field of "fv" once it has been told that the value is ready.

4.09 Composition Routine

When two or more transparent overlapping sheets overlap (such as transparent overlapping sheets # 1 and # 2 in figure 34B), each event E is finally delivered to the receiving application (application # 3B of figure 34C) after having been processed by all clear overlay sheets in front to back processing order (clear overlay # 2 and then clear overlay # 1). As a result, Action A can describe an operation that includes contributions from some or all of the transparent overlay sheets. Said action is mentioned as if it were a composition of the actions that would be determined by each of the transparent overlapping sheets individually.

In the current implementation, each transparent O overlay is responsible for combining its action with the action that the event has accumulated at the time that O receives the event. The new action can be calculated from the existing action through one of a variety of procedures, including append, pre-append, delete, order alteration, argument alteration, or coordinate alteration. Each of these procedures is described below.

4.09.01 Annexation

A tool that appends your action, simply adds the list of action orders to the end of the list of existing orders to create a compound action. For example, let a transparent overlay sheet O receive an event E whose coordinates <x, y> are contained by the tool T of O, and let the tool T be a tool that changes the colors of the contour lines. Individually, T's reaction to event E would be to generate the action:

((SelectShape <x, y>) (SetLineColor blue))

Suppose, however, that E has already passed through a tool that has specified an action to set a fill color to red:

((SelectShape <x, y>) (SetFillColor red))

If T is an addition tool, it added its command list to the existing command list to form a longer action ((SelectShape <x, y>) (SetFillColor red)) (SelectShape <x, y>) (SetLineColor blue ))

4.09.02 Pre-annexed

Pre-append is like append except that the new command list is appended to the beginning of the existing list. If the order of the orders matters, the prefix may produce a different result than the append. For example, if event E arrives with an existing order:

(RotateShape 45)

ES 2 153 407 T3 that rotates a shape by 45 degrees and if the tool T will generate the command in an ordinary way:

(ScaleShape 2 1) that sizes a shape by increasing its size by 2 in the x direction, and by 1 in the y direction, then the final order will be:

((ScaleShape 2 1) (RotateShape 45)) which has a different effect than ((RotateShape 45) (ScaleShape 2 1)).

4.09.03 Deletion

The T tool will be able to remove one or more orders from the event. For example, T can protect the drawing under it from being edited by deleting any command that can edit that scene. If T received the event:

((SelectShape <x, y>) (SelectLineColor blue)) can remove the (SetLineColor blue) command that will have modified the underlying drawing, but you can leave the (SelectShape <x, y>) command that will only change the currently selected object from that drawing. The resulting order will be:

(SelectShape <x, y>)

4.09.04 Alteration of an order

A tool T will be able to change the names of the orders used in some or all of the received actions. For example, T will be able to change all calls to SetFillColor to the SetFancyFillColor call and all calls to SetLineColor to SetFancyLineColor, allowing a user to test the new figuration commands while still using familiar transparent overlay tools on transparent overlays that are in front of the transparent overlay sheet O. If said tool T receives the command:

((SelectShape <x, y>) (SetFillColor red) (SelectShape <x, y>) (SetLineColor blue)) this will produce the command:

((SelectShape <x, y>) (SetFancyFillColor red) (SelectShape <x, y>) (SetFancyLineColor blue))

4.09.05 Alteration of arguments

In the case of argument alteration, T modifies the values ​​specified after the command name of a command. For example, T will be able to modify any of the colors received by the previous tools to make them more intense. In the event that said tool receives the command (SetLineColor blue), this will produce the command:

(SetLineColor vivid-blue)

4.09.06 Alteration of coordinates

Coordinate alteration is a special case of argument alteration. In this case, the T tool modifies the specified <x, y> coordinates in one order. This is particularly useful in the case that T contains a visual filter that modifies the view of the drawing that is after O. For example, if T increases the drawing below, then the coordinates from the tools in front of T must have a reductive transformation applied so that the commands from those tools will be applied to the object that actually appears through T. Thus, if T increases the size of the drawing after it by a factor of 2, it must dimension the coordinates by a factor of 0.5. If the order received is:

(SelectShape <x, y>)

You will produce:

SelectShape (0.5 * <x, y>)

4.09.07 About Marked Coordinates

The transparent overlay event routine identifies those <x, y> coordinates in the command list that are "marked for translation". In general, the coordinates of the cursor will be marked in this way. This mark will be kept by the previous composition routines, so that when the input commands contain marked coordinates, the corresponding coordinates in the commands produced by composition will also be marked. 4.10 Basic output handling

In addition to delivering the input events to the application programs behind them, the transparent overlay sheets also modify the display of these application programs. In particular, most of the tools display a visible icon to indicate the limits of the active region of that tool. In addition, the tools can contain visual filters that filter the view of the application, modifying the shapes, colors, sizes and other graphic properties. While the basic operation of the visual filters is described in the previously referenced co-annexed application by Stone et al., The following sections describe how the transparent overlay mechanism controls the activation of the visual filters.

Commonly, the MMM output is composed of the sheet applications. To support the visual filters, the normal display refresh composition has been extended to allow information to flow down and through the tree structure as well as up. For example, if the tools in Figure 34B contain one or more visual filters, if any of these visual filters is placed on top of the graphic editor, each visual filter must examine the content of the graphic editor (which is the brother of the visual filter in the graphic editor). hierarchy) in order to draw himself.

4.10.01 Calculation of the region of exchange

Some parts of the screen refresh routine provided by the transparent overlay sheets are similar to the input handling routine, but with the data passed in the opposite direction. As shown in Figure 34C, the clear overlay routines ensure that input events pass through the clear overlay in order from the front clear overlay to the back before being delivered to the target application. To calculate how much of the screen should be redrawn when the application of content # 3B is modified, the information about the modified part of application # 3B is passed along the path shown in figure 39, which is the path reverse of the path of FIG. 34C.

When the change region information is passed from node to node in this structure of

ES 2 153 407 T3 the tree, the region of change is translated into the coordinate system of each reception node so that it is easily understood by this node. Additionally, each transparent overlay node can potentially modify the region of change based on current view filters that are visible on that transparent overlay. For example, if a given transparent overlay O includes a tool T that contains a magnification filter, O can increase the size of the screen region that will be affected by the original change to application # 3B. Or it increases the size of the region of the change data structure appropriately and passes the changed data structure to the next node in the chain.

4.10.02 Screen update

In the absence of visual filters, producing a new screen image that represents both the application and the transparent overlapping sheets is carried out by producing a drawing of each node in the hierarchy in pre-ordering from bottom to bottom first and backwards. and cross-sectional post-ordering (that is, each application is drawn both before and after its children are drawn). For the example in figure 39, the nodes should be drawn in the order: Beginning (application # 1) Beginning and Ending (application # 3B) Ending (application # 3) Beginning and Ending (transparent overlay sheet # 1) Beginning and Ending (application # 4) Start and Finish (transparent overlay # 2) Finish (application # 1).

However, if a transparent overlay contains one or more visual filters, each of those visual filters may require that part of the hierarchy be redrawn using the transformation that characterizes that visual filter. The part of the hierarchy that is redrawn includes the parent of the transparent overlay sheet and all the siblings of the transparent overlay that lie below the transparent overlay. For example, if the transparent overlay sheet # 1 includes a visual filter, the nodes for application # 2, for application # 3A, for application # 3B, for application # 3 and for application # 1 will be redrawn ( in that order). This fragment of the tree structure is shown in figure 40.

5.0 Advantages

This section summarizes several advantages of the numerous embodiments of the clear overlay. Although some of these advantages were explicitly stated in connection with the above description of various tools, it is useful to publicize these advantages in one place. 5.01 Quick completion of tasks

By combining multiple steps in one step, clear overlay tools can save time. As described below, the transparent buttons activated by the mouse press combine the choice of order and the selection of the operand in a single movement. With the addition of a visual filter, these buttons can also create custom views in the region of the operand, without requiring any keystrokes to explicitly turn these views on and off. In addition, both the user's gaze and the cursor can remain in the work area. This keeps the user focused on the task, and saves time compared to systems that require the user to move the cursor from here to there between the work area and the pallets that are placed outside on the side.

5.02 Fewer temporary modes

With the transparent overlay sheet, an operation is available as long as the cursor is over a tool that facilitates the operation. Essentially, the transparent overlay provides spatial order modes. Spatial modes are often more desirable than temporary modes as the user can easily see what current mode is in (for example, by the label above the tool) and how to exit it (for example, move the cursor out of it). tool).

5.03 Better graphical feedback, fewer errors

Often times, systems with temporary modes provide feedback to the user that aids in the successful completion of the current order. For example, in text editing mode in a graphical editor, the editor can highlight all editable text strings. Clear overlay tools can provide this same type of feedback, but within the limits of the tool. In fact, with multiple tools on the screen at the same time, several different types of feedback can be provided, each in its own tool. Also, by increasing a tool to the full size of the editor, the user can get the same feedback available with the temporary modes. When several tools are visible at the same time, the feedback in each of them serves a double role. It helps the user to make proper use of the tool and helps the user to choose the correct tool.

In addition, the use of visual filters in the tools provides a kind of feedback that is not provided in the traditional way; These filters show a part of the scene in a modified view while a normalized view of the rest of the scene is shown as context. A visual filter can show hidden information, like a hidden vertex. Furthermore, it can help to identify those objects that are appropriate operands for a given operation. For example, a tool that changes the border colors of objects could show only the edges of the shape. Such feedbacks will make it difficult for the user to mistakenly attempt to use said tool for a different purpose, such as changing the fill colors.

5.04 Easy customization

In an application whose tools make exclusive use of a region of screen space, the potential for user customization is limited. Even if the user can create new buttons, pins and other interaction elements, there is very little space to put them. However, the super26 blade

ES 2 153 407 T3 made transparent provides essentially unlimited space; Not only can the tools occupy the entire work area, but they can be placed outside the screen to be moved within the screen when needed. In addition, because the transparent overlay tools and visual filters can be overlapped, these provide an easy way for users to build custom macros.

5.05 Reduced learning time

Because the tools can combine several steps of a task into one step, users no longer need to learn these steps individually. For example, instead of learning how to select an object and apply a color to it, the user learns to use a transparent button activated by clicking the mouse button for the color. Likewise, novice users become experts in a natural way. A skilled user knows where various tools are located, knows how to compose tools effectively, and has learned to put tools in place quickly. A novice user becomes an expert by learning the spatial arrangement of tools, learning to compose tools, and repeating commonly used movements until they become part of the user's motor memory. These tasks require a little conscious effort. 5.06 Application Independent Tools

Some user commands are common for enough applications to be performed via keyboards, such as the common cut and paste keys. When the user presses one of these keys, the window manager directs the keystroke to the current user application. The transparent overlay represents a new substrate on which said independent orders of the application can be placed. In particular, if a transparent overlay sheet can slide from application to application, its tools can be applied to any of the applications behind it whenever it makes sense, without any need for a distinguished current application. For example, transparent buttons activated by clicking the mouse to change the color can change the colors of objects in different editors.

5.07 That can be used with one or both hands

Many graphical user interfaces require the user to coordinate the use of both hands. For example, the user may need to hold down the Control key or the Shift key while simultaneously using the mouse. While the clear overlay can be used more quickly with two hands, it can also be used with one hand to place the clear overlay and then place the cursor sequentially. This can be done by users with a disabled hand or by healthy users using one hand for another task such as holding a phone or a cup of coffee.

5.08 Different display sizes

Modern computers come equipped with screens of many different sizes, from pocket size to wall size. Transparent overlay interfaces can span this size range. Because clear overlay tools shift and grow in size, they can be used on miniature displays that have little or no room for fixed position tool palettes. In addition, on very large screens, the user can move the tools to any part of the screen, making it unnecessary to enter through the screen to access a fixed menu region.

6.0 Conclusion

In conclusion it can be seen that the present invention provides a new style of user interface, the view-through interface, based on a visual description of a transparent overlay sheet with active tool definition regions. The through-view interface offers a new design space for user interfaces based on spatial modes rather than temporal modes and provides a natural medium for two-handed interaction. Because the interface is movable and covers the application area, it does not take up permanent screen space and can be conveniently adapted to a wide variety of screen sizes. Because clear overlay tools are selected and brought into the work area simply by moving the clear overlay, the user's attention can be kept focused on the work area. Because the operations and views are spatially defined, the user can work without changing the global context.

Additionally, the transparent overlay sheet of the present invention, when used with visual filters, allows operations on a hidden state of the application, such as the equations of a spreadsheet, the grouping of objects in a graphical editor or the position of water pipes in an architectural model. The user is able to view and edit this hidden state in a way that does not occupy permanent screen space and does not require memorization of commands.

Additionally, the through-view interface provides a new paradigm to support an open software architecture. Because transparent overlay tools can be moved from one application to another, rather than being tied to a single application window, they provide an interface to the common functionality of various applications and can encourage more applications to provide common functionality.

Furthermore, because transparent overlay sheets can contain an unlimited number of tools, they provide a valuable new substrate on which users can create their own custom tools and tool sets, compose macros by overlapping tools, and drag tools27

ES 2 153 407 T3 while together in new configurations. Actually, a transparent overlay can even be used to edit another.

Although the foregoing is a complete description of specific embodiments of the invention, various modifications, alternative constructions and equivalents thereof may be used. For example, while the above description of the transparent overlay tools emphasizes graphical editing applications, many of the tools described can potentially be used in any screen-based application, including spreadsheets, text editors, editors. multimedia, paint programs, 3D figure modelers, circuit editors, scientific viewers, or motorcycle support tools. Transparent overlay sheets can be used on animated media, including video and computer graphics. This is a particularly compelling application area because people who watch moving media are particularly reluctant to view views from the workpiece to menus outside on the side because they could miss an important event. In the context of animated media, the media itself can provide the event that triggers a button. For example, to make an object orange in a succession of frames of an animation, the user could hold down a button where the object appears. As a new frame is drawn, the button will be applied to the object at that position in the frame, automatically. In addition, while the use of clear overlay tools has been emphasized previously in connection with application programs, clear overlay sheets can be used in window systems to place windows and to set window system parameters. such as window border colors, keyboard macros, and the like.

Therefore, the foregoing description should not be taken as limiting the scope of the invention as defined in the claims. 7.0 Bibliography

[Bier86] EricA. Bier and Maureen C. Stone. Quick drag. In Proceedings of Siggraph '86 (Dallas, August), Computer Graphics, Vol. 20, No 4, ACM, 1986, pp. 233-240.

[Bier88] Eric A. Bier Rapid trawling: Interactive geomometric design in two and three dimensions. Report No. UCB / CSD 88/416, April 28, 1988, Computer Science Division, Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA 94720. Also available as Xerox PARC technical report EDL-89-2.

[Bier90] Eric A. Bier and Aaron Goodisman. Documents as user interfaces. In R. Furuta (ed). EP90, Proceedings of the International Conference on Electronic Publishing, Document Manipulation and Typography, Cambridge University Press, 1990, pp. 249-262. (The earliest paper on EmbeddedButtons)

[Bier9la] Eric A. Bier. Recessed buttons: documents as user interfaces. In Proceedings of the ACM SIGGRAPH Symposium on User Interface Software and Technology (South Carolina, November), ACM, 1991, pp. 45-53 [Bier91b] Eric A. Bier and Steve Freeman. MMM: a user interface architecture for shared editors on a single screen. In Proccedings or the ACM SIGGRAPH Symposium on User Interface Software and Technology (Hilton Head, SC, November 11-13), ACM, 1991, pp. 79-86.

[Bier92] Eric A.Bier. Recessed buttons: Help buttons in documents. ACM Transactions on Information Systems, Vol. 10, No. 4. October 1992, pp. 381-407.

[English90] Paul M. English, Ethan S. Jacobson, Robert A. Morris, Kimbo B. Mundy, Stephen D. Pelletier, Thomas A. Polucci, and H. David Scarbro. An extensible object-oriented system for active documents. R. Furuta (ed) EP90, Proceedings of the International Conference on Electronic Publishing, Document Manipulation and Typography, Cambridge University Press, 1990, pages 263-276.

[Goldberg91] DavidGoldberg and Aaron Goodisman. Stylus user interfaces for text manipulation. In Proccedings oó the ACM Symposium on User interface Software and Technology (UIST '91, Hilton Head, South Carolina, November), 1991, pp. 127-135. [Hopkinsg91] Don Hopkins. The design and realization of menus of sectors. Dr. Dobb's Journal. Vol. 16, No. 12, December 1991, pages 16-26 [Kurlander92] David Kurlander and Steven Feiner. Interactive search and substitution based on restriction. In Proceedings of CHI '92 (Monterey, California, May), Human Factors in Computing Systems, ACM, New York, 1992, pp. 609-618.

[Kurtenbach91] Gordon Kurtenbach and Williarn Buxton. Results in the combination of marking techniques and direct manipulation. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '91, South Carolina, November), ACM, 1991, pp. 137-144. [Ousterhout90] JK Ousterhout. Tcl: An embeddable command language. In winter USENIX Conference Proceedings, 1990.

[Pier88] Ken Pier, Eric A. Bier, and Maureen C. Stone. An Introduction to Gargoyle: An Interactive Illustration Tool. Proceedings of the Intl. Conf. On Electronic Publishing, Document Manipulation and Typography (Nice, France, April). Cambridge Univ. Press, (1988), pp. 223-238.

[Rubine91] Dean Rubine. Specification of details by means of example. In Proccedings of ACM SIGGRAPH '91, Computer Graphics, vol 25, No. 4, July 1991, pages 329-337.

[Swinehart86] Daniel C. Swinehart, Polle T. Zellweger, Richard J. Beach, Robert B. Hagmann. A structural view of the Cedar programming environment. ACM Transactions on Programming Languages ​​and Systems, vol. 8, No. 4, 1986, pages 419-490. Also available as Xerox PARC Technical Report CSL-86-1.