Compound application presentation across multiple devices.

(54) Title: PRESENTATION OF APPLICATION OF COMPOUND THROUGH MULTIPLE DEVICES.

(54) Title: COMPOUND APPLICATION PRESENTATION ACROSS MULTIPLE DEVICES.

(57) Summary

The presentation of transformation string output in devices is described. The transform chain includes one or more presentation nodes that drive the output representation, such that a change in a value at the presentation node causes the representation to occur. In response to a change in the value of the presentation node, the presentation system identifies features or various devices that can be used to represent the corresponding output. The display system identifies an appropriate device based on the identified characteristics of that device. The display system then facilitates rendering of at least one version of the output on the selected device, perhaps still transforming the output into a form suitable for the selected device. In some embodiments, the transformation chain may be a compound transformation chain constructed by joining multiple constituent transformation chains.

(57) Abstract

Presentation of transformation Chain output on devices. The transformation Chain includes one or more presentation nodes that drive rendering of output, such that a change in a value in the presentation node causes rendering to occur. In response to a change in value of the presentation node, the presentation system identifies characteristics or various devices that may be used to render the corresponding output. The presentation system identifies an appropriate device based on the identified characteristics of that device. The presentation system then facilitates rendering of at least a version of the output on the selected device, perhaps even transforming the output in a manner suitable for the selected device. In some embodiments, the transformation Chain may be a compound transformation Chain constructed by joining multiple constituent transformation chains.

PRESENTATION OF COMPOUND APPLICATION THROUGH MULTIPLE DEVICES

BACKGROUND

Computer technology has revolutionized the way we work, play and communicate. As computer technology has advanced, so has the diversity of devices that modalize such computer systems or display content from a computer system. For example, a computer system can take the form of a server stand, a desktop computer, a laptop, a tablet, a smartphone, a video game console, a clock, a refrigerator, a smart home, and the like.

Along with the diversity of computer systems, the types of devices that can be used to process information input and output to a computer have also diversified. For example, output devices can include displays such as projectors, television monitors, three-dimensional displays, laptops, tablet computers, telephones, and the like. Output devices can also include sound output, such as speakers. The output devices could also include actuators, lights, valves, and the like. Input devices could include keyboards, pointing devices (such as a mouse), touch screens, microphones, video cameras, still cameras, 3-dimensional position detectors, global positioning system monitors, light sensors, accelerometers, thermometers, compasses, and the like.

Computer systems and associated input and output devices have become very prolific and highly mobile. Often, in any given location, there may be a large number and variety of devices present. For example, in an average conference room that is completely full, there could be overhead projectors, television screens, laptops, tablets, smartphones, microphones, cameras, lighting, and the like. The conventional paradigm is that each device runs in its own application or displays content from a single application. When applications interact, they often interact as separate applications that interact through an application program interface.

The subject matter claimed here is not limited to modalities that solve problems or that only work in environments such as those previously described. Rather, this background is only provided to illustrate an exemplary area of ​​technology where some of the modalities described herein can be practiced.

BRIEF DESCRIPTION OF THE INVENTION

At least some of the modalities described here refer to facilitating the presentation of transformation chain output in devices. The transform chain includes one or more presentation nodes that drive the output representation, such that a change in a value in the presentation node causes the presentation to occur. In response to a change in the value of the presentation node, the presentation system identifies the characteristics or various devices that can be used for rendering the corresponding output. The display system identifies an appropriate device based on the identified characteristics of that device. The display system then facilitates the rendering of at least one version of the output on the selected device, perhaps still transforming the output into a form suitable for the selected device. In some embodiments, the transformation chain can be a composite transformation chain constructed by joining multiple constituent transformation chains.

This Brief Description is not intended to identify the main characteristics or essential functions of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the aforementioned features and other advantages can be obtained, a more specific description of various embodiments will be presented with reference to the accompanying drawings. Understanding that these drawings represent only sample modalities and should not, therefore, be considered as limiting the scope of the invention, the modalities will be described and explained in additional detail and specificity by using the attached drawings where:

Figure 1 abstractly illustrates a computing system in which some of the modalities described here can be used;

Figure 2 abstractly illustrates a simple transformation chain in which there is but a single link coupling a single data source and a single data target and where a transformation represented by the link is performed automatically by a value in the source data as input to generate a value in the data target;

Figure 3 abstractly illustrates another simple transformation chain example where a transformation is performed using input values ​​from three data sources to generate output values ​​at two data targets;

Figure 4 illustrates a transformation chain in the form of a combination of the transformation chain of Figure 2 and the transformation chain of Figure 3;

Figure 5 illustrates an example environment in which the principles described herein may work and that includes various devices associated with the constituent transform chains of a compounding application, and that also includes input devices and output devices;

Figures 6A through 6D each illustrate illustrative transform chains that may be associated with the respective devices in Figure 5 (arrows through which data does not flow absent joining with another transform chain are illustrated with an X, and dependency elements that are not nodes in the transformation chain itself are illustrated with dotted borders);

Figure 7A shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A and 6B;

Figure 7B shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A and 60;

Figure 7C shows an enlarged transformation chain representing the joining of the transformation chains of Figures 6B and 60;

Figure 7D shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A and 6D;

Figure 8A shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A, 6B and 60;

Figure 8B shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A, 6B and 6D;

Figure 8C shows an enlarged transformation chain representing the junction of the transformation chains of Figures 6A, 60 and 6D;

Figure 9 illustrates an augmented transformation chain representing the joining of the transformation chains Figures 6A, 6B,

6C and 6D;

Figure 10 shows a flow chart of a method for preparing the facilitation of a compound application;

Figure 11 abstractly illustrates a system that can be used to perform the method of Figures 10, 12, 13 and 14, and that includes a library of transform string class definitions and device records, as well as a presentation service. ;

Figure 12 shows a flow chart of a method for joining two instances of transform chain classes;

Figure 13 shows a flow chart of a method for decoupling a transformation chain from a larger transformation chain;

Figure 14 shows a flow chart of a method for representing changes in presentation of one or more transform chain nodes in an appropriate device;

Figures 15A through 15J illustrate different user interfaces that can be experienced in a scenario referred to herein as a device scenario, in which a user is going to request a number of devices; and

Figures 15K through 150 illustrate various user interfaces that can be found to change the compound application used in the device scenario of Figures 15A through 15J.

DETAILED DESCRIPTION

At least some of the modalities described here refer to facilitating the presentation of transformation chain output in devices. The transform chain includes one or more presentation nodes that drive the output representation, such that a change in a value in the presentation node causes the presentation to occur. In response to a change in the value of the presentation node, the presentation system identifies the characteristics or various devices that can be used for rendering the corresponding output. The display system identifies an appropriate device based on the identified characteristics of that device. The display system then facilitates rendering of at least one version of the output on the selected device, perhaps still transforming the output into a form suitable for the selected device. In some embodiments, the transformation chain can be a composite transformation chain constructed by joining multiple constituent transformation chains.

Some introductory discussion of a computing system will be described with respect to that of Figure 1. Then, the compound device application technology will be described in the following Figures.

Computer systems are already taking more and more a wide variety of forms. Computer systems can be, for example, handheld devices, household appliances, laptop computers, desktop computers, central computers, distributed computing systems, or even devices that have not traditionally been considered a computer system. In this description and in the claims, the term computer system is broadly defined as including any device or system (or combination of them) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having therein computer-executable instructions that can be executed by the processor. Memory can take any form and can depend on the nature and form of the computer system. A computer system can be distributed across a network environment and can include several constituent computer systems.

As illustrated in Figure 1, in its most basic configuration, a computer system 100 typically includes at least one hardware processing unit 102 and memory 104. Memory 104 can be physical system memory, which can be volatile, not volatile, or some combination of the two. The term memory can also be used herein to refer to non-volatile mass storage such as physical storage medium. If the computer system is distributed, the processing, memory and / or storage capacity can be distributed as well. As used here, the term executable module or executable component can refer to software objects, paths, or methods that can be executed on the computer system. The different components, modules, engines and services described in this document can be implemented as objects or processes that are executed in the computer system (for example, as independent sequences).

In the description that follows, modalities are described with reference to the acts that are performed by one or more computer systems. If such acts are implemented in software, one or more processors of the associated computer system that performs the act direct the operation of the computer system in response to having executed computer-executable instructions. For example, such computer-executable instructions can be modeled on one or more computer-readable media that form a computer program product. An example of such an operation involves data manipulation. Computer executable instructions (and manipulated data) can be stored in memory 104 of computer system 100. Computer system 100 may also contain communication channels 108 that allow computer system 100 to communicate with other message processors via, for example, network 110.

The computer system 100 can also include output display components, such as displays, speakers, lights, actuators, or the like. The computer system 100 may also include input components, such as a keyboard, a pointing device (such as a mouse or track pad), voice recognition devices, and possibly also physical sensors (eg, thermometers, systems global positioning system, light detectors, compasses, accelerometers, etc.).

The modalities described herein may comprise or utilize a general purpose or special purpose computer including equipment hardware, such as one or more processors and system memory, as discussed in greater detail below. The modalities described also include physical and other computer-readable media for transporting or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that stores computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, for example, and without limitation, embodiments of the invention may comprise at least two different classes of computer-readable media: computer storage media and transmission media.

Computer storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium that can be used to store program code media. desired in the form of computer-executable instructions or data structures and accessed by a general-purpose or special-purpose computer

A network is defined as one or more data links that allow the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or supplied over a network or other communications connection (whether wired, wireless, or a combination of wired or wireless) to a computer, the computer appropriately views the connection as a transmission medium. The transmission media may include a network and / or data links that can be used to transport means of desired program code in the form of computer executable instructions or data structures and accessed by a general-purpose computer or special purpose. Combinations of all of the above should also be included in the scope of computer-readable media.

Furthermore, after reaching the various computer system components, program code media in the form of computer-executable instructions equipment or data structures can be automatically transferred from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (for example, a NIC), and then , eventually transferred to a computer system RAM and / or less volatile computer storage media in a computer system. Therefore, it should be understood that computer storage media can be included in computer system components that also (or even primarily) use transmission media.

Computer-executable instructions include, for example, instructions and data that, when executed on a processor, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions. functions. Computer-executable instructions can be, for example, binary files or even instructions that undergo some translation (such as compilation) before direct execution by processors, such as intermediate format instructions such as assembly language, or even the source code. Although the subject has been described in language specific to structural features and / or methodological acts, it is understood that the subject matter defined in the appended claims is not necessarily limited to the features described or the acts described above. Rather, the described features and acts are described as illustrative ways to implement the claims.

Those skilled in the art will appreciate that the invention can be practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems. , microprocessor-based or programmable consumer electronics, network PCs, minicomputers, central computers, 25 mobile phones, PDAs, pagers, routers, switches, and the like. The invention can also be practiced in distributed system environments where local and remote systems, which are linked (either by wired data links, wireless data links, or by a combination of wired and wireless data links) through of a network, both perform tasks. In a distributed system environment, program modules can be located on local and remote memory storage devices.

The principles described here work through a chain of transformation. A transform chain is a set of interconnected nodes that each can represent data sources or data targets. There are links between the nodes, each link represents a transformation. For any given link, the associated transformation receives copies of values ​​from one or more data sources located at an input end of the link, and generates resulting values ​​that are provided in one or more data targets located at the output end of the link. . For any given transformation, when a value in one or more of the data sources at its input end changes, the transformation is automatically reevaluated, potentially resulting in changes in the value (s) of one or more data targets at the end. transformation output.

In one embodiment, regardless of how complex the transformation chain is, the transformation can be built from declarative sentences expressing equations, rules, constraints, simulations, or any other type of transformation that can receive one or more values ​​as input and provide one or more resulting values ​​as output. An example of a transform chain is a spreadsheet program, where any of the cells can be a data source or a data target. An equation (that is, a transformation) can be associated with any cell to cause that cell to be a data target where the results of the equation are placed.

Just as an example, Figure 2 illustrates a simple transform chain 200 in which there is but a single link 220. In the drawing notation used throughout this description, a link will be illustrated as an arrow, with the input end is represented as the tail of the arrow, and the end end is represented as the head of the arrow. In cases where there are multiple data sources at the input end of the link, the arrow will be represented with multiple tails. Copies of the data source (s) values ​​in the arrow queue (s) represent input to the transformation. In cases where there are multiple data targets affected by value (s) resulting from the transformation, the arrow will be represented with multiple heads. The values ​​of the data targets in the arrowhead (s) represent the output of the transformation.

For example, Figure 2 illustrates a simple string transformation 200 that includes a data source 201, a data target 202, and an individual link 220. Link 220 represents a transformation performed on a copy of the value 211 in the data source. data 201 to generate a value of 212 in data target 202. If the value 211 changes, the transformation represented by link 220 is potentially reevaluated resulting in a change in value 212 in data target 202.

Figure 3 shows another example of a simple transformation chain 300 that includes three data sources 301, 302, and 303, two data targets 304 and 305, and a single link 320. Link 320 represents a transformation performed on copies of the values ​​within data sources 301, 302, and 303 to generate the values ​​in the 304 and 305 data targets. If any of the values ​​within data sources 301, 302, or 303 change, the transform link 320 is re-evaluated potentially resulting in a change in the values ​​at one or more of data targets 304 and 305.

Figure 4 illustrates another example of transform chain 400, and illustrates the principle that transform chains can build on each other where a data source to one link can be a data target on another link, in order to create even more complicated transform chains. For example, transformation chain 400 includes one instance 401 of transformation chain 200, and one instance of 402 of transformation chain 300. In this case, data target 202 of link 220 is also data source 301 of link 320. If the value with data source 201 changes, the transformation represented by link 220 is re-evaluated potentially resulting in a change in the value in data target 202, which is also a data source 301 for the next link 320. Similarly, a change in the value of data source 301 would result in transformation link 320 being re-evaluated, potentially resulting in a change in the values ​​in one or more of data targets 304 and 305. The data targets 304 and 305 can also represent data sources for other links. Consequently, in complex transformation chains, a value change could cause value changes propagated across multiple nodes in a transformation chain through appropriate automated re-evaluation of transformations within the transformation chain.

Although the example transformation chain 400 includes only two links, the transformation chains can be quite complex and involve innumerable nodes and associated links connecting the innumerable nodes. The principles described here can work regardless of the complexity of the transformation chains.

Figure 5 illustrates a room 500 where there are six devices 501 to 506. The ellipses represent 507 flexibility in the number of devices that are present in room 500. In fact, there may be devices that go out and enter room 500 very dynamically . Although not necessary, each of the devices 507 can be structured as described for the computer system 100 of Figure 1.

Environment 500 need not be a physical environment in which all devices are located in the same proximity, although this is often the case. Environment 500 can be considered as any environment in which there is a group of devices through which one or more users can provide input and multiple devices through which multiple users can be provided output.

Some of the devices (eg 501 to 504 devices) help in the formation of what is just more 5 cooperating devices with cooperating applications. Instead, devices 501 to 504 each are related to a component of a compounding application. As any of the devices 501 through 504 leaves environment 500, the compounding application becomes smaller, thus resulting in changed application functionality. On the other hand, since devices (such as 501 to 504 devices) allow application components to enter the 500 environment, the composite application actually gets larger, which actually changes the functionality and functionality. very structure of the compound application. In accordance with the principles described herein, the transformation chain of one device can be linked with the transformation chain of other devices, resulting in a large transformation chain that can more efficiently utilize the increased pool of devices.

Device 505 represents an output device that can be used in environment 500, but does not necessarily contribute a transformation chain to the larger transformation chain a compounding application. For example, device 505 could be a large screen display. Device 506 represents an input device that can be used in environment 500, but does not necessarily contribute a transformation chain to the larger transformation chain of compounding. For example, device 506 could be a microphone. The presence of devices 501 to 504 causes portions of the respective associated transformation chains to contribute compounding to the larger transformation chain. However, devices 501 to 504 may also have input and output capabilities that can be utilized by the compounding application as a whole. Environment 500 can optionally include an external system 510, described below.

Figures 6A through 6D illustrate exemplary transform chain instances or classes 600A through 600D. The instances will have the same structure as the classes, and in this way the illustrated shapes are considered to represent the transformation classes as well as the transformation instances. However, the instances will have a particular instance state associated with each of one or more of the nodes in the transformation chain. Consequently, elements 600A to 600D can be referred to as transformation chain classes or transformation chain instances. The term transform chain will generally be used to refer to transform chain classes as their associated transform chain instances. As an example, transform chain instances 600A through 600D could be associated with respective devices 501 through 504.

The illustrative transformation chains 600A to 600D are relatively simple in order to avoid obscuring the general principles described here with an overly complex example. That said, the principles described here apply regardless of the complexity of the transform chain, and regardless of the number of transform chains and associated devices that are within the environment and make up the composite application.

In the notation of Figures 6A to 6D, nodes belonging to transformation class 600N (where N runs from A to D) are represented using the suffix N. For example, in Figure 6A, transformation chain 600A includes nodes 601A, 602A, 603A and 604A. The remaining elements 601B, 601C and 601D do not end with the suffix A, and therefore are not nodes within the transformation chain 600A. Instead, items 601B, 601C, and 601D represent the dependencies on other transform chains.

Throughout Figures 6A to 6D, 7A to 7D, 8A to 8C and 9, to highlight those elements that are dependency elements, rather than nodes in the same transformation chain, the dependency elements are represented with limits of dotted lines. Data does not flow from a node to a dependency element, unless the transform chain is joined by another transform chain that includes a node represented by the dependency element. The fact that data cannot flow through a particular transformation is represented through the figures by the link that is marked with an X.

For example, element 601B in transform chain 600A represents a dependency on node 601B in transform chain 600B. The dependency element 601B is bordered with dotted lines and all links leading to or from that dependency element 601B are marked with an X, since at this stage, the transformation chain 600A is not linked to the transformation chain 600B. Element 601C in transformation chain 600A represents a dependency on node 601C in transformation chain 600C. Element 601D in transform chain 600A represents a dependency on node 601D in transform chain class 600D.

For its part, the transform chain instance 600A can function as an application. For example, the value of data source 601A can be used to form a transformed result as the value of data target 604A. In addition, the values ​​of data sources 601A and 602 A can be transformed to produce the data target value 603A. If the transform chain instance 600A is on its own, the transforms leading to and from elements 601B, 601C, and 601D are not evaluated.

The transformation chain 600B includes three nodes 601B, 602B, and 603B. However, transform chain 600B also includes dependency elements 601A, 602A, 601C, and 603C that refer to a node in a different transform chain. Again, the transform chain instance 600B can function independently as a single application. For example, the value from data source 601B can be provided through a transformation to generate the resulting value for the data target

602Β. The 602B data source value can be provided through a transformation to generate the resulting value for the 603B data target.

Although the transform chain instances 600A and 600B can operate independently, Figure 7A shows a linked transform chain 700A, which includes transform chain 600A linked to transform chain 600B. The dependency elements in each of the transformation chains are now replaced with the actual node mentioned. For example, dependency element 601B of Figure 6A is now node 601B, and dependency element 601A of Figure 6B is now node 601A. All nodes that have the suffix A or B are nodes within the transformation chain 700A, and only nodes that have the suffixes C or D are dependency elements. For example, nodes 601A, 602A, 603A, 604A, 601B, 602B, and 603B are nodes within the augmented transform chain 700A, and the functionality of the composite application becomes somewhat better than the sum of the functionality of the individual transformation chains 600A and 600B as such.

The transformation chain 600C includes three nodes 601C, 602C and 603C. However, the transformation chain 600C also includes dependency elements 603A, 601B and 603B that refer to a node in a different transformation chain. Again, the transform chain instance 600C can function independently as a single application. For example, the value for data source 601C can be provided by a transformation to generate the resulting value for data target 602C. Likewise, the value from data source 601C can also be provided through a transformation to generate the resulting value for data target 6030.

Although the transform chain instances 600A and 600C can operate independently, Figure 7B shows a linked transform chain 700B that includes transform chain 600A linked to transform chain 600C. The dependency elements in each of the transformation chains are now replaced with the current node refers to the extent to which the dependency element refers to a node within any of the transformation chains 600A or 600C. Now all nodes that have the suffix A or C are nodes within the transformation chain, and only those nodes that have the suffixes B or D are dependency elements. For example, nodes 601A, 602A, 603A, 604A, 601C, 602C, and 603C are nodes within the augmented transform chain 700B. The functionality of the composite application becomes better than the sum of the capabilities of the individual 600A and 600C transform chain instances.

Figure 7C shows a linked transformation chain 700C including the transformation chain class 600B linked with the transformation chain class 600C. The dependency elements in each of the transformation chains are replaced with the actual node contemplated to the extent that the dependency element refers to a node within any of the transformation chains 600B or 600C.

Now all nodes that have the suffix B or C are nodes within the transformation chain, and only those nodes that have suffixes A or D are dependency elements. For example, nodes 601B, 602B, 603B, 601C, 602C, and 603C are nodes within the 700C augmented transform chain and the functionality of the composite application becomes better than the sum of the functionalities of the chain instances. individual transformation 600B and 600C.

Figure 8A shows a linked transformation chain 800A, which includes transformation chains 600A, 600B and 600C also linked. The dependency elements in each of the transformation chains are replaced with the current node referred to to the extent that the dependency element refers to a node within any of the transformation chains 600A, 600B or 600C. Note that all the illustrated nodes are actually nodes in the transformation chain, except in the case of dependency element 601D. The functionality of the compounding application becomes better than the sum of the functionality of the individual transform chains 600A, 600B and 600C.

Transform chain 600D includes two nodes 601D and 602 D. However, transform chain 600D also includes a single dependency element 603A that references a node in a different transform chain class 600A. Again, instances of the 600D transform string class can function independently as a single application. For example, the value for data source 601D can be provided by a transformation to generate the resulting value for data target 602D.

Although the transformation chain instances 600A and 600D can operate independently, Figure 7D shows a joined transformation chain 700D that includes the transformation chain 600A joined with the transformation chain 600D. The dependency elements in each of the transformation chains are now replaced with the actual referenced node to the extent that the dependency element refers to a node within any of the transformation chains 600A or 600D. Now all the nodes that have the suffix A or D are nodes within the transformation chain, and only those nodes that have the suffixes B or C are dependency elements. For example, nodes 601 A, 602A, 603A, 604A, 601D, and 602D are nodes within the augmented transform chain 700D and the functionality of the compounding application becomes somewhat better than the sum of the functions of the chain of individual transformation 600A and 600D.

Note that Figures 7A to 7D illustrate all possible permutations involving two and only two of the transformation chains 600A, 600B, 600C and 600D. Transform chains 600B and 600D are not directly linked in a combination of two transform chains, as neither transform chain has a dependency element referring to a node in the other transform chain. Also, transformation 600C and 600D are not directly linked in a combination of two transformation chains, as neither has a dependency reference to the other.

Figure 8 illustrates one of three possible combinations of three and only three transformation chains 600A, 600B, 600C and 600D. In particular, Figure 8A shows an augmented transformation chain 800A combining the transformation chains 600A, 600B and 600C. Figure 8B shows an augmented transformation chain 800B combining transformation chains 600A, 600B, and 600D (in which all nodes are part of the transformation chain, except dependency elements 601C and 603C). Figure 8C shows an augmented transformation chain 800C combining transformation chains 600A, 600C and 600D (in which all nodes are part of the transformation chain, except dependency elements 601B and 603B). Note that there is no combination of transformation chains 600B, 600C and 600D illustrated since the transformation chain 600D does not include dependency references to the transformation chain 600B (or vice versa), or to the transformation chain 600C (or vice versa) . Figure 9 illustrates a combined transformation chain 900 that includes all transformation chains 600A, 600B, 6000 and 600D combined.

Therefore, given the transformation chains 600A, 600B, 600C and 600D associated with their respective devices 501 to 504 in the environment, there are 8 possible compound applications that can be formed (corresponding to the transformation chains of Figures 7A to 7D , Figures 8A to 80, and Figure 9). Thus, as the transformation chains of various devices are joined in and decoupled from the environment, the transformation chain itself changes, and the structure of the composite application in this way changes. For example, a change in the value of data source 601A could have a very different impact on the transformation chain as the effects of that change are automatically propagated through one or more transformations, depending on whether the data source 601A is within the 600a transformation chain alone, within the 700A transformation chain, within the 700B transformation chain, within the 700D transformation chain, within the 800A transformation chain, within the 800B transformation chain, within the 800C transformation chain, or within the 900 transformation chain.

As an example, assume that device 501 first enters the resulting environment in the transformation chain 600A used. Device 502 then enters the environment resulting in transformation chain 600B joining transformation chain 600A, resulting in transformation chain 700A. Thus, the operation transformation chain changes from transformation chain 600A to transformation chain 700A. Now suppose that third device 503 enters the environment resulting in transformation chain 600C joining transformation chain 700A, thus resulting in transformation chain 800A. Therefore, the operation transformation chain changes from the 700A transformation chain to the 800A transformation chain. Now suppose that device 500B exits. The transformation chain 700B would then become operational. Now suppose that device 504 enters the environment resulting in transformation chain 600D joining the transformation chain

700Β, resulting in transformation chain 8000 becoming the operation transformation chain. Now suppose that device 500C drops out, resulting in transformation chain 700D being operational. Now suppose that device 501 drops out, resulting in transformation chain 600D being operational.

Finally, device 504 exits, shutting down the transformation chain in the environment. In this scenario, the operational application structure changed (and therefore functionality changes) seven times as follows: 1) starting with transformation chain 600A, 2) next, changing to transformation chain 700A, 3) next, changing to transformation chain 800A, 4) next, changing to transformation chain 700B, 5 ) next, changing to transformation chain 800C, 6) next, changing to transformation chain 700D, 7) next, ending with transformation chain 600D.

Figure 10 shows a flow chart of a method 1000 to prepare for the facilitation of a compound application represented by multiple linked transformation chain instances.

Method 1000 can be performed by any of devices 501 to 507 that are within environment 500. Alternatively, or in addition, method 1000 can be performed external to devices 501 to 507, such as external system 510 of Figure 5 For example, external system 510 could be a service provided in a cloud computing environment. Regardless of where method 1000 is performed, Figure 11 illustrates a system 1100 that can perform method 1000. Consequently, regardless of whether system 1100 is embedded in one or more of devices 501 through 507 or if the System 1100 is an external system 510 of Figure 5, or combinations thereof, method 1000 will now be described with frequent reference to system 1100 of Figure 11.

Method 1000 includes two acts 1001 and 1002 that do not have temporal dependencies. For example, the method includes identifying dependencies between different transformation chain classes (act 1001). These dependencies are essentially authorizations created by the author of the class who generated the class in order to allow instances of different transform chain classes to interoperate (subject to subsequent instance-based approval).

For example, system 1100 includes a logic component 1101 and a transformation class definition library 1110 which is illustrated as including six transformation string class definitions 1111 to 1116. However, the ellipses 1117 represent that the Transform string class definition library 1110 can include any number of transform string class definitions. As an example, suppose that transformation chain class definitions 1111, 1112, 1113 and 1114, respectively, define transformation chain classes 600A, 600B, 600C and 600D of Figures 6A to 6D. Since each transform chain class defines dependency elements, the system 1100 can use library 1110 to identify dependencies between the different transform chain classes.

For example, by looking at dependency element 601B in transform chain 600A, and dependency element 601A in transform chain 600B, logic component 1101 can identify a dependency between nodes 601A and 601B that would exist if the chain classes of transformation 600A and 600B were joined, or if the instances of the class were joined. The logic component 1101 can infer that the authors of the transformation chains 600A and 600B both consent to authorize their union (assuming that users also authorize the instance level at run time) on these dependency elements.

Method 1000 also includes identifying (act 1002) devices that are associated with instances of any of the transform chain classes. The devices can themselves identify the 1100 system perhaps with the device identifier, as well as potentially any kind of transform string that the device wants to use. Alternatively, the system 1100 may have a particular default transform class associated with each device and therefore perhaps receive the device identifier. There may be some transform chain classes that a device is authorized to use and other transform chain classes that the device is not authorized to use.

For example, system 1100 also includes device records 1120 that include device records as associated with various transform string classes. For example, device records 1120 include five device records 1121 through 1125, although ellipses 1126 represent that device records 1120 can include any number of device records. In addition, the device records 1120 may be persisted for a long time and / or may only be kept for a short time. Regardless, the system 1100 identifies multiple devices and finds the associated transform chains for those devices. Just as an example, perhaps devices 501 through 504 in Figure 5 are associated with registers 1121 through 1124. By that association, the system 1100 is able to detect that the transform string instances 600A, 600B, 600C, 600D (respectively, defined by the transform string class definitions 1111, 1112, 1113, and 1114) are, respectively, associated with these devices 501 to 504.

In some embodiments, the system 1100 itself executes a representation of an instance of the particular transform string class on behalf of the corresponding device. For example, by identifying that transform string class 600A is associated with device 501, system 1100 may operate a representation of transform string instance 600A for device 501. Accordingly, as inputs are received from the device, the system 1100 identifies the change with a particular node in the transform chain, and propagates the chains along the transform chain. This is known as a practical approach. This practical approach has some advantages in that the processing can be downloaded to devices or systems, or clouds that have greater processing capacity than the device that is providing the input itself.

In another aspect referred to here as the hands-free approach, the device is associated with the transform class itself by executing the instance of the class. When a dependency encounters another transform chain associated with another device, data can flow directly (for example, through the peer-to-peer network) to that device if that other device is also docked in the hands-free approach. Alternatively, the device could indirectly flow the data to the other device (for example, through an external system or a cloud). If the external system 510 is executing the rendering of the transform chain class instance for that other device, the device may instead be able to flow the data into the external system 510 for further propagation to the rest of the transformation chain.

Figure 12 shows a flow diagram of a method 1200 to join two instances of transformation classes. First, the transformation chain classes are confirmed to be of the type that can be joined (act 1201). For example, if the transform chain classes do not refer to each other, then perhaps there is no class-based authorization to join the instances of the respective class. For example, the transformation chain class 600D cannot be bound to the transformation chain class 600B or the transformation chain class 600C, because the transformation chain class 600D does not refer to elements in the transformation class. transformation chain 600B or 600C. However, the transformation chain class 600D can be joined with the transformation chain class 600A because they mutually contain dependency references to each other.

In this example, however, although transformation chain class 600D cannot be directly linked to transformation chain classes 600B and 600C, transformation chain class 600D can be linked with transformation chains 700A, 700B, and 800A although transformation chains 700A, 700B and 800A include one or both transformation chains 600B and 600C.

However, in an alternative mode, the author of a transformation chain class can specify additional restrictions on the union of other transformation chain classes. For example, an author might state a general restriction that the union of a certain transformation class instance is not allowed if the transformation class instance is already a compound transformation chain and / or if the compound transformation chain has a particular constituent transformation chain. For example, when two transform chain instances are joined, the system 1100 can keep track of the transform class identities that were used to build the composite transform chain up to that point. That list can be used to confirm whether the conditions for class-based authorization have been met.

The author could also express restrictions on the granularity of a single dependency. For example, in dependency element 601B of transformation chain class 600A, the author could state that the join is authorized in that dependency element only if the transformation chain into which it was joined does not include a chain class of identified transformation created by a competitor. The author could also control the data that went out of the transformation chain to another joined transformation chain by writing constraints or conditions in the transformation that could join the dependency together (for example, between nodes 601A and dependency element 601B).

However, although the transform chain class can interoperate, that does not mean that the user wants their particular instance of that transform chain class to be joined with other instances of other transform chain classes. After all, the data itself (for example, the state of the instance) can be user-sensitive. Consequently, method 1200 also includes the determination that instances of different transformation chain classes were joined (act 1202).

The join criteria for authorizing two instances of different transform chain classes to join can include one or more of the following: whether or not the user is on a list of meeting attendees, a relationship (for example, family, friends, social networks or similar) of the users of the different devices, a communication capacity (for example, near field) between the devices, a proximity of the respective devices (for example, in the same conference room), the request of the users of the same.

For example, the join criteria could include some business criteria such as the associated users of the instances are on the same computer. As another example, a device could be a kiosk in a commercial space or a hotel, where a cli uses the kiosk and a shop assistant or the concierge can automatically use their device to join their transformation chain with that of the kiosk with in order to interact well with the client using the composite application. Conditions can be applied to join criteria. For example, a pushbutton device might be able to join a customer's application if the concierge does not round (perhaps detected by the concierge is not actively using the application to join customers, or they are off the network).

Unless the instance of the transform chain class associated with the device starts in a default state defined by the class, in the practical approach, the device can then load the instance data (for example, the values ​​in each one of the nodes of the transformation chain associated with the device) to system 1100 (act 1203). In the hands-free approach, perhaps the instance data only at the boundary nodes is loaded into the 1100 system.

The transform chain instance is then joined with the rest of the transform chain (act 1204). For example, this can be achieved by searching the source transform chain instance for dependency elements that correspond to the nodes in the target transform chain instance. Once said dependency element is found, that dependency element is replaced with the actual identified node in the target transformation chain instance, thus creating a real link between source and target transformation chain instances. This repeats for all dependency items found in the source transform chain instance that identifies a node in the target transform chain instance. If there are dependency items found in the source transform chain instance that do not correspond to a node in the target transform chain instance, then the dependency items remain dependency chain items in the transform chain instance combined.

Note that in the examples of Figures 6A to 6D, none of the transformation chain classes 600A to 600D includes dependency elements that refer to the same class. However, the principles described here can work even if an instance of a particular transform chain class may have a dependency element referring to another node in another instance of the same particular transform chain class. Also, if allowed by a particular node it is joined to, multiple nodes from one or more different transform chain instances can join the particular node.

Once joined, the transformation chain grows, and the data flows into the augmented transformation chain (act 1205), as if the transformation chain was originally created in augmented form in the first place. For example, when an instance of the transformation chain class 600A is joined to an instance of the transformation chain class 600B, the result is a single instance of the transformation chain class 700A. Once an instance of the transformation chain class 600A is joined to an instance of the transformation chain class 6000, the result is a single instance of the transformation chain class 700B. Once an instance of the transformation chain class 600B is joined to an instance of the transformation chain class 6000, the result is a single instance of the transformation chain class 700C. Once an instance of the transformation chain class 600A is joined to an instance of the transformation chain class 600D, the result is a single instance of the transformation chain class 700D.

An instance of the transformation chain class 800A can be formed by either joining instances of the transformation chain classes 700A and 6000, or by joining instances of the transformation chain class 600A and 7000. An instance of the class of the transform chain 800B can be formed by joining instances of transform chain classes 700A and 600D. An instance of the transformation chain class 800C can be formed by joining together instances of the transformation chain classes 700B and 600D. This join operation can be done by joining many times to create a large transformation chain in situations where there are many devices available for collaboration in a given environment.

Consequently, once joined (act 1204), data can flow freely (act 1205) within the augmented transformation chain, even across what used to be the boundaries between constituent transformation chains. For example, in transform chain 700A of Figure 7A, data can flow freely (without the user of an application programming interface and without function calls) from node 601A to node 601B (via transform ( es) corresponding precisely how data can flow from node 601A to node 604A (via corresponding transformation (s)).

Consequently, the system can be used to link transformations associated with a wide variety of devices in a relatively convenient and automated way to increase the transformation chain of a compound application that is shared among multiple devices. This allows for a wide variety of scenarios.

For example, consider people arriving for a meeting in a conference room. When the first user to enter the conference room with a laptop and a smartphone, a transformation chain instance associated with the smartphone is joined to the laptop transformation chain instance in such a way that a single application It is running effectively through the early smartphone and laptop users. A second user enters the conference room at a later time. The second user is also invited to the meeting, and in order for the user's smartphone transformation chain to join the existing composite transformation chain to further increase the transformation chain. A television screen then automatically turns on the output of some of the data associated with the board. A third user comes in with a tablet-type computer and is also a meeting guest. Thus, the transformation chain associated with the tablet computer joins the compound transformation chain to further augment the transformation chain. Thus, as more devices join an environment suitable for joining transformation chains, the application that is running on all devices is in augmented reality, thus changing the functionality of the application itself. Several users can share in the environment in such a way that multiple users are observing each one of at least one or some of the devices.

Figure 13 shows a flow chart of a method 1300 for decoupling transform chains. Since there is a method to join transform chains when devices enter an environment, there is a method to decouple transform chains when devices leave the environment. First, the system determines that an instance of a transformation chain class can be decoupled (act 1301). In response, the flow of data is interrupted to and from that instance (act 1302). The undocking criteria used to determine that the instance should be undocked could include any criteria, but as an example, they can include one or more of a proximity of the device associated with the instance with other devices in the environment, a meeting completion state , a communication capability between the device associated with the instance and other devices in the environment, the user's request, and so on. Thus, transformation chains can change dynamically as devices enter and leave the environment.

In one embodiment, system 1100 includes a presentation service 1150. One or more nodes of the executing transform chain instance can conduct rendering on one or more surrounding devices. Also, one or more devices can provide input to one or more nodes in the transform chain instance. For example, suppose that a transform chain instance 900 runs in environment 500 in Figure 5, and that devices 501 through 504 are associated with the respective transform chain class instances 600A through 600D, thus resulting in the instance of the augmented transform class 900. Referring to Figure 9, assume that nodes 601A and 601C are input nodes and nodes 604A, 603B, 602 C, and 602D are output nodes. Presentation service 1150 can select which devices can provide input to nodes 601A and 601C, and if so, which transformations will be performed. In one example, the devices are associated with their respective portions of their transform chains such that the input nodes within that respective portion are provided by the corresponding device. Thus, by default, input node 601A, which was originally part of transform chain 600A, could be provided with input from device 501 (after potentially some transform (s)). Also, in the default case, input node 601C, which was originally part of the transform chain

600C, can be provided with input from device 503 (after potentially some transformation (s)). The 1150 presentation service could also select the devices to render the output from nodes 604A, 603B, 602C and 602D, and what 5 transformations (if any) should be performed.

Figure 14 shows a flow chart of a method 1400 for outputting a transform chain in a multi-device environment. The method 1400 can be performed by the system 1100 of Figure 11, or perhaps by the presentation service 1150 of Figure 11. The logic component 1101 determines (act 1401) that a certain node (presentation node) of an instance of a transform chain class has changed. For example, in the example, nodes 604A, 603B, 602C, and 602D are output nodes, or presentation nodes.

The logic component also identifies characteristics (act 1402) of various candidate devices in which at least one version of that changed output can be displayed. For example, logic component 1101 may query device register 1120 and / or inquire about devices directly in order to determine the rendering capacity of each device. The characteristics of each device could also include a juxtaposition of each user with respect to the device.

Next, the system selects (act 1403) at least one of the candidate devices in which at least the version of the representation is represented according to the identified characteristics. The system makes the selection in order to maximize the use of the output. For example, if the output is intended for all users, and all users are physically present, the output could be rendered on a larger screen. If the output is interactive and requires input from a particular device 5, it is possible that the device will be selected for output.

The system determines whether any transformations are applied (decision block 1404) for rendering prior to actual device output. The transformations that take into consideration the suitability of the selected device (s), where the transformations can vary depending on the device (s) selected for representation. Examples of transformations include changing a representation type (for example, text-to-speech, speech-to-text, video-to-text, text-to-video, and so on). Transformations can also include cinematization of the output. For example, a video can be created from input, in which a backdrop is provided and perhaps changed accordingly, and moving elements move in and out of the video.

If transformations are to be carried out (Yes in decision block 1404), then those transformations are carried out (act 1405). Finally, the representation is provided in the selected device (act 1406). Consequently, the compound transform chain can take the input from any device in any form, and transform the input, if necessary, into a form that is recognized by an input node of the compound transform chain. Thus, this 25 input is propagated throughout the transformation chain. If the values ​​of one or more presentation nodes change, an appropriate device can be selected to represent the output. Consequently, the most suitable device for rendering can be selected regardless of which device was associated with the portion of the transform chain that contains the display node. For example, changes in the value of the display node 604A need not be represented exclusively, or even across the entire device 501 that corresponds to the transform chain instance 600A. In various modalities, representation can involve converting information to some physical action that includes some type of movement. For example, the representation may include opening or closing a door, opening or closing a valve, unlocking or locking a door, turning a television on or off, and so on.

A detailed scenario (hereinafter referred to as the device scenario) will be described with respect to Figures 15A to 15M (collectively referred to as Figure 15). This scenario is provided just like an infinite variety of scenarios that are triggered by the general principles outlined in this document. In this scenario, a user is ordering different computing devices. The scenario involves four devices 1501, 1502, 1503, and 1504, although the scenario begins in Figure 15A with only two devices 1501 and 1502 being visible. Each of the four devices 1501 to 1504 participates in providing input and receiving output for a compound application.

Figure 15A introduces two devices 1501 and 1502 that are laptop computers. Assume that these two devices 1501 and 1502 are initially the only devices in the operating environment, and therefore their respective transformation chains have been merged. The transform chain associated with device 1501 allows navigation and selection of a manufacturer. The transform chain associated with device 1502 allows navigation of a list of devices offered by the manufacturer, and also allows the user to change a count of any of the devices in the list.

More specifically, device 1501 allows the user to scroll horizontally through a series of manufacturers, and select a manufacturer. In Figure 15A, device 1501 displays only three of those manufacturers labeled Vendor 7, Vendor 8, and Vendor 9, with the user having selected Vendor 8. On the other hand, device 1502 displays a horizontally scrollable list of devices that are provided by the selected vendor on device 1501. In Figure 15A, since Vendor 8 is selected on device 1501, device 1502 displays a list of Vendor provided devices 8. Each item in the device list includes an image 1505 of the device, a name 1506 of the device, a cost per unit 1507 of the device, and an account 1508 of the device being ordered. The account for each device can be controlled by a corresponding scroll control.

Figure 15B shows a scenario state that has progressed from the state of Figure 15A, in which the user has scrolled to the left the list of manufacturers and also selected Vendor 4. Consequently, the output from device 1502 automatically changes to display a list of the device manufactured by Vendor 4. To do so, data automatically flowed (without the use of an application program interface) from the transform chain associated with device 1501 to the transform chain associated with device 1502.

Figure 15C shows a state of the scenario that has progressed from the state of Figure 15B, in which the user has scrolled to the left in the list of manufacturers and also Vendor 2 selected. Consequently, the output from device 1502 automatically changes to display a list of devices manufactured by Vendor 2. By doing so, the data automatically flows back from the transform chain associated with device 1501 to the transform chain associated with device 1502.

Figure 15D shows a state of the scenario that has progressed from the state of Figure 15C, in which the user has used the scroll controls 1511 and 1512 displayed on the device 1502 in order to change an order count for two devices. offered by the Seller 2. Figure 15E shows a state of the scenario that has progressed from the state of Figure 15D, in which the user used device 1502 to scroll right through the list of devices offered by Vendor 2 and also uses the controls Scroll 1513 and 1514 presented on device 1502 in order to change an order account for two additional devices offered by Vendor 2. Therefore, at this stage, the user has entered memorandum accounts for four devices offered by Seller 2.

Figure 15F shows a stage state that has progressed from the state of Figure 15E, in which a third device 1503 is introduced into the environment. For example, the third device 1503 could be associated with a transform chain that shows a maximum unit cost and a total cost for the entire order. Once device 1503 is introduced into the environment, that portion of the transform chain is joined with the existing compound transform chain, thus changing the functionality of the compound application, to now send such cost data to device 1503 . Once the transform chain instance is attached, the data representing the accounts, and unit costs flows to that portion of the transform chain, causing device 1503 to be immediately populated. Note that the flows also occur in the opposite direction, as device 1503 indicates a maximum unit price, and therefore the list of devices displayed on device 1502 is restricted to any device below the maximum unit price. In this case, all of the devices listed above are below the maximum unit price, and therefore there is no change in the displays on device 1502.

Figure 15G shows a scenario state that has progressed from the state of Figure 15F, in which the user has used scroll control 1515 on device 1502 to enter an account for another device offered by Vendor 2. The Account data and unit cost data flow in the transformation chain portion to the corresponding device 1503, which results in a change in the total cost displayed on device 1503.

Figure 15H shows a scenario state that has progressed from the state of Figure 15G, in which the user has adjusted down from the maximum unit price to $ 987 using scroll control 1516 on device 1503. That maximum unit price change has derived from the transform chain portion associated with device 1503 to the transform chain portion associated with device 1502, causing various devices offered by Vendor 2 to be removed from the list. If the user returns the higher amount, the removed items will be automatically added back to the list, perhaps returning device 1502 to the state illustrated in Figure 15G.

Figure 151 shows a stage state that has progressed from the state of Figure 15G, in which a fourth device 1504 is introduced into the environment. For example, the fourth device 1504 could be associated with a transform chain that displays a maximum device weight. Once device 1504 is introduced into the environment, that portion of the transformation chain is joined with the existing composite transformation chain, thus changing the functionality of the composite application, to now output such cost data. Once the transform chain instance is joined, the data representing the maximum weight flows from the portion of the transform chain associated with device 1504 to the portion of the transform chain associated with device 1502. Device 1502 It responds by displaying overweight warnings 1518A, 1518B and 1518C and 1518D associated with each device that is heavier than the maximum weight. Currently, in Figure 151, the weight is only 606 grams, and therefore all devices are listed with such a warning.

Figure 15J shows a stage state that has progressed from the state of Figure 151, in which the user has used the displacement control 1516 on the device 1504 in order to increase the maximum weight to more than 2 kilograms. The maximum weight change is caused to flow from the portion of the transform chain associated with device 1504 to the portion of the transform chain corresponding to device 1502, resulting in the extraction of overweight warnings 1518C and 1518D (overweight warnings 1518A and 1518 B remain).

That concludes the scenario from the user's perspective. Figures 15K through 150 illustrate that at edit time, the author can declaratively change the transformations to change the functionality of the composite application. In Figure 15K, user interface element 1520 is illustrated as including a declarative transformation that filters out devices that have a manufacturer's suggested retail price that is less than the value indicated by slider 1516. Figure 15L demonstrates that this restriction has been removed and therefore the currently displayed devices are not filtered by the value of slider 1516.

In Figure 15M, the user interface element 1521 is illustrated as defining a highlight color used for the manufacturer's suggested retail price. Figure 15N illustrates that the highlight color is now dependent on whether or not the manufacturer's suggested retail price is higher than the value indicated by slider 1516. If so, the highlight color changes to, for example, green . Figure 150 shows that, as slider 1516 is incremented in this situation, the behavior of the compounding application has changed from a state in which devices are removed from the list if their suggested manufacturer retail price is higher than the value of a 1516 slider, to a state in which the manufacturer's suggested retail price (compared to the value of the 1516 slider) triggers the highlighting of the 15 manufacturer's suggested retail price.

Accordingly, the rendering of transform string output has been described in devices, such that a change in a value in the display node causes rendering to occur. In response to a change in the value of the presentation node, the presentation system identifies features or various devices that can be used to represent the corresponding output. The display system identifies an appropriate device based on the identified characteristics of that device. The display system then facilitates rendering of at least one version of the output in the selected device, perhaps still transforming the output into a form suitable for the selected device. In some embodiments, the transformation chain may be a compound transformation chain constructed by joining multiple constituent transformation chains.

The present invention can be embodied in other specific ways, without departing from its spirit or essential characteristics. The modalities described will be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and equivalence scale of the claims are encompassed within their scope.