Method and system for automatically determining differences in a user interface throughout a development cycle

<u>Cross-Reference to Related Application</u>

The present invention is related to a patent application entitled "Method and System for Masking Dynamic Regions in a Mobile Device User Interface to Enable Testing of User Interface Consistency," filed concurrently with this application. The related application is assigned to the assignee of the present patent application and hereby incorporated by reference.

<u>Background of the Invention</u>

Complexity of mobile devices such as cellular phones, personal data assistants (PDAs), handheld computers, and others, continues to increase. The market for such devices is steadily becoming a worldwide market, requiring the devices to provide content in a number of different languages and on a number of different platforms. Each different platform may further include a variety of options that result in different user interface outputs depending on the options selected.

The user interface of the device provides the mechanism through which the content of an application on the device is displayed to a user. A user interface provides a set of commands or menus through which a user communicates with a program. A command-driven interface is one in which you enter commands. A menu-driven interface is one in which you select command choices from various menus displayed on the screen.

The user interface is one of the most important parts of any program because it determines how easily you can make the program do what you want. A powerful program with a poorly designed user interface has little value. Graphical user interfaces (GUIs) that use windows, icons, and pop-up menus have become standard on personal computers. Accordingly, for the variety of languages that exist, the user interface is adapted to provide the content and commands in the language required. In addition, a matrix of other differences may exist between user interfaces. For example, a user interface may be displayed in portrait or landscape, in different resolutions, with or without a software input panel (SIP), and with a host of other variables. What is needed is a method for automatically verifying that a user interface is consistent in its display of content despite a change in the language, platform, or other display variables that may be present for a particular user interface output.

<u>Summary of the Invention</u>

The present invention provides a method and system for automatically determining the differences in a mobile device user interface throughout a development cycle. The present invention involves running a test automation on a target device to exercise that target device's user interface. While the user interface is exercised, an extension is activated that takes snapshots of the user interface output automation execution while it is in selected states. The snapshots are stored and uploaded to a server where snapshots of different automation executions are then compared on a pixel-by-pixel basis. The differences between snapshots are output in a visual format where the pixels that are different between snapshots are highlighted. The visual format of the differences between snapshots provides the user with a method to readily determine if errors exist in the user interface of the target device. Since the differences between snapshots are analyzed on a pixel-by-pixel basis, the present invention operates for devices operating under any language and on any platform.

Along with the automatic comparison of the snapshots, a robust system of data manipulation is provided to assist in the analysis of the user interface throughout a development cycle. In testing the user interface for accuracy, the tester is presented with a number of options for analysis of the data, including but not limited to, a fast test of the accuracy of the snapshots between automation executions that returns a pass/fail variable. Other options are available and other embodiments of the present invention are presented in the figures and detailed description provided below.

<u>Brief Description of the Drawings</u>

<ul id="ul0001" list-style="none" compact="compact"><li>FIGURE 1 illustrates an exemplary computing device that may be used in one exemplary embodiment of the present invention.</li><li>FIGURE 2 illustrates an exemplary mobile device that may be used in one exemplary embodiment of the present invention.</li><li>FIGURE 3 illustrates exemplary snapshots in accordance with the present invention.</li><li>FIGURE 4 illustrates an exemplary function block diagram of a system for automatically determining differences in a user interface in accordance with the present invention.</li><li>FIGURE 5 illustrates an exemplary functional block diagram of a database used in the system of FIGURE 4 in accordance with the present invention.</li><li>FIGURE 6 illustrates a logical flow diagram for a client side process of automatically determining differences in a user interface in accordance with the present invention.</li><li>FIGURE 7 illustrates a logical flow diagram for an upload process of automatically determining differences in a user interface in accordance with the present invention.</li><li>FIGURE 8 illustrates a logical flow diagram for a server side process of automatically determining differences in a user interface in accordance with the present invention.</li></ul>

<u>Detailed Description</u>

The present invention now will be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments for practicing the invention. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Among other things, the present invention may be embodied as methods or devices. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.

Illustrative Operating Environment

FIGURE 1 shows an exemplary computing device that may be included in system <b>100</b> for implementing the invention. Computing device <b>100</b> illustrates a general operating environment that may apply to the present invention. In a very basic configuration, computing device <b>100</b> typically includes at least one processing unit <b>102</b> and system memory <b>104</b>. Processing unit <b>102</b> includes existing physical processors, those in design, multiple processors acting together, virtual processors, and any other device or software program capable of interpreting binary executable instructions. Depending on the exact configuration and type of computing device, the system memory <b>104</b> may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. System memory <b>104</b> typically includes an operating system <b>105</b>, one or more program modules <b>106</b>, and may include program data <b>107.</b> This basic configuration is illustrated in FIGURE 1 by those components within dashed line <b>108</b>.

Computing device <b>100</b> may also have additional features or functionality. For example, computing device 100 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIGURE 1 by removable storage <b>109</b> and non-removable storage <b>110</b>. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. System memory <b>104,</b> removable storage <b>109</b> and non-removable storage <b>110</b> are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device <b>100.</b> Any such computer storage media may be part of computing device <b>100</b>. Computing device <b>100</b> may also have input device(s) <b>112</b> such as keyboard, mouse, pen, stylus, voice input device, touch input device, etc. Output device(s) <b>114</b> such as a display, speakers, printer, etc. may also be included. All these devices are known in the art and need not be discussed at length here.

Computing device <b>100</b> may also contain communications connection(s) <b>116</b> that allow the device to communicate with other computing devices <b>118</b>, such as over a network. Communications connection(s) <b>116</b> is an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.

FIGURE 2 shows an alternative operating environment for a mobile device substantially for use in the present invention. In one embodiment of the present invention, mobile device <b>200</b> is integrated with a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

In this embodiment, mobile device <b>200</b> has a processor <b>260</b>, a memory <b>262</b>, a display <b>228</b>, and a keypad <b>232.</b> Memory <b>262</b> generally includes both volatile memory (e.g., RAM) and non-volatile memory (e.g., ROM, Flash Memory, or the like). Mobile device <b>200</b> includes an operating system <b>264</b>, which is resident in memory <b>262</b> and executes on processor <b>260.</b> Keypad <b>232</b> may be a push button numeric dialing pad (such as on a typical telephone), a multi-key keyboard (such as a conventional keyboard), or may be not be included in the mobile device in deference to a touch screen or stylus. Display <b>228</b> may be a liquid crystal display, or any other type of display commonly used in mobile computing devices. Display <b>228</b> may be touch-sensitive, and would then also act as an input device.

One or more application programs <b>266</b> are loaded into memory <b>262</b> and run on operating system <b>264.</b> Examples of application programs include phone dialer programs, e-mail programs, scheduling programs, PIM (personal information management) programs, word processing programs, spreadsheet programs, Internet browser programs, and so forth. Mobile device <b>200</b> also includes non-volatile storage <b>268</b> within the memory <b>262</b>. Non-volatile storage <b>268</b> may be used to store persistent information which should not be lost if mobile device <b>200</b> is powered down. The applications <b>266</b> may use and store information in storage <b>268</b>, such as e-mail or other messages used by an e-mail application, contact information used by a PIM, appointment information used by a scheduling program, documents used by a word processing application, and the like. A synchronization application also resides on the mobile device and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the storage <b>268</b> synchronized with corresponding information stored at the host computer.

Mobile device <b>200</b> has a power supply <b>270</b>, which may be implemented as one or more batteries. Power supply <b>270</b> might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.

Mobile device <b>200</b> is also shown with two types of external notification mechanisms: an LED <b>240</b> and an audio interface <b>274.</b> These devices may be directly coupled to power supply <b>270</b> so that when activated, they remain on for a duration dictated by the notification mechanism even though processor <b>260</b> and other components might shut down to conserve battery power. LED <b>240</b> may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. Audio interface <b>274</b> is used to provide audible signals to and receive audible signals from the user. For example, audio interface <b>274</b> may be coupled to a speaker for providing audible output and to a microphone for receiving audible input, such as to facilitate a telephone conversation.

Mobile device <b>200</b> also includes a radio <b>272</b> that performs the function of transmitting and receiving radio frequency communications. Radio <b>272</b> facilitates wireless connectivity between the mobile device <b>200</b> and the outside world, via a communications carrier or service provider. Transmissions to and from the radio <b>272</b> are conducted under control of the operating system <b>264</b>. In other words, communications received by the radio <b>272</b> may be disseminated to application programs <b>266</b> via the operating system <b>264,</b> and vice versa.

The radio <b>272</b> allows the mobile device <b>200</b> to communicate with other computing devices, such as over a network. The radio <b>272</b> is one example of communication media. Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.

Automatically Determining Differences in a User Interface

FIGURE 3 illustrates exemplary snapshots in accordance with the present invention. The snapshots included are master snapshot <b>310</b>, also referred to as a baseline snapshot, new snapshot <b>320</b>, which refers to the snapshot most recently taken, and difference snapshot <b>330,</b> which provides a visual representation of the difference between the master snapshot and the new snapshot.

Each of the exemplary snapshots shown includes graphical elements and textual elements. In other embodiments, the snapshots may include other elements, such as animations, that may also be captured when the snapshots of the user interface are taken. In the example shown, master snapshot <b>310</b> includes graphical elements <b>312, 314,</b> and <b>318</b> and textual element <b>316.</b> New snapshot <b>320</b> includes graphical elements <b>312, 318</b>, and <b>324</b> and textual elements <b>316</b> and <b>322.</b> Upon visual inspection of master snapshot <b>310</b> and new snapshot <b>320</b>, differences are noticeable. The differences between master snapshot <b>310</b> and new snapshot <b>320</b> are visually represented by difference snapshot <b>330</b> on a pixel-by-pixel basis. For each pixel that is different between master snapshot <b>310</b> and new snapshot <b>320,</b> a black pixel is depicted in difference snapshot <b>330</b>. In one embodiment, a confirmation message such as text reading "no difference" is provided if there is no difference between master snapshot 310 and new snapshot <b>320.</b>

For example, graphical element <b>314</b> is included in master snapshot 310 and not included in new snapshot <b>320.</b> Due to the difference, the pixels corresponding to graphical element <b>314</b> are represented as black pixels in difference snapshot <b>330.</b> Similarly, textual element <b>322</b> and graphical element <b>324</b> are included in new snapshot <b>320</b> but do not appear in master snapshot <b>310</b>. As a result, the pixels that correspond to textual element <b>322</b> and graphical element <b>324</b> are depicted as black pixels in difference snapshot <b>330</b> to illustrate that master snapshot <b>310</b> and new snapshot 320 differ with respect to these elements. The result in difference snapshot <b>330</b> is a readily understandable visual depiction of the differences between master snapshot <b>310</b> and new snapshot <b>320</b>. It is understood that the color of the pixels in difference snapshot <b>330</b> is selectable, and black pixels are referred to merely as an example.

Also illustrated in each snapshot shown in FIGURE 3, is mask <b>340</b>. A mask may be selectively included within each snapshot to exclude portions of the user interface from comparison. For example, a clock element included in the user interface changes over time. As a result the clock element would be shown as a difference between master snapshot <b>310</b> and each subsequent snapshot taken during a development cycle. Masking the portion of the user interface that corresponds to the clock animation avoids the false positives that result from the difference in each snapshot. A description of masking of dynamic regions of a user interface is included in the related application entitled "Method and System for Masking Dynamic Regions in a Mobile Device User Interface to Enable Testing of User Interface Consistency", which is incorporated herein by reference.

FIGURE 4 illustrates an exemplary function block diagram of a system for automatically verifying a user interface in accordance with the present invention. The system <b>400</b> includes test automation <b>410,</b> user interface (UI) target device <b>420,</b> host device <b>430,</b> and server <b>440.</b> UI target device <b>420</b> includes device extension <b>422</b> and device file system <b>424.</b> Host device <b>430</b> includes export tool <b>432</b>. Server <b>440</b> includes import service <b>442</b>, database <b>444,</b> and web application <b>446.</b>

Test automation <b>410</b> provides the functionality for instructing UI target device <b>420</b> to exercise its user interface. Test automation <b>410</b> calls device extension <b>422.</b> The operation of test automation <b>410</b> and device extension <b>422</b> is described in greater detail in the discussion of FIGURE 6 below. Essentially test automation <b>410</b> exercises the user interface while device extension <b>422</b> takes snapshots. Device extension <b>422</b> then stores the snapshots in the form of bitmap files and extensible markup language (XML) files in device file system <b>424.</b>

Export tool <b>432</b> of host device <b>430</b> uploads the bitmap files and the associated XML files to server <b>440</b> from device file system <b>424.</b> In addition, export tool <b>432</b> may provide an indication of a pass/fail status of the test. The pass/fail status may then be transmitted to test automation <b>410</b> in response to a call from test automation 410. The pass/fail assessment may also be transmitted to database <b>444</b> of server 440.

Import service <b>442</b> processes the results as they are uploaded to server <b>440.</b> The results are stored in database <b>444</b>. The architecture of database <b>444</b> is described in greater detail in the discussion of FIGURE 5 below. Web application <b>446</b> is then able to display the results across a network. The results are displayed so that a user is able to manipulate the results to refine their analysis. For example, a user is able to track known bugs that occur in the user interface output by tracking the differences that occur between snapshots of different automation executions. The user is also able to verify and update individual snapshots while analyzing the visual representation of the differences through the difference snapshots that are produced.

In another embodiment, export tool <b>432</b> may be included in UI target device <b>420,</b> eliminating the need for host device <b>430</b>. In addition, test automation <b>410</b> may also be included in UI target device <b>420</b> such that the run of the test automation and production of the snapshots is contained on UI target device <b>420</b>. The results may then be directly transmitted to server <b>440</b> for display. In addition, the results may be displayed directly on UI target device <b>420</b> by incorporating database <b>444</b> on UI target device <b>420</b> as well.

FIGURE 5 illustrates an exemplary functional block diagram of a database used in the system of FIGURE 4 in accordance with the present invention. Database <b>444</b> includes snapshot keys table <b>510,</b> snapshots table <b>520</b>, difference table <b>530,</b> default mask table <b>540</b>, and snapshot mask table <b>550.</b>

Snapshot key table <b>510</b> includes a list of snapshot keys that identify each group of snapshots stored on database <b>500</b> as the snapshots are generated. In one embodiment, each snapshot key is defined according to information extracted from the XML file generated when the snapshot was taken. Accordingly, the key is generated using information regarding the device edition, the platform used, the language used, the color depth of the device, the screen size, and other unique aspects related to the snapshot.

Snapshots table <b>520</b> includes a list of the snapshots taken by filename and a count incremented to indicate that this is the latest confirmed snapshot of a development cycle. The filename of the individual snapshots in snapshot table <b>520</b> is associated with at least one of the snapshot keys listed in snapshot key table <b>510</b>. As new snapshots are generated, the new snapshots are added into snapshots table <b>520.</b> Alternatively, when a snapshot corresponds to and matches a snapshot within snapshots table <b>520,</b> the previous snapshot is replaced with the latest confirmed snapshot, and the count is incremented.

Difference table <b>530</b> includes the difference snapshots. The new snapshots having a particular snapshot key are compared with previous snapshots having the same key. If no difference exists between a new snapshot and a previous snapshot having the same key, the previous snapshot is replaced with the latest confirmed snapshot. If a difference does exist, the difference snapshot is included within difference table <b>530</b>.

Default mask table <b>540</b> includes a list of default masks to be applied to all snapshots corresponding to a particular product and screen size. Snapshot mask table <b>550</b> includes both the default masks that correspond to a generated snapshot listed in snapshots table <b>520</b> and other masks that are user-defined. The use of default mask table <b>540</b> and snapshot mask table <b>550</b> is described in the related application entitled "Method and System for Masking Dynamic Regions in a Mobile Device User Interface to Enable Testing of User Interface Consistency", which is incorporated herein by reference.

In another embodiment, for each snapshot stored in snapshots table <b>520</b>, a value of the snapshot is calculated. The value of the snapshot is referred to as its cyclical redundancy check (CRC) value, and is generated from the bitmap of the snapshot. The calculated CRC value provides a "fast check" mechanism whereby snapshots may be compared quickly by their CRC values.

FIGURE 6 illustrates a logical flow diagram for the client side process of automatically determining differences in a user interface in accordance with the present invention. Process <b>600</b> starts at block <b>602</b> where a UI target device includes a device extension and is loaded with a test automation . Processing continues at block <b>604</b>.

At block <b>604,</b> the test automation that is loaded on the UI target device is activated. The test automation exercises the user interface. During each automation execution, the user interface is exercised according to a test automation so that menus are pulled down, screen content is changed, and other activities are performed that affect the user interface output. Exercising the user interface through a variety of activities provides confirmation that the user interface is operating correctly across a range of possible executions. Processing continues at block <b>606</b>.

At block <b>606,</b> a device extension residing on the UI target device as shown in FIGURE 4 is initiated. The device extension takes snapshots of the user interface of the UI target device. The snapshots are taken throughout a automation execution for multiple automation executions to generate a development cycle for the user interface. In an additional embodiment, the device extension may take multiple snapshots of the user interface for each interval of the build. For example, some mobile devices make use of a software input panel (SIP) that may dynamically change the elements of the screen. The device extension therefore takes a snapshot of the user interface with the SIP and without the SIP for each interval in the automation execution. In another example, a device may allow for portrait to landscape screen rotation. In this example, the device extension takes a snapshot of the user interface in portrait and in landscape orientations. Once the device extension is initiated, processing continues at block <b>610</b>.

At block 610, the device extension stores the snapshots as bitmap and XML files in the file system of the UI target device. It is appreciated that the snapshots may be stored in various data forms other than in bitmap and XML that are well known in the art and are not described further herein. In one embodiment, when the snapshots of the user interface are recorded, a bitmap file of the snapshot image is recorded along with an extensible markup language (XML) file. The XML file includes information such as the language, screen size, SKU (i.e. edition), and other aspects of the target device from which the snapshot is taken.

The following is an exemplary portion of an XML file that may be generated in accordance with the present invention: <ul id="ul0002" list-style="none" compact="compact"><li>&lt;?xml version="1.0" ?&gt; <ul id="ul0003" list-style="dash" compact="compact"><li></li><li>&lt;VisualDiff-FileImportInfo&gt; <ul id="ul0004" list-style="none" compact="compact"><li>&lt;Source&gt;<b>Shell_Notifications_TestApp</b>&lt;/Source&gt;</li><li>&lt;Area&gt;<b>ShellAPI</b>&lt;/Area&gt;</li><li>&lt;SIP&gt;<b>0</b>&lt;/SIP&gt;</li><li>&lt;Title&gt;<b>Shell_Notifications_Bad_SHNOTIFICATION_Struct</b></li><li>&lt;/Title&gt;</li><li>&lt;Product&gt;<b>Magneto-PocketPC Phone</b>&lt;/Product&gt;</li><li>&lt;LCID&gt;<b>0409</b>&lt;/LCID&gt;</li><li>&lt;ColorDepth&gt;<b>16</b>&lt;/ColorDepth&gt;</li><li>&lt; ScreenX&gt;<b>240</b>&lt;/ScreenX&gt;</li><li>&lt;ScreenY&gt;<b>320</b>&lt;/ScreenY&gt;</li><li>&lt;BuildNum&gt;<b>14020</b>&lt;/BuildNum&gt;</li><li>&lt;Platform&gt;<b>x86</b>&lt;/Platform&gt;</li><li>&lt;Filename&gt;<b>VD_ShellAPI_Shell_Notifications_Bad_SHNOT</b></li><li><b>IFICATION_Struct_Magneto-PocketPCPhone04091624 0320_14020_x8601272004211634.bmp</b>&lt;/Filename&gt;</li><li>&lt;/VisualDiff-FileImportInfo&gt;</li></ul></li></ul></li></ul>

As shown in this example, the XML file includes a number of variables related to the display of the UI target device. With this information from the XML file, a particular snapshot may be uniquely identified within the database. After the bitmap and XML files are stored in the device file system, processing continues at decision block <b>612.</b>

At decision block <b>612,</b> a determination is made whether the automation execution is complete and all the snapshot have been taken with their associated files stored. If the automation execution is not complete, processing returns to block <b>608</b> where snapshots continue to be collected. If however, the automation execution is complete, processing proceeds to block <b>616</b> where process <b>600</b> ends.

In an alternative embodiment, the bitmap and XML files continue to be stored in the device file system through multiple automation executions, such that all the snapshots for comparison are stored prior to uploading or analyzing the files.

FIGURE 7 illustrates a logical flow diagram for the upload process of automatically determining differences in a user interface in accordance with the present invention. Process <b>700</b> starts at block <b>702</b> where the bitmap and XML files corresponding to the snapshots of a automation execution are stored in the UI target device file system. Processing continues at block <b>704.</b>

At block <b>704,</b> the bitmap and XML files are retrieved from the UI target device by an export tool located on host device. The export tool runs a application for extracting the files from the UI target device. Once the files are extracted, processing continues at decision block 706.

At decision block <b>706,</b> a determination is made whether to have the export tool calculate the cyclical redundancy check (CRC) value for the bitmap of the snapshot. An option may be included such that the user chooses whether to have the CRC calculated. If the CRC is calculated, processing moves to block <b>708</b>. If however, the determination is made not to calculate the CRC, processing advances to decision block <b>710</b>.

At block <b>708</b>, the user may optionally perform a fast compare between snapshots based on the CRC. This "fast check" option is described in greater detail above with relation to FIGURE 5. Processing then continues at decision block <b>710</b>.

At decision block <b>710,</b> a determination is made whether generate a pass/fail variable. The pass/fail variable indicates whether, on initial examination, the user interface snapshots contain differences when compared. An option may be included such that the user chooses whether to generate the pass/fail variable at this time. If the pass/fail variable is generated, processing moves to block <b>712</b>. If however, the determination is made not to generate the pass/fail variable, processing advances to block <b>714.</b>

At block <b>712,</b> the pass/fail variable is output to the test automation and the database of the server as shown in FIGURE 4. The pass/fail variable is output to the test automation to provide a tester with feedback of the success or failure of the test during at the conclusion of the test. The pass/fail variable is output to the database of the server to provide a record of the pass/fail result that may be accessed on the server through a network. Once the pass/fail variable is output, processing continues at block <b>714.</b>

At block <b>714</b>, the bitmap and XML files are uploaded to the server by the export tool. The export tool prepares the data as necessary for transmission and then transmits the data to the server via a network protocol. Network protocols are well know in the art and are not further described herein. Process <b>700</b> then moves to block <b>716</b> where the process ends.

FIGURE 8 illustrates a logical flow diagram for the server side process of automatically determining differences in a user interface in accordance with the present invention. Process <b>800</b> starts at block <b>802</b> where the bitmap and XML files have been uploaded to the server. Processing continues at block <b>804</b>.

At block <b>804,</b> the incoming files from the export tool are processed by the import service as shown in FIGURE 4. The files are processed according to the network protocol used and other formatting considerations. Processing continues at block <b>806.</b>

At block <b>806,</b> the database of the server is updated as the snapshots are compared with previous snapshots of other builds as necessary. The architecture of the database is described in greater detail in the discussion of FIGURE 5 above. Each of the tables in the database is updated as necessary for the incoming snapshots from the UI target device. Once the database is updated, processing continues at block <b>808.</b>

At block <b>808,</b> the results of the database update are output by a web application that provides the user with options for manipulation of the data. In one embodiment, the user has options to add and delete masks as well as make other changes for further analysis of the data. Processing continues at decision block <b>810</b>.

At decision block <b>810</b>, a determination is made whether the user has decided to make a change to the data stored in the database. If the user has decided make a change, processing moves to block <b>812.</b> However, if the user has decided not to make a change to the data, processing advances to block 814 where the process ends.

At block <b>812,</b> the changes chosen by the user are instituted in the database and the updated results are output by the web application for analysis. For example, the user may have decided to change the baseline snapshot of group of snapshots from an earlier snapshot to a later snapshot. The snapshots table described in FIGURE 5 is then updated by deleting the earlier snapshot and replacing it with the later snapshot. Once any changes by the user are instituted in the database, processing proceeds to block <b>814</b> where process <b>800</b> ends.

In an alternative embodiment, process <b>700</b> of FIGURE 7 may be eliminated and processes <b>600</b> and <b>800</b> of FIGURES 6 and 8 may be combined such that the automation process is performed by the UI target device and the UI target device also displays the results from the testing process.

The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.