Computer implemented systems and methods for testing the usability of a software application

FIELD

0001The technology described in this patent document relates generally to software performance analysis. More specifically, computer-implemented systems and methods are provided for testing the usability of a software application.

BACKGROUND AND SUMMARY

0002Usability testing relates generally to the process of collecting human performance data on the task workflow and user interface design for a software application. The goal of usability testing is often to determine user problem areas in the software interface before the product is released and to set human performance benchmarks for assessing productivity improvements in the software over time. In a typical usability study, a user sits in front of a designated computer and is given a list of tasks to try to perform with the software package being studied. The study facilitator observes the participant as he or she attempts to complete the task and makes performance measurements. Performance measurements may, for example, be based on the time it takes the participant to complete the task, whether the task is completed successfully, the number and nature of errors made by the user, and/or other data. Based on these observed performance measures, problem areas in the user interface or task workflow are identified and recommendations are made for usability improvements. This type of study, however, is typically time intensive for the usability engineers and is limited in the number of studies that can feasibly be performed for each software application.

0003In accordance with the teachings described herein, systems and methods are provided for testing the usability of a software application. A test interface may be provided that executes independently of the software application under test. A task may be assigned via the test interface that identifies one or more operations to be performed using the software application under test. One or more inputs may be received via the test interface to determine if the task was performed successfully.

BRIEF DESCRIPTION OF THE DRAWINGS

0004<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram depicting an example system for testing the usability of a software application.

0005<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram depicting an example system for testing the usability of a plurality of software applications.

0006<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram depicting an example usability testing system in a network environment.

0007<figref idref="DRAWINGS">FIG. 4</figref> depicts a system configuration in which the usability testing program is located on the same computer as the software application under test.

0008<figref idref="DRAWINGS">FIG. 5</figref> is a flow diagram depicting an example method for performing an automated usability study for a software application.

0009<figref idref="DRAWINGS">FIG. 6</figref> is a diagram illustrating examples of a test interface for an automated usability testing system.

0010<figref idref="DRAWINGS">FIG. 7</figref> is a flow diagram depicting an example method for testing the usability of a software application.

0011<figref idref="DRAWINGS">FIGS. 8-13</figref> depict an example test interface for an automated usability testing system.

DETAILED DESCRIPTION

0012<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram depicting an example automated usability testing system <b>10</b> for testing the usability of a software application <b>12</b>. The system <b>10</b> includes a usability testing program <b>14</b> that executes independently of the software application <b>12</b> under test. The usability testing program <b>14</b> accesses test configuration data <b>16</b> and generates a test interface <b>18</b>. The test configuration data <b>16</b> is specific to the software application <b>12</b> under test, and is used by the usability testing program <b>14</b> to generate the test interface <b>18</b>. The test configuration data <b>16</b> may, for example, be configured by one or more persons facilitating a usability study for the software application <b>12</b>. The test interface <b>18</b> is accessible by the test participant, but executes independently of the software application <b>12</b>. For example, the test interface <b>18</b> may be accessible over a computer network, such as the Internet or a company intranet. In this manner, the test interface <b>18</b> may be provided to numerous test participants to perform large-scale usability testing of the software application <b>12</b>.

0013In operation, the usability testing program <b>14</b> presents one or more tasks via the test interface <b>18</b> which are to be performed by the test participant in order to evaluate usability. The test interface <b>18</b> then receives input to determine whether the tasks were completed successfully. For example, the test interface <b>18</b> may present a question that can be answered upon successful completion of a task, and then receive an input with the answer to the question to determine if the task was successfully completed. The test interface <b>18</b> may also provide the test participant with an input for indicating that the task could not be successfully performed and possibly for identifying the cause of the failed performance. In another example, the test interface <b>18</b> may provide one or more inputs for determining the time that it takes to complete each task. For example, the time for completing a task may be measured by requiring the test participant to enter a first input (e.g., click on a first button) in the test interface <b>18</b> before beginning the task and entering a second input (e.g., click on a second button) when the task is completed, with the usability testing program <b>14</b> recording time stamp data when each input is entered. Additional inputs to the test interface <b>18</b> may also be provided to collect other usability data and/or user feedback.

0014<figref idref="DRAWINGS">FIG. 2</figref> is a block diagram depicting an example automated usability testing system <b>30</b> for testing the usability of a plurality of software applications <b>32</b>-<b>34</b>. <figref idref="DRAWINGS">FIG. 2</figref> illustrates that the usability testing program <b>36</b> may be used to perform simultaneous usability studies on multiple software applications <b>32</b>-<b>34</b>. In order to facilitate multiple studies, the testing system <b>30</b> may include a plurality of test configuration data stores <b>38</b>-<b>40</b>, which are used by the usability testing program <b>36</b> to generate a test interface <b>42</b>-<b>44</b> specific to each software application <b>32</b>-<b>34</b> under test. The test interfaces <b>42</b>-<b>44</b> execute independently of the software applications <b>32</b>-<b>34</b>, and may be accessed by numerous test participants, for example over a computer network. In this manner, a large number of studies may be conducted simultaneously in a cost-effective manner, with each study including a broad base of participants.

0015<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram depicting an example automated usability testing system <b>50</b> in a network environment. As illustrated, the usability testing program <b>52</b> and test configuration data stores <b>54</b> may be located on a first computer <b>56</b> or set of computers (e.g., a network server), which is configured to communicate with a second computer <b>58</b> (e.g., a network client) over a computer network <b>60</b>. Using this configuration, the usability testing program <b>52</b> may be accessed via the computer network <b>60</b> to display the test interface <b>62</b> on the second computer <b>58</b> along with the software application <b>64</b> under test. For example, the usability testing program may be a web-based application and the test interface <b>62</b> may be displayed using a web browser application executing on the second computer <b>58</b>.

0016<figref idref="DRAWINGS">FIG. 4</figref> depicts a system configuration <b>70</b> in which the usability testing program <b>72</b> is located on the same computer <b>74</b> as the software application <b>76</b> under test. The usability testing program <b>72</b> and test configuration data <b>80</b> may, for example, be installed along with the software application <b>78</b> on one or more isolated computers <b>74</b> used for usability testing. In another example, the usability testing program <b>72</b> may be loaded onto multiple computers within an organization and the test configuration data <b>80</b> may be loaded to the computers (e.g., over a computer network) to configure the usability testing program <b>72</b> to generate a test interface <b>82</b> for a specific software application <b>78</b>. In this manner, the usability testing program <b>72</b> could be used to test multiple software applications within the organization by loading different test configuration data <b>80</b>. The usability testing program <b>72</b> and test interface <b>82</b>, although operating on the same computer in this example, execute independent of the software application <b>78</b> under test.

0017<figref idref="DRAWINGS">FIG. 5</figref> is a flow diagram depicting an example method <b>90</b> for performing an automated usability study for a software application. The method begins at step <b>92</b>. At step <b>94</b>, an introduction is presented to a test participant, for example using a test interface that executes independent of the software application under test. The introduction may describe the purpose of the usability study and provide instructions to the participant. At step <b>96</b>, participant training information is presented to the test participant. The participant training information may, for example, be in the form of one or more practice tasks to familiarize the participant with the testing system.

0018The usability study is performed at step <b>98</b>. The usability study may require the participant to complete one or more identified tasks using the software application under test and provide information relating to the performance of the tasks via a test interface. The information provided by the participant may be recorded for use in assessing the usability of the software application under test. Upon completion of the usability study, a survey may be presented to the test participant at step <b>100</b>. The survey may, for example, be used to acquire additional information from the test participant regarding software usability, user satisfaction, demographics, task priority, and/or other information. The method then ends at step <b>102</b>.

0019It should be understood that similar to the other processing flows described herein, one or more of the steps and the order in the flowchart may be altered, deleted, modified and/or augmented and still achieve the desired outcome.

0020<figref idref="DRAWINGS">FIG. 6</figref> is a diagram illustrating examples of a test interface for an automated usability testing system. The diagram illustrates three examples <b>110</b>, <b>112</b>, <b>114</b> of software interfaces that may appear on a test participant's computer screen during a usability test. The first example <b>110</b> depicts a test introduction displayed within a web browser interface <b>116</b>. The usability test may, for example, be initiated via a network connection (e.g., by accessing a web site), and the introduction page <b>116</b> may be displayed on the web browser upon initiating the test. The introduction page <b>116</b> may, for example, describe the intent of the study, the software application being tested, and an overview of the test interface.

0021The second example <b>112</b> depicts the test interface <b>118</b> displayed on the computer screen next to an interface <b>120</b> for the software application under test. In this example, the test interface <b>118</b> appears on the computer screen as a tall, thin column alongside the application window <b>120</b>, enabling the test participant to simultaneously view both the test interface <b>118</b> and the application window <b>120</b>. The arrangement of the test interface <b>1</b><b>18</b> on the computer screen with respect to the application window may, for example, be automatically performed by the usability testing program, but could be performed manually in other examples. As illustrated in the third example <b>114</b>, the usability testing information is provided to the test participant via the test interface <b>118</b>, which executes independently of the software application <b>120</b> under test.

0022<figref idref="DRAWINGS">FIG. 7</figref> is a flow diagram depicting an example method <b>130</b> for testing the usability of a software application. In step <b>132</b>, a begin task input (e.g., clicks on a “Begin” button) is received from the test participant to reveal a description of a first task to be performed using the software application under test. The begin task input also causes the method to begin timing the amount of time that it takes the test participant to complete the task. For example, time stamp data may be recorded when the test participant clicks on a “Begin” button to mark the time that the task is started. The test participant then attempts to complete the task using the software application under test at step <b>134</b>. At decision step <b>136</b>, if the task is successfully completed, then the method proceeds to step <b>138</b>. Else, if the task cannot be completed by the test participant, then the method proceeds to step <b>140</b>.

0023Upon successfully completing the assigned task, an answer to a validation question is received from the test participant at step <b>138</b>. The validation question is presented to the user to verify successful completion of the task. For example, the validation question may request an input, such as a data value or other output of the software application, which can only be determined by completing the task. After the answer to the validation question is input, the test participant enters a task completion input (e.g., clicks on a “Done” button) at step <b>142</b> to indicate that the task is completed and to stop measuring the amount of time taken to complete the task. For example, if a first time stamp is recorded when the test participant clicks on a “Begin” button and a second time stamp is recorded when the test participant clicks on a “Done” button, then the first and second time stamps may be compared to determine the amount of time taken by the test participant to complete the assigned task. Once the task completion input is received, the method proceeds to step <b>150</b>.

0024If the test participant is unable to complete the assigned task, then a task failure input (e.g., an “I quit” button) is entered at step <b>140</b>. The task failure input causes the method to stop measuring the amount of time taken on the task (e.g., by recording a second time stamp), and step-by-step instructions for completing the task are presented to the participant at step <b>144</b>. The step-by-step instructions may be presented in an additional user interface window. After reviewing the step-by-step instructions, the test participant inputs one or more comments at step <b>146</b> to indicate which one or more steps in the task caused the difficulty. At step <b>148</b>, the test participant closes the additional window with the step-by-step instructions, and the method proceeds to step <b>150</b>.

0025At step <b>150</b>, an input is received from the test participant to indicate the perceived importance of the task, for example using a seven point Likert scale. Another input is then received from the test participant at step <b>152</b> to rate the test participant's satisfaction with the user experience of the task, for example using a seven point Likert scale. At step <b>154</b>, a textual input is received from the test participant to provide comments, for example regarding the task workflow and user interface. A next task input is then received from the test participant (e.g., by clicking a “next task” button) at step <b>156</b>, and the method proceeds to decision step <b>158</b>. If additional tasks are included in the usability test, then the method returns to step <b>132</b> and repeats for the next task. Otherwise, if there are no additional tasks, then the method proceeds to step <b>160</b>. At step <b>160</b>, a final input may be received from the test participant before the test concludes, for example the participant may fill out an end of session survey.

0026<figref idref="DRAWINGS">FIGS. 8-13</figref> depict an example test interface for an automated usability testing system. The test interface is generated by a usability testing program, which executes independently of the software application under test. For example, the testing system has no programmatic interaction with the software application under test, nor does it require the collection of system events or event logging from the operating system. Rather, the usability testing program records information entered by the test participant within the test interface. Because of this separation between the software application under test and the testing system, the usability testing system may be used to conduct automated studies on web or desktop applications without the need to install monitoring software on the participant's computer. In this manner, the usability testing system may be used to perform large-scale testing and to improve the reliability of measures beyond that possible in a typical quality lab environment.

0027With reference first to <figref idref="DRAWINGS">FIG. 8</figref>, the example test interface <b>200</b> and the software application <b>202</b> under test are displayed side-by-side in two windows on a computer screen. Within the example test interface <b>200</b>, the testing system has displayed a practice task <b>204</b> to enable the test participant to become familiar with the test interface <b>200</b> and the usability testing process. In the illustrated example, the practice task <b>204</b> requires the test participant to log into the software application <b>202</b> under test using the displayed user name and password. Before beginning the displayed task <b>204</b>, the test participant clicks on a begin task input <b>206</b>, which begins measuring the amount of time taken to perform the assigned task <b>204</b>. The begin task input <b>206</b> may, for example, cause the usability testing system to record timestamp data to indicate the time that the test participant began performing the assigned task. When the task is completed, the test participant clicks on a task completion input <b>208</b>. The task completion input <b>208</b> may, for example, cause the usability testing system to record timestamp data to indicate the time that the test participant finished performing the task. In addition, the task completion input <b>208</b> may cause the test interface <b>200</b> to display the next step in the usability test. For instance, in the illustrated example, survey questions and survey input fields <b>210</b> are displayed after the task completion input <b>208</b> is entered. When the practice task <b>204</b> is completed and the survey information <b>210</b> is entered, the test participant may proceed to the first task in the usability test by pressing the “next task” input <b>212</b>.

0028<figref idref="DRAWINGS">FIG. 9</figref> illustrates the beginning of the first task in the example usability test. To begin the task, the test participant clicks on the begin task input <b>214</b>. The first task <b>216</b> is then displayed to the test participant in the test interface <b>200</b>, as illustrated in <figref idref="DRAWINGS">FIG. 10</figref>. Also displayed in <figref idref="DRAWINGS">FIG. 10</figref> is a validation question <b>218</b> and a validation input field <b>220</b> for entering an answer to the validation question <b>218</b> upon successful completion of the assigned task. The validation question <b>218</b> preferably can only be answered upon successful completion of the assigned task, as illustrated in <figref idref="DRAWINGS">FIG. 11</figref>. For instance, in the illustrated example, the assigned task of finding and opening the report called “Annual Profit by Product Group” must be performed successfully to answer the validation question <b>218</b>, which relates to a data value within the report. When the assign task is completed and the validation input <b>220</b> is entered, the test participant may click on the task completion input <b>222</b> to record the time on task and to move onto the next phase of the usability test. For example, <figref idref="DRAWINGS">FIG. 11</figref> illustrates survey questions and survey input fields <b>226</b> that are displayed after the test participant clicks the task completion input <b>222</b>.

0029Alternatively, if the test participant is unable to complete the task, he or she may click on the task failure input <b>224</b> to end the task and to display step-by-step instructions <b>228</b> for performing the task. Example step-by-step instructions are illustrated in <figref idref="DRAWINGS">FIG. 12</figref>. As shown in <figref idref="DRAWINGS">FIG. 12</figref>, the step-by-step instructions may be displayed in a separate window from the test interface <b>200</b>. The reason or reasons that the task could not be performed successfully will typically be evident to the test participant once the step-by-step instructions are reviewed. The instruction window <b>228</b> may, therefore, also include a field <b>230</b> for inputting information indicating one or more reasons why the task was not successfully performed.

0030After the usability test is completed, the test interface <b>200</b> may display one or more additional survey questions, as illustrated in the example of <figref idref="DRAWINGS">FIG. 13</figref>.

0031This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention may include other examples that occur to those skilled in the art.

0032It is further noted that the systems and methods described herein may be implemented on various types of computer architectures, such as for example on a single general purpose computer or workstation, or on a networked system, or in a client-server configuration, or in an application service provider configuration.

0033It is further noted that the systems and methods may include data signals conveyed via networks (e.g., local area network, wide area network, internet, etc.), fiber optic medium, carrier waves, wireless networks, etc. for communication with one or more data processing devices. The data signals can carry any or all of the data disclosed herein that is provided to or from a device.

0034Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform methods described herein. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein.

0035The systems' and methods' data (e.g., associations, mappings, etc.) may be stored and implemented in one or more different types of computer-implemented ways, such as different types of storage devices and programming constructs (e.g., data stores, RAM, ROM, Flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.

0036The systems and methods may be provided on many different types of computer-readable media including computer storage mechanisms (e.g., CD-ROM, diskette, RAM, flash memory, computer's hard drive, etc.) that contain instructions for use in execution by a processor to perform the methods' operations and implement the systems described herein.

0037The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.