Performing software testing based on grouping of tests using test list entity

RELATED APPLICATIONS

p-0002The present application is related to co-filed U.S. patent application Ser. No. 12/789,199 entitled “Mechanism for Dynamic Software Testing Using Test Entity” and U.S. patent application Ser. No. 12/789,228 entitled “Mechanism for Performing Dynamic Software Testing Based on Test Result Information Retrieved in Runtime Using Test Result Entity”, which are assigned to the assignee of the present application.

TECHNICAL FIELD

p-0003The embodiments of the invention relate generally to software testing and, more specifically, relate to performing dynamic software testing based on grouping of tests using a test list entity.

BACKGROUND

p-0004It is commonly known that administrative tasks include software testing various entities (e.g., software components, network protocols, etc.) using various script executing platforms (e.g., Windows PowerShell® platform) that use test execution scripts (e.g., PowerShell script) and are powered by extensible automated engines, such as PowerShell® invocation engine (e.g., System.Management.Automation.dll), and software frameworks (e.g., .NET framework). However, complications occur when testing is to be performed based on various (software and/or hardware) configuration settings, and the testing process becomes even more convoluted, and even unmanageable, when one or more configuration settings are to be modified. This is because today's testing solutions require testing team members (e.g., software developers) to build a test execution platform or infrastructure with multiple rules/regulations and configuration setting variants to accommodate each test and its corresponding hardware and software configuration settings, separately and independently, which, at best, requires constant supervision and rerunning of failed tests and, at worst, becomes overly complex and unmanageable.

BRIEF DESCRIPTION OF THE DRAWINGS

p-0005The invention will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the invention. The drawings, however, should not be taken to limit the invention to the specific embodiments, but are for explanation and understanding only.

p-0006<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates a host machine for employing a test list entity enabling a test execution script at a script execution platform to dynamically run software tests in groups according to one embodiment of the invention;

p-0007<figref idrefs="DRAWINGS">FIG. 2A</figref> illustrates a transaction sequence for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention;

p-0008<figref idrefs="DRAWINGS">FIG. 2B</figref> illustrates a system employing a test list entity for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention;

p-0009<figref idrefs="DRAWINGS">FIG. 3A</figref> illustrates a method for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention;

p-0010<figref idrefs="DRAWINGS">FIG. 3B</figref> illustrates a transaction sequence for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention;

p-0011<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates a method for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention; and

p-0012<figref idrefs="DRAWINGS">FIG. 5</figref> is a block diagram illustrating one embodiment of a computer system.

DETAILED DESCRIPTION

p-0013Embodiments of the invention provide for performing dynamic software testing based on grouping of tests using a test list entity. A method of embodiments of the invention includes causing execution of a first group of first tests based on a first set of configuration settings, and executing the first group includes enabling a test execution script to run the first tests according to the first set of configuration settings, and the test execution script is hosted at a test execution platform of a computer system. The method further includes facilitating creation, in runtime, of a second group of second tests based on a second set of configuration settings that is different from the first set of configuration settings, and causing execution, in runtime, the second group of second tests, and executing the second group includes enabling the test execution script to run the second tests according to the second set of configuration settings.

p-0014The embodiments of the present invention are used to provide dynamic software testing that is based on groups of tests using a test list entity. A number of groups may be created with each group having a number of software tests based on any number of hardware or software configuration settings. Using the test list entity, an entire group of tests (or even multiple groups, simultaneously) can be executed. When a group of tests is executed, the test list entity enables a test execution script to run the grouped tests according to their corresponding properties and configuration settings. Test list entity provides the test execution script the necessary ability to seamlessly continue to detect various configuration settings and adjust and behave accordingly. These groups of tests may be predetermined or created during runtime. This group-based testing technique keeps software testing dynamic and efficient and capable of being performed on a single testing platform or infrastructure that is simple and manageable. In other words, this efficient use of grouping of tests eliminates the need for creating labor-intensive, complex, and unmanageable test execution infrastructure so that each test is individually run.

p-0015<figref idrefs="DRAWINGS">FIG. 1</figref> is a block diagram of a host machine <b>100</b> for employing a test list entity <b>114</b> enabling a test execution script at a script execution platform to dynamically run software tests in groups according to one embodiment of the invention. Host machine <b>100</b> includes a base hardware platform <b>102</b> that comprises a computing platform, which may be capable, for example, of working with a standard operating system <b>108</b> (e.g., Microsoft® Windows®, Linux®, etc.). Operating system <b>108</b> serves as an interface between hardware or physical resources of the host machine <b>100</b> and a software framework <b>110</b> (e.g., Microsoft .NET® Framework). In some embodiments, base hardware platform <b>102</b> may include a processor <b>104</b>, memory devices <b>106</b>, network devices, drivers, and so on. Host machine <b>100</b> includes a server computer system and/or a client computer system (e.g., a client device).

p-0016Software framework <b>110</b> may provide features (e.g., user interface, database connectivity, network communication, etc.) allowing to address common programming problems and a virtual machine to manage the execution of software programs written for the software framework <b>110</b>. Script execution platform or environment <b>112</b> (e.g., Windows® PowerShell®) supports a test execution script (e.g., PowerShell® script) and is built on software framework <b>110</b>. Script execution platform <b>112</b> may include an extensible automation engine that includes a command-line shell and a corresponding test execution scripting language, such as PowerShell script. As aforementioned, script execution platform <b>112</b> is built on software framework <b>110</b> to provide an environment for the user (e.g., software developers/administrators) to perform local and remote administrative tasks (including software testing). Using the PowerShell script as a test execution script, these administrative tasks may be performed by execution of command-lets (“cmdlets”) that are specialized classes (e.g., .NET classes) implementing a particular operation and are combined using scripts, which are composition of cmdlets and imperative logic, standalone applications, or by instantiating regular classes. For brevity and simplicity, PowerShell platform, PowerShell Script, .NET Framework, and Microsoft Windows are used as examples throughout this document.

p-0017Script execution platform <b>112</b> provides a hosting mechanism for the execution script that can be used by a user to produce and run software tests. For example, when testing a software component, the user defines certain properties (e.g., rules, regulations, script behavior, testing time limit, etc.) and software and/or hardware configuration settings relating to the test by inserting such properties and settings in the execution script prior to running the test. The user then initiates the test and the execution script runs the test according to the defined properties and configuration settings. In one embodiment, script representation logic <b>116</b> of script execution platform <b>112</b> provides logical script representation of testing (e.g., test scripts, test results, test history, hardware and/or software configuration information (both to be viewed and modified), etc.) shown as an example with reference to <figref idrefs="DRAWINGS">FIG. 3B</figref>. The user further uses the logical script representation to perform certain administrative tasks, such as amending test scripts, starting or stopping tests, modifying configuration settings, viewing and analyzing results, etc. In one embodiment, some or all of the testing information may be provided and the aforementioned administrative tasks can be performed using a Graphical User Interface (GUI)-based representation. Test list entity <b>114</b> may serve as a database interface to access a database coupled to the host machine <b>100</b> for relevant testing information (e.g., test scripts, test results, test history, test statistics (such as pass/fail percentage) based on test history), and serve as a user interface to, in communication with script representation logic <b>116</b>, provide access to logical script representation (as shown in <figref idrefs="DRAWINGS">FIG. 3B</figref>) or through a GUI-based representation.

p-0018Test list entity <b>114</b>, in one embodiment, facilitates listing or grouping of any number of software tests of different corresponding configuration settings. These grouping can be predetermined and done prior to running tests or determined and performed in runtime. For example, the runtime grouping of tests may be performed by a user in response to analysis or comparison of the results of various tests in runtime such as while one or more tests being run by the execution script. Test list entity <b>114</b> enables the execution script to execute these groups of tests by running the grouped tests according to their corresponding properties and configuration settings. To maximize test coverage, a group of tests may include multiple tests to be run on multiple hardware or software configuration settings and this context of the group may be defined by the tests themselves based on their predefined execution policies and preference or previous test execution results (on different platforms and product versions). This runtime grouping of tests and their execution using a single script execution platform <b>112</b> and its execution script as enabled by test list entity <b>114</b> eliminates the need for building separate, complex, and unmanageable testing infrastructures for running of these tests and their configuration settings.

p-0019Test list entity <b>114</b> further enables test execution script being hosted at script execution platform <b>112</b> to run and rerun various tests based on changing configuration settings (e.g., software or hardware configuration settings) in runtime. Test list entity <b>114</b> enables the test execution script to define itself, adjust itself, and decide its own behavior based on runtime-modification of hardware and/or software configuration settings relating to various tests. Test list entity <b>114</b>, in one embodiment, uses a single configuration variant (as opposed to requiring a separate one for each test and/or hardware/software configuration setting) having the ability to dynamically adjust, in runtime, according to various test properties and the changing configuration settings. Test list entity <b>114</b> enables the test execution script to make decisions on how a test is to be executed by it so that the testing infrastructure may remain simplified and manageable (e.g., the execution script limits itself in focusing on the available resources to satisfy the current resting requirements). Test list entity <b>114</b> may provide an additional ability to, automatically, create test replications and run them with changing hardware/software configuration settings, in runtime, to provide the ability to compare results of different tests on multiple hardware/software configuration settings.

p-0020Throughout this document, the term “in runtime” at least refers to performing a task (e.g., modifying hardware or configuration settings, running or rerunning a test according to the modified configuration setting, accessing and obtaining test result information from a database, enabling an execution script at the script execution platform <b>112</b> to run/rerun a test based on the obtained test information, etc.) without pausing or stopping the testing system, such as without having to stop the execution script from running one or more tests that the execution script is in the process of running or build a complex text infrastructure to accommodate a modified configuration settings, or the like.

p-0021<figref idrefs="DRAWINGS">FIG. 2A</figref> illustrates a transaction sequence for performing dynamic software testing based on grouping of tests using a test list entity <b>114</b> according to one embodiment of the invention. As an initial matter, the words “group” and “list” are used interchangeably throughout this document. In the illustrated embodiment, a user wishes to run two tests, test A <b>212</b> and test B <b>214</b>, to test a network protocol. However, each of the two tests A <b>212</b> and B <b>214</b> is to be run based on two different software configuration settings A1 <b>216</b>, A2 <b>218</b> and B1 <b>220</b>, B2 <b>222</b>, respectively. Further, both tests A-B <b>212</b>-<b>214</b> are to be run on two different hardware configuration settings X <b>202</b> and Y <b>204</b>. Two test groups, test group X <b>208</b> and test group Y <b>210</b>, are formed either as predetermined or formed during runtime. Test group X <b>208</b> includes tests A <b>212</b> and B <b>214</b> with their corresponding software configuration settings A1-A2 <b>216</b>-<b>218</b> and B1-B2 <b>220</b>-<b>222</b>, respectively, and hardware configuration setting X <b>202</b>. Similarly, test group Y <b>210</b> includes the same two tests A-B <b>212</b>-<b>214</b> and their corresponding software configuration settings A1-A2 <b>216</b>-<b>218</b> and B1-B2 <b>220</b>-<b>222</b>, but a different hardware setting Y <b>204</b>.

p-0022In one embodiment, to run the two tests <b>212</b>-<b>214</b> on hardware configuration setting <b>202</b>, the user merely types in one or more command lines using a script representation <b>228</b> (or click on an icon (e.g., Test List X) using a GUI-based representation) to run the first group of tests A-B <b>212</b>-<b>214</b> on all their software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> on hardware configuration setting X <b>202</b>. Upon the user's initiation of the running of test group X <b>208</b>, test list entity <b>114</b> enables test execution script <b>206</b> (e.g., PowerShell script) by, for example, having test entity of other logic/modules <b>234</b> referencing test execution logic <b>230</b> of test execution script <b>206</b> to run tests A-B <b>212</b>-<b>214</b> and dynamically accommodate, in runtime, the four different software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> associated with tests A-B <b>212</b>-<b>214</b> running on hardware configuration setting X <b>202</b>.

p-0023In one embodiment, a user, using command lines in script representation <b>228</b>, generates groups of tests A-B <b>212</b>-<b>214</b>, such as test list X <b>208</b> and Y <b>210</b>, while test list logic <b>224</b> of test list entity <b>114</b> facilitates and recognizes the user's grouping of tests A-B <b>212</b>-<b>214</b> into test list X <b>208</b> and test list Y <b>210</b>, and references test entity invocation logic (also referred to as “test logic” or “test invocation logic”) of other logic/modules <b>234</b> which further references test execution logic <b>230</b> to run tests A-B <b>212</b>-<b>214</b> in groups as defined and initiated by the user. For example, when the user initiates the execution of test list X <b>208</b>, the test entity invocation logic of other logic/modules <b>234</b>, in working with test list logic <b>224</b>, dictates software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> and hardware configuration setting X <b>202</b> for tests A-B <b>212</b>-<b>214</b> to the test execution logic <b>230</b> through command-line arguments or configuration objects. Test execution logic <b>230</b> executes test list X <b>208</b> by running the two tests A-B <b>212</b>-<b>214</b> based on various configuration settings <b>216</b>-<b>222</b>, consecutively or concurrently in any combination by, for example, separate threads (e.g., virtual machines) as defined by the user. The user may use script representation (or GUI-based representation) <b>228</b> to perform various tasks, such as generating and initiating execution of test lists X-Y <b>208</b>-<b>210</b>, using it as command-line interface to generate command-line arguments, modify existing and/or generate new hardware and/or software configuration settings, view and analyze results of test lists X-Y <b>208</b>-<b>210</b> and/or previously-run tests that can be accessed and retrieved from database <b>226</b>, and the like.

p-0024With the grouping of tests A-B <b>212</b>-<b>214</b> into test list X-Y <b>208</b>-<b>210</b>, using test list entity <b>114</b>, instead of initiating the execution of the two tests A-B <b>212</b>-<b>214</b> four times (once for each software configuration setting <b>216</b>-<b>222</b>), the user merely types in one or more command lines using script representation <b>228</b> or clicks, once, on an icon (e.g., Test List X) representing the two tests A-B <b>212</b>-<b>214</b> and its four software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> and hardware configuration setting X <b>202</b>. Similarly, the user then initiates the same two tests A-B <b>212</b>-<b>214</b> with the same four software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> on hardware configuration setting Y <b>204</b> by typing in one or more command lines or clicking another icon (e.g., Test List Y) representing this second group of tests A-B <b>212</b>-<b>214</b>. Using this technique, a total of eight tests (both tests A-B <b>212</b>-<b>214</b> four times each in test list X <b>208</b> and Y <b>210</b>) are run by merely two execution initiations representing the two test lists <b>208</b>-<b>210</b>.

p-0025In one embodiment, a user may choose to group any number of tests with any number of hardware and/or software configuration settings, in advance, by using the script (or GUI-based) representation <b>228</b>. In generating groups or lists of tests, such as test lists X-Y <b>208</b>-<b>210</b>, the user may use any other relevant testing data (e.g., previous tests, test history, test statistics, previous configuration settings, etc.) of, for example, previously run tests of the current software component that is being tested or other components (including older versions of the current component) by accessing the testing information on database <b>226</b> through script representation <b>228</b>.

p-0026In another embodiment, the user may access and analyze certain (old and/or new) relevant testing information stored at database <b>226</b> during running of a relevant test (e.g., running tests J and Q or a standalone running of test A <b>212</b> or the like), such as through test result logic of other logic/modules <b>234</b>, and decide to form groups of tests in runtime, such as while running the relevant test and without stopping any other tests from running or pausing the test system itself. In one embodiment, test result logic of other logic/modules <b>234</b> accesses and retrieves the relevant information from database <b>226</b> which the user reviews and analyzes to decide whether to generate any groups or lists of tests and, if yes, whether to generate them in runtime or upon conclusion of the current running test(s). Using the illustrated embodiment, let us suppose test A <b>212</b> is running with software configuration setting A1 <b>216</b> using hardware configuration setting X <b>202</b>. During the running of test A <b>212</b>, the user reviews and analyzes some relevant testing information (e.g., prior history or analysis of this and/or other relevant tests and components) and decides to add another software configuration setting A2 <b>218</b> to test A <b>212</b>, in runtime, by accessing and using script (GUI-based) representation <b>228</b> to draft this new scenario, i.e., adding a second software configuration setting <b>218</b> to test A <b>212</b>. Regarding this second configuration setting <b>218</b>, it may have been generated in advance (such as in anticipation of associating it to test A <b>212</b>) or in runtime after reviewing the relevant testing information. As aforementioned, the user may utilize the script representation (or GUI-based representation) <b>228</b> to generate groups of tests and initiate their execution. Furthermore, a group of lists of tests may be created for a single run or multiple runs, as necessitated or desired.

p-0027This grouping of tests and their group of execution significantly reduces the number of times the user has to initiate the running of tests with multiple hardware and/or software configuration settings. Referring back to the illustrated embodiment, without the benefit of test list entity <b>114</b>, the user would have had to initiate the two tests A-B <b>212</b>-<b>214</b> at least eight times corresponding to the number of software configuration settings <b>216</b>-<b>222</b>. For brevity and simplicity, merely two tests A-B <b>212</b>-<b>214</b> based on four software configuration settings A1-A2 <b>216</b>-<b>218</b>, B1-B2 <b>220</b>-<b>222</b> and further based on two hardware configuration settings X-Y <b>202</b>-<b>204</b> are illustrated, but it is contemplated that a real-life testing scenario can be of far greater complexity, which further highlights the benefit of employing test list entity <b>114</b>.

p-0028<figref idrefs="DRAWINGS">FIG. 2B</figref> illustrates a system <b>280</b> employing a test list entity <b>114</b> for performing dynamic software testing based on grouping of tests using a test list entity <b>114</b> according to one embodiment of the invention. This illustration is a simplified block diagram of system <b>280</b> employing test list entity <b>114</b> and other relevant components to perform various tasks as discussed in <figref idrefs="DRAWINGS">FIG. 2A</figref>. System <b>280</b> (e.g., host machine including server computer system and/or client computer system) running on a standard operating system (e.g., Linux, Windows, etc.), includes a script execution platform or environment <b>112</b> (e.g., PowerShell platform), built on a software framework (e.g., .NET Framework), supporting a test execution script <b>206</b> (e.g., PowerShell script). Script execution platform <b>112</b> includes script representation logic <b>116</b> to provide a script representation of testing information (e.g., test scripts, test history, test statics, hardware and/or configuration settings, etc.) for the user to access, analyze, and manipulate. Script representation <b>228</b> is provided via a host machine that includes a server machine and/or a client machine (e.g., client device). In one embodiment, testing and other relevant information are provided using a GUI-based representation (as opposed to script representation <b>228</b>) and GUI-based logic (as opposed to script representation logic <b>116</b>).

p-0029Script execution script includes test execution logic <b>220</b> to facilitate execution of tests. Test result entity <b>114</b> includes other logic and modules <b>224</b> including test entity invocation logic that works in communication with test execution logic <b>220</b> to execute tests (such as by having test entity invocation logic referencing test execution logic <b>220</b>) regardless of changes (including runtime modifications) made to software and/or hardware configuration settings. Other logic or modules <b>224</b> may include test list logic for performing listing or grouping of tests, etc. Interface logic <b>222</b> enables test entity to provide a user interface to script representation (or GUI-like representation) and further provide a data interface to database <b>226</b> that stores testing information.

p-0030Test list entity <b>114</b> facilitates and recognizes grouping or listing of various tests and their multiple hardware and/or software configuration settings. As mentioned with regard to <figref idrefs="DRAWINGS">FIG. 2A</figref>, a user may use script representation <b>228</b> (via a user interface, a display device) to generate one or more groups of tests either in advance or in runtime. Test list entity <b>114</b> in recognizing groups of tests, and upon initiation of the execution of these groups of tests, having its test list logic <b>224</b> enable test entity invocation logic of other logic/modules <b>234</b> to reference test execution logic <b>230</b> to perform execution of these groups of tests, such as running the tests in groups as formed by the user. These tests can be run consecutively or concurrently as defined by the user. Testing information as well as results obtained from execution of groups of tests may be provided via script representation (or GUI-based representation) <b>228</b> as provided using script representation logic <b>116</b>.

p-0031Test list entity <b>114</b> provides, via test entity invocation logic of other logic/modules <b>234</b> referencing test execution logic <b>230</b>, test execution script <b>206</b> the ability to run tests in groups and not only in accordance with their initial configuration settings, but also based on their modified configuration settings (if any) in runtime. In one embodiment, an abstraction layer for test list entity <b>114</b> is achieved by providing a “setup” argument in an execution constructor of test execution logic <b>230</b>. Test list entity <b>114</b> may have a number of execution overloads. The setup parameter of test list entity <b>114</b> is an execution parameter that represents different physical devices (which may have different hardware profiles) on which the test execution script <b>206</b> is executed. Test list entity <b>114</b> does not include an execution script; rather, it references the test execution script <b>206</b>, while the test execution script <b>206</b> acts as a black-box so script changes do not influence test list entity <b>114</b>. In one embodiment, other logic/modules <b>224</b> include test entity invocation logic that references the test execution logic <b>220</b> to (re)run groups of tests by dictating the tests' groupings (or listings) as well as their new and modified (hardware and/or software) configuration settings to the test execution logic <b>230</b> through command-line arguments or configuration objects.

p-0032Modifying or using different hardware configuration settings could be of any nature, such as two computer systems having different processors (e.g., using a 1 GHz processor as opposed to using a 2 GHz processor), memory sizes (e.g., using a 256 MB Random Access Memory (RAM) as opposed to using a 512 MB RAM), etc. Examples of modifying or using different software configuration settings may include changing or modifying any number of software components of the product being tested or other software applications running on the host machine, changing virtual machine speed, number of virtual machines in iteration, using different operating systems for the same test, etc.

p-0033Test list entity <b>114</b> further includes interface logic <b>222</b> to provide a user interface to script representation <b>228</b> and a database interface to database <b>226</b>. Database <b>226</b> may include relevant test information, such as tests scripts of previously-run tests of other components or previous version of the current component that is being currently tested, test history, test statistics based on test history, previous hardware and/or software configuration settings, etc.

p-0034Software testing involving test list entity <b>114</b> includes software testing of any nature, such as ranging from testing modules and components of a software application to testing complex and large-scale benchmarks, network scalability, internal releases of products (e.g., software package), antivirus scan loading, time drifting, user performance-related components, etc. Further, these tests can be performed for any number of reasons, such as investigating performance issues, performing load/stress testing, investigating regression issues relating to different product versions, etc. As an example, these testing techniques can be used to test a number of aspects of Red Hat Enterprise Virtualization (RHEV) products.

p-0035<figref idrefs="DRAWINGS">FIG. 3A</figref> illustrates a method for performing dynamic software testing based on grouping of tests using a test list entity <b>114</b> according to one embodiment of the invention. Method <b>300</b> may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof, such as firmware or functional circuitry within hardware devices. In one embodiment, method <b>300</b> is performed by test list entity <b>114</b> of <figref idrefs="DRAWINGS">FIG. 1</figref>.

p-0036Method <b>300</b> begins at block <b>305</b> with the querying tests by build and revision. The term “build” is a software term that relates to interval releases or versions of a product before the product is released to the public, while the term “revision” is refers to the changes or revisions made to the build; for example, each build may have a couple of revisions. At block <b>310</b>, a query is performed for those tests that are not build and revision, such as a test that did not run on a previous version of the product and is limited to its current version. At block <b>315</b>, previous test results are queried from a database. For example, test results of a previously-run test that is relevant to or the same as the ones requested in block <b>305</b> and <b>310</b> is requested. At block <b>320</b>, the ancestor identification for the requested previously-run test is selected.

p-0037A runtime access and retrieval of relevant test information (e.g., including test results, actual tests, historical test statistics, properties, and configuration settings, etc.) from a database may be performed using another logic <b>325</b>, such as test result logic of other logic and modules referenced throughout this document, which may be separate logic of test list entity <b>114</b> or integrated within a single logic also having test entity invocation logic and test list logic <b>224</b>. The results are then analyzed at block <b>330</b>. A user may access the results through a script or GUI-based representation on a display device via a user interface. The user read, study, review, and analyze the results and then, based on the findings of these analysis perform a range of decisions. For example, in one embodiment, based on relevant testing information showing an inhering problem with a particular component of the current version of the product that is being tested, the user may choose to modify the software or hardware configuration or a particular logic of the component to achieve favorable testing results. In another embodiment, the user may use the analysis to rerun previous tests of previous versions of the current component (that is being tested) to troubleshoot the current components. For example, if the rerunning of the previous tests of the previous version is successful, the user may obtain a clue from those tests to improve results of the current test being performed to test the current component (since the current component is merely the latest version of the same components). If, for example, the user decides, based on the analysis of the relevant testing information, to modify software and/or hardware configuration settings, they are modified, at block <b>365</b>, as represented by configuration object <b>370</b>. The modified configuration settings are then saved at block <b>380</b> and the current hardware/software configuration settings are updated, accordingly, at block <b>375</b>.

p-0038In one embodiment, one or more test groups or lists are created as predetermined by the user (e.g., software developer) at block <b>335</b> or generated, in runtime, as determined, for example, in light of the analyzed results at block <b>330</b>. If a group of tests is generated in runtime, it goes through the process of test selection (blocks <b>340</b>-<b>355</b>) through test entity invocation logic <b>360</b> (e.g., test entity) and software and/or hardware configuration setting modification (block <b>365</b>-<b>380</b>). Either way, once a group or a number of groups have been created, they are executed by enabling the test execution script to run the tests that are represented by these groups (according to their corresponding properties and hardware/software configuration settings) at block <b>385</b>. As aforementioned, the user may simply initiate the group test using a script representation (e.g., type in a command line) using a GUI-based representation (e.g., click an icon on the screen) to initiate the execution of these groups of tests.

p-0039<figref idrefs="DRAWINGS">FIG. 3B</figref> illustrates a transaction sequence for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention. At block <b>388</b>, which parallels block <b>335</b> of <figref idrefs="DRAWINGS">FIG. 3A</figref>, a group or list of tests is created by an exemplary name of “Benchmark_Windows” using the cmdlet “Create-TestList” in a test execution script, such as the PowerShell script. At blocks <b>390</b> and <b>392</b>, tests are added to the group. In block <b>390</b>, an array of tests is added to the list using a default configuration setting. RunOnce configuration is used when default configuration is retrieved, modified and passed to List/Test execution interface (of test list entity) as in block <b>392</b> (e.g., this configuration is to be used only once); otherwise, use default configuration as in block <b>390</b>. For example, if (configuration retrieved+modified and then passed to List/Test execution interface): used “RunOnce” execution [configuration is NOT persistent] else-if: (configuration retrieved+modified+called $conf.update( )), conf is saved and so called “default configuration” else-if: ($test executed/added to list with no configuration specified), used “default configuration” [which is persistent]. In block <b>392</b>, however, test are being created and added to the group with modifying configuration settings (see point number <b>4</b>) and these tests are added without RunOnce configuration so that these tests can run multiple times with various hardware/software configuration settings. In this case, the modification of the configuration setting and the adding of the test to the group are being performed in runtime.

p-0040At block <b>394</b>, paralleling block <b>385</b> of <figref idrefs="DRAWINGS">FIG. 3A</figref>, illustrates various test list entity execution interfaces. Test list entity provides the necessary ability to the test execution script to run these tests (within the group) back-to-back according to the properties and configuration settings as set forth in the test execution script. At block <b>396</b>, the results of the group tests are retrieved for analysis and further decision-making, while at block <b>398</b>, the tests themselves are retrieved.

p-0041<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates a method for performing dynamic software testing based on grouping of tests using a test list entity <b>114</b> according to one embodiment of the invention. Method <b>400</b> may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof, such as firmware or functional circuitry within hardware devices. In one embodiment, method <b>400</b> is performed by test list entity of <figref idrefs="DRAWINGS">FIG. 1</figref>.

p-0042Method <b>400</b> begins at block <b>410</b> where a group (group A) of multiple tests (tests A) is generated. Group A is predetermined and generated based on the information already available to a user. At block <b>420</b>, group A is executed. The execution of group A means running of tests A. This execution of group A may be initiated by the user by, in one embodiment, typing in one or more command lines using a script representation, or by, in another embodiment, clicking on an appropriate group icon using a GUI-like representation and upon that initiation, the test list entity enables the test execution script to run each of tests A according to their properties and hardware and/or software configuration settings as defined in the test execution script. Each of the first tests runs as many times as defined in its corresponding properties.

p-0043In runtime, at block <b>430</b>, a determination is made as to whether another group (group B) of tests (tests B) be created. The decision may be based on a combination of already available information and new testing information (such as results or progress of tests A of group A, historical statistics of these or other relevant tests, etc.). If group B is not to be created, the testing process may end upon completion of the execution of group A at block <b>460</b>; otherwise, group B is generated and populated with tests B, in runtime, at block <b>440</b>. Group B may be populated with any number of new tests with new configuration settings, new tests with some or all of old configuration settings (such as the ones from group A), some or all of old tests (such as tests A) with some or all of new configuration settings, and any combination thereof. Group B is executed at block <b>450</b>. Group B's execution includes test list entity enabling the test execution script to run tests B as set forth in group B and according to their properties and configuration setting as defined using the test execution script. As with the first tests, the test execution script is enabled by the test list entity to the second tests according to their corresponding properties and configuration settings. This group generation and testing process continues with block <b>430</b>.

p-0044<figref idrefs="DRAWINGS">FIG. 5</figref> is a block diagram of a computer system <b>500</b> for performing dynamic software testing based on grouping of tests using a test list entity according to one embodiment of the invention. Within the computer system <b>500</b> is a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

p-0045The exemplary computer system <b>500</b> includes a processing device <b>502</b>, a main memory <b>504</b> (e.g., read-only memory (ROM), flash memory, random access memory (RAM), dynamic RAM (DRAM) such as synchronous DRAM (SDRAM), DRAM (RDRAM), etc.), a static memory <b>506</b> (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory <b>518</b> (e.g., a data storage device in the form of a drive unit, which may include fixed or removable machine-accessible or computer-readable storage medium), which communicate with each other via a bus <b>530</b>.

p-0046Processing device <b>502</b> represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device may be complex instruction set computing (CISC) microprocessor, reduced instruction set computer (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device <b>502</b> may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processing device <b>502</b> is configured to execute the processing logic <b>526</b> for performing the operations and methods discussed herein.

p-0047The computer system <b>500</b> may further include a network interface device <b>508</b>. The computer system <b>500</b> also may include a video display unit <b>510</b> (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)) connected to the computer system through a graphics port and graphics chipset, an alphanumeric input device <b>512</b> (e.g., a keyboard), a cursor control device <b>514</b> (e.g., a mouse), and a signal generation device <b>516</b> (e.g., a speaker).

p-0048The data storage device <b>518</b> may include a machine-accessible storage medium (or a computer-readable storage medium) <b>528</b> on which is stored one or more sets of instructions <b>522</b> (e.g., test list entity) embodying any one or more of the methodologies or functions described herein. The test list entity may also reside, completely or at least partially, within the main memory <b>504</b> (e.g., test list entity (instructions) <b>522</b>) and/or within the processing device <b>502</b> (e.g., test list entity (processing logic) <b>526</b>) during execution thereof by the computer system <b>500</b>, the main memory <b>504</b> and the processing device <b>502</b> also constituting machine-readable storage media. Further, for example, the test list entity instructions <b>522</b> may be transmitted or received over a network <b>520</b> via the network interface device <b>508</b>.

p-0049The machine-readable storage medium <b>528</b> may also be used to store the test list entity (instructions) <b>522</b> persistently. While the machine-accessible storage medium <b>528</b> is shown in an exemplary embodiment to be a single medium, the term “machine-accessible storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-accessible storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instruction for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-accessible storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.

p-0050Test list entity modules <b>532</b>, components and other features described herein (for example in relation to <figref idrefs="DRAWINGS">FIG. 1</figref>) can be implemented as discrete hardware components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the modules <b>532</b> can be implemented as firmware or functional circuitry within hardware devices. Further, the modules <b>532</b> can be implemented in any combination hardware devices and software components.

p-0051In the above description, numerous details are set forth. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

p-0052Some portions of the detailed descriptions above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

p-0053It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “creating”, “executing”, “selecting”, “querying”, “enabling”, “storing”, “saving”, “running”, “rerunning”, “determining”, “selecting”, “modifying”, “performing”, “analyzing” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

p-0054The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a machine readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, magnetic-optical disks, ROMs, compact disk ROMs (CD-ROMs), RAMs, erasable programmable ROMs (EPROMs), electrically EPROMs (EEPROMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

p-0055The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

p-0056The present invention may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the present invention. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., ROM, RAM, magnetic disk storage media, optical storage media, flash memory devices, etc.), a machine (e.g., computer) readable transmission medium (non-propagating electrical, optical, or acoustical signals), etc.

p-0057Whereas many alterations and modifications of the present invention will no doubt become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that any particular embodiment shown and described by way of illustration is in no way intended to be considered limiting. Therefore, references to details of various embodiments are not intended to limit the scope of the claims, which in themselves recite only those features regarded as the invention.