Probationary software tests

TECHNICAL FIELD

This disclosure relates to testing of software.

BACKGROUND

In software development it may be useful to implement integrated, unit, and/or other tests for software code. These tests may be run with respect to the code under development, for example, within an Integrated Development Environment (“IDE”) in order to ensure the code under development is properly written, functions correctly with respect to existing code, and so on. An IDE may be an application or process that facilitates software code development in various known ways. In certain embodiments, an IDE may operate with various source control (or version control) management processes and systems and may be in communication with source control repositories and/or other software code storage systems. IDE's may vary among individuals and organizations and may include various types of functionality. A unit test may be a test in which individual units of source code, and/or various modules are tested, often in conjunction with various relevant control data and procedures. An integrated test may be a test in which individual software modules are combined and tested as a group.

In certain instances, code tests may be run within a Continuous Integration (“CI”) environment (e.g., as facilitated by a CI application or process), which may facilitate the integration of the development work of various developers and testing of code in various ways. In certain instances, a CI environment may be associated with regular (e.g., daily) merging of various developer workspaces into a shared mainline. A CI environment may additionally/alternatively be associated, for example, with the compiling of source code from various developers, the linking of various code files, the packaging of object/binary code in various ways, the execution of various software tests and/or test suites, and/or various other functionality relating to software code development. In certain instances a CI environment may facilitate automatically running one or more suites of tests (such as integrated and unit tests) with respect to each new build of code, in order to identify whether any new changes to the codebase may have caused a regression in code functionality and/or various other issues.

BRIEF SUMMARY OF THE DISCLOSURE

According to one aspect of the disclosure, a computer-implemented method includes identifying, by one or more computing devices, a continuous integration environment. The method includes identifying, by the one or more computing devices, a first software test associated with the continuous integration environment. The method includes determining, by the one or more computing devices, a probationary status for the first software test, the probationary status indicating, at least in part, a potential lack of reliability for the first software test.

One or more of the following features may be included. The method may include identifying a second software test associated with the continuous integration environment, the second software test having a promoted status. The method may include presenting, in a first format, first test result information associated with running the first software test. The method may include presenting, in a second format, second test result information associated with the second software test. The first format may be distinct from the second format based upon, at least in part, the probationary status of the first software test and the promoted status of the second software test. One or more of the first test result information and the second test result information associated with the second software test may include an aggregate pass/fail rate.

The method may include determining a promoted status for the first software test based upon, at least in part, identifying a sequence of successful runs for the first software test. The method may include determining a promoted status for the first software test based upon, at least in part, identifying a code consistency over a sequence of runs for the first software test. Identifying the probationary status may be based upon, at least in part, storing first test result information associated with running the first software test in a test result database and comparing the first test result information with other test result information in the test result database in order to, at least in part, identify an indicator of the potential lack of reliability for the first software test. Identifying the probationary status may be based upon, at least in part, identifying the first software test as a previously-promoted software test and determining an intermittent reliability of the first software test.

According to another aspect of the disclosure, a computer program product resides on a computer readable storage medium that has a plurality of instructions stored on it. When executed by a processor, the instructions cause a processor to perform operations including identifying a continuous integration environment. The operations include identifying a first software test associated with the continuous integration environment. The operations include determining a probationary status for the first software test, the probationary status indicating, at least in part, a potential lack of reliability for the first software test.

One or more of the following features may be included. The operations may include identifying a second software test associated with the continuous integration environment, the second software test having a promoted status. The operations may include presenting, in a first format, first test result information associated with running the first software test. The operations may include presenting, in a second format, second test result information associated with the second software test. The first format may be distinct from the second format based upon, at least in part, the probationary status of the first software test and the promoted status of the second software test. One or more of the first test result information and the second test result information may include an aggregate pass/fail rate.

The operations may include determining a promoted status for the first software test based upon, at least in part, identifying a sequence of successful runs for the first software test. The operations may include determining a promoted status for the first software test based upon, at least in part, identifying a code consistency over a sequence of runs for the first software test. Identifying the probationary status may be based upon, at least in part storing, in a test result database, test result information associated with running the first software test and comparing the test result information with other test result information in the test result database in order to, at least in part, identify an indicator of the potential lack of reliability for the first software test. Identifying the probationary status may be based upon, at least in part, identifying the first software test as a previously-promoted software test and determining an intermittent reliability of the first software test.

According to another aspect of the disclosure, a computing system includes one or more processor devices and one or more memory architectures coupled with the one or more processor devices. The one or more processor devices are configured to identify a continuous integration environment. The one or more processor devices are configured to identify a first software test associated with the continuous integration environment. The one or more processor devices are configured to determine a probationary status for the first software test, the probationary status indicating, at least in part, a potential lack of reliability for the first software test.

One or more of the following features may be included. The one or more processor devices may be configured to identify a second software test associated with the continuous integration environment, the second software test having a promoted status. The one or more processor devices may be configured to present, in a first format, first test result information associated with running the first software test. The one or more processor devices may be configured to present, in a second format, second test result information associated with the second software test. The first format may be distinct from the second format based upon, at least in part, the probationary status of the first software test and the promoted status of the second software test. One or more of the first test result information and the second test result information may include an aggregate pass/fail rate.

The one or more processor devices may be configured to determine a promoted status for the first software test based upon, at least in part, one or more of identifying a sequence of successful runs for the first software test and identifying a code consistency over a sequence of runs for the first software test. Identifying the probationary status may be based upon, at least in part storing, in a test result database, test result information associated with running the first software test and comparing the test result information with other test result information in the test result database in order to, at least in part, identify an indicator of the potential lack of reliability for the first software test. Identifying the probationary status may be based upon, at least in part, identifying the first software test as a previously-promoted software test and determining an intermittent reliability of the first software test.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> is a diagrammatic view of a Probationary Test process coupled to a distributed computing network;

<figref idref="DRAWINGS">FIG. 2</figref> is a flowchart of a process executed by the Probationary Test process of <figref idref="DRAWINGS">FIG. 1</figref>;

<figref idref="DRAWINGS">FIG. 3</figref> is a diagrammatic view of an implementation of the Probationary Test process of <figref idref="DRAWINGS">FIG. 1</figref>.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As will be appreciated by one skilled in the art, the present invention may be embodied as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present invention may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium.

Any suitable computer usable or computer readable medium may be utilized. The computer usable medium may be a computer readable signal medium or a computer readable storage medium. A computer-usable, or computer-readable, storage medium (including a storage device associated with a computing device or client electronic device) may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device. In the context of this document, a computer-usable, or computer-readable, storage medium may be any tangible medium that can contain or store a program for use by or in connection with the instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program coded embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in an object oriented programming language such as Java, Smalltalk, C++ or the like. However, the computer program code for carrying out operations of the present invention may also be written in conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

In certain instances, a particular software test (e.g., a unit test or integrated test) may run successfully in a particular IDE (e.g., an IDE associated with a particular developer), but may not run successfully within a CI environment (e.g., with respect to a particular build of a complete codebase) and/or another code development environment. This may result, for example, from differences between the IDE environment (and/or the code tested therein) and the CI (or other) environments and/or other factors, such as a tendency of the particular test to fail intermittently for various reasons. The uncertainty surrounding these (and/or other) issues may result, for example, in certain tests (e.g., tests that are new to a CI or other environment) being somewhat unreliable with respect to indicating the quality of code functionality and/or various aspects of the tested code. This may be problematic with respect to code development, for example, as it may be important to maintain a suite of CI (or other) tests with a 100% nominal pass rate, so that if a test failure is detected such failure may reliably be viewed as an indicator of a potential functional regression of code (i.e., of a defect introduced through a prior build) rather than, for example, an indicator of a faulty or otherwise unreliable test.

A Probationary Test (“PT”) process (or application) may address these and/or various other issues. For example, a PT process may determine that a particular test has a “probationary” status or a “promoted” status, which may indicate, respectively, whether the particular test is to be viewed as a potentially unreliable test or not. In certain embodiments, a PT process may determine that a test previously determined to have a probationary status may merit an upgrade to a promoted status, as may be appropriate, for example, if the test to be upgraded is determined to have exhibited appropriately sufficient reliability. In certain embodiments, a PT process may display information relating to probationary tests differently from information relating to promoted tests, as may, for example, alert a user to the potential differences in reliability of the respective tests, and/or various other differences. In certain embodiments, a PT process may facilitate demotion of a test from promoted status to probationary status, as may be appropriate, for example, based upon identifying various indications of unreliability associated with the test that is to be demoted. Various other functionality may also be provided, as further discussed below.

Referring now to <figref idref="DRAWINGS">FIG. 1</figref>, a PT process may be coupled to a computer or computer network. For example, server PT process <b>10</b> may reside on and may be executed by server computer <b>12</b>, which may be connected to network <b>14</b> (e.g., the Internet or a local area network). Examples of server computer <b>12</b> may include, but are not limited to: a personal computer, a server computer, a series of server computers, a mini computer, and/or a mainframe computer. Server computer <b>12</b> may be a web server (or a series of servers) running a network operating system, examples of which may include but are not limited to: Microsoft® Windows Server®; Novell® Netware®; or Red Hat® Linux®, for example. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Novell and NetWare are registered trademarks of Novell Corporation in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.)

The instruction sets and subroutines of server PT process <b>10</b>, which may be stored on storage device <b>16</b> coupled to server computer <b>12</b>, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into server computer <b>12</b>. Storage device <b>16</b> may include but is not limited to: a hard disk drive; a tape drive; an optical drive; a RAID array; a random access memory (RAM); and a read-only memory (ROM).

Server computer <b>12</b> may execute a web server application, examples of which may include but are not limited to: Microsoft® IIS, Novell® Web Server™, or Apache® Web Server, that allows for access to server computer <b>12</b> (via network <b>14</b>) using one or more protocols, examples of which may include but are not limited to HTTP (i.e., HyperText Transfer Protocol), SIP (i.e., session initiation protocol), and the Lotus® Sametime® VP protocol. (Webserver is a trademark of Novell Corporation in the United States, other countries, or both; Apache is a registered trademarks of Apache Software Foundation in the United States, other countries, or both; Lotus and Sametime are registered trademarks of International Business Machine Corp. in the United States, other countries, or both.) Network <b>14</b> may be connected to one or more secondary networks (e.g., network <b>18</b>), examples of which may include but are not limited to: a local area network; a wide area network; or an intranet, for example.

Client PT processes <b>20</b>, <b>22</b>, <b>24</b>, <b>26</b> may reside on and may be executed by client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, and/or <b>34</b> (respectively), examples of which may include but are not limited to personal computer <b>28</b>, laptop computer <b>30</b>, a data-enabled mobile telephone <b>32</b>, notebook computer <b>34</b>, personal digital assistant (not shown), smart phone (not shown) and a dedicated network device (not shown), for example. Client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b> may each be coupled to network <b>14</b> and/or network <b>18</b> and may each execute an operating system, examples of which may include but are not limited to Microsoft® Windows®, Microsoft Windows CE®, Red Hat® Linux®, or a custom operating system.

The instruction sets and subroutines of client PT processes <b>20</b>, <b>22</b>, <b>24</b>, <b>26</b>, which may be stored on storage devices <b>36</b>, <b>38</b>, <b>40</b>, <b>42</b> (respectively) coupled to client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b> (respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b> (respectively). Storage devices <b>36</b>, <b>38</b>, <b>40</b>, <b>42</b> may include but are not limited to: hard disk drives; tape drives; optical drives; RAID arrays; random access memories (RAM); read-only memories (ROM); compact flash (CF) storage devices; secure digital (SD) storage devices; and memory stick storage devices.

In an embodiment, the PT process may be a server-side process (e.g., which may be implemented via server PT process <b>10</b>), in which all of the functionality of the PT process may be executed on a server computer (e.g., server computer <b>12</b>). In an embodiment, the PT process may be a client-side process (e.g., which may be implemented via one or more of client PT processes <b>20</b>, <b>22</b>, <b>24</b>, <b>26</b>), in which all of the functionality of the PT process may be executed on a client computing device (e.g., one or more of client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b>). In an embodiment, the PT process may be a hybrid server-client process (e.g., which may be implemented by server PT process <b>10</b> and one or more of client PT processes <b>20</b>, <b>22</b>, <b>24</b>, <b>26</b>), in which at least a portion of the functionality of the PT process may be implemented via server computer <b>12</b> and at least a portion of the functionality of the PT process may be implemented via one or more client computing devices (e.g., one or more of client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b>).

In certain embodiments, a PT process may be a stand-alone process. In certain embodiments, a PT process may operate as part of, or in conjunction with, one or more other processes and/or may include one or more other processes. For example, in certain embodiments, a PT process may be included in (or may operate in conjunction with) a Continuous Integration (“CI”) application (or process), which may generally facilitate implementation of a continuous integration environment, and/or various other software development environments or tools (not shown). A CI environment may, for example, facilitate the merging of the code development work of various developers and the testing of code in various ways. For example, in certain instances a CI environment may facilitate automatically running a suite of tests (such as integrated and unit tests) with respect to new builds of the relevant code, in order to identify whether any new changes to the codebase may have caused a regression in code functionality and/or various other issues.

A CI application (or process) may operate (and/or reside) on a client device (e.g., client CI application <b>44</b>, operating on client electronic device <b>28</b>; client application CI <b>46</b>, operating on client electronic device <b>30</b>; client CI application <b>48</b>, operating on client electronic device <b>32</b>; or client CI application <b>50</b>, operating on client electronic device <b>34</b>). A client PT process (e.g., client PT process <b>20</b>) or a server PT process (e.g., server PT process <b>10</b>) may be in communication with a client CI application (e.g., client CI application <b>44</b>) or may be part of a client CI application.

A CI application may additionally/alternatively operate (and/or reside) on a server device (e.g., server CI application <b>52</b>, operating on server computer <b>12</b> or another server CI application (not shown), operating on another server computer (not shown)). A server PT process (e.g., server PT process <b>10</b>) or a client PT process (e.g., client PT process <b>20</b>) may be in communication with a server CI application (e.g., server CI application <b>52</b>) or may be a part of a server CI application.

Users <b>54</b>, <b>56</b>, <b>58</b>, <b>60</b> may access a PT process in various ways. For example, these users may access server PT process <b>10</b> directly through the device on which a client process (e.g., client PT processes <b>20</b>, <b>22</b>, <b>24</b>, <b>26</b>) is executed, namely client electronic devices <b>28</b>, <b>30</b>, <b>32</b>, <b>34</b>. Users <b>54</b>, <b>56</b>, <b>58</b>, <b>60</b> may access server PT process <b>10</b> directly through network <b>14</b> and/or through secondary network <b>18</b>. Further, server computer <b>12</b> (i.e., the computer that executes server PT process <b>10</b>) may be connected to network <b>14</b> through secondary network <b>18</b>, as illustrated with phantom link line <b>62</b>. Users <b>54</b>, <b>56</b>, <b>58</b>, <b>60</b> may also access a client or server CI application (or process) in similar ways.

The various client electronic devices may be directly or indirectly coupled to network <b>14</b> (or network <b>18</b>). For example, personal computer <b>28</b> is shown directly coupled to network <b>14</b> via a hardwired network connection. Further, notebook computer <b>34</b> is shown directly coupled to secondary network <b>18</b> via a hardwired network connection. Laptop computer <b>30</b> is shown wirelessly coupled to network <b>14</b> via wireless communication channel <b>64</b> established between laptop computer <b>30</b> and wireless access point (“WAP”) <b>66</b>, which is shown directly coupled to network <b>14</b>. WAP <b>66</b> may be, for example, an IEEE 802.11a, 802.11b, 802.11g, 802.11n, Wi-Fi, and/or Bluetooth device that is capable of establishing wireless communication channel <b>64</b> between laptop computer <b>30</b> and WAP <b>66</b>. Data-enabled mobile telephone <b>32</b> is shown wirelessly coupled to network <b>14</b> via wireless communication channel <b>68</b> established between data-enabled mobile telephone <b>32</b> and cellular network/bridge <b>70</b>, which is shown directly coupled to network <b>14</b>.

As is known in the art, all of the IEEE 802.11x specifications may use Ethernet protocol and carrier sense multiple access with collision avoidance (i.e., CSMA/CA) for path sharing. The various 802.11x specifications may use phase-shift keying (i.e., PSK) modulation or complementary code keying (i.e., CCK) modulation, for example. As is known in the art, Bluetooth is a telecommunications industry specification that allows e.g., mobile phones, computers, and personal digital assistants to be interconnected using a short-range wireless connection.

For the following discussion, client PT process <b>20</b> will be described for illustrative purposes. It will be understood that client PT process <b>20</b> may, for example, interact and/or communicate with a server PT process such as server PT process <b>10</b> and/or may be executed within one or more applications that allow for communication with other server and/or client PT processes. This is not intended to be a limitation of this disclosure, as other configurations are possible (e.g., PT process <b>20</b> may include stand-alone client processes and/or stand-alone server processes). For example, some implementations may include one or more of client PT processes <b>22</b>, <b>24</b>, <b>26</b> or server PT process <b>10</b> in place of or in addition to client PT process <b>20</b>.

Referring now also to <figref idref="DRAWINGS">FIG. 2</figref>, there is shown a diagrammatic view of an example process that may be implemented by a PT process, e.g., client PT process <b>20</b>. Client PT process <b>20</b> may identify <b>200</b> a first software test. For example, PT process <b>20</b> may identify a software test associated with an identified CI environment (e.g., may be a test forming part of a test suite that is applied to various builds of a codebase associated with the CI environment).

PT process <b>20</b> may determine <b>202</b> a probationary status for the identified <b>200</b> test. A probationary status may indicate, for example, that a particular test may not be fully reliable with respect to a particular CI environment, a particular code build, a particular test suite, and so on. For example, in various instances, tests may be recently added to a CI environment or a test suite associated with a CI environment and/or may have been recently updated (e.g., by revision of various code segments representing the relevant test logic). In certain embodiments, such recent and/or recently-updated (and/or various other) tests may appropriately be considered as somewhat unreliable (e.g., based upon their newness). For example, until a test has been appropriately vetted within a particular CI environment, it may be possible that the test may pass with respect to a particular code build (i.e., return a result indicating that tested code is acceptable with respect to the tested functionality/characteristic or otherwise) even though a pass may not have been appropriate (e.g., because the tested code does not actually exhibit acceptable functionality/characteristics, and so on). Similarly, such a test may sometimes fail (i.e., return a result indicating that tested code is not acceptable with respect to the tested functionality/characteristic or otherwise) even though a fail may not have been appropriate (e.g., because the tested code actually is acceptable with respect to the tested functionality/characteristic, and so on).

Such unreliability in tests (e.g., within a CI environment, or otherwise) may be problematic to software development. For example, it is generally preferred that a code build achieve a 100% nominal pass rate with respect to a test or suite of tests, so that if changes are made to the codebase a failed test may be relied upon to indicate a regression in the quality/functionality of the code rather than, for example, an issue with the failed test itself. As such, if a test cannot be relied upon to exhibit a 100% pass rate for suitable code, that test (and, for example, an associated test suite) may not provide fully useful information to code developers, development managers, and so on. As such, for example, it may be useful to determine <b>202</b> a probationary status for particular tests, in order to appropriately designate those tests (and related test results) as not fully reliable. A “probationary” status for a test may stand in contrast, for example, to a “promoted” status for a test, which may indicate a greater degree of reliability for a particular test. In this way, for example, code developers may be appropriately alerted/directed to more assiduously address test failures when such failures may be more likely to indicate actual problems with the codebase (e.g., rather than problems with the test itself).

It will be understood that, in certain embodiments, PT process <b>20</b> may determine <b>202</b> various levels of probationary status. For example, tests that may be brand new to a CI environment may be considered highly probationary, tests that have exhibited some reliability may be considered semi-probationary/semi-promoted, and tests that have been determined to be fully reliable may be considered to be fully promoted. Various other configurations and levels may also be possible.

In certain embodiments, a probationary status may be determined <b>202</b> for a test that may have previously been determined to have a promoted status. For example, it may be useful to remove/downgrade the promoted status of a test that, despite its promoted status, may be exhibiting (or may have exhibited) various signs of unreliability. For example, PT process <b>20</b> may identify <b>204</b> a test that has a promoted status (e.g., based upon an earlier determination by PT process <b>20</b>). PT process <b>20</b> may then determine <b>202</b> a (new) probationary status for the identified <b>204</b> promoted test based upon, for example, determining <b>206</b> an intermittent reliability of the identified <b>204</b> test. For example, if a promoted (or other) test is exhibiting an intermittent pattern of pass/fail results with respect to a particular codebase(s) or build(s)—e.g., if the test sometimes passes and sometimes fails, without apparent connection to whether relevant tested code has been updated/changed—this may indicate that the test has become unreliable with respect to providing useful information about the functionality/quality of the tested code. As such, it may be appropriate to determine <b>202</b> a probationary status for such a test, even though the test may have been previously determined to have a promoted status.

PT process <b>20</b> may additionally/alternatively determine <b>202</b> a probationary status for an identified <b>204</b> promoted test based upon various other factors. For example, PT process <b>20</b> may determine that a previously-promoted test has been upgraded and/or otherwise edited since the previous code build/test run. Accordingly, because the upgraded/edited test may not necessarily exhibit the same reliability as the previous version of the test, it may be appropriate to determine <b>202</b> a probationary status for such a test. In certain embodiments, determining <b>202</b> a probationary status (e.g., for an identified <b>204</b> promoted test) may be based upon, at least in part, various user input. For example, PT process <b>20</b> may determine <b>206</b> an intermittent reliability for an identified <b>204</b> promoted test (and/or that the test has been recently updated) and may alert an administrator or other user associated with the relevant CI environment of such intermittent reliability (and/or recent update). This may result, for example, in user input indicating that the identified <b>204</b> promoted test should be determined <b>202</b> to have a probationary status, that the test should be flagged for further investigation, or that the test should remain a promoted test.

In certain embodiments, a probationary status may be determined <b>202</b> based upon analysis of various test results (and/or other) information. For example, PT process <b>20</b> may store <b>208</b> test result information associated with running an identified <b>202</b> test in a test result database (e.g., a database associated with storage device <b>36</b>), and may compare <b>210</b> the stored <b>208</b> test result information with other test result information in the database (e.g., test result information from previous execution of various related tests). PT process <b>20</b> may compare <b>210</b> the stored <b>208</b> test result information with other test result information in order to, at least in part, identify an indicator of the potential lack of reliability of the first software test. For example, test result information may include identifying information for various tests (e.g., package, class and test names, whether the test has been previously determined to be probationary and/or promoted, and so on), the time a particular test was run, a pass/fail outcome for a test, the time a particular test was added to the relevant CI environment or test suite, the time a particular test was edited or updated, and/or various other information associated with various tests and/or test runs. Based upon comparing <b>210</b> test result information from various tests PT process <b>20</b> may, for example, determine that a particular test (e.g., an identified <b>200</b> test) is relatively new and/or has been updated/edited relatively recently, with respect to other tests associated with a relevant CI environment (e.g., with respect to the date the various tests were introduced to the CI environment, the number of times the various tests have been run, the last time the various tests were edited or otherwise transformed, and so on), which may indicate that the test may be potentially unreliable (or may be less reliable than a more established test). For example, in a simple case, a test that is brand new to a CI environment may have no corresponding information in a test result database, so comparison <b>210</b> of test result information (e.g., identifying information) for that test may indicate that the test is brand new to the CI environment (i.e., because such comparison <b>210</b> may indicate that there is no information in the test result database from earlier test runs of the new test). As also noted above, a relatively new test to a CI environment (and/or a newly edited test), for example, may not be as reliable with respect to relevant test results as tests that have been associated with the CI environment for a longer time (and/or have not been edited for some time). As such, it may be appropriate for PT process <b>20</b> to determine <b>202</b> a probationary status for such a test.

In certain embodiments, PT process <b>20</b> may determine <b>202</b> a probationary status for a previously-promoted test (e.g., a test that has been associated with a relevant CI environment for some time) based upon comparing <b>210</b> test result information. For example, based upon comparing <b>210</b> test result information, PT process <b>20</b> may determine <b>202</b> that a test that had been designated as “promoted” may have been significantly edited since the last execution of a relevant test suite, and/or may exhibit various other relevant characteristics. Accordingly, PT process <b>20</b> may determine <b>202</b> a probationary status for such a previously-promoted test.

As also noted above, determining <b>202</b> a probationary status for an identified <b>200</b> test may be associated with different treatment within a CI environment (and/or otherwise) of the identified <b>200</b> probationary test and associated test results, versus promoted (i.e., non-probationary) tests and associated test results. This may be useful, for example, in order to facilitate alerting appropriate users of the potential unreliability of probationary versus promoted tests and/or various other relevant information.

For example, PT process <b>20</b> may identify a test with a determined <b>202</b> probationary status and may also identify <b>212</b> a test with a promoted status. PT process <b>20</b> (alone and/or in conjunction with another application or process, such as CI application <b>44</b>) may run <b>214</b> both the identified <b>200</b>, <b>212</b> tests (e.g., in parallel, in series, and/or in various other ways) and may present <b>216</b> test result information from the tests to a user. In certain embodiments, PT process <b>20</b> may present <b>216</b> test result information associated with the identified <b>200</b> probationary test (e.g., aggregate pass/fail rates <b>222</b>) in a first format <b>218</b> and may present <b>216</b> test result information associated with the identified <b>212</b> promoted test (e.g., aggregate pass/fail rates <b>222</b>) in a second format <b>220</b>. (It will be understood that aggregate pass/fail rates <b>222</b> may represent, individually and/or collectively, the pass/fail rate for a particular test over a plurality of test runs.) For example, test result information associated with the probationary test may be presented <b>216</b> in a separate portion of a relevant user interface from test result information associated with a promoted test, in a different font or with different prominence, and/or in various different formats <b>218</b>, which may, for example, indicate to a user the probationary status of the relevant test(s). This may, accordingly, facilitate various users focusing on various test results in appropriate ways (e.g., addressing primary code correction efforts on errors identified by promoted tests, revising the code of probationary tests as appropriate, and so on).

Based upon a demonstrated reliability or consistency of a probationary test and/or various other factors, it may be appropriate, in certain embodiments, to upgrade the designation of a particular unit test from probationary to promoted. As such, for example, in certain embodiments PT process <b>20</b> may determine <b>224</b> a promoted status for the identified <b>200</b> software test. PT process <b>20</b> may determine <b>224</b> a promoted status for the identified <b>200</b> software test in a variety of ways and/or based on various types of information. For example, PT process <b>20</b> may determine <b>224</b> a promoted status for a test based upon a sequence <b>226</b> of successful runs of the test or determining code consistency <b>228</b>. For example, if a probationary test has been successfully run (e.g., has passed or has otherwise successfully operated) for a particular number of test runs (e.g., three or more runs), over a particular number of code builds (e.g., three or more code builds) and so on, it may be appropriate to treat the test as having exhibited reasonable reliability and to determine <b>224</b> a promoted (or otherwise elevated) status for the test.

Similarly, for example, in certain embodiments PT process <b>20</b> may determine <b>224</b> a promoted status for an identified <b>200</b> test based upon identifying a code consistency over a sequence of test runs (or code builds and so on). For example, if a probationary test has not been edited (e.g., updated, rewritten in whole or part, and so on) in some time, this may, depending on the appropriate context, indicate that the probationary test may be considered appropriately reliable and that it may accordingly be upgraded to a promoted status. For example, the fact that a particular probationary test has not been changed by relevant developers for a certain number of test runs, code builds, and so on, may indicate that the relevant developers are satisfied with the performance of the test (e.g., because they have not determined that it is necessary to correct the test through modification of its code). As such, for example, it may be appropriate to determine <b>224</b> a promoted status for that test.

It will be understood that determining <b>224</b> a promoted status for a previously-probationary test (and/or determining <b>202</b> a probationary status for a previously-promoted code, and so on) may be executed with or without user input. For example, in certain embodiments PT process <b>20</b> may flag (e.g., through presentation <b>216</b> of test results associated with the test in a particular format <b>218</b>) a probationary test that has exhibited a sequence <b>226</b> of successful runs, which may prompt an appropriate administrator to designate whether the test should be promoted or not. Similarly, in certain embodiments PT process <b>20</b> may flag a promoted test that has exhibited intermittent reliability, which may prompt an appropriate administrator to designate whether the test should be demoted or not. As part of such “flagging” (and/or otherwise), for example, PT process may present various relevant information to an administrator or other user in order to facilitate receiving appropriate user input regarding determining <b>202</b>, <b>224</b> a probationary or promoted status (and/or for other reasons, such as to inform relevant users of the reasons for an automatic determination <b>202</b>, <b>224</b>). For example, PT process may present various information relating to a determined <b>206</b> intermittent reliability (e.g., may present <b>216</b> the frequency of test failure, the amount of time since the last time a test passed, the number/extent of code builds and/or other code updates that have occurred between a tests having passed and then failed (or vice versa), and so on) and/or various other information relevant to determining <b>202</b>, <b>224</b> a relevant test status.

Referring now also to <figref idref="DRAWINGS">FIG. 3</figref>, various code developers may compose and/or edit code in various IDEs, such as IDEs <b>300</b>, <b>302</b> and <b>304</b>. Code from these IDEs may be combined into various builds within CI environment <b>306</b> (e.g., with daily or other frequency). CI environment <b>306</b> may include various tests and/or test suites, which may be associated with probationary status <b>308</b> or promoted status <b>310</b> through the functionality of PT process <b>20</b>. For example, unit test <b>312</b>, which may have been newly added to a CI environment <b>306</b> test suite may be determined <b>202</b> to have a probationary status based upon, for example, its relative newness with respect to other tests associated with CI environment <b>306</b>. Similarly, unit test <b>314</b>, which may have been included in a relevant test suite for several builds and test runs, may be determined <b>224</b> to have a promoted status based upon, for example, its relative seniority with respect to other tests associated with CI environment <b>306</b>. (It will be understood that determination <b>202</b>, <b>224</b> of test status may not always be associated with relative age (e.g., with respect to a particular CI environment), reliability, and so on. For example, PT process <b>20</b> may determine <b>202</b> a probationary status based upon an absolute amount of time that a particular test (or revision of a test) has been associated with a particular CI environment, an absolute measure of reliability, and so on.)

Upon execution of a test suite (and/or based upon various other events, such as a user request for test information), PT process <b>20</b> (alone and/or in conjunction with another application or process) may present various information relating to various software tests associated with CI environment <b>306</b>. For example, PT process <b>20</b> may present <b>216</b> test result information <b>316</b>. In certain embodiments, presenting <b>216</b> test result information <b>316</b> may include presenting test result (or other) information that is associated with probationary tests differently (e.g., in a different format <b>218</b>) than test result (or other) information that is associated with promoted tests. For example, with respect to test results <b>316</b>, results from promoted tests may be presented <b>216</b> in a bold (and/or otherwise more prominent) format <b>220</b>, whereas results from probationary tests may be presented in a grayed out (and/or otherwise less prominent) format <b>218</b>. In certain embodiments, other configurations may also be possible. For example, PT process <b>20</b> may present <b>216</b> as a default only test results from promoted tests, presenting <b>216</b> test results from probationary tests, for example, only upon receiving an appropriate user input.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiment was chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

A number of embodiments and implementations have been described. Nevertheless, it will be understood that various modifications may be made. Accordingly, other embodiments and implementations are within the scope of the following claims.