System and method for wireless network performance measurement and management using remote devices

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 15/161,172, titled “SYSTEM AND METHOD FOR WIRELESS NETWORK PERFORMANCE MEASUREMENT AND MANAGEMENT USING REMOTE DEVICES” filed on May 20, 2016, the entire specification of which is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

Field of the Art

The disclosure relates to the field of electronics networking, and more particularly to the field of wireless network service quality measurement and management.

Discussion of the State of the Art

All must recognize that the advent and later widespread use of computer networking has revolutionized the operation of virtually all business installations from smallest outlet to warehouse, factory and corporate headquarters. At first it was slow speed wired service which was somewhat constricting, difficult to set up and required the laying of possibly miles of sometimes very inconvenient wiring but speeds improved rapidly and most corporations have become fully dependent on networking and cloud access to function. More recently, the availability, better speed and increased reliability of wireless networking has once again begun large scale changes in corporate operations. Now one may have continuous network connectivity with all of the corporate resources and the cloud, but may now move around the corporate landscape at will depending on the mobility of the device being used. Device mobility has greatly changed at the same time. While wireless networking is indeed a large step forward, it does not come without drawbacks as networks may become over-subscribed without much recognizable warning, one does not need to worry that there are enough ports on the switch to accommodate everyone, but the capacity constraint is still there. Also wireless coverage is greatly affected by environmental factors, building infrastructure such as walls and beams, radio interference either chronic from other office equipment or items employees bring or transient cloud cover, sun spots, humidity, etc. Coverage at peripheral areas of a building complex so that all employees have fast optimal service without “leaking” your network outside of the corporate campus to where it is a beacon for hacking are all important considerations. There are many tests that may be run that probe an aspect of wireless network service delivery and quality, but knowing which ones to run, how often to run them, from where to run them, and how to interpret the results, are daunting factors. Budgeting such endeavors is also no small consideration.

What is needed is a system and method that will comprehensively test a company's wireless network equipment and coverage, retrieve data from all available equipment, coordinate third party parts of the test system, automatically run important tests without intervention, use end user devices or testing equipment that is already or may easily be placed throughout the coverage map of the tested network and will transform and then display the data in such a way that all concerned with network health and function may understand and show to those in charge of the purse strings to resolve issues may easily comprehend, for example graphs and minimal service levels, possibly with remedy suggestions when something does go wrong.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in a preferred embodiment of the invention, a system and method for wireless network performance measurement and management using remote devices.

To provide a means to measure wireless network service quality from a plurality of locations and access points within that network, the inventors conceived and reduced to practice a system where wireless network connected mobile devices such as smart phones and tablets may be used in conjunction with a central data analysis and test control server to measure multiple parameters that reflect wireless network service quality. To participate, a given mobile device downloads and installs a generic test suite application and enters an identifying code which has been provided separately. Device sends this code to the central data analysis and test control server. The central server responds with the coding and configuration for tests to be run. These tests may include wireless signal strength, wireless current channel, wireless frequency band, data throughput, network capacity currently in use, radio attachment latency and success, ping latency, remote resource access latency and remote resource download bandwidth among many other parameters possible. Test parameters define details of the tests, including networks to be tested and target servers to be used in the test. Test parameters also determine whether tests are run periodically, continuously or initiated by user only. Tests performed by the test suite may be chosen to reflect the needs and purpose of individual clients or test campaign at the central data analysis and test control server and then pushed to the remote devices just prior to the start of testing. The identification code allows customizing a standard device or a group of standard devices with a generally available software for the purpose of the customized test. The identification code is also submitted with the test data points and allows storing and analyzing the data centrally for a group of devices which belong to certain organization.

The wireless network service quality test system may be configured to receive information from third party through an application programming interface “API”. This data may include data from devices such as wireless access points to receive information that may include the manufacturer and model of the access point, map of the area, access point location in the map, GPS coordinates of the access points, location of certain device at certain time as positioned by the network, current attached clients, current error logs and current firmware and software level, among other information. This information may be used to better complete the dataset both during routine testing and in case troubleshooting analytics are needed.

The API may also be used to send information such as warnings, alarms and wireless network performance data to third party devices such as general network managers, hospital equipment and supervisory devices dependent on effected wireless networking or mobile devices of network administrative personnel. These data or messages may be routine, or informative or may signify that service level of a particular aspect or aspects of a monitored wireless network has dropped below a pre-designated minimal level required by the customer. Alternatively the API may provide all raw measurement data to other applications for further processing. Analytics capability of the wireless network service quality test system may also be used to predict the root cause of the service reduction and suggest possible remedial action, if desired.

Finally, data collected by the remote service quality test devices that are part of the system and analyzed by the central data analysis and test control server may be represented and displayed to best suit the customer's needs. Some examples are showing one or more selected service quality parameters graphically over a pre-selected time period as a percentage of known service target level of the tested network, possibly also depicting pre-decided minimal service levels. Another possibility may be a color-coded topographical type map of the coverage area of the tested network showing signal levels or network throughput as differing colors as a function of location. Yet another possibility is to display adherence to pre-determined service levels with colored cells on a map or floor plan. This would make use of location reporting functions of the test devices, possibly GPS based, or location data gleaned from the wireless access points or other electromagnetic signal transmitters based on signal levels. These may use radio frequencies, microwave frequencies or even optical light. As well, propagation time and comparative target signal strength of the test device may be utilized to determine the location. Location may also be entered by user of the device by pointing out a location on a map or floor plan. Many other representations are possible, dependent on the needs and goals of the customer. Data may be encoded in such a way so as to be used by another electronic system such as a third party network manager.

According to a preferred embodiment of the invention, A system for wireless network performance measurement and management using remote devices, comprising: a central analysis and control server comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device, a mobile wireless device comprising at least a processor and a memory, and a wireless network testing software application stored in the memory and operating on the processor of the wireless device. The central analysis and control server: transmits wireless network testing software application to at east one wireless device, receives result data of wireless network service quality testing performed by at least one wireless device, and analyzes test data of wireless network service quality testing using pre-determined transformations to demonstrate key aspects of service quality; and represents transformed result data as best suited for further steps towards overall wireless service quality management. The mobile wireless device: comprises a plurality of mobile wireless device types routinely operated by a plurality of customer's employees, performs sets of wireless network service quality tests due to the downloaded wireless network testing software application, is capable of performing tests in the background while a user performs other conventional wireless device related functions; and transmits wireless network service quality test data to central analysis and control server. The wireless network testing software application: programs mobile wireless device to carry out pre-designed, directed active examination of wireless network functions and passive monitoring of network parameters to establish level of wireless network service quality, and allows display of wireless network service quality tests on the wireless mobile device executing it.

According to another embodiment of the invention, at least one remote wireless test device is a network enabled mobile device such as a smartphone or tablet running a specifically programmed wireless network test suite framework as an application. Remote wireless test devices may be wireless network enabled laptop computers, desktop computers, cable modems, ADSL routers, Wi-Fi access points, mobile network base stations, network switches, plug in devices (USB, or similar) in addition to those mentioned. A large plurality of remote wireless test devices may be employed during network service quality monitoring of a single wireless network. Dedicated network target devices may be used to provide additional operational data as part of wireless network service quality tests. Information from one or more third party network infrastructure devices may be integrated into wireless network service quality test results to provide additional operational information. Data collected by remote wireless test device and analysis functions employed by central analysis and control server may differ based upon a client's specifications. The wireless network service quality parameters collected by remote wireless test device come from a set of wireless network parameters that include: wireless network signal strength, wireless network channel, wireless network signal frequency band, wireless network standard in use, connected wireless access point, network radio attachment time and success rate, network data throughput, voice quality, web page load time and success rate and ping latency to pre-selected target, RF characteristics of the test signal like retry rates and data rates among other parameters known to the art. The central analysis and control server may accept and store wireless network user comments concerning service quality and third party application service quality data.

According to another embodiment of the invention, the wireless network testing software application tests programmed occur on a repetitive periodic basis wake sleeping wireless device, an active test is loading a web page or logging into a certain site active tests like throughput and voice quality target custom endpoints, the transmission of test results are delayed with result data stored locally if connectivity to the central analysis and control server is unavailable, results are submitted when connection is restored, additional network configuration data is imported from network equipment and then correlated with test data at central analysis and control server, service level agreement target thresholds are used to determine sufficient wireless network service quality, while one network is used by the wireless device, a second network may be tested in the background, test configuration parameters define which networks are tested, if wireless device cannot connect to one or more of the defined networks, absence of network connection is recorded and the active tests deferred.

According to another embodiment of the invention, a system for wireless network performance measurement and management using remote devices, comprising customer specific test customizations such as correct test profiles and SLA thresholds are downloaded from a central server to the mobile wireless device when the user enters a code. Customer specific test customizations such as correct test profiles and SLA thresholds are downloaded from a central server to the mobile wireless device when the user enters pre-existing profile credentials which causes the user to be added to a specific organization for data aggregation. Passive user network traffic monitoring, observed network responsiveness and network access data collected while user uses the mobile wireless device. Tests include passive measurement for collection wireless radio signal related and network traffic load data. Active tests are postponed when user runs tasks on the mobile wireless device that result in network usage or CPU load above pre-set threshold. At least a subset of test data from individual mobile wireless devices is be correlated with similar wireless mobile devices in a central database. User feedback concerning wireless network service quality in integrated within the wireless network testing software application, user feedback collection triggered at discretion of an administrator of the central analysis and control server or by pre-determined SLA violations determined by the mobile wireless device application. At least one data set which includes location information is layered over a scalable map. A pre-programmed roaming test monitors mobile wireless device connections and transitions between cells and networks.

According to another embodiment of the invention, A method for wireless network performance measurement and management using remote devices, comprising the steps of: a) receiving a wireless network testing software application encoding pre-designed wireless network test functions using a wireless device from a central analysis and control server after entering an identity code; b) employing the wireless network test functions to both actively probe and passively monitor parameters impacting wireless network service quality level using monitoring devices, at least in part, consumer level mobile wireless devices; c) transmitting data resultant from wireless network test functions to the central analysis and control server; d) transforming wireless network test function data using central analysis and control server and formatting it for display or further action as predetermined by test administrators

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention according to the embodiments. It will be appreciated by one skilled in the art that the particular embodiments illustrated in the drawings are merely exemplary, and are not to be considered as limiting of the scope of the invention or the claims herein in any way.

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating an exemplary system architecture for measuring and reporting wireless network service quality using remote devices, according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 2</figref> is a flow diagram illustrating an exemplary method for measuring and reporting wireless network service quality using remote devices, according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 3</figref> is a flow diagram illustrating an exemplary method by which the system is programmed with customer specific test suites according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram illustrating an exemplary system architecture for central analysis and control server to analyze network service quality using remote devices, according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 5</figref> is a flow diagram illustrating an exemplary method by which central analysis and control server may interact with other components of the system and analyze network performance test data according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 6</figref> is a diagram illustrating information that is processed by central analysis and control server analytics engine according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram illustrating an exemplary system architecture for mobile wireless service quality test device employed to analyze network service quality using remote devices, according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 8</figref> is a flow diagram illustrating an exemplary method by which mobile wireless network service quality device may operate to collect store and transmit test suite data according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 9</figref> is a diagram illustrating methods by which users of wireless networks being tested might submit their perceived experience to the wireless network service quality test system according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 10</figref> is a diagram illustrating both passive and active testing of more than one wireless network by wireless network service quality test devices according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 11</figref> Is a list of methods by which the location of individual wireless network service quality test devices may be obtained according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 12</figref> is a diagram displaying two example formats of wireless service quality test output according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 13</figref> is a flow diagram of a possible notification, warning and alarm mechanism to be used when network performance is found to be below customer established service level minimums according to a preferred embodiment of the invention.

<figref idref="DRAWINGS">FIG. 14</figref> is a block diagram illustrating an exemplary hardware architecture of a computing device used in an embodiment of the invention.

<figref idref="DRAWINGS">FIG. 15</figref> is a block diagram illustrating an exemplary logical architecture for a client device, according to an embodiment of the invention.

<figref idref="DRAWINGS">FIG. 16</figref> is a block diagram showing an exemplary architectural arrangement of clients, servers, and external services, according to an embodiment of the invention.

<figref idref="DRAWINGS">FIG. 17</figref> is another block diagram illustrating an exemplary hardware architecture of a computing device used in a plurality of embodiments of the invention.

<figref idref="DRAWINGS">FIG. 18</figref> is a diagram illustrating change of access point during wireless network user roaming according to an embodiment of the invention.

<figref idref="DRAWINGS">FIG. 19</figref> is a table of network parameters that may be tested according to an embodiment of the invention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferred embodiment of the invention, a system and method for wireless network performance measurement and management using remote devices.

One or more different inventions may be described in the present application. Further, for one or more of the inventions described herein, numerous alternative embodiments may be described; it should be appreciated that these are presented for illustrative purposes only and are not limiting of the inventions contained herein or the claims presented herein in any way. One or more of the inventions may be widely applicable to numerous embodiments, as may be readily apparent from the disclosure. In general, embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the inventions, and it should be appreciated that other embodiments may be utilized and that structural, logical, software, electrical and other changes may be made without departing from the scope of the particular inventions. Accordingly, one skilled in the art will recognize that one or more of the inventions may be practiced with various modifications and alterations. Particular features of one or more of the inventions described herein may be described with reference to one or more particular embodiments or figures that form a part of the present disclosure, and in which are shown, by way of illustration, specific embodiments of one or more of the inventions. It should be appreciated, however, that such features are not limited to usage in the one or more particular embodiments or figures with reference to which they are described. The present disclosure is neither a literal description of all embodiments of one or more of the inventions nor a listing of features of one or more of the inventions that must be present in all embodiments.

Headings of sections provided in this patent application and the title of this patent application are for convenience only, and are not to be taken as limiting the disclosure in any way.

Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more communication means or intermediaries, logical or physical.

A description of an embodiment with several components in communication with each other does not imply that all such components are required. To the contrary, a variety of optional components may be described to illustrate a wide variety of possible embodiments of one or more of the inventions and in order to more fully illustrate one or more aspects of the inventions. Similarly, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may generally be configured to work in alternate orders, unless specifically stated to the contrary. In other words, any sequence or order of steps that may be described in this patent application does not, in and of itself, indicate a requirement that the steps be performed in that order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred. Also, steps are generally described once per embodiment, but this does not mean they must occur once, or that they may only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some embodiments or some occurrences, or some steps may be executed more than once in a given embodiment or occurrence.

When a single device or article is described herein, it will be readily apparent that more than one device or article may be used in place of a single device or article. Similarly, where more than one device or article is described herein, it will be readily apparent that a single device or article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other embodiments of one or more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be appreciated that particular embodiments may include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of embodiments of the present invention in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those having ordinary skill in the art.

Conceptual Architecture

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating an exemplary system architecture <b>100</b> for measuring and reporting wireless network service quality using remote test devices <b>115</b>, according to a preferred embodiment of the invention. Under the embodiment, data concerning factors that affect the service quality levels of a wireless network of interest <b>120</b> which may include but are not limited to steady state signal strength, variability in signal strength, signal to noise ratio, packet retry rate, used data rate, current percentage of network traffic versus network capacity, radio attachment latency and success rate, resource request latency, web page load time and success rate, access to certain applications over network connection, voice quality, video quality, packet loss, jitter, location of the test device, and ping response time, among others known to those skilled in the art, are collected by wireless network connected remote test device <b>115</b> which most often will be an end-user's mobile device such as a smart phone or tablet running a pre-programmed network test framework application. Remote network test device <b>115</b>, may also, under some circumstances, be a desktop workstation, a laptop computer, a kiosk, an Internet of Things (IoT) device, a wireless access point, modems with wireless (“Wi-Fi”) or optical (“Li-Fi”) capability such as asynchronous digital subscriber link (“ADSL”) or cable modems, or dedicated sensor devices for this purpose. Other test device possibilities exist; those put forth here are meant only as selected examples. A plurality of remote network test devices <b>115</b> may be used in each network service quality test.

While simple spot tests of a network are possible using only the remote network test device with the un-programmed network test framework application, the data collected in this fashion is not associated with data collected from concurrent testing run by other mobile devices and the data are not stored in such a way as to be later used to diagnose network issues or render time based status results of a network. To run continuous or recurring ongoing tests that may be used in further analysis and conclusion generation, the client must have an account which holds, among other information the specific network tests to run, the duration of the data collection for each test, where appropriate, the periodicity at which each test should be run, the number of recurrences for each test, the transformations to be run on each returned data set and potentially, the manner in which results are to be presented, to name a few examples of parameters that may comprise a test from the many known within the field. In this embodiment all of these parameters, with the programming functions to execute the desired network evaluation are stored in a cloud-based test code and configuration server <b>116</b> and, prior to testing, information, including a unique test suite ID, is transferred to a site's central analysis and control server <b>111</b> from which remote network test devices <b>115</b> to be used are programmed and configured. All data collected as part of the test suite will have the unique test suite ID when sent to central analysis and control server <b>111</b> and may be stored in a data store <b>113</b> long-term as such including the unique ID for retrieval and future transformation by central analysis and control server <b>111</b>. Central analysis and control server <b>111</b> also provides an API <b>112</b> that allows it to share output from test analysis to third party network equipment <b>124</b> such as a network management server, to name one of a plurality of examples, and to retrieve important information from wireless network equipment such as maps with access point locations, wireless network firmware and software versions, network configuration information, traffic volume and quality parameters from network equipment, wireless access point <b>121</b> parameters where such data as wireless standards being supported (examples: b, g, a, n, ac), security protocols enforced (examples: WPA, WPA2, WPA Enterprise), access point RF radio transmitted signal strength, broadcast bands enabled (examples: 2.4 GHz, 5 GHz), collisions, retransmissions and percent capacity in use, to list a few members of a larger set of attributes, may be important to diagnosing wireless network service quality issues, proposing remedies, and sending notifications, capabilities for which the analytics engine of central analysis and control server <b>111</b> is programmed should network service levels fall below customer service level expectations.

Based upon programming and test system setup, the remote network test devices may attempt to connect to and download or upload resources to company internal dedicated target test devices <b>122</b>. These devices may be connected to the same networked by cable or connected wirelessly. They may also be connected to another segment or network within the company to allow testing of a wider range of infrastructure components. There may also be other devices present on a company's network infrastructure that are critical to operations that are therefore tested <b>123</b>. An example may be wired or wirelessly connected printers that the company uses which, if not available for jobs, may cause process delays or work stoppages; some may print manufacturing orders, others invoices and still others general duty jobs. Another example may be wirelessly connected or wired scanners. Service requirements for these resources are expected to be high and action in event of significant reduction or loss of function needed swiftly. Similarly, remote devices performing test may be programmed to automatically log in to certain applications like warehouse control systems or sales management systems to test availability of access to them and possibly perform a test query to record overall service availability and response times. Measurement of service to external cloud <b>130</b> resources <b>131</b>, <b>132</b>, <b>133</b> is equally important and testing connectivity and resource availability from external dedicated targets <b>131</b>, discretionary targets <b>133</b>, perhaps a government printer or software as service systems that the company might use. The ability to reach a wide number of high volume web server web pages <b>132</b>, for example FACEBOOK™, SKYPE™, CHROME™ and GOOGLE™ to list a few known to those skilled in the art, as well as customer critical web server web pages is also a good indicator of wireless network service health as it relates to traversing firewalls and gateways, and the health of web servers of specific concern to the customer, if included.

Devices accessing network use several services which control access to resources and provide basic connectivity information. DHCP services allocate IP addresses to devices connecting to network. Authentication and authorization services like Radius provide a wide variety of protocols for authenticating the users prior to allowing access to resources. DNS services allowing use of URLs instead of numeric IP addresses. Device with the test suite use and tests these services and report qualities like success rates, delays and errors with the services. This information is stored and analyzed for further actions.

Devices programmed for test may perform measurements in the background measuring signal levels, data rates, retry rates and throughputs at different times. While the device moves, signal levels vary or when instructed by network, device may change connection to another access point as illustrated in <figref idref="DRAWINGS">FIG. 18</figref>. Assessing characteristics of the roaming behavior can be done be measuring signal level, signal to noise ratio, used data rate, throughput, packet loss and other parameters before and after the roaming event <b>1800</b>. Measuring the time gap between the last data packet from previous access point and the first packet from the new access point provides information on the packet flow interruptions, which are especially important for real time traffic like voice. Collecting this information allows comparing behavior of different device manufacturers, device models, software versions, access points and different network settings and optimizing the service for best user experience.

Resolving certain issues may require more detailed information than the normal test suite can provide. Central analysis and control server may initiate a more detailed test procedure to selected devices. This may include observing and measuring terminal behavior continuously, collecting logs from the terminal, collecting logs triggered by failure to meet predetermined service parameters, entering a special test mode or root/jail break mode which provides more information or recording the device display views at the time of the issue. Central control and analysis server may also be used to ask user to perform certain task.

Test controls define when tests are performed. Test controls include a list of networks which are to be tested. In the absence of an included network, no active test will be attempted. This allows collection of data only from networks of interest, minimized terminal battery consumption and bandwidth consumption if network is metered.

The system also has capability to accept and analyze user experience concerning wireless network service quality during testing <b>115</b>. Either as a matter of normal test suite function or due to the return of test results showing borderline or deficient service quality levels, users may be asked to comment either being given multiple preprogrammed answers from which to choose or through freeform written response which is interpreted by the central analysis and control server <b>111</b>. Extra data provided by receiving user experience feedback provides a more complete picture of impact of lower service levels on actual user experience.

It should be noted that placement components of the invention in this embodiment were made for explanatory reasons. Some components, for example the central analysis and control server, could reside elsewhere without changing its function. The central analysis and control server would function similarly as a corporation local device or as a cloud device, possibly a software as a service appliance subscribed to by the client corporation.

Detailed Description of Exemplary Embodiments

<figref idref="DRAWINGS">FIG. 2</figref> is a flow diagram illustrating an exemplary method <b>200</b> for measuring and reporting wireless network service quality using remote devices, according to a preferred embodiment of the invention. To perform reproducible testing at least one, probably more than one, of remote wireless network testing devices <b>115</b> as depicted in <figref idref="DRAWINGS">FIG. 1</figref> must be programmed with the suite of tests to be run <b>201</b> as depicted in <figref idref="DRAWINGS">FIG. 1</figref><b>116</b>, <b>111</b>, <b>115</b>. These remote test devices <b>115</b> then run the tests specified in test suite <b>202</b>, which are attached to a unique test suite specific ID, such as but not limited to: wireless signal strength at test device location; wireless network BSSID; ability and latency logging onto the wireless network; negotiated wireless standard; signal level, wireless signal band or bands offered; ping round trip times; ability to upload and download data to LAN resident and cloud based dedicated targets, if available, measuring latency, data bandwidth, voice quality, video quality, packet loss, jitter; ability to access specified web pages and retrieve resources; ability to access the company's wireless phone carrier, if applicable; and ability to contact dedicated or critical phone targets and upload or download either voice or other data based on the needs of the customer. There are of course other tests known to the art and no feature precludes the invention from performing any of them should a customer desire. While there are circumstances where the results of one or more wireless network tests are stored on the remote wireless network test device such as inability to functionally access the target wireless network or tests in the programmed test suite that specify that a second RF radio based wireless network be probed where only one RF radio exists on the test device, where, in both cases results are stored and transmitted, each associated with the test suite's ID upon re-establishment of connection with the central analysis and control server <b>111</b> as depicted in <figref idref="DRAWINGS">FIG. 1</figref> routinely; however, test result data are transmitted to central analysis and control server <b>111</b> with the test suite specific ID at time of test completion <b>203</b> such that interim results are as up-to-data as possible. At the central analysis and control server, test result data from remote wireless service quality test devices <b>115</b> may be combined with information retrieved from third party sources <b>204</b> such as wireless access points <b>121</b>, <b>141</b>, that serve the tested wireless networks <b>120</b>, <b>140</b>. Examples of information provided by wireless access points may be, but is not limited to: RF Radio transmit strength, BSSID of the access point, wireless standards supported, wireless standards (example: Wi-Fi b, g, a, n, ac) in use as well as device or devices using each standard, percent bandwidth capacity under which the access point is currently operation and any recent warnings or failures in the access point's logs. Information may also be retrieved from other third party devices <b>124</b> as available. Central analysis and control server <b>111</b> may also receive user perception data <b>205</b> concerning the function of the wireless network during testing, either through pre-designed multiple choice questionnaires or interpreted freeform text descriptions, this is especially probable if service deficiencies are determined by other tests in the suite as knowing user experience during those instances may be especially helpful. Within central analysis and control server <b>111</b>, data received from previously mentioned sources and possibly other sources specific to the needs or the test suite customer, are transformed using pre-programmed analysis functions to produce a representation of all tested aspects of network function. While these representations may be purely numerical, such a display is very difficult to quickly analyze and results are thus usually depicted as a proportion of 100% function with customer-decided minimum service levels as line graphs over a time period of interest <b>206</b>. Given possession of floorplans and wireless access point maps, the system may also display service quality, coverage, congestion, failures, errors, user feedback and similar factors as colored topographical like maps to make interpreting the data and possible problem areas as easy as possible. Drops in service quality below certain limits may have profound consequences on customer operations and therefore central analysis and control server <b>111</b> has APIs to communicate with third party alarm and control systems <b>207</b>. For example, over the past decade the proportion of hospital monitors and actuators that are wirelessly connected has greatly increased due to obvious convenience. However, if wireless service quality falls below a certain level within parts or all of the hospital for some reason, other methods for connect and control of that equipment is needed. The API of the central analysis and control server may communicate with the hospital's administration system to issue the needed advisories and alarms per hospital protocol. The API would, of course, function similarly in many other less drastic situations, but the server also includes analytics programming that allows it to combine data from different sources and different test suite which may uncover small irregularities in previous service levels or current equipment function such that a specific root cause and remedy for an issue at hand may be proposed by the central analysis and control module with the data <b>206</b>. It may be that at least one test in the test suite is meant to recur a certain number of time, possibly indefinitely. The recurrence of tests and their time of run are controlled within the programming <b>208</b> and tests will continue to repeat automatically for the length programmed.

<figref idref="DRAWINGS">FIG. 3</figref> is a flow diagram illustrating an exemplary method <b>300</b> by which the system may programmed with customer specific test suites according to a preferred embodiment of the invention. As previously mentioned, it is possible for an unregistered system user to download the wireless network service quality framework application <b>301</b> onto a compatible remote device (see <figref idref="DRAWINGS">FIG. 1, 115</figref>) and run basic network service quality tests <b>302</b>. While the results of the tests may be individually reviewed on the remote device <b>302</b>, it may not be uploaded to a central analysis server (see <figref idref="DRAWINGS">FIG. 1, 111</figref>) for further analysis or combined with other tests that might be run at approximately the same time either on the same remote test device or on another remote test device. Nor may the results be saved for later specific retrieval <b>303</b>, and are therefore, under these conditions only and isolated snapshot of network function. If a user wants to take full advantage of the capabilities of the system <b>304</b> she must sign up for service, in this embodiment the user signs up for <b>100</b> remote test device service <b>305</b>. At this point an account for the user is created on the central server, a unique ID code is created for her and she is consulted concerning the specific tests that would be most useful under her current wireless network operating conditions and perceived issues <b>306</b>. This results in a suite of tests being created for the user connected to her unique account ID. When prepared to start testing the user need only enter her unique account ID into the running framework application running on each of her remote network test devices <b>307</b>, which will result in her pre-programmed test suite being downloaded <b>308</b> and run <b>310</b> on each of her remote network test devices and this data, associated with the unique account ID may be stored <b>309</b> and used with data bearing the same ID to perform in depth, coordinated analysis by the central server <b>312</b>. With the unique ID, it is now also possible to create more complex testing protocols that are run on each remote network testing device associated with the user's ID <b>311</b>, single tests may be run, continuous tests may be run, tests may be run periodically in the background without intervention, or tests may be uploaded and then triggered by central server. Finally, as the testing runs mature at the user's location, the tests may be modified <b>313</b> to better reflect the user's needs and these revised tests seamlessly uploaded to the remote network test devices from the central server <b>314</b> without remote test device operator intervention.

<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram illustrating an exemplary system architecture for central analysis and control server to analyze network service quality using remote devices, according to a preferred embodiment of the invention. As seen, <b>400</b> central analysis and control server <b>111</b>, and <b>410</b> has many components expected for a computing device <figref idref="DRAWINGS">FIG. 5, 20</figref> such as, but not limited to: operating system <b>420</b>, memory <b>430</b>, CPU <b>455</b> I/O controller module <b>465</b>, video card <b>470</b>, storage drive <b>495</b> and wired network access module <b>475</b>. In addition to these components, the central analysis and control drive also has a wireless service quality test control module <b>450</b>, which receives customer specified test suites and modification parameters from a centralized possibly cloud based wireless network service quality test coding and parameter data store <b>490</b> then uses that programming to configure a plurality of mobile wireless network service quality test devices <b>115</b>, which run the test suites and collect the correct, customer-requested data. Wireless network performance test data are sent back to the wireless service quality test transform module <b>440</b> of central analysis and control server <b>410</b> where specific preprogrammed algorithms are performed against them to establish performance levels using multiple aspects of wireless network operations additional data received from third party sources, such as wireless access points within the probed wireless network and received through application programming interface (API) module <b>460</b> may also take part in the calculations. The analytics engine module <b>445</b> may also take part in the routine calculations. Results of the test suite may be shown to end users on system interaction display <b>480</b>, which may range from a dedicated display and keyboard on the central server to a serial or network attached workstation, in one or more formats specified by the customer prior to test setup <figref idref="DRAWINGS">FIG. 12</figref>.

Deficiencies in network performance compared to customer service level requirements may invoke the collection of additional information through both the mobile wireless network service quality test device <b>115</b> and communication with third party devices through the central server's API <b>460</b>. All gathered data may then be submitted to the central server's analytics engine module where it, data obtained from wireless equipment manufacturers and vendors, specialized predictive programming and data stored from past test wireless network runs may be used to formulate a root cause. Notifications, warnings and alarms specified by the customer will also be implemented at this point. All data will be saved in data store <b>485</b> for possible further analysis and record keeping.

<figref idref="DRAWINGS">FIG. 5</figref> is a flow diagram illustrating an exemplary method <b>500</b> by which central analysis and control server may interact with other components of the system and analyze network performance test data according to a preferred embodiment of the invention. In this embodiment, the central analysis and control server <b>111</b>, <b>410</b> first programs a plurality of mobile network test devices <b>115</b> with test suite and connected parameters <b>501</b>. These test may recur automatically at set time intervals indefinitely, recur a predetermined number of times either regularly or pseudo-randomly, or occur one time only. Mobile network test devices then carry out those tests and send the wireless network performance data back to the centralized analysis and control server <b>502</b>. The centralized server then applies a plurality of customer specified preprogrammed analytical functions on that data <b>504</b> and may also employ data obtained directly from third party wireless network equipment such as, but not limited to: wireless access points and signal amplifiers and repeaters <b>503</b>. A major focus of analysis is to determine whether there are any wireless network performance parameter that fall below pre-established customer service quality level minimums <b>505</b>. When all parameters are found to be at or above pre-established customer service quality level minimums, results are sent to long term storage <b>508</b> and they are presented as per specified customer rendering choices. However, if any of the parameters specified by the customer to have a minimum acceptable service quality level are found to function below that level <b>506</b> customer required, notifications warnings may be automatically sent, and, based on severity and equipment probed, alarms may be activated and messages sent to third party equipment monitoring and control systems through the central server's API <b>112</b>, <b>460</b>. All results are placed into long term storage <b>508</b>.

<figref idref="DRAWINGS">FIG. 6</figref> is a diagram illustrating information that is processed by central analysis and control server analytics engine according to a preferred embodiment of the invention. Within the embodiment <b>600</b>, central server analytics engine module <b>445</b> retrieves and makes use of many data from a plurality of sources to predict wireless network equipment that may be cycling towards failure or after a failure or significant deficiency in service has been discovered though testing, steps that may remedy the issue. Information employed comes from a set that includes the test result database <b>485</b>, test configuration controls <b>601</b> and other sources that may include, but are not limited to, WLAN management data—tag-specific access points reporting errors in log, non-responsive <b>602</b>, mobile network management data <b>603</b>—to inform on wireless phone coverage issues, location service data (whether test device is indoors or outdoors)—test devices outside the building may show poor service, possibly by design <b>604</b>, wired network management data—issues with wired network service may significantly affect resource availability outside of wireless service performance <b>605</b>, deep packet inspection data—streams of packets from host, or malformed packets from host may be easily halted and service quality restored <b>606</b>, application performance management data—heavily used mobile wireless network test device may issue inaccurate performance data <b>607</b>, application data <b>608</b> and current alarm and notification status—In times of issue have notifications been sent, alarms activated <b>609</b>. As described above root cause and potential remedies may be offered by the central analytics engine module <b>601</b>.

<figref idref="DRAWINGS">FIG. 7</figref> is a block diagram illustrating an exemplary system architecture for mobile wireless service quality test device employed to analyze network service quality using remote devices, according to a preferred embodiment of the invention <b>700</b>. The mobile wireless network service quality test device of the embodiment incorporates many of the components present in any handheld mobile device such as: antenna <b>711</b>, RF transceiver and amplifiers <b>712</b>, speaker <b>723</b>, CPU and assorted device controllers <b>713</b>, display controller <b>714</b>, input controller <b>715</b>, operating system <b>717</b>, memory <b>721</b>, mobile network controller <b>716</b>—found in all wireless phone and data network connecting devices and an LCD display, not depicted for simplicity. The device operates as a wireless network performance probe through use of wireless network service quality test framework application <b>718</b> which in itself can run a plurality of simple network performance tests as a standalone device and when programmed with specific customer wireless network test suites by the system's central analysis and control server <b>450</b> can run a larger number of tests and send them back to the central server for further analysis, storage and reporting. Tests are run using the networking resources of the device through the wireless access circuitry present <b>711</b>, <b>712</b>, <b>713</b>, <b>716</b>. Results of each test which may include, but are not limited to steady state signal strength, variability in signal strength, signal to noise ratio, current percentage of network traffic versus network capacity, radio attachment latency, resource request latency, location of the test device, and ping response time are collected and transmitted to the central analysis server by direction of wireless test data collection and delivery module <b>719</b>. The wireless network test device may also disconnect from the primary network temporarily to perform such tests as packet analysis and to scan the radio bandwidths used by the customer's wireless network for interference, multi-path effects. Similarly, the wireless network test device <b>710</b> may disconnect from the internal wireless network to measure the signal strength, connection latency and data rate of the customer's mobile phone provider's voice and data connection. Those skilled in the art may quickly recall other tests to be run, the list provided here is illustrative and should not be thought as limiting to the invention. When not connected to the central analysis and control server <b>410</b>, the mobile wireless network test device uses its resident network test application data store <b>720</b>. Once connection to the central analysis and control server is reestablished, the wireless test data collection and delivery module <b>719</b> retrieves the stored data from the network test application data store <b>720</b> and transmits it to the central analysis and control server <b>410</b>. Under certain circumstances, the mobile wireless test device may send information inquiries to known high volume web sites such as, but not limited to FACEBOOK™, SKYPE™, CHROME™ and GOOGLE™ using wireless test application inbound API <b>722</b> as part of a test suite.

<figref idref="DRAWINGS">FIG. 8</figref> is a flow diagram illustrating an exemplary method by which mobile wireless network service quality device may operate to collect store and transmit test suite data according to a preferred embodiment of the invention <b>800</b>. The mobile wireless network service quality device receives coding and parameters for a customer directed network performance test suite from the central analysis and coding server <b>801</b> which it stores <b>802</b> and uses in conjunction with a preloaded resident test framework application <b>803</b>. The tests of the test suite uploaded may be executed interactively or programmed to run in the background, possibly in an automatic recurring fashion that continues indefinitely or is finite in their repetition <b>803</b>. Single execution tests, are of course, also possible. Occasionally, a test or group of tests cause mobile wireless network test device to temporarily disconnect from primary wireless network and thus central analysis and control server, the target of collected wireless network performance data. Unexpected disruptions in primary network connection can also cause temporary loss of contact with central analysis and control server <b>804</b>. Under these conditions wireless performance test data collected by mobile wireless network test device are stored in a test framework initiated data store <b>805</b>. All wireless network performance data is eventually transmitted to the central analysis and control server <b>410</b> for transformation and long term storage <b>806</b>.

<figref idref="DRAWINGS">FIG. 9</figref> is a diagram illustrating methods by which users of wireless networks being tested might submit their perceived experience to the wireless network service quality test system according to a preferred embodiment of the invention <b>900</b>. It is often helpful when executing equipment performance tests, wireless network service quality tests in this case, to also know how a particular service level is perceived by the equipment users. Correlation between very positive performance test readouts and a similar user perception of service quality solidifies the validity test. Similarly, understanding the effect of what is considered mediocre, or unacceptable performance test readouts on user perception is equally important, if the test results are poor but the user response is positive at the same time, the tests employed may be inaccurate or may not be calibrated correctly. The invention may allow the collection of user experience feedback as part of wireless network service quality testing. In the current embodiment of the invention, user wireless network performance experience is collected using an applet that is part of the wireless network testing framework on the wireless network test device <b>910</b>. A part of the applet displays a multiple choice request for user wireless network performance experience where the user can rate current network performance from one of five choices: “Excellent”, “Good”, “Sufficient”, “Not Sufficient” and “Very Poor” <b>911</b>. The second portion of the applet instructs the user to describe any issues she is having on the wireless network in freeform text <b>912</b> and then gives space to do that <b>913</b>. Response text is interpreted and mapped to standardized terms by the central analytics engine as part of overall testing <b>914</b>. The choice from the first portion <b>911</b> of the applet are similarly used to measure general wireless network performance.

<figref idref="DRAWINGS">FIG. 10</figref> is a diagram illustrating both passive and active testing of more than one wireless network by wireless network service quality test devices according to a preferred embodiment of the invention <b>1000</b>. The system <b>1001</b> has the ability to run tests on two separate networks within a single test suite. In the case of testing the corporation's wireless phone provider network connection <b>1003</b>, this testing may occur actively, attempts to connect to designated phone exchanges, to download designated data or to connect to designated servers and web pages to list a few illustrative examples, or passively, measurements of signal strength at test location, signal to noise ratios, radio and connect acknowledgement latency. Results may be reported to the phone service provider's quality management as well as client corporate analysts once connection with the primary wireless network <b>1004</b> is reestablished at the conclusion of mobile network testing <b>1003</b>. When two separate wireless networks <b>1004</b> that require use of the RF radio is planned, this may be accomplished in two ways. First, the remote test device may have simultaneous dual RF radio capability, in which case the second wireless network not depicted for simplicity may be probed and the resultant data immediately sent to the central analysis and control server <b>410</b> for inclusion in the analysis results made available to the end user <b>480</b>. Alternatively, a remote test device with only single RF capability may disconnect from the first network <b>1004</b>, connect to the second network to be tested and while probing the network, store the resultant data internally. Upon completion of inspection of the second network, the remote test device may disconnect from the second network, re-connect to the first network <b>1004</b> ad send all of the stored result data from the second network to the central analysis and control server <b>410</b> for inclusion in the test results of the current test suite and presentation to the end users <b>480</b>. A second network service quality test device <b>1002</b> with both wireless <b>1004</b> network and wired network <b>1005</b> connection capabilities may be used to execute interactive and background network performance tests, either active such as data throughput, retry number, and ping return time or passive such as deep packet inspection, packet capture and protocol analysis, packet drop rates and wireless radio channel usage is manners similar to those described above. Again, data would be sent to the central analysis and control server <b>410</b> as soon as a connection is reestablished.

<figref idref="DRAWINGS">FIG. 11</figref> Is a list of methods by which the location of individual wireless network service quality test devices may be obtained according to a preferred embodiment of the invention <b>1100</b>. The ability to position each wireless network test device at a specific location within the geography of the network greatly augments the value of the data collected as such things as coverage maps and access point identification can be made. The GPS system <b>1101</b> is an extremely accurate, automated method to establish test device location and also allows determination of test device movement, which can affect test data as the orientation of the device antenna changes and altitude which may play a role in specific circumstances. GPS signals can be weak or erratic indoors where may wireless network performance tests occur, then, other methods such as location determination by proximity to known Wi-Fi SSID/BSSID or visible light based data transmitter (“Li-Fi”) or infrared sensor <b>1102</b>, Wi-Fi location service, RTLS or similar where test device location is determined by differential signal levels, packet propagation delay or signal direction determination through use of a directional antenna pattern <b>1103</b>. Test device location may also be determine using a magnetic location service which has been calibrated for a particular building or floor plan <b>1104</b>. Less reliable, as it relies on manual response and a user's perception of the precise location she finds herself is to have the user identify her location on a floorplan map <b>1105</b> or verbally provide her location in an uncoached, freeform manner <b>1106</b>.

<figref idref="DRAWINGS">FIG. 12</figref> is a diagram displaying example forms of wireless service quality test output according to a preferred embodiment of the invention <b>1200</b>. The invention can display both interim and final results of wireless service quality analysis in a large plurality of ways and display method may be significantly affected by the desires and needs of individual customers. Multiple exemplary display methods from one or more embodiments of the invention are shown in <figref idref="DRAWINGS">FIG. 12A</figref>, and <figref idref="DRAWINGS">FIG. 12B</figref>. The first example, <b>1201</b> illustrates the use of line graphs to show one aspect of wireless network performance, file download from a remote network server to mobile network service quality devices <b>710</b> on two separate wireless networks <b>1204</b>, <b>1205</b>. The x-axis denotes time in hours with six hour intervals specifically labeled <b>1203</b>. The y-axis denotes download speed in megabits per second. It can be seen that download speed varies significantly over time for both tested network <b>1</b><b>1204</b> and tested network <b>2</b><b>1205</b> and variance is cyclic for both networks. The variance is highly synchronized between the two networks with valleys of bandwidth occurring largely concurrently for both network. It should also be noted that for both tested network <b>1</b><b>1204</b> and network <b>2</b><b>1205</b>, these lows fall below service quality minimum set by the customer for this activity <b>1206</b> where warnings may be sent out. Notifications may thus be sent to administrators, technicians, and managers per the customer's problem handling plan. Further, during the test period shown, the performance quality of network <b>2</b><b>1205</b> falls below the level designated by the customer as “critical” on multiple occasions. These events may lead to more drastic notification of a wider group of people within the customer company and may also lead to the activation of one or more alarms to signify critical status as is illustrated in <figref idref="DRAWINGS">FIG. 13</figref>.

A second example shows a pseudo-topographical map which may be superimposed over the floorplan of the area served by a wireless network, or may depict individual wireless access point designations as markers and which relies on the location of wireless network service quality devices. This map conveys three parameters important to wireless network performance: shading represents congestion and retransmission levels with darker shading denoting diminishing service quality; The height of the graph denotes wireless signal strength; and the lines denote user reported wireless service experience with denser groupings indicating diminishing user service quality experience. The pin at <b>1215</b> denotes a wireless access point and the expected high signal strength is seen <b>1214</b>. <b>1213</b> shows that although the signal strength is somewhat diminished with distance from the access point <b>1215</b> and test measured congestion and packet retransmission level appear relatively high, user experience is good to excellent. Two apparent problem areas for this network appear to be at <b>1216</b> and <b>1217</b> where test measured signal strengths are low, test measured network congestion and packet retransmission levels are high and user reported service quality experience is very poor. These results allow the company to take action if desired. Legend <b>1212</b> represents a compass bearing to give an external reference.

A third example shown in <figref idref="DRAWINGS">FIG. 12B, 1220</figref> depicts a map of functional wireless network coverage for a specific customer <b>1221</b>. This coverage map is the result of programmatic transformation by the central control and analysis server followed by display in a format that may be specific to or modified for the needs of the customer. This example should not be seen to limit the capabilities of the invention to display of network parameter data using maps or the complexity that such display may attain but is meant only to introduce such map use in an easy to visualize method.

A fourth example is a set of graphs illustrating test results from a subset of the network parameters from a plurality of possible parameters which include ping response time <b>1230</b>; web page download time <b>1240</b>; network packet throughput <b>1260</b>; voice over IP voice quality <b>1270</b>; TCP download throughput <b>1250</b>; and TCP upload throughput <b>1280</b> from a specific network device. All graphs measure service parameter quality as a function of time <b>1232</b>, <b>1242</b>, <b>1252</b>, <b>1262</b>, <b>1272</b>, <b>1282</b>. The graphs for ping response time <b>1230</b>, web page download time <b>1240</b>, network packet throughput <b>1260</b> and voice over IP voice quality <b>1270</b> report those parameters with respect to customer service level agreement (SLA) requirements <b>1231</b>, <b>1241</b>, <b>1261</b>, <b>1271</b> with service level plotted as the percentage of that SLA <b>1233</b>, <b>1243</b>, <b>1263</b>, <b>1273</b>. Each graph shows a percentage of customer SLA that would lead to warning notifications <b>1234</b>, <b>1244</b>, <b>1264</b>, <b>1274</b> being sent as illustrated in <figref idref="DRAWINGS">FIG. 13</figref>. Some of the graphs plot parameters on which the customer relies to have functioning at or very near 100% of their SLA level <b>1240</b>, <b>1260</b><b>1270</b> whereas the embodiment is set to warn those designated only after ping response drops below approximately 80% of the customer's SLA <b>1234</b>. All four graphs also have a critical service level set <b>1235</b>, <b>1245</b>, <b>1265</b>, <b>1275</b>. Service levels below these critical set points may cause more extreme notifications to be sent, possibly to a larger group of designated customer representatives and alarms may also be activated <b>1300</b>. As an example of such a situation, it can be seen in the ping response time graph <b>1230</b> that service level not only falls below the level where warnings may be sent <b>1234</b>, but falls below <b>1236</b> the customer's “critical” service level <b>1235</b>. This may lead to alarms being activated at that time, possibly per the process illustrated in <figref idref="DRAWINGS">FIG. 13</figref>. The last two graphs <b>1250</b> and <b>1280</b> are plotted using time <b>1252</b>, <b>1282</b>, but use the less derived measure of Mbits/s for the y-axis <b>1251</b>, <b>1281</b> with the extent of the y-axis fitted closely to the maximal throughput <b>1253</b>, <b>1283</b> in both of these situations, warnings <b>1254</b>, <b>1284</b> and critical notifications <b>1255</b>, <b>1285</b> are both set such that a network failure would be required, or were not set by the customer. SLA and MBits/s are two examples of measurements that may be graphed by the invention but are not exhaustive. The invention may be used to display any measurement known to those skilled in the art and appropriate to the parameter being displayed.

<figref idref="DRAWINGS">FIG. 13</figref> is a flow diagram of a possible notification, warning and alarm mechanism to be used when network performance is found to be below customer established service level minimums according to a preferred embodiment of the invention <b>1300</b>. Wireless service service quality test results once analyzed <b>1301</b> by the central analysis and control server <b>410</b>, may be compared to customer service level requirements <b>1302</b>. Often all results will be above customer minimums and those results may be display according customer specifications <b>1303</b>. Otherwise, network wide notifications <b>1304</b> will be sent to those representatives designated by the company according to the customer's notification policy. Messages or readouts identifying the faulty equipment or suggesting remedial steps as calculated by the central analytics engine module <b>445</b> may also be sent or displayed <b>1305</b>. If there are customer alarms set for the determined deficiencies <b>1306</b>, signals to activate those alarms will be sent <b>1307</b>. Again, messages or readouts identifying the faulty equipment or suggesting remedial steps as calculated by the central analytics engine module <b>445</b> may be sent of displayed <b>1308</b>. It is also possible that the analyzed test results show that only a specific portion of the wireless network tested is impacted <b>1309</b>, if the customer has policy for such circumstances those will be followed <b>1310</b>, if not the information may be included in any general network notifications. Once again, messages or readouts identifying the faulty equipment or suggesting remedial steps as calculated by the central analytics engine module <b>445</b> may be sent of displayed. In all cases the test data will be displayed on the interactive terminal <b>480</b> and placed in long-term storage <b>1312</b>.

<figref idref="DRAWINGS">FIG. 18</figref> is a diagram illustrating a change of access point during wireless network user roaming, according to an embodiment of the invention <b>1800</b>. Roaming within an area of network coverage by a user with a mobile device along a spatial dimension <b>1802</b> may present special challenges for network service quality, especially during certain network tasks. The issue may occur at a point where signal strength <b>1801</b> of a first access point <b>1804</b> decreases and signal strength of a second, adjacent access <b>1803</b> point increases to the point where the user's network device switches connection from the first access point to the second access point at a particular time <b>1817</b>. As illustrated, this event <b>1817</b> may be configured such that the signal level from the first access point <b>1804</b> will have decreased significantly below <b>1806</b> the available signal strength <b>1805</b> of the second access point <b>1803</b>, towards which the user is traveling, before the wireless device switches access points at time <b>1817</b> so as to minimize rapid, repetitive, switches of the device from one to the other access point. The point of the roaming event <b>1817</b> at access point coverage boundaries may be mapped, and is affected by several factors including but not limited to environment (such as weather), mobile or transient obstructions, user device characteristics, and radio interference from other equipment.

The rate of occurrence of roaming events <b>1817</b> is minimized as, during each event <b>1817</b>, there is a period of time, usually quite small, where the user device is breaking down the connection to the first access point <b>1807</b> and establishing the connection with the second access point <b>1809</b>; during this time packets may be lost <b>1808</b>, incurring an effective “roaming delay”. Often, this event will go unnoticed by the user as the lost packets are retransmitted and the packet stream at the point of the roaming event <b>1817</b> reassembled with no noticeable effect. Some applications, such as (but not limited to) voice over IP, are highly sensitive to packet loss and loss of coherence or stutter may occur at points of roaming delay <b>1808</b>. It is also possible that operation of one or more of the access points in a customer's network may lead to either static or transient elongation of roaming delay events <b>1808</b>, which may lead the customer to implement an embodiment of the invention that tests this parameter at critical boundaries within the network.

<figref idref="DRAWINGS">FIG. 19</figref> is a table <b>1900</b> of network parameters that may be tested according to an embodiment of the invention. There are many network parameters that are measurable, many of which may be of great importance to customers reliant on knowing and maintaining the service quality level of their wireless networks at one or more sites, and which can be tested and analyzed by embodiments of the invention. These measurable and analyzable parameters range from level <b>1</b><b>1901</b> of the OSI model to level <b>7</b><b>1902</b> of that model. A non-exhaustive listing of testable parameters at each OSI level are listed in table <b>1900</b>. These range from radio spectrum analysis of bands associated with wireless network function and properties of the radio signals at the network site at the physical level; beaconing, and traffic volume at the network level; and packet loss and throughput of UDP, TCP and ICMP packets at the transport level, just to list a few examples from the already somewhat limited example listing <b>1900</b>. This list <b>1900</b> is purely illustrative and the absence of a parameter from the listed parameters does not at all imply that the invention is incapable of probing its functional level. The invention is designed to probe any measurable parameter known to those skilled in the art and requested by customer specification.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented on hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into network applications, on a specially constructed machine, on an application-specific integrated circuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of the embodiments disclosed herein may be implemented on a programmable network-resident machine (which should be understood to include intermittently connected network-aware machines) selectively activated or reconfigured by a computer program stored in memory. Such network devices may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols. A general architecture for some of these machines may be described herein in order to illustrate one or more exemplary means by which a given unit of functionality may be implemented. According to specific embodiments, at least some of the features or functionalities of the various embodiments disclosed herein may be implemented on one or more general-purpose computers associated with one or more networks, such as for example an end-user computer system, a client computer, a network server or other server system, a mobile computing device (e.g., tablet computing device, mobile phone, smartphone, laptop, or other appropriate computing device), a consumer electronic device, a music player, or any other suitable electronic device, router, switch, or other suitable device, or any combination thereof. In at least some embodiments, at least some of the features or functionalities of the various embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, virtual machines hosted on one or more physical computing machines, or other appropriate virtual environments).

Referring now to <figref idref="DRAWINGS">FIG. 14</figref>, there is shown a block diagram depicting an exemplary computing device <b>10</b> suitable for implementing at least a portion of the features or functionalities disclosed herein. Computing device <b>10</b> may be, for example, any one of the computing machines listed in the previous paragraph, or indeed any other electronic device capable of executing software- or hardware-based instructions according to one or more programs stored in memory. Computing device <b>10</b> may be configured to communicate with a plurality of other computing devices, such as clients or servers, over communications networks such as a wide area network a metropolitan area network, a local area network, a wireless network, the Internet, or any other network, using known protocols for such communication, whether wireless or wired.

In one embodiment, computing device <b>10</b> includes one or more central processing units (CPU) <b>12</b>, one or more interfaces <b>15</b>, and one or more busses <b>14</b> (such as a peripheral component interconnect (PCI) bus). When acting under the control of appropriate software or firmware, CPU <b>12</b> may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine. For example, in at least one embodiment, a computing device <b>10</b> may be configured or designed to function as a server system utilizing CPU <b>12</b>, local memory <b>11</b> and/or remote memory <b>16</b>, and interface(s) <b>15</b>. In at least one embodiment, CPU <b>12</b> may be caused to perform one or more of the different types of functions and/or operations under the control of software modules or components, which for example, may include an operating system and any appropriate applications software, drivers, and the like.

CPU <b>12</b> may include one or more processors <b>13</b> such as, for example, a processor from one of the Intel, ARM, Qualcomm, and AMD families of microprocessors. In some embodiments, processors <b>13</b> may include specially designed hardware such as application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field-programmable gate arrays (FPGAs), and so forth, for controlling operations of computing device <b>10</b>. In a specific embodiment, a local memory <b>11</b> (such as non-volatile random access memory (RAM) and/or read-only memory (ROM), including for example one or more levels of cached memory) may also form part of CPU <b>12</b>. However, there are many different ways in which memory may be coupled to system <b>10</b>. Memory <b>11</b> may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, and the like. It should be further appreciated that CPU <b>12</b> may be one of a variety of system-on-a-chip (SOC) type hardware that may include additional hardware such as memory or graphics processing chips, such as a QUALCOMM SNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly common in the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to those integrated circuits referred to in the art as a processor, a mobile processor, or a microprocessor, but broadly refers to a microcontroller, a microcomputer, a programmable logic controller, an application-specific integrated circuit, and any other programmable circuit.

In one embodiment, interfaces <b>15</b> are provided as network interface cards (NICs). Generally, NICs control the sending and receiving of data packets over a computer network; other types of interfaces <b>15</b> may for example support other peripherals used with computing device <b>10</b>. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like. In addition, various types of interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radio frequency (RF), BLUETOOTH™, near-field communications (e.g., using near-field magnetics), 802.11 (Wi-Fi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) or external SATA (ESATA) interfaces, high-definition multimedia interface (HDMI), digital visual interface (DVI), analog or digital audio interfaces, asynchronous transfer mode (ATM) interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale (POS) interfaces, fiber data distributed interfaces (FDDIs), and the like. Generally, such interfaces <b>15</b> may include physical ports appropriate for communication with appropriate media. In some cases, they may also include an independent processor (such as a dedicated audio or video processor, as is common in the art for high-fidelity A/V hardware interfaces) and, in some instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in <figref idref="DRAWINGS">FIG. 14</figref> illustrates one specific architecture for a computing device <b>10</b> for implementing one or more of the inventions described herein, it is by no means the only device architecture on which at least a portion of the features and techniques described herein may be implemented. For example, architectures having one or any number of processors <b>13</b> may be used, and such processors <b>13</b> may be present in a single device or distributed among any number of devices. In one embodiment, a single processor <b>13</b> handles communications as well as routing computations, while in other embodiments a separate dedicated communications processor may be provided. In various embodiments, different types of features or functionalities may be implemented in a system according to the invention that includes a client device (such as a tablet device or smartphone running client software) and server systems (such as a server system described in more detail below).

Regardless of network device configuration, the system of the present invention may employ one or more memories or memory modules (such as, for example, remote memory block <b>16</b> and local memory <b>11</b>) configured to store data, program instructions for the general-purpose network operations, or other information relating to the functionality of the embodiments described herein (or any combinations of the above). Program instructions may control execution of or comprise an operating system and/or one or more applications, for example. Memory <b>16</b> or memories <b>11</b>, <b>16</b> may also be configured to store data structures, configuration data, encryption data, historical system operations information, or any other specific or generic non-program information described herein.

Because such information and program instructions may be employed to implement one or more systems or methods described herein, at least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein. Examples of such nontransitory machine-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory (as is common in mobile devices and integrated systems), solid state drives (SSD) and “hybrid SSD” storage drives that may combine physical components of solid state and hard disk drives in a single hardware device (as are becoming increasingly common in the art with regard to personal computers), memristor memory, random access memory (RAM), and the like. It should be appreciated that such storage means may be integral and non-removable (such as RAM hardware modules that may be soldered onto a motherboard or otherwise integrated into an electronic device), or they may be removable such as swappable flash memory modules (such as “thumb drives” or other removable media designed for rapidly exchanging physical storage devices), “hot-swappable” hard disk drives or solid state drives, removable optical storage discs, or other such removable media, and that such integral and removable storage media may be utilized interchangeably. Examples of program instructions include both object code, such as may be produced by a compiler, machine code, such as may be produced by an assembler or a linker, byte code, such as may be generated by for example a JAVA™ compiler and may be executed using a Java virtual machine or equivalent, or files containing higher level code that may be executed by the computer using an interpreter (for example, scripts written in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may be implemented on a standalone computing system. Referring now to <figref idref="DRAWINGS">FIG. 15</figref>, there is shown a block diagram depicting a typical exemplary architecture of one or more embodiments or components thereof on a standalone computing system. Computing device <b>20</b> includes processors <b>21</b> that may run software that carry out one or more functions or applications of embodiments of the invention, such as for example a client application <b>24</b>. Processors <b>21</b> may carry out computing instructions under control of an operating system <b>22</b> such as, for example, a version of MICROSOFT WINDOWS™ operating system, APPLE OSX™ or iOS™ operating systems, some variety of the Linux operating system, ANDROID™ operating system, or the like. In many cases, one or more shared services <b>23</b> may be operable in system <b>20</b>, and may be useful for providing common services to client applications <b>24</b>. Services <b>23</b> may for example be WINDOWS™ services, user-space common services in a Linux environment, or any other type of common service architecture used with operating system <b>21</b>. Input devices <b>28</b> may be of any type suitable for receiving user input, including for example a keyboard, touchscreen, microphone (for example, for voice input), mouse, touchpad, trackball, or any combination thereof. Output devices <b>27</b> may be of any type suitable for providing output to one or more users, whether remote or local to system <b>20</b>, and may include for example one or more screens for visual output, speakers, printers, or any combination thereof. Memory <b>25</b> may be random-access memory having any structure and architecture known in the art, for use by processors <b>21</b>, for example to run software. Storage devices <b>26</b> may be any magnetic, optical, mechanical, memristor, or electrical storage device for storage of data in digital form (such as those described above, referring to <figref idref="DRAWINGS">FIG. 14</figref>). Examples of storage devices <b>26</b> include flash memory, magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implemented on a distributed computing network, such as one having any number of clients and/or servers. Referring now to <figref idref="DRAWINGS">FIG. 16</figref>, there is shown a block diagram depicting an exemplary architecture <b>30</b> for implementing at least a portion of a system according to an embodiment of the invention on a distributed computing network. According to the embodiment, any number of clients <b>33</b> may be provided. Each client <b>33</b> may run software for implementing client-side portions of the present invention; clients may comprise a system <b>20</b> such as that illustrated in <figref idref="DRAWINGS">FIG. 15</figref>. In addition, any number of servers <b>32</b> may be provided for handling requests received from one or more clients <b>33</b>. Clients <b>33</b> and servers <b>32</b> may communicate with one another via one or more electronic networks <b>31</b>, which may be in various embodiments any of the Internet, a wide area network, a mobile telephony network (such as CDMA or GSM cellular networks), a wireless network (such as Wi-Fi, WiMAX, LTE, and so forth), or a local area network (or indeed any network topology known in the art; the invention does not prefer any one network topology over any other). Networks <b>31</b> may be implemented using any known network protocols, including for example wired and/or wireless protocols.

In addition, in some embodiments, servers <b>32</b> may call external services <b>37</b> when needed to obtain additional information, or to refer to additional data concerning a particular call. Communications with external services <b>37</b> may take place, for example, via one or more networks <b>31</b>. In various embodiments, external services <b>37</b> may comprise web-enabled services or functionality related to or installed on the hardware device itself. For example, in an embodiment where client applications <b>24</b> are implemented on a smartphone or other electronic device, client applications <b>24</b> may obtain information stored in a server system <b>32</b> in the cloud or on an external service <b>37</b> deployed on one or more of a particular enterprise's or user's premises.

In some embodiments of the invention, clients <b>33</b> or servers <b>32</b> (or both) may make use of one or more specialized services or appliances that may be deployed locally or remotely across one or more networks <b>31</b>. For example, one or more databases <b>34</b> may be used or referred to by one or more embodiments of the invention. It should be understood by one having ordinary skill in the art that databases <b>34</b> may be arranged in a wide variety of architectures and using a wide variety of data access and manipulation means. For example, in various embodiments one or more databases <b>34</b> may comprise a relational database system using a structured query language (SQL), while others may comprise an alternative data storage technology such as those referred to in the art as “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLE BIGTABLE™, and so forth). In some embodiments, variant database architectures such as column-oriented databases, in-memory databases, clustered databases, distributed databases, or even flat file data repositories may be used according to the invention. It will be appreciated by one having ordinary skill in the art that any combination of known or future database technologies may be used as appropriate, unless a specific database technology or a specific arrangement of components is specified for a particular embodiment herein. Moreover, it should be appreciated that the term “database” as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system. Unless a specific meaning is specified for a given use of the term “database”, it should be construed to mean any of these senses of the word, all of which are understood as a plain meaning of the term “database” by those having ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or more security systems <b>36</b> and configuration systems <b>35</b>. Security and configuration management are common information technology (IT) and web functions, and some amount of each are generally associated with any IT or web systems. It should be understood by one having ordinary skill in the art that any configuration or security subsystems known in the art now or in the future may be used in conjunction with embodiments of the invention without limitation, unless a specific security <b>36</b> or configuration system <b>35</b> or approach is specifically required by the description of any specific embodiment.

<figref idref="DRAWINGS">FIG. 17</figref> shows an exemplary overview of a computer system <b>40</b> as may be used in any of the various locations throughout the system. It is exemplary of any computer that may execute code to process data. Various modifications and changes may be made to computer system <b>40</b> without departing from the broader scope of the system and method disclosed herein. Central processor unit (CPU) <b>41</b> is connected to bus <b>42</b>, to which bus is also connected memory <b>43</b>, nonvolatile memory <b>44</b>, display <b>47</b>, input/output (I/O) unit <b>48</b>, and network interface card (NIC) <b>53</b>. I/O unit <b>48</b> may, typically, be connected to keyboard <b>49</b>, pointing device <b>50</b>, hard disk <b>52</b>, and real-time clock <b>51</b>. NIC <b>53</b> connects to network <b>54</b>, which may be the Internet or a local network, which local network may or may not have connections to the Internet. Also shown as part of system <b>40</b> is power supply unit <b>45</b> connected, in this example, to a main alternating current (AC) supply <b>46</b>. Not shown are batteries that could be present, and many other devices and modifications that are well known but are not applicable to the specific novel functions of the current system and method disclosed herein. It should be appreciated that some or all components illustrated may be combined, such as in various integrated applications, for example Qualcomm or Samsung system-on-a-chip (SOC) devices, or whenever it may be appropriate to combine multiple capabilities or functions into a single hardware device (for instance, in mobile devices such as smartphones, video game consoles, in-vehicle computer systems such as navigation or multimedia systems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems or methods of the present invention may be distributed among any number of client and/or server components. For example, various software modules may be implemented for performing various functions in connection with the present invention, and such modules may be variously implemented to run on server and/or client components.

The skilled person will be aware of a range of possible modifications of the various embodiments described above. Accordingly, the present invention is defined by the claims and their equivalents.