Peered Proctoring

CROSS-REFERENCE TO RELATED APPLICATIONS

0001The present application is a continuation-in-part and claims the priority benefit of U.S. patent application Ser. No. 12/850,136, filed on Aug. 4, 2010, entitled “Optimized Data Stream Upload,” the disclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

00021. Field of the Invention

0003The present invention generally relates to streaming multimedia data. More specifically, the present invention concerns managing the real-time distribution of a live data stream such as a live video data stream from a computing device.

00042. Description of Related Art

0005Examinations are used to determine the ability of an exam taker such as a student or prospective practitioner as it pertains to proficiency in a particular subject or skill set. For example, a student might take an exam to determine whether the student possesses requisite knowledge in a particular subject that might be related to receiving a degree or certificate. A prospective practitioner of law or medicine similarly might sit for an examination to determine their competence as it pertains to practicing in that profession.

0006Students or prospective practitioners have historically gathered at the designated locale for an examination on a proscribed date and time. Examination materials are then handed out by a testing authority and the exam begins. During the allotted time, the exam takers read questions and provide answers on a provided answer sheet or in a ‘blue book.’ Throughout the course of examination, a teacher or proctor keeps careful watch over the exam takers to ensure that no instances of cheating are taking place. While a single proctor may be able to observe a small group of exam takers, such observation becomes more difficult for a larger exam taking pool or for a group of exam takers utilizing laptop computers or other computing devices.

0007The increased popularity of distance learning has also complicated proctoring of examinations. The distance learning instructional model delivers education material and information to students who are not physically ‘on site’ at an education facility. Distance learning provides access to learning opportunities when the source of the information and the student are separated by time or distance if not both. Thousands of distance learners may be involved in a particular distance learning program or course at any given time.

0008Distance learning is no different than any other educational program in that there is a need to verify the qualifications of students through examination. Because distance learners are not collectively gathered at a physical learning institution such as a university, the distance learning program often requires that the students attend an examination center—which defeats a purpose of distance learning—or administers an examination online. An online examination is difficult to proctor as an exam taker could be taking an examination in one window of a web browser while looking up answers in another window via the Internet. An exam taker could also utilize a ‘chat’ or ‘messaging’ application to relay questions to and receive answers from a knowledgeable third-party. The value of online examinations is, therefore, questionable and calls into question the overall value of the corresponding class or degree program.

0009Techniques for remotely monitoring the visual and/or audio environment surrounding the exam taker, through the use of an environment capture device such as a video camera or other capture device, are disclosed in commonly-owned U.S. patent application Ser. No. 12/723,666, the disclosure of which is incorporated herein by reference. In order to ensure the integrity of the exam process, the real-time audio and/or video data streams of the testing environment can be provided over a network to a remote proctor who is responsible for remotely observing the administration of the exam.

0010Uploading the data stream and subsequently providing it in real-time to a remote proctor is challenging for a number of reasons including high bit rates, delay, and loss sensitivity. In particular, the data stream is vulnerable to upload restrictions and/or disruptions along the ‘first mile’ from the test taker's computing device into the network, which can cause packet loss and delay of data in the data stream. This can be due to physical upload bandwidth limitations through the use of dial-up modems, or other upload restrictions such as network service provider limitations. As a result, the so-called ‘first mile’ can become a bottleneck, which can render further downstream techniques ineffective, since these downstream techniques cannot improve the bit rate and stream integrity of the initially uploaded data stream.

0011Similarly, the data stream is vulnerable to download restrictions and/or disruptions along the ‘last mile’ from the network to the remote proctor's computing device. As a result, the so-called ‘last mile’ can also become a bottleneck, which can render the previous upstream techniques for improving the bit rate and stream integrity ineffective. These download restrictions and disruptions also hamper the observation of the exam taker during the administration of the exam, which may call into question the integrity of the exam process.

0012There is a need in the art for managing the distribution of a data stream, which addresses first mile and last mile vulnerability issues. There is a similar need for improved remote proctoring of large scale examinations such that a small number of proctors can properly maintain the integrity of the testing environment, notwithstanding the large number of exam takers.

SUMMARY OF THE CLAIMED INVENTION

0013Systems and methods described herein utilize a distributed server network to allow for the real-time distribution of copies of a data stream uploaded from a computing device. The uploaded data stream corresponds to an environment surrounding a user (exam taker) of the computing device during execution of a testing routine for an examination. Providing copies of the data stream in real-time allows proctors as well as other third party test assessment authorities (e.g., a professor) to ‘peek in’ on the exam taker's environment during the exam. As a result, the environment surrounding the exam taker can be monitored in real-time to determine whether any visual or audio activity in the environment constitutes activity not in accordance with the exam protocol. Following an affirmative determination that the exam taker violated an exam taking protocol, a proctor or other third party test assessment authority can then make a final or early decision regarding disciplinary action.

0014The management of the distribution of the copies utilizing the distributed server network also allows for the real-time sharing and shifting of proctoring responsibilities, as well as those of the third party test assessment authorities. Advantageously, this provides for remote proctoring of large scale examinations such that a small number of proctors and assessment authorities can properly maintain the integrity of the testing environment, notwithstanding the large number of exam takers.

0015A method is described herein for managing distribution of a data stream from a distributed server network. The method includes receiving a data stream into the distributed server network. The data stream corresponds to an environment surrounding a user of a first computing device during execution of a testing routine for an examination. The method also includes receiving a request to transmit the data stream to a second computing device. The request is received during execution of the testing routine. The data stream is then transmitted from a server in the distributed server network to the second computing device in response to the request. The data stream is transmitted during execution of the testing routine.

0016A computer readable storage medium as described herein has embodied thereon a program, the program being executable by a processor to perform a method as described above managing distribution of a data stream from a distributed server network.

BRIEF DESCRIPTION OF THE DRAWINGS

0017<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system for providing optimization of entry and subsequent real-time distribution of a data stream uploaded into a distributed server network.

0018<figref idref="DRAWINGS">FIG. 2</figref> illustrates a method for implementing an online proctored examination.

0019<figref idref="DRAWINGS">FIG. 3</figref> illustrates a method for uploading a data stream of the examination environment surrounding the exam taker during an online proctored examination.

0020<figref idref="DRAWINGS">FIG. 4</figref> is illustrates a method for selecting an entry server to receive an encoded data stream.

0021<figref idref="DRAWINGS">FIG. 5</figref> illustrates a method for transmitting an uploaded data stream of an examination environment to a proctor or other third party assessment authority during an online proctored examination.

0022<figref idref="DRAWINGS">FIG. 6</figref> illustrates a first interface that may be utilized in proctoring an online examination.

0023<figref idref="DRAWINGS">FIG. 7</figref> illustrates a second interface that may be utilized in proctoring an online examination upon detecting aberrant behavior in the interface of <figref idref="DRAWINGS">FIG. 6</figref>.

DETAILED DESCRIPTION

0024Systems and methods described herein utilize a distributed server network to allow for the real-time distribution of copies of a data stream uploaded from a computing device. The uploaded data stream corresponds to an environment surrounding a user (exam taker) of the computing device during execution of a testing routine for an examination. A video camera device or other visual and/or audio environment capture device may be used to generate the data stream. This data stream is encoded and uploaded into the distributed server network. The distributed server network can then transmit copies of the data stream for playback on the computing device of a remote proctor, as well as other third party test assessment authorities (e.g., a professor), during the administration of the examination.

0025Providing copies of the data stream in real-time allows proctors as well as other third party test assessment authorities to ‘peek in’ on the exam taker's environment during the exam. As a result, the environment surrounding the exam taker can be monitored in real-time to determine whether any visual or audio activity in the environment constitutes activity not in accordance with the exam protocol. Following an affirmative determination that the exam taker violated an exam taking protocol, the proctor or other third party test assessment authority can then make a final or early decision regarding disciplinary action.

0026The management of the distribution of the copies utilizing the distributed server network also allows for the real-time sharing and shifting of proctoring responsibilities, as well as those of the third party test assessment authorities. Advantageously, this provides for remote proctoring of large scale examinations such that a small number of proctors and third party authorities can properly maintain the integrity of the testing environment, notwithstanding the large number of exam takers.

0027<figref idref="DRAWINGS">FIG. 1</figref> illustrates a system <b>100</b> for providing optimization of entry and subsequent real-time distribution of a data stream uploaded into a distributed server network such as a content delivery network (CDN) <b>162</b>. The system <b>100</b> of <figref idref="DRAWINGS">FIG. 1</figref> includes computing devices <b>110</b>, <b>112</b>, <b>114</b> that may be utilized by a user (exam taker) to take an examination, CDN <b>162</b> for administering an examination and for routing exam data including real-time data streams via a communications network <b>150</b>, a central office proctoring center <b>180</b>, and geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b>.

0028The computing devices <b>110</b>, <b>112</b>, <b>114</b> may be any sort of computing device as is known in the art. The computing devices <b>110</b>, <b>112</b>, <b>114</b> include memory for storage of data and software applications, a processor for accessing data and executing applications, and input and output devices that allow for user interaction. The computing devices <b>110</b>, <b>112</b>, <b>114</b> further include components that facilitate communication over the communications network <b>150</b> such as an RJ-45 connection for use in twisted pair based 10baseT networks or a wireless network interface card allowing for connection to a radio-based communication network (e.g., an 802.11 wireless network).

0029The computing devices <b>110</b>, <b>112</b>, <b>114</b> may each be a general purpose computing device such as a desktop or laptop computer. The computing devices <b>110</b>, <b>112</b>, <b>114</b> may belong to a particular user rather than being a computing device dedicated to exam taking as might be found in a examination center. Thin client or netbook client devices may be implemented in the context of computing devices <b>110</b>, <b>112</b>, <b>114</b> as might mobile computing devices such as smart phones.

0030Communication network <b>150</b> may be a local, proprietary network (e.g., an intranet) and/or may be a part of a larger wide-area network. The communication network <b>150</b> may be a local area network (LAN), which may be communicatively coupled to a wide area network (WAN) such as the Internet. The Internet is a broad network of interconnected computers and servers allowing for the transmission and exchange of Internet Protocol (IP) data between users connected through a network service provider. Examples of network service providers are the public switched telephone network, a cable service provider, a provider of digital subscriber line (DSL) services, or a satellite service provider. Communication network <b>150</b> allows for communication between the various components of system <b>100</b>.

0031The computing device <b>110</b> is representative of the computing devices <b>110</b>, <b>112</b>, <b>114</b>. As described below, a video camera device <b>120</b> or other visual and/or audio environment capture device is used to monitor the exam taking environment surrounding the user (exam taker) of the computing device <b>110</b> during administration of an examination. The video camera device <b>120</b> generates a real-time data stream of the exam taking environment. This data stream is encoded and uploaded by the computing device <b>110</b> into the CDN <b>162</b>, which in turn transmits the data stream in real-time to the central office proctoring center <b>180</b>, and/or one or more geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b>.

0032A proctor or other third party assessment authority may then view the data stream via a streaming technique in real-time (i.e., during administration of the examination) utilizing a viewing application (e.g., view application <b>196</b>), and determine if any visual activity in the examination environment constitutes activity not in accordance with the exam protocol. The term “streaming” as used herein refers to the transmission of data in the data stream by the CDN <b>162</b> to the computing device of an end user such as proctor, without requiring that the data stream first be uploaded in its entirety to the CDN <b>162</b>.

0033The CDN <b>162</b> is coupled to the communications network <b>150</b>. The CDN includes a CDN data center <b>170</b> and a plurality of entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. In the system <b>100</b> illustrated in <figref idref="DRAWINGS">FIG. 1</figref>, six entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> are shown. Notwithstanding the six illustrated servers, the system <b>100</b> may include more of less entry servers.

0034The CDN data center <b>170</b> also manages the real-time upload and distribution of data streams of the examination environments from the computing devices <b>110</b>, <b>112</b>, <b>114</b>, as well as the storing of the data streams for retrieval and playback in a non-streaming fashion. The CDN data center <b>170</b> includes an application server <b>172</b>, database server <b>174</b>, and media server <b>176</b>. The application server <b>172</b>, database server <b>174</b>, and media server <b>176</b> are each a computing device and include memory, a processor for accessing data and executing applications, and components to facilitate communication over communications network <b>150</b>. The application server <b>172</b> includes an upload/download optimization module <b>173</b> stored in memory and executed by a processor to invoke its corresponding functionality.

0035The optimization module <b>173</b> is executable to handle assignment requests from the computing devices <b>110</b>, <b>112</b>, <b>114</b> to upload data streams such as those captured by the environment capture devices during administration of an exam. These assignment requests are analyzed by the optimization module <b>173</b> to select one or more “optimum” entry servers in the plurality of entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. The “optimum” entry server or servers then serve as an entry point or points for the data stream into the CDN <b>162</b>. The database server <b>174</b> stores entry server information such as data indicating the physical or network location of each of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. This information can then be used by the optimization module <b>173</b> to select the entry point or points for the data stream.

0036The optimization module <b>173</b> is also executable to handle requests to provide the data stream to the central office monitoring center <b>180</b>, and distributed proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b>. These requests are analyzed by the optimization module <b>173</b> to select one or more “optimum” entry servers in plurality of entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. The “optimum” entry server or servers then serve as the source or sources of the data stream out of the CDN <b>162</b>.

0037The optimization module <b>173</b> in the illustrated embodiment includes a rate allocation algorithm and a network/location proximity algorithm. The rate allocation algorithm is used to determine the sending rate or upload bandwidth from each the computing devices <b>110</b>, <b>112</b>, <b>114</b> to assist in minimizing packet loss in the uploaded data streams. Similarly, the rate algorithm is used to determine the receiving rate or download bandwidth to each of the central office monitoring center <b>180</b> and distributed proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b> to assist in minimizing packet loss in the downloaded data streams.

0038The network/location proximity algorithm is used to determine the location of the computing devices <b>110</b>, <b>112</b>, <b>114</b> to assist in minimizing the probability of packets arriving late to the selected entry server. Similarly, the network/location proximity algorithm is used to determine the location of the distributed proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b> to assist in minimizing the probability of packets arriving there late. The optimization module <b>173</b> is described in further detail below with respect to <figref idref="DRAWINGS">FIGS. 3 to 5</figref>.

0039The application server <b>172</b> also operates as an origin server for the exam content during the administration and proctoring of an examination. The exam content includes the uploaded data streams.

0040The exam content may also include exam data such as the actual exam (e.g. prompts and questions) or other data concerning an exam, as well and answers to the questions provided by the exam takers during an examination. Registration information of the exam takers, such as a name or examination identification number as well as a password, may also be stored in the database server <b>174</b>. Other registration information might include a geographic location or address provided by the exam taker and proctor via a graphical user interface.

0041Biometric information such as a visual image of the exam taker may also be stored in the database server <b>174</b> and compared against a previously stored and known ‘good’ image of the exam taker. A similar comparison may be made with respect to a voice print. Retinal scans and finger prints, subject to the presence of the appropriate peripheral device, may also be stored and used for verifying exam taker identity. These peripheral devices may be implemented in the context of the use of video camera device <b>120</b>, microphone <b>130</b>, or other environment capture device.

0042The media server <b>176</b> handles the requests for the non real-time playback of the uploaded data streams, which may be stored in the database server <b>174</b> as well as the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>.

0043The entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>, in conjunction with the application server <b>172</b> and database server <b>174</b>, provide high-performance upload and delivery of the exam content to the exam takers and the proctors.

0044Each of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> is a computing device which includes memory, a processor for accessing data and executing applications, and components to facilitate communication over communications network <b>150</b>. The application server <b>172</b> handles the distribution of copies of the exam content to the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. The exam content can then be stored and transmitted to the exam takers and proctors by way of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> using a variety of different load balancing techniques, instead of being transmitted directly by the application server <b>172</b>. The distribution of the exam content to the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> may include on-demand or push-based mechanisms that move the exam content from the database server <b>174</b> to the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. In addition, the distribution of the uploaded data streams may consist of on-demand or push-based mechanisms that move the uploaded data streams between entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>, without requiring that the uploaded data stream first enter the CDN data center <b>170</b>.

0045The entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> are arranged at various physically remote locations or regions. For example, entry server <b>160</b>-<b>1</b> may be located in North America, entry server <b>160</b>-<b>2</b> may be located in Asia, and entry server <b>160</b>-<b>3</b> may be located in Europe. The entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> may be arranged at the “edge” of the communication network <b>150</b> so that the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> can be located physically close to the exam takers and proctors. Due to this physical proximity, the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> can provide faster, higher quality data transmissions to achieve better load balancing, lower latency, and higher throughput in delivering the data streams than may be otherwise achieved using solely the application server <b>172</b>.

0046Central office proctoring center <b>180</b> is an operations center with computing devices staffed with one or more proctors or other test assessment authorities observing the data streams of various examination environments for exam takers at one or more examination sites. These examination sites may be physically remote from the central office proctoring center <b>180</b>. Examination sites can be examination centers dedicated to the offering of examinations, traditional classroom settings, as well as personal space such as a home or office workspace. The geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b> include computing devices which may belong to a particular proctor or other test assessment authority, rather than being a computing device dedicated to proctoring as might otherwise be found in the central office proctoring center <b>180</b>. The proctors and other test assessment authorities at the central office proctoring center <b>180</b> and geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b> may observe and analyze a variety of different types of information to help ensure the integrity of the examination. The observation and analysis of information is described in further detail below with respect to secure testing application <b>140</b> and video camera device <b>120</b>.

0047The computing devices <b>110</b>, <b>112</b>, <b>114</b> are secured in order to prevent access to files or other types of data such as notes, outlines, and exam preparation material during an examination, as well as preventing access to applications that themselves allow for access to data. The computing devices <b>110</b>, <b>112</b>, <b>114</b> may be secured for the taking of an exam as described in commonly-owned U.S. patent application Ser. No. 12/571,666, the disclosure of which has been previously incorporated by reference.

0048The computing device <b>110</b>, <b>112</b>, <b>114</b> may be secured through the download and subsequent installation of a secure testing application (e.g., secure testing application <b>140</b> installed on computing device <b>110</b>). Secure testing application <b>140</b> may be downloaded from application server <b>172</b> or another computing device coupled to communications network <b>150</b>. Secure testing application <b>140</b> may also be installed from a computer-readable storage device such as a CD-ROM. The secure testing application <b>140</b> may then be stored in memory at the corresponding computing device <b>110</b> and executed by a processor to invoke its corresponding functionality.

0049Secure testing application <b>140</b> is a security application software that prevents computing device <b>110</b> from accessing certain data or applications that might otherwise be in violation of examination regulations or protocols as identified by application server <b>172</b>. The security application software <b>140</b> causes the computing device <b>110</b> to operate in a secure mode by introducing certain changes to the system registry such that only those applications or files deemed necessary or appropriate by the exam administrator and as embodied in a corresponding exam protocol may be allocated address space, loaded into memory and ultimately executed by the computing device <b>110</b>.

0050For example, an exam protocol for a particular examination may deny access to a web browser, e-mail client, and chat applications such that an exam taker may not electronically communicate with other individuals during the examination. This particular protocol may be downloaded to the computing devices <b>110</b> from the CDN <b>162</b> along with exam data. The secure testing application <b>140</b> then operates in accordance with the downloaded testing protocol such that certain applications are not allowed to be loaded and executed.

0051Similar prohibitions or permissions may apply to hardware components of the computing device <b>110</b> as well as any number of hardware peripherals that might be introduced to the computing devices <b>110</b>. Examples of such peripherals that might be introduced include a second computer monitor, docking stations, a traditional full-sized keyboard as might be used with a laptop computer. Other peripherals might include thumb drives, ‘time-shift’ recording devices that offer TiVo®-like functionality, as well as any number of other plug-and-play peripherals.

0052The secure testing application <b>140</b> may also operate in conjunction with the CDN <b>162</b> to properly execute an exam routine for the given examination event. For example, the exam routine may allow for the user to have access to all questions at any given time such that the user may answer and not answer questions at their leisure and subsequently return to any questions at a later time for further review. The exam routine may alternatively require the exam taker to lock in an answer or set of answers and have the same reported to the CDN <b>162</b> prior to receiving a subsequent question.

0053The secure testing application <b>140</b> may also observe activity on the computing device <b>110</b> during administration of an examination. If an exam taker attempts to make changes to the system registry that were implemented by the secure testing application <b>140</b>, the secure testing application <b>140</b> may identify and report these attempts to the central office monitoring center <b>180</b>, and distributed proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b>.

0054The secure testing application <b>140</b> operates in conjunction with video camera device <b>120</b> or other visual and/or audio capture device to monitor the exam environment surrounding the exam taker of the computing device <b>110</b>. Video camera device <b>120</b>, which may be a commercially available web camera or other image acquisition device, generates a real-time data stream of the exam environment. If the exam taker leaves their seat or another individual enters the exam area during the course of the examination, the video camera device <b>120</b> will capture this visual information. The video camera device <b>120</b> provides the data stream to the secure testing application <b>140</b>. Other external devices may also be used to gather environment data that can be provided in the encoded data stream uploaded to the CDN <b>172</b>, such as a microphone <b>130</b> or other environment capture device.

0055As described in more detail below, the secure testing application <b>140</b> encodes and uploads the captured data stream into the CDN <b>162</b>. The CDN <b>162</b> in turn delivers the data stream to the central office monitoring center <b>180</b>, and/or one or more geographically distributed proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b>, during the administration of the examination.

0056The computing device <b>191</b> at the proctoring center <b>192</b> is representative of computing devices in the central office proctoring center <b>180</b> and geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b>. The computing device <b>191</b> may be any sort of computing device as is known in the art. The computing device <b>191</b> include memory for storage of data and software applications, a processor for accessing data and executing applications, and input and output devices that allow for user interaction. The computing device <b>191</b> further includes components that facilitate communication over the communications network <b>150</b>.

0057The computing device <b>191</b> may be a general purpose computing device such as a desktop or laptop computer. Thin client or netbook client devices may be implemented in the context of computing device <b>191</b> as might mobile computing devices such as smart phones.

0058The computing device <b>191</b> includes a viewing application <b>196</b> stored in memory and executed by a processor to invoke its corresponding functionality. The viewing application <b>196</b> operates in conjunction with the CDN <b>162</b> to request, receive and display real-time data streams of the examination environment surrounding exam takers. The viewing application <b>196</b> is executable to request that the CDN <b>162</b> transmit data streams to the computing device <b>191</b>, as well as to request that the CDN <b>162</b> transmit data streams to other proctor's computing devices.

0059The viewing application <b>196</b> may for example be a commercially available application such as a web browser, or any other software application which can be utilized for proctoring an online examination as described herein. The viewing application <b>196</b> may alternatively be created by the testing administrator or otherwise developed specifically for proctoring of the online exam.

0060Similar to the discussion above with regards to the secure testing application <b>140</b>, in some embodiments the viewing application <b>196</b> may cause the computing device <b>191</b> to operate in a secure mode.

0061A proctor or other test assessment authority at the central office monitoring center <b>180</b> and/or the proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b> may then view the data stream provided by the CDN <b>162</b> and determine if any visual activity constitutes activity not in accordance with the exam protocol. The proctor may then log the information for further assessment by the actual exam administrator (e.g., the professor or professional association administrating the examination) or make a direct inquiry of the exam taker as to the nature of the observed behavior, provide a warning as to terminate that behavior, or otherwise intervene in the testing routine to resolve the aberrant behavior.

0062The proctor or other test assessment authority viewing the data stream may also request that the CDN <b>162</b> temporarily or permanently transmit the data stream to another computing device, such as one or more of central office monitoring center <b>180</b> and the remote proctor monitoring centers <b>190</b>, <b>192</b>, <b>194</b>. This allows for real-time sharing and shifting of responsibilities among the proctors and the third party test assessment authorities. As a result, a small number of proctors and assessment authorities can properly maintain the integrity of the testing environment, notwithstanding the large number of exam takers.

0063As an example, a proctor may be responsible for simultaneous observation of a number of data streams of the examination environments surrounding different exam takers. Upon detection of possible aberrant behavior by a particular exam taker, the proctor may request that the data streams of the other exam takers currently be received also be transmitted to the computing device of one or more proctors. In doing so, the proctor can more closely analyze the possible aberrant behavior by the particular exam taker, without allowing the remaining exam takers to go unobserved. This request may be generated automatically based on user input to the viewing application <b>196</b>, such as the selection by the proctor that the data stream be ‘exploded’ into a larger view (like in <figref idref="DRAWINGS">FIG. 7</figref>).

0064As another example, a proctor may request that the data stream of a test taker exhibiting possible aberrant behavior be transmitted to the computing device of another proctor or third party test assessment authority (e.g., a professor) for the purpose of validating or making an affirmative determination that the exam taker in fact violated an exam taking protocol. Following a determination that an exam taker violated an exam taking protocol, the proctor or other third party test assessment authority can then make a final or early decision regarding disciplinary action.

0065<figref idref="DRAWINGS">FIG. 2</figref> illustrates a method for implementing an online proctored examination. In step <b>210</b>, an account is created by an exam taker. In step <b>220</b>, an exam taker registers for and/or schedules an examination. In step <b>230</b>, an exam taker engages in biometric enrollment and authentication. In step <b>240</b> the exam is delivered and proctoring commences at step <b>250</b>. The aforementioned steps may be carried out as described in co-pending U.S. patent application Ser. No. 12/723,667 entitled “Secure Online Testing,” the disclosure of which is incorporated herein by reference.

0066Proctoring step <b>250</b> takes place over the course of the examination and may be implemented as part of step <b>240</b>. The proctoring step <b>250</b> utilizes a variety of security technologies and processes to deter and detect aberrance during the exam process. In particular, the exam taking environment surrounding the exam taker is monitored in real-time by a proctor utilizing a live video feed of the exam taker. The live video feed is captured utilizing a video camera device or other visual and/or audio environment capture device, and uploaded and distributed in real-time by the CDN <b>162</b> utilizing the techniques described herein.

0067<figref idref="DRAWINGS">FIG. 3</figref> illustrates a method <b>300</b> for uploading a data stream of the examination environment surrounding the exam taker during an online proctored examination. At step <b>305</b>, the computing device <b>110</b> transmits an assignment request to the application server <b>172</b>. The assignment request is a request for identification information of an entry server of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>, which will serve as the entry point for the data stream into the CDN <b>162</b>. The transmission of the assignment request may be initiated automatically by the secure testing application <b>140</b> upon successful completion of the enrollment and authentication of the exam taker. The assignment request will be described in more detail with respect to <figref idref="DRAWINGS">FIG. 4</figref>.

0068At step <b>310</b>, the application server <b>172</b> receives the assignment request. At step <b>315</b> the assignment request is analyzed to select an entry server of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>, which will be entry point for the data stream. This determination is made based on one or more estimated performance metrics for a network path from the computing device <b>110</b> to the selected entry server of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>. The performance metrics can be computed using an estimated physical location of the computing device <b>110</b>, as well as entry server information such as the physical location of each of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> stored in the database server <b>174</b>. The analysis of the assignment request and the selection of the particular entry server in step <b>315</b> are described in further detail below with respect to <figref idref="DRAWINGS">FIG. 4</figref>.

0069At step <b>320</b>, the application server <b>172</b> transmits assignment data to the computing device <b>110</b> requesting that the computing device <b>110</b> upload the data stream to the selected entry server. The assignment data may include data representing an IP address of the selected entry server, or other unique identifying information of the selected entry server which can be used to facilitate connection between the computing device <b>110</b> and the selected entry server.

0070At step <b>325</b>, the computing device <b>110</b> receives the assignment data. At step <b>330</b>, the secure testing application <b>140</b> launches the examination on the computing device <b>110</b>. The secure testing application <b>140</b> may then provide the exam taker with instructions concerning the positioning of the video camera device <b>120</b> or other visual and/or audio environment capture device. Alternatively, this process may be undertaken at the enrollment and authentication stage.

0071The video camera device <b>120</b> operates in conjunction with the secure testing application <b>140</b> to generate a real-time data stream of the examination environment. This data stream is provided to secure testing application <b>140</b>, which in turn encodes the data stream into an encoded data stream. The data stream is encoded in a format dependent upon the application, and in the illustrated embodiment is encoded as defined by the well-known H.264/AVC standard. The encoded data stream includes not only the image and audio data itself, but also the information to enable a decoder to decode the data stream, and information about the structure of the encoded data and the compression tools used during encoding. These streams may be referred to as “bitstreams,” but the units of the encoded data stream in various embodiments can be bits, bytes, picture fields, frames, or any other unit that is appropriate for the embodiment.

0072At step <b>335</b>, the computing device <b>110</b> transmits the encoded data stream in real-time to the selected entry server using the assignment data received at step <b>325</b>. The selected entry server acts as the entry point for the encoded data stream into the CDN <b>162</b>, and receives the data in the encoded data stream at step <b>340</b>. The CDN <b>162</b> can then in turn transmit copies of the encoded data stream in real-time to the central office proctoring center <b>180</b>, and/or one or more geographically distributed proctoring centers <b>190</b>, <b>192</b>, <b>194</b>, during the administration of the examination. A proctor may then view the data stream via a streaming technique, and determine if any visual activity in the examination environment constitutes activity not in accordance with the exam protocol.

0073<figref idref="DRAWINGS">FIG. 4</figref> is a method for selecting an entry server to receive the encoded data stream. At step <b>400</b>, the assignment request is analyzed to determine the geographic location of the computing device <b>110</b>. In the illustrated embodiment, the geographic location of the computing device <b>110</b> is determined by inspecting the IP address of the computing device <b>110</b> which is included in the assignment request. In alternative embodiments, other techniques for determining the geographic location of the computing device <b>110</b> may be used. For example, metadata in the assignment request may include the geographic location or address provided by the exam taker during registration or verification stages, or other metadata which can be used to identify the location of the computing device <b>110</b>. In some embodiments, the geographic location or address of the computing device <b>110</b> may be determined utilizing the address information provided by the corresponding exam taker, which may for example be stored in the database server <b>174</b>. As described above, this address information may be provided by the exam taker during the registration or verification stages. Alternatively, the exam taker may be prompted to provide such information prior to the launch of the examination.

0074At step <b>405</b>, a set of one or more entry servers of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> are identified as possible entry points for the encoded data stream into the CDN <b>162</b>. These identified entry servers may be a subset of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b>, which may be selected based on proximity to the determined geographic location of the computing device <b>110</b>. For example, if it is determined that the computing device <b>110</b> is located in San Francisco, Calif., the set of entry servers may be those which are closest to San Francisco, Calif.

0075At step <b>410</b>, expected performance metrics for different network paths from the computing device <b>110</b> to each of the identified entry servers are calculated. For example, if the set includes entry servers <b>160</b>-<b>1</b>, <b>160</b>-<b>2</b> and <b>160</b>-<b>3</b>, then the network paths from the computing device <b>110</b> through the network <b>150</b> to each of the entry servers <b>160</b>-<b>1</b>, <b>160</b>-<b>2</b> and <b>160</b>-<b>3</b> are evaluated. These expected performance metrics may include bandwidth, packet loss, and delay for each of the various network paths. In some embodiments, these expected performance metrics are estimated without requiring an analysis of the actual performance metrics of the network paths. In such a case, relative values of the expected performance metrics may be determined by comparing the geographic location of the computing device <b>110</b> to the geographic and/or network locations of the identified entry servers. For example, the network path for the geographically closest entry server may be assigned the lowest estimated delay value. Alternatively, the actual performance metrics of the networks may be collected and analyzed.

0076At step <b>415</b>, the performance metrics for the set of identified entry servers are compared to one another. The selected entry server to act as the entry point for the encoded data stream is then selected from the set based on this comparison. This selected entry server is selected as the “optimum” entry point into the CDN <b>162</b> for the encoded data stream, to allow for an optimal first mile experience for the data stream. The criteria for the selection of the “optimum” entry server can vary from embodiment to embodiment. The criteria may, for example, be performed by weighting each of the expected performance metrics for a given network path, and selecting the entry server based on comparison of the weighted results. The criteria may, for example, be based on minimizing delay and/or packet loss of the uploaded data stream into the CDN <b>162</b>. Other and/or additional criteria such as upload bandwidth may also or alternatively be used.

0077The selection of the particular entry server may be based for example on which of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> is the closest to the computing device <b>110</b> in terms of network proximity. Network proximity may for example be represented by the delay or number of handoffs for the different network paths from the computing device <b>110</b> to each entry server of the set. In such a case, the selected entry server may correspond to the network path with the least number of handoffs.

0078The selection of the entry server may alternatively be based for example on which of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> is closest to the computing device <b>110</b> in terms of physical proximity. In such a case, the selected entry server may be the entry server which is geographically the closest to the computing device <b>110</b>.

0079A backup entry server of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> may also be selected as an additional entry point for the data stream into the CDN <b>162</b>. The backup entry server may be selected for example as the second most “optimum” entry point into the CDN <b>162</b>, using the same criteria used to select the “optimum” entry point. In such a case, the assignment data transmitted to the computing device <b>110</b> by the application server <b>172</b> will further request that the computing device <b>110</b> also transmit the data stream to the selected backup entry server.

0080The utilization of the techniques for the optimization of the upload of a data stream as described herein provide for high data stream quality with low startup latency, as well ensuring network friendliness and upload fairness. In addition, the techniques can be implemented within existing CDN infrastructures.

0081<figref idref="DRAWINGS">FIG. 5</figref> illustrates a method <b>500</b> for transmitting an uploaded data stream of the examination environment to a proctor or other third party test assessment authority during an online proctored examination.

0082At step <b>505</b>, the application server <b>172</b> receives a request to transmit a particular data stream to a computing device during the administration of the examination. In the illustrated example, the request for the data stream is from the computing device <b>191</b>. As described above, alternatively the request may be a request that a data stream already being transmitted to the computing device <b>191</b> also be transmitted to a different computing device, so that proctoring responsibilities can be shared and/or shifted in real-time.

0083The request to transmit the data stream may for example be initiated based on user input provided to the viewing application <b>196</b> executing on the computing device <b>191</b>. Alternatively, the request may for example be generated by the application server <b>172</b> on behalf of the computing device <b>191</b>. For example, the request to begin transmitting the data stream to the computing device <b>191</b> may be automatically initiated by the application server <b>172</b> upon successful completion of the enrollment and authentication of the exam taker.

0084In response to the request, at step <b>510</b> the application server <b>172</b> selects an entry server in the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> to transmit the data stream in a streaming fashion to the computing device <b>191</b>. The application server <b>172</b> may select the entry server to transmit the data stream by analyzing the request in a manner similar to the techniques described above for selecting an “optimum” entry server which is the entry point for the data stream. For example, this determination can be made based on one or more estimated performance metrics for a network path from the selected entry server to the computing device <b>191</b>. The performance metrics can be computed using an estimated physical location of the computing device <b>191</b>, as well as entry server information such as the physical location of each of the entry servers <b>160</b>-<b>1</b> to <b>160</b>-<b>6</b> stored in the database server <b>174</b>.

0085Various techniques may be used to determine the geographic location of the computing device <b>191</b>. For example, the geographic location may be determined by inspecting the IP address of the computing device <b>191</b> which is included in the request. As another example, metadata in the request may include the geographic location or address provided by the proctor during login to begin a proctoring session, or other metadata which can be used to identify the location of the computing device <b>191</b>. In some embodiments, the geographic location or address of the computing device <b>191</b> may be determined utilizing the address information provided by the proctor, which may for example be stored in the database server <b>174</b>. Alternatively, the proctor may be prompted to provide such information prior to the beginning of a proctoring session.

0086At step <b>515</b>, the data stream is transmitted by the selected entry server of the CDN <b>172</b> to the computing device <b>191</b> associated with the proctor. The data stream can then be displayed by the viewing application <b>196</b>, so that the proctor can monitor the examination environment surrounding the exam taker in real-time to detect possible aberrant behavior.

0087During the course of an examination, the proctor associated with the computing device <b>191</b> may be responsible for the simultaneous observation of a number of data streams of the examination environments surrounding different exam takers. Upon detection of possible aberrant behavior by a particular exam taker, the proctor may provide user input to the viewing application <b>196</b> requesting that the data streams of the other exam takers also be transmitted to the computing devices of one or more other proctors. The viewing application <b>196</b> transmits this request to the application server <b>172</b>. The application server <b>172</b> then determines which computing devices are available to receive the data streams of the other exam takers, and selects one or more appropriate entry servers to transmit the data streams of the other exam takers. In doing so, the proctor can more closely analyze the possible aberrant behavior by the particular exam taker, without allowing the other exam takers to go unobserved. This selection of the entry servers to transmit the data streams of the other exam takers can be similar to the techniques described above for selecting an “optimum” entry server to transmit the data stream to the computing device <b>191</b>.

0088The proctor may also request that the data stream of a test taker exhibiting possible aberrant behavior be transmitted to the computing device of another proctor or third party test assessment authority (e.g., a professor) for the purpose of validating or making an affirmative determination that the exam taker in fact violated an exam taking protocol. Following a determination that an exam taker violated an exam taking protocol, the proctor or other third party test assessment authority can then make a final or early decision regarding disciplinary action.

0089Upon detection of aberrant behavior by an exam taker, the proctor may intervene in the testing routine to resolve the aberrant behavior. The proctor may intervene by providing user input to the viewing application <b>196</b> which then transmits an indication to the application server <b>172</b> that aberrant behavior has been detected. In response to the indication, the application server <b>172</b> operates in conjunction with secure testing application <b>140</b> to resolve the aberrant behavior. For example, the application server <b>172</b> may transmit a request to the computing device <b>110</b> that the testing routine for the examination is no longer allowed to continue. Alternatively, the examination may be allowed to continue, but under a flag of caution requiring further analysis of the data stream during grading of the examination.

0090<figref idref="DRAWINGS">FIG. 6</figref> illustrates an interface <b>600</b> that can be utilized in proctoring an online examination as might be observed on a computing device at the central office proctoring center <b>180</b> or the proctoring centers <b>190</b>, <b>192</b>, <b>194</b>. Interface <b>600</b> may allow for simultaneous observation of a number of data streams of the examination environment surrounding different exam takers. As shown in <figref idref="DRAWINGS">FIG. 6</figref>, the data stream of a single examination environment <b>610</b> is being observed from a total of twelve available examination environments <b>620</b>. As illustrated in <figref idref="DRAWINGS">FIG. 6</figref>, the examination environment <b>610</b> being monitored exhibits aberrant behavior as reflected by alert <b>630</b>. Aberrant behavior may be automatically detected, or generated in response to proctor input. <figref idref="DRAWINGS">FIG. 6</figref> also illustrates a session ID <b>640</b>, which is unique to the exam taker; a proctor identification <b>650</b>, which identifies a proctor responsible for observing the examination session; as well as a start and end time <b>660</b> for the examination session. All of this information may be utilized in generating assessment data or logs following completion of the examination. In some instances, aberrant behavior may result in the data stream of the examination environment being ‘exploded’ into a larger view (like in <figref idref="DRAWINGS">FIG. 7</figref>) in the case where the proctor is responsible for monitoring a large number of exam takers.

0091Upon the exhibition of aberrant behavior as reflected by alert <b>630</b> in <figref idref="DRAWINGS">FIG. 6</figref>, the specific examination session may be singled out for further investigation through the interface <b>700</b> illustrated in <figref idref="DRAWINGS">FIG. 7</figref>.

0092<figref idref="DRAWINGS">FIG. 7</figref> illustrates a second interface <b>700</b> that can be utilized in proctoring an online examination. The interface <b>700</b> may be launched in response to detecting aberrant behavior observed in the interface <b>600</b> of <figref idref="DRAWINGS">FIG. 6</figref>. The interface <b>700</b> of <figref idref="DRAWINGS">FIG. 7</figref> (like that of interface <b>600</b> of <figref idref="DRAWINGS">FIG. 6</figref>) displays real-time video of the examination environment <b>610</b> of the exam taker. Recording of the video may take place upon detection of aberrant behavior for the purpose of validating or providing requisite evidence related to addressing disciplinary activity following an affirmative determination that an exam taker violated an exam taking protocol. In some instances the aberrant behavior may simply be that the examination environment needs to be modified to ensure proper proctoring, which could include raising the light level or decreasing background noise (e.g., closing a window). A proctor may provide this information to an exam taker.

0093The interface <b>700</b> of <figref idref="DRAWINGS">FIG. 7</figref> also illustrates a current alert log <b>720</b> that identifies the specific aberrant behavior that led to the automated alert <b>630</b> in the interface <b>600</b> of <figref idref="DRAWINGS">FIG. 6</figref>. The proctor may log the outcome of their determination related to the aberrant behavior in response log <b>730</b>. Response log <b>730</b> allows a proctor to identify the particular behavior that was at issue (e.g., an audio problem or multiple people being present) (<b>732</b>) and the results of monitoring the aberrant behavior (<b>734</b>), which could include clearing the alert as a false alert, terminating the examination, or allowing the exam to continue. A proctor may also launch an on-demand verification of audio, visual, or keystroke analytics. Notes related to the incident may also be maintained in notes section <b>736</b> to further detail the specific incident. In some instances, the proctor may launch a live chat session with the exam taker while maintaining real-time observation.

0094The interface <b>700</b> may also maintain additional information such as a historical alert log <b>740</b> that maintains a running list of all aberrant behavior for the exam taker in question as well as security information <b>750</b>, session information <b>760</b>, and testing program information <b>770</b>. Security information <b>750</b> may display specific information about an exam taker, including biometric information such as a photograph. Session information <b>760</b> may display information such as the name of the exam taker, the number of exam items answered, the number of breaks taken, and so forth as illustrated in <figref idref="DRAWINGS">FIG. 7</figref>. Information concerning specific protocols related to the examination may be identified in exam program information window <b>770</b>.

0095Logging of aberrant behavior may be tied to audio and video feeds of the examination environment. In such instances, a proctor may simply log the unusual behavior but leave it to the exam assessment authority as to the ultimate disciplinary behavior. Providing audio and video context tied to the alert may be useful in this regard.

0096Computer-readable storage media refer to any non-transitory storage medium and that may allow for instructions to be provided to a central processing unit (CPU) for execution. Such media can take many forms, including, but not limited to, non-volatile and volatile media such as optical or magnetic disks and dynamic memory, respectively. Common forms computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM disk, digital video disk (DVD), any other optical medium, RAM, PROM, EPROM, a FLASHEPROM, and any other memory chip or cartridge.

0097Computer-readable storage medium, which are non-transitory in nature, differ from the various forms of transmission media may be involved in carrying one or more sequences of one or more instructions to a CPU for execution. A bus carries the data to system RAM, from which a CPU retrieves and executes the instructions. The instructions received by system RAM can optionally be stored on a fixed disk either before or after execution by a CPU. Various forms of storage may likewise be implemented as well as the necessary network interfaces and network topologies to implement the same.

0098While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the invention to the particular forms set forth herein. To the contrary, the present descriptions are intended to cover such alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.