Neural processing unit

CROSS REFERENCE TO RELATED APPLICATION

p-0002This application claims the benefit under 35 U.S.C. §119(e) of the following provisional application, which is incorporated herein by reference in its entirety: U.S. Ser. No. 61/346,441, entitled “Neural Processing Unit,” filed May 19, 2010.

FIELD

p-0003The subject matter described herein relates to data processing and, in particular, neural-based processing.

BACKGROUND

p-0004Neural-based data processing may be implemented based on a variety of neural models defining the behavior of neurons, dendrites, and/or axons. In some instances, neural-based data processing may be implemented using an immense numbers of parallel neurons and connections between those neurons. This parallel nature of neural-based processing makes it well suited for processing tasks, such as for example data processing, signal processing, prediction, classification, and the like.

SUMMARY

p-0005The subject matter disclosed herein provides methods, apparatus, and articles of manufacture for neural processing.

p-0006In one aspect, there is provided a method. The method may include reading, from a first memory, context information stored based on at least one connection value; reading, from a second memory, an activation value matching the at least one connection value; sending, by a first processor, the context information and the activation value to at least one of a plurality of microengines to configure the at least one microengine as a neuron; and generating, at the at least one microengine, a value representative of an output of the neuron.

p-0007Embodiments of the method include one or more of the features described herein including one or more of the following features. The reading from the first memory may further include reading a data structure including context information stored in memory serially based on connection values for neurons implemented at the plurality of microengines. The data structure may include a plurality of blocks, each of the plurality of blocks including a type defining execution at the neuron, a first address representative of a location in the first memory where the context information including the at least one connection value is stored, and a second address representative of a location in the second memory where the activation value is stored. The data structure may include a plurality of sequential blocks, each of the plurality of sequential blocks including a connection value and a neuron type defining a corresponding neuron implemented at one of the plurality of microengines. The sequential blocks may be sent to at least one of the plurality of microengines as a packet, and each of the plurality of microengines may include at least one processor and at least one memory. The data structure may include a plurality of sequential blocks, each of the sequential blocks including a neuron type and a plurality of connection values for a corresponding neuron. The first processor may be coupled to the first memory to enable reading from the first memory. The value may be generated at the at least one microengine based on the context information and the activation value without accessing the first memory and the second memory to obtain additional context information. The first memory and the second memory may be implemented in the same memory of an integrated circuit. The generated value may be sent to at least the second memory.

p-0008Articles are also described that comprise a tangibly embodied machine-readable medium operable to cause one or more machines (e.g., computers, etc.) to result in operations described herein. Similarly, systems are also described that may include a processor and a memory coupled to the processor. The memory may include one or more programs that cause the processor to perform one or more of the operations described herein.

p-0009The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

p-0010In the drawings,

p-0011<figref idrefs="DRAWINGS">FIG. 1</figref> depicts a block diagram of a neuron;

p-0012<figref idrefs="DRAWINGS">FIG. 2</figref> depicts a block diagram of a neural processing unit (NPU);

p-0013<figref idrefs="DRAWINGS">FIGS. 3A-B</figref> depict examples of data structures for sequentially storing context information;

p-0014<figref idrefs="DRAWINGS">FIG. 4</figref> depicts a process for configuring a neuron based on the context information;

p-0015<figref idrefs="DRAWINGS">FIG. 5</figref> depicts another block diagram of a neural processing unit;

p-0016<figref idrefs="DRAWINGS">FIG. 6</figref> depicts a block diagram of a system including a plurality of neural processing units;

p-0017<figref idrefs="DRAWINGS">FIG. 7</figref> depicts another process for configuring a neuron;

p-0018<figref idrefs="DRAWINGS">FIG. 8</figref> depicts an example of a microengine configured to implement a neuron based on context information; and

p-0019<figref idrefs="DRAWINGS">FIG. 9</figref> depicts a block diagram of a system including a plurality of neural processing units.

p-0020Like labels may refer to the same or similar elements.

DETAILED DESCRIPTION

p-0021The subject matter described herein relates to a neural processing unit (NPU) configured by at least one packet including context information. As used herein, context information refers to information for configuring a processor as a neural processing unit. Moreover, some, if not all, of the context information may be stored sequentially, based on connection values, in memory to facilitate processing by the neural processing unit.

p-0022Before explaining the details of the neural processing unit, the following provides a description of the processing performed by a neuron implemented using the neural processing unit.

p-0023<figref idrefs="DRAWINGS">FIG. 1</figref> depicts a system <b>100</b> including a neuron Aj <b>150</b> which may be implemented by a neural processing unit. Although <figref idrefs="DRAWINGS">FIG. 1</figref> depicts a single neuron <b>150</b>, system <b>100</b> may include a plurality of neurons as well.

p-0024The neuron <b>150</b> may generate an output Aj(t) <b>170</b> based on activation values Ai(t−1) (which correspond to A₀-A₈) <b>160</b>A-I, connections Wij <b>165</b>A-I (which are labeled c_ojthrough c_8j), and input values <b>110</b>A-I (labeled S₀-S₈). The input values <b>110</b>A-I may be received from the outputs of other neurons, from memory, and/or from one or more sensors providing a value, such as for example a voltage value. The activation values Ai(t−1) may be received from memory and may correspond to an output, such as for example a previous activation value of a previous cycle or epoch (e.g., at t−1) of system <b>100</b>, although the activation value(s) may be provided by a host computer as well. The connections Wij <b>165</b>A-I (also referred to as weights, connection weights, and connection values) may be received from memory and/or provided by a host computer.

p-0025To illustrate by way of an example, at a given time, t, each one of the activation values <b>160</b>A-I is multiplied by one of the corresponding connections <b>165</b>A-I. For example, connection weight c_oj<b>165</b>A is multiplied by activation value A₀<b>160</b>A, connection weight c_1j<b>165</b>B is multiplied by activation value A₁<b>160</b>B, and so forth. The products (i.e., of the multiplications of the connections and activation values) are then summed, and the resulting sum is operated on by a basis function K to yield at time t the output A_j(t) <b>170</b> for node A_j<b>150</b>. The outputs <b>170</b> may be used as an activation value at a subsequent time (e.g., at t+1).

p-0026System <b>100</b> may include a plurality of neurons, such as for example neuron <b>150</b>, and each of the neurons may be implemented on the neural processing units described herein. Moreover, the neurons may be configured in accordance with a neural model, an example of which is as follows:

p-0027<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mrow><msub><mi>A</mi><mi>j</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mi>K</mi><mo>[</mo><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mi>n</mi></munderover><mo>⁢</mo><mrow><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>t</mi><mo>-</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow><mo>*</mo><msub><mi>W</mi><mi>ij</mi></msub></mrow></mrow><mo>]</mo></mrow></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mi>Equation</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mn>1</mn></mrow></mtd></mtr></mtable></math></maths>

p-0028wherein

p-0029K corresponds to a basis function (examples of which include a sigmoid, a wavelet, and any other basis function),

p-0030Aj(t) corresponds to an output value provided by a given neuron (e.g., the j^thneuron) at a given time t,

p-0031Ai(t−1) corresponds to a prior output value (or activation value) assigned to a connection i for the j^thneuron at a previous time t−1,

p-0032Wij represents the i^thconnection value for the j^thneuron,

p-0033j varies in accordance with the quantity of neurons and identifies a given neuron,

p-0034i varies from 0 to n−1, and

p-0035n corresponds to the number of connections to the neuron.

p-0036Although the description herein refers to Equation 1 as an example of a neural model, other models may be used as well to define the type of neuron. Moreover, in some implementations, each connection may be associated with one of a plurality of neuron types. For example, connections Wij <b>165</b>A-C may implement a first neural model corresponding to a first type of neuron, and connections Wij <b>165</b>D-E may implement a second neural model corresponding to a second type of neuron. In this example, the context information would include the connection values and information representative of the types of neurons.

p-0037<figref idrefs="DRAWINGS">FIG. 2</figref> depicts an example implementation of a neural processing unit <b>200</b> configured to operate as a neuron, such as for example neuron <b>150</b> described above with respect to <figref idrefs="DRAWINGS">FIG. 1</figref>. The neural processing unit <b>200</b> includes a first memory, such as for example sequential memory <b>205</b>, a processor <b>210</b>, a second memory, such as for example static random access memory <b>215</b> (labeled SRAM), and one or more processors, such as for example one or more microengines <b>220</b>A-E configured to implement neurons.

p-0038Although <figref idrefs="DRAWINGS">FIG. 2</figref> depicts a single neural processing unit <b>200</b>, in some implementations, a system may include a plurality of neural processing units. For example, a plurality of neural processing units may be implemented on an integrated circuit and/or application specific integrated circuit to provide a neural processing system.

p-0039The sequential memory <b>205</b> may be implemented as any type of memory, such as for example random access memory, dynamic random access memory, double data rate synchronous dynamic access memory, flash memory, ferroelectric random access memory, mechanical, magnetic disk drives, optical drives, and the like. Sequential memory <b>205</b> may include context information comprising some, if not all, of the information required to configure a microengine, such as for example microengine <b>220</b>A, as a neuron. For example, the sequential memory <b>205</b> may include a data structure including one or more of the following: the identity of the neuron (e.g., which one of the j neurons is being configured); the connection values Wij for each connection i, an indication of the basis function K being used, and/or previous activation values Ai(t−1).

p-0040In some implementations, the context information is obtained from sequential memory <b>205</b> in a packet format. The term packet refers to a container including the context information and/or a pointer to the context information. The packets provide the connection values and other information (e.g., instructions to configure a type of neuron, an indication of the basis function K, the identity of the j^thneuron, etc.) but the previous activation values Ai(t−1) are obtained from another memory, such as for example static random access memory <b>215</b>. The packets read from sequential memory <b>205</b> may include context information configured as the data structure described below with respect to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>. In some implementations, sequential memory <b>205</b> may also receive context information from a host computer.

p-0041The processor <b>210</b> may be implemented as any type of processor, such as for example, a central processing unit configured to handle a very long instruction word (VLIW), although other type of processors may be used as well. The processor <b>210</b> may retrieve context information (formatted as one or more packets) from sequential memory <b>205</b>.

p-0042One or more additional neural processing units (also referred to as clusters) may receive context information from processor <b>210</b> and/or provide context information to processor <b>210</b> via connection <b>230</b>. The processor <b>210</b> may also store and/or retrieve intermediate values, such as for example previous activation values Ai(t−1) from static random access memory <b>215</b>.

p-0043The processor <b>210</b> may route packets including the context information obtained from memory <b>205</b> and any intermediate values (e.g., previous activation values Ai(t−1) obtained from static random access memory <b>215</b>) to a microengine to configure the microengine as a neuron.

p-0044In some implementations, context information in memory <b>205</b> is organized in sequential blocks, as described further below with respect to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>. The context information in memory <b>205</b> is read from the sequential blocks in memory <b>205</b> to configure the microengines as neurons. Moreover, if the system includes a plurality of neurons, the processor <b>210</b> may route a set of packets including context information to each of the microengines <b>220</b>A-E to configure each of the microengines as a neuron operating in accordance with a model, such as for example the neural model of Equation 1 above. In addition, the configuration and execution of the microengines may be repeated over time for each of the connections, neurons, and/or epochs.

p-0045Each of the microengines <b>220</b>A-E may be implemented as a processor, such as for example a central processing unit, a reduced instruction set processor, and the like. In implementations using the reduced instruction set processor, the functionality of the reduced instruction set processor may be limited, reducing thus the space/size used on a chip, such as for example an integrated circuit. In any case, the microengines <b>220</b>A-E may each be configured by context information provided by processor <b>210</b> to enable a neuron to be implemented at a microengine.

p-0046<figref idrefs="DRAWINGS">FIG. 3A</figref> depicts an example of a data structure <b>300</b> stored in sequential memory, such as for example sequential memory <b>205</b>. In the example of <figref idrefs="DRAWINGS">FIG. 3A</figref>, the sequential memory is configured to store blocks <b>310</b>A-B of context information which can be read from, and/or written to, sequentially until the end of data structure <b>300</b>, and then reading and/or writing would resume at the beginning of memory, such as for example block <b>310</b>A. The blocks <b>310</b>A-B correspond to each of the plurality of neurons of the neural processing system.

p-0047For example, for a given neuron, such as the j^thneuron, a block <b>310</b>A may store context information. Specifically, the block <b>310</b>A may include information defining the type of neuron <b>320</b>. The type <b>320</b> defines the kind of neuron and how to execute the neuron. For example, the type <b>320</b> may define the neural model, defining Equation 1 or the basis function K being used by the j^thneuron. In this example, a neuron type may have a corresponding code set which is loaded into a microengine to enable the microengine to process the context information and generate an output. Moreover, in some implementations, a plurality of neuron types are supported, such that the system configures a microengine with a corresponding code set for the type of neuron indicated in the context information.

p-0048The context information in block <b>310</b>A may also include the address <b>330</b> being used in sequential memory to store block <b>310</b>A. The address <b>330</b> enables write backs to sequential memory if the contents of block <b>310</b>A are changed, such as for example by altering weights for learning and plasticity. The context information in block <b>310</b>A may also include the activation address <b>340</b> associated with a given connection of the neuron of block <b>310</b>A. The activation address <b>340</b> may, for example, enable loading from static random access memory <b>215</b> activation values for connections being used in block <b>310</b>A. The block <b>310</b>A may also include the connection values being used <b>350</b>A-N. For example, for a given neuron j having 10 connections, the block <b>310</b>A would include 10 connection values corresponding to W_0jto W_9j.

p-0049The data structure <b>300</b> may include other blocks, such a block <b>310</b>B and the like, for each of the neurons being implemented by the neural processing system. The use of data structure <b>300</b> may allow sequential reading of context information (which is formatted in a packet-based format) for each neuron, and then configuring and executing those neurons at microengines. In some implementations, the data structure <b>300</b> may reduce, if not eliminate, inefficient, random, memory reads by the microengine to memories <b>205</b> and <b>215</b> during microengine execution of neurons.

p-0050<figref idrefs="DRAWINGS">FIG. 3B</figref> depicts another example of a data structure <b>305</b> stored in sequential memory. In the example of <figref idrefs="DRAWINGS">FIG. 3B</figref>, each block <b>390</b>A-C may include a single connection value for a neuron. Specifically, block <b>390</b>A includes the first connection <b>350</b>A and other context information <b>320</b>-<b>340</b> for configuring a first neuron, block <b>390</b>B includes a first connection value <b>333</b> and other context information for configuring a second neuron, and so forth until all of the first connection values of a set of neurons have configured for execution. Once executed, the output activation values may be stored in for example static random access memory <b>215</b>, and then the next set of connections for the neurons is processed. Block <b>390</b>C includes the second connection <b>350</b>B and other context information for configuring the first neuron. Once configured with the second connection values, the second set of connections for the neurons is processed, yielding another set of output activation values. This process may be repeated for each of the connections to the neurons until all of the connections are processed, at which time the process repeats starting at the first connection at block <b>390</b>A of data structure <b>305</b>.

p-0051In some implementations, the data structures <b>300</b> and <b>305</b> may reduce the quantity of memory accesses when a processor executes a neural model and retrieves data required to execute that model. For example, in a typical microprocessor not configured in accordance with the data structures described herein, the microprocessor would require extensive random fetches of data from memory in order to execute the neural model of Equation 1 due to for example the indexing from 0 to n, 1 to i, and 1 to j. In contrast, the microengines described herein may, in some implementations, reduce, if not eliminate, the random fetches from memory by serially, sequencing the context information in memory as depicted in the example data structures <b>300</b> and <b>305</b>. Moreover, the random fetches of system <b>200</b> may, in some implementations, be limited to processor <b>205</b> retrieving activation values from static random access memory <b>215</b>. In some implementations, a first processor, such as for example processor <b>210</b> handles all of the memory fetches from memory <b>205</b> and <b>215</b> which are associated with the indexing noted above, and second processors, such as for example the microengines, implement the neurons without accessing memory <b>205</b> and <b>215</b>. Moreover, the microengines may be configured to operate the neurons using data accessed from its register memory (which is described further below with respect to <figref idrefs="DRAWINGS">FIG. 5</figref>). In addition, the system including the first and second processors may, in some implementation, facilitate efficient processing, especially in the context of the sparse matrixes associated with, for example, neural models, such as for example Equation 1.

p-0052<figref idrefs="DRAWINGS">FIG. 4</figref> depicts a process <b>400</b>, which may be implemented by the neural processing systems described herein. The description of <figref idrefs="DRAWINGS">FIG. 4</figref> also refers to <figref idrefs="DRAWINGS">FIG. 2</figref> and <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>.

p-0053At <b>492</b>, processor <b>210</b> may read sequentially from memory <b>205</b>. This sequential reading may include reading from a block of consecutive addresses in memory <b>205</b> at least one of a plurality of packets including context information. For example, the data may be read sequentially in blocks as described with respect to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>. The context information may include information for configuring a microengine as a neuron in accordance with a neural model, such as for example Equation 1. For example, the packets received from sequential memory <b>205</b> may provide the connection values and other information (e.g., instructions indicating neuron type, activation values, etc.) to configure a neuron.

p-0054At <b>493</b>, processor <b>210</b> may also read from static random access memory <b>215</b> the previous activation values Ai(t−1). Referring to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>, the processor <b>210</b> may receive a packet including a connection value <b>350</b>A, a neuron type <b>320</b>, an address <b>330</b> in sequential memory, and an address <b>340</b> in static random access memory <b>215</b>. The processor <b>210</b> may then use the address <b>340</b> to locate static random access memory <b>215</b>, and then use a lookup table to determine an activation value that corresponds to the connection value <b>350</b>A.

p-0055At <b>494</b>, the processor <b>210</b> may forward the context information and the activation value to a microengine to configure the microengine. For example, the processor <b>210</b> may forward to microengine <b>220</b>A at least one packet including the context information (e.g., connection value <b>350</b>A, a neuron type <b>320</b>, and the like) and the activation value obtained from static random access memory <b>215</b>. When the microengine receives the at least one packet, the microengine <b>220</b>A may configure itself based on the neuron type (e.g., by loading a code set/instructions for the neuron type indicated by neuron type <b>320</b>) and then execute the neuron using the connection value <b>350</b>A, activation value, and other context information which may be provided to the microengine.

p-0056At <b>496</b>, the configured at least one microengine generates for a given time t an output, such as for example Aj(t). The output Aj(t) may also be provided to processor <b>210</b>, which may route the output Aj(t) to static random access memory <b>215</b> or other neural processing units <b>230</b>. The process <b>290</b> may be repeated for each of the connections, neurons, and/or epochs of a system.

p-0057<figref idrefs="DRAWINGS">FIG. 5</figref> depicts another example implementation of a neural processing unit <b>500</b>. Neural processing unit <b>500</b> is similar to system <b>200</b> in some respects but further includes a memory interface <b>505</b>, an application interface <b>510</b>, and a dispatcher <b>520</b>. The memory interface <b>505</b> is coupled to sequential memory <b>205</b> and processor <b>210</b>, and the application interface <b>510</b> is coupled to processor <b>210</b> and dispatcher <b>520</b>, which is further coupled to microengines <b>220</b>A-E.

p-0058The memory interface <b>505</b> controls access to sequential memory <b>205</b>. For example, the memory interface <b>505</b> may sequentially index into memory <b>205</b> to retrieve the next packet of context information which is passed to the processor <b>210</b>.

p-0059The processor <b>210</b> may be further configured as a router. When the processor <b>210</b> receives context information in packet form from sequential memory <b>205</b> and/or memory <b>215</b>, the processor <b>210</b> may then route the packet-based context information to a microengine to configure the microengine as a neuron. After the neuron is executed, the packet processor <b>210</b> may also receive an output value Aj(t) generated by a microengine. The received output value Aj(t) may then be provided to other neural processing units via connections <b>565</b>A-D and/or stored in memory, such as for example static random access memory <b>215</b>.

p-0060During an initial load of data from a host computer to configure system <b>500</b>, the processor <b>210</b> may move blocks of data from the host computer to, for example, sequential locations in sequential memory <b>205</b> and other locations, such as for example static random access memory <b>215</b>, other adjacent neural processing units via connections <b>565</b>A-D, and/or one or more of the microengines <b>220</b>A-E.

p-0061During execution of a neuron at a microengine, the processor <b>210</b> may match a connection weight to an activation value. For example, the processor <b>210</b> may receive from sequential memory <b>205</b> a packet including a connection weight Wij for the i^thconnection of the j^thneuron. For the connection, the processor <b>210</b> may then match the connection weight Wij to the previous activation value Ai(t−1), which is stored in static random access memory <b>215</b>. In some implementations, a lookup table is used to match each of the connection weights Wij to corresponding activation values Ai(t−1) stored in static random access memory <b>215</b>.

p-0062The application interface <b>510</b> provides an interface to each of the microengines <b>220</b>A-E. In some implementations, the application interface <b>510</b> may fetch from static random access memory <b>215</b> an activation value that matches a connection value included in a received packet. The address of the matching activation value may be included in a packet received from processor <b>210</b>. For example, the address of the matching activation address may be stored in the packet as a neuron static random access memory activation address <b>340</b>. The packet including context information (e.g., neuron type, connection value, activation value, and the like) is then forwarded to a microengine.

p-0063The dispatcher <b>520</b> provides packet handling and queuing for packets exchanged among the application interface <b>510</b> and the microengines <b>220</b>A-E. In some implementations, the dispatcher <b>520</b> selects a destination microengine for a packet including context information. The dispatcher <b>520</b> may also load the microengine including the registers of the microengine with context information and may send output data from the microengines <b>220</b>A-E to other neural processing units, static random access memory <b>215</b>, and/or sequential memory <b>205</b>.

p-0064The neural processing unit <b>500</b> may be connected to other neural processing units via connections <b>565</b>A-D (labeled North, East, West, and South). For example, neural processing unit <b>500</b> may have connections <b>565</b>A-D to four other neural processing units, such as for example a neural processing unit north of unit <b>500</b>, a neural processing unit south of unit <b>500</b>, a neural processing unit east of unit <b>500</b>, and a neural processing unit west of unit <b>500</b>). Moreover, each of the other neural processing units may be coupled to four other neural processing units, and, in some implementations, each of the neural processing units may be implemented on one or more application specific integrated circuits.

p-0065<figref idrefs="DRAWINGS">FIG. 6</figref> depicts an example of a system <b>600</b> including four neural processing units <b>660</b>A-D. Each of the neural processing units may include sequential memory <b>632</b> (labeled DDR<b>2</b>), a memory controller <b>634</b>, a memory interface <b>636</b>, an application interface <b>640</b>, a static random access memory <b>642</b>, a dispatcher <b>644</b>, and a plurality of microengines <b>646</b> (labeled NME).

p-0066In the implementation depicted at <figref idrefs="DRAWINGS">FIG. 6</figref>, each of the neural processing units <b>660</b>A-D are coupled to router <b>610</b>, which may be implemented as described above with respect to processor <b>210</b>. However, router <b>610</b> may, in some implementations, be further configured as a non-blocking, crossbar packet router providing multiple, parallel paths among inputs and outputs, such as for example memory interfaces and application interfaces.

p-0067The following provides a description of the elements within neural processing unit <b>660</b>A, but the other neural processing units <b>660</b>B-D may be configured in a manner similar to neural processing unit <b>660</b>A. Moreover, system <b>600</b> including neural processing units <b>660</b>A-D may be implemented on a chip, such as for example an application specific integrated circuit (ASIC), and, although only four neural processing units <b>660</b>A-D are depicted at <figref idrefs="DRAWINGS">FIG. 6</figref>, system <b>600</b> may include other quantities of neural processing units as well.

p-0068The sequential memory <b>632</b> may be implemented as described above with respect to sequential memory <b>205</b>. In the example of <figref idrefs="DRAWINGS">FIG. 6</figref>, the sequential memory <b>632</b> may be implemented as double data rate synchronous dynamic access memory, although other types of memory may be used as well. The sequential memory <b>632</b> may be electrically coupled to memory controller <b>634</b> to allow reads from, and writes to, sequential memory <b>632</b>.

p-0069The memory controller <b>634</b> may control reading and writing to sequential memory <b>632</b>. The context information may be stored in sequential addresses of sequential memory <b>632</b>, and the context information may be read from, or written to, memory <b>632</b> in a packet-based format. When a packet-based format is implemented, the packets may be provided to, or received from, the router <b>610</b> via an electrical connection to the memory interface <b>636</b>. Moreover, the memory controller <b>634</b> may, in some implementations, provide an interface that generates packets from data obtained from memory, such as for example memory <b>632</b> and sends the generated packets to the router <b>610</b>. The memory controller <b>634</b> may also accept packets from the router <b>610</b> and write the contents of packets to the memory <b>632</b>. Different types of memory, ranging from static random access memory, dynamic random access memory to, more persistent, optical storage mechanisms may be used at memory <b>632</b> but regardless of the type of memory being used, the memory controller <b>634</b> handles packet and addresses packets to memory.

p-0070The memory interface <b>636</b> may be implemented in a manner similar to memory interface <b>505</b> described above. In the example of <figref idrefs="DRAWINGS">FIG. 6</figref>, the memory interface <b>636</b> may buffer packets sent to, or received from, the memory controller <b>634</b>.

p-0071The router <b>610</b> may be electrically coupled to each of the application interfaces at <b>660</b>A-D and to connections <b>692</b>A-C. Connections <b>692</b>A-C may provide connections to other devices, such as for example other neural processing units, memory, host computers, and the like. In some implementations, the connection <b>692</b>C may be implemented as a PCI interface to allow transferring data to (and from) the router <b>610</b> at speeds of up to 132 megabits per second. The connection <b>692</b>C may also handle loading, debugging, and processing data for the system <b>600</b>. For example, connections <b>692</b>A-C may be used to couple system <b>600</b> to a host computer. The host computer may provide context information including activation values, receive output values generated by the microengines, and provide code to each of the microengines to configure a microengine as a neuron.

p-0072The application interface <b>640</b> may be implemented in a manner similar to application interface <b>510</b>. In the example at <figref idrefs="DRAWINGS">FIG. 6</figref>, the application interface <b>640</b> may, however, be electrically coupled to static random access memory <b>642</b> and dispatcher <b>644</b>. The static random access memory <b>642</b> may be implemented in a manner similar to static random access memory <b>215</b>, and the dispatcher <b>644</b> may be implemented in a manner similar to dispatched <b>520</b>. The dispatcher <b>644</b> is electrically coupled to a plurality of microengines <b>646</b> (labeled NME), which may be implemented in a manner similar to microengines <b>210</b>A-E.

p-0073<figref idrefs="DRAWINGS">FIG. 7</figref> depicts a process <b>700</b>, which may be implemented by a neural processing systems described herein. The description of process <b>700</b> will also refer to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref> and <b>6</b>.

p-0074At <b>793</b>, memory controller <b>634</b> may read data from sequential addresses of memory <b>632</b>. This reading operation may include reading at least one of a plurality of packets including context information for configuring a microengine as a neuron in accordance with a neural model. The memory controller <b>634</b> may provide the read packet(s) to memory interface <b>636</b>, where the packet(s) may be queued before being provided to router <b>610</b>. The memory controller <b>634</b> may also control writing data, such as for example packets received from router <b>610</b>, to memory <b>632</b>.

p-0075At <b>794</b>, the router <b>610</b> may receive from memory interface <b>636</b> at least one of the plurality of packets and then provide the received packets to one of the application interfaces at <b>660</b>A-D. For example, the router <b>610</b> may route the at least one packet including a connection weight Wij for the i^thconnection of the j^thneuron to application interface <b>640</b>.

p-0076At <b>797</b>, the application interface may fetch the matching activation value from memory. For example, the application interface <b>640</b> may match the connection weight to the previous activation value Ai(t−1) and then fetch the matching activation value from memory, such as for example static random access memory <b>642</b>. For each packet received, the application interface <b>640</b> may read the connection weight Wij included in the packet and then determine a matching activation value stored in static random access memory <b>642</b>. As noted, application interface <b>640</b> may determine a match based on a lookup table indicting which activation value to fetch.

p-0077At <b>798</b>, the application interface, such as for example application interface <b>640</b>, may then provide the context information (e.g., connection weight Wij, the matching activation value Ai(t−1, and the like) to a dispatcher, such as for example dispatcher <b>644</b>. Next, the dispatcher <b>644</b> provides this context information to one of the microengines <b>646</b> to configure the microengine as a neuron.

p-0078At <b>799</b>, the configured microengine generates an output, such as for example Aj(t). The output Aj(t) may be provided to dispatcher <b>644</b> and application interface <b>640</b>, which may provide the output Aj(t) to static random access memory <b>642</b> or other neural processing units <b>660</b>B-D. The process <b>700</b> may be repeated for each of the connections of a neuron and repeated for each neuron of a neural processing system. Moreover, process <b>700</b> may be repeated for a plurality of epochs.

p-0079<figref idrefs="DRAWINGS">FIG. 8</figref> depicts an example of a microengine <b>800</b>, which may be used at microengines <b>220</b>A-E and/or microengines <b>646</b>. The microengine <b>800</b> may include a register memory <b>820</b>, a central processing unit <b>830</b>, and program memory <b>850</b>. The microengine <b>800</b> may be electrically coupled to dispatcher <b>520</b> to allow microengine <b>800</b> to receive packets including context information, activation values, and the like from dispatcher <b>520</b> and to provide output activations to dispatcher <b>520</b>.

p-0080In some implementations, the microengine <b>800</b> receives a packet including context information, such as for example block <b>390</b>A depicted at <figref idrefs="DRAWINGS">FIG. 3B</figref>. The microengine <b>800</b> stores block <b>390</b>A in register <b>820</b>. The microengine <b>800</b> may then access program memory <b>850</b> to obtain instructions, such as for example program code, to configure the microengine in accordance with the neural type <b>320</b> indicated by the context information of block <b>390</b>A stored in register <b>820</b>. Next, the microengine <b>800</b> executes the instructions using the context information (e.g., activation value and connection value) included in the block stored at register <b>820</b>. The output is then sent to dispatcher <b>520</b>, where it is further routed to another device, such as for example static random access memory, another neural processing unit, and/or a host computer. The output may be used as an activation for a subsequent time.

p-0081The register <b>820</b> may receive from the dispatcher <b>520</b> context information structured for example as described above with respect to <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>. The register <b>820</b> may also include a control status register (CSR), an accumulating register, a program counter (PC), and a number of scratchpad registers. The register <b>820</b> may be implemented to include sufficient storage space to store at least data structures <b>300</b> and <b>305</b> and/or at least one of the blocks of the data structures <b>300</b> and <b>305</b> depicted at <figref idrefs="DRAWINGS">FIGS. 3A-B</figref>. The register <b>820</b> may also be implemented in a ping-pong configuration including two identical register banks to allow the dispatcher <b>520</b> to write into one of the banks of register <b>820</b> while the central processing unit <b>830</b> reads and executes from the other banks of register <b>820</b>.

p-0082The microengine <b>800</b> may include a set of instructions (e.g., code) defining a set of possible neural models that can be implemented at the microengine. Thus, the set of instructions (which may be stored in program memory <b>850</b>) may be used to configure and code the microengines to operate as at least one of a plurality of neuron types. Moreover, the set of instructions may be ported among microengines to facilitate configuration. The code for microengine <b>800</b> may also use an assembler to handle an assembly language program and turn that program into a binary code file for loading into the microengine. For example, a neural assembler may be invoked via a command line to take an assembly code program and turn the assembly code into a binary file for loading into the microengine.

p-0083<figref idrefs="DRAWINGS">FIG. 9</figref> depicts a system <b>900</b>, referred to herein as a hive. The neural processing units (each labeled as NPU) may be interconnected in a two-dimensional grid layout. In some implementations, the two-dimensional grid structure may increase processing power, when compared to multiple processors sharing a single memory bus. The interconnections among neural processing units may be implemented as electrical interconnections providing high-speed, serial data lines. The host interface <b>990</b> may interface the neural processing units and a host computer <b>992</b>. For example, the host interface <b>990</b> may pass packets into the hive, read packets exchanged among the neural processing units, and intercept packets sent among neural processing units. The neural processing units may each have a unique identifier to enable locating and/or addressing each neural processing unit.

p-0084The subject matter described herein may be embodied in systems, apparatus, methods, and/or articles depending on the desired configuration. In particular, various implementations of the subject matter described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

p-0085These computer programs (also known as programs, software, software applications, applications, components, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

p-0086Similarly, systems are also described herein that may include a processor and a memory coupled to the processor. The memory may include one or more programs that cause the processor to perform one or more of the operations described herein.

p-0087Moreover, although the systems herein are described within the context of neural processing, the systems described herein may be used in other environments including, for example, finite element analysis and filter bank processing. Furthermore, the term set may refer to any quantity including the empty set.

p-0088Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and/or variations may be provided in addition to those set forth herein. For example, the implementations described above may be directed to various combinations and subcombinations of the disclosed features and/or combinations and subcombinations of several further features disclosed above. In addition, the logic flow depicted in the accompanying figures and/or described herein does not require the particular order shown, or sequential order, to achieve desirable results. Other embodiments may be within the scope of the following claims.