Accelerated mathematical engine

[Cross reference to related applications]

This application claims priority to U.S. Provisional Application No. 62/536399 (20150-2154P (P0822-1PUS)) entitled "Accelerated Mathematical Engine" filed on July 24, 2017, and is the inventor. Featuring Peter Joseph Bannon, Kevin Altair Hurd, and Emil Talpes. The aforementioned patent documents are incorporated herein by reference in their entirety.

This application is also a commonly owned U.S. patent application no. Claims priority to 710433(20150-2154(N0822-1NUS)). Each of the aforementioned patent documents is incorporated herein by reference in its entirety.

TECHNICAL FIELD This disclosure relates to accelerated math engines that operate on large amounts of data, and more particularly to accelerated math engines that perform complex convolution operations based on matrix multiplication operations.

Those skilled in the art will recognize the ever-increasing demands for speed and performance on general processors and systems used to perform time-sensitive and complex mathematical operations. Because these common systems are used to process large amounts of data and perform complex mathematical operations, the computational resources and speed of calculations are limited by the existing common hardware that performs these calculations. Limited by design features. For example, general purpose computing devices and processors that perform matrix operations may not be able to perform these operations in a timely manner under certain circumstances. Many traditional multipliers that perform digital signal processing operations rely on a series of software and hardware matrix manipulation procedures (such as address generation, transposition, bitwise addition and shifting) that can be used in time-sensitive systems. can become a bottleneck. These operation steps often require using the processor's arithmetic functions to produce intermediate results, which waste computation time as intermediate results are stored and fetched from various locations to complete the operation. become.

FIG. 1 shows an example of a conventional multiplier system. Multiplier system 100 is a scalar machine that includes a computation unit 102, registers 104, cache 106, and memory 108. During operation, computing unit 102 uses registers 104 and cache 106 to retrieve data stored in memory 108. Typically, the computational unit 102 may perform various computational procedures including matrix multiplication on the input matrices to obtain the resulting matrix, for example by converting the multiplication to addition and outputting the result to some internal register. A microprocessor such as a CPU or GPU.

For example, the dot product representing the output pixels of an image is typically produced by dot multiplying the individual matrix elements of two matrices to get partial results, and then adding the final results to get the final dot product. be done. Multiplication of individual matrix elements, or scalar multiplication, is typically performed on individual data elements by dividing the dot multiplication into a series of individual sub-operations. As a result, partial products must be stored and fetched into one or more of registers 104, cache 106, and memory 108 to complete a single arithmetic operation.

Applications that require computations such as convolutions often require software functions to be incorporated into the computation unit 102 and used to convert the convolution operation to an alternative matrix multiplication operation. This is accomplished by rearranging and reformatting the data into two matrices so that they can be directly used for matrix multiplication. However, there is no mechanism to efficiently share or reuse data in scalar machine 100 because the data required to perform each scalar operation must be re-stored and re-fetched from registers many times. The complexity and management overhead of these operations increases significantly as the amount of image data subject to the convolution operation increases.

The inability to reuse much of the data within the scalar machine 100, coupled with the added inefficient steps of storing and fetching intermediate results from registers 104, cache 106, and memory 108 to complete arithmetic operations, These are just some of the drawbacks of existing systems such as multiplier system 100.

<p>Therefore, what is needed is a high computational throughput system and method that can perform matrix mathematical operations quickly and efficiently.</p>

Reference will now be made to embodiments of the invention, examples of which are illustrated in the accompanying drawings. These figures are intended to be illustrative rather than limiting. While the invention will be generally described in the context of these embodiments, it will be understood that they are not intended to limit the scope of the invention to these particular embodiments. Items in figures may not be drawn to scale.

<figref num="1">An example of a conventional multiplier system is shown.</figref>

<figref num="2">4 illustrates an example matrix processor architecture for performing arithmetic operations in accordance with various embodiments of the present disclosure.</figref>

<figref num="3">3 shows details of an example configuration of the matrix processor architecture shown in FIG. 2; FIG.</figref>

<figref num="4">4 illustrates an exemplary multiply-add circuit implementation of the logic circuit shown in FIG. 3. FIG.</figref>

<figref num="5">4 illustrates an example convolution operation according to various embodiments of the present disclosure.</figref>

<figref num="6">4 illustrates details of an example convolution operation according to various embodiments of the present disclosure.</figref><figref num="7">4 illustrates details of an example convolution operation according to various embodiments of the present disclosure.</figref><figref num="8">4 illustrates details of an example convolution operation according to various embodiments of the present disclosure.</figref>

<figref num="9">4 illustrates an example deconvolution operation according to various embodiments of the present disclosure.</figref>

<figref num="10">FIG. 3 illustrates a process for performing arithmetic operations to make a convolutional neural network faster, according to various embodiments of the present disclosure.</figref>

In the following description, for purposes of explanation, specific details are set forth to provide an understanding of the invention. However, it will be apparent to one skilled in the art that the invention may be practiced without these details. Furthermore, those skilled in the art will appreciate that the embodiments of the invention described below may be implemented in a variety of ways, such as in a process, apparatus, system, device, or method on a tangible computer-readable medium. .

The configurations or modules shown in the figures are illustrative of exemplary embodiments of the invention and are intended to avoid obscuring the invention. Also, although throughout this description, configurations are described as separate functional units that may include subunits, those skilled in the art will appreciate that the various configurations, or portions thereof, may be divided into separate configurations or integrated into a single It will be appreciated that they may be integrated together, including within a system or configuration. Note that the features or operations described herein can be implemented as a configuration. The configuration can be implemented in software, hardware, or a combination thereof. Many configurations are formed through the interconnection of many sub-configurations. If these logically distinct subconfigurations can be aggregated and combined with other subconfigurations to provide functionality similar or identical to that described here at the aggregate configuration level, then the You can choose subconfigurations that are logically different in behavior (for example, an active high signal can be replaced by an active low signal, an AND gate can be replaced by an inverting input NOR gate, etc.).

Furthermore, connections between configurations or systems in the figures are not limited to direct connections. Rather, data between these configurations may be modified, reformatted, or modified by intermediate configurations. Also, additional or fewer connections may be used. It is also understood that the terms "coupling," "connection," or "communicating coupling" include direct connections, indirect connections through one or more intermediate devices, and wireless connections.

References herein to "one embodiment," "preferred embodiment," "embodiments," or "some embodiments" refer to the specific features, structures, or characteristics described in connection with the embodiment. , or means that a feature is included in at least one embodiment of the invention and may be included in more than one embodiment. Additionally, the appearances of the above phrases in various places in the specification are not necessarily all referring to the same embodiment or embodiments.

The use of particular terms in various places herein is for illustration purposes only and is not to be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; use of these terms refers to a grouping of related services, functions, or resources that may be distributed or aggregated.

The terms "comprising", "including", "provided" and "having" are understood to be open terms, and any list below is an example, and the items listed are is not intended to be limited to and may include subsets or supersets of items, as well as additional items. Any headings used herein are for organizational purposes only and will not be used to describe or limit the scope of any claims. Each document mentioned in this patent document is incorporated herein by reference in its entirety.

Additionally, those skilled in the art will appreciate that (1) certain steps may optionally be performed; (2) the steps may not be limited to the particular order set forth herein; and (3) certain steps may be performed in different orders. (4) Certain steps can be performed simultaneously.

Although the embodiments herein are primarily described in the context of convolution, those skilled in the art will appreciate that deconvolution and other matrix operations can also be configured as matrix-matrix multiplication operations, and that the principles of the present invention also apply to deconvolution as well. You will understand that it can be applied to convolutions. Additionally, other types of mathematical operations may be performed in accordance with various embodiments of the present disclosure.

FIG. 2 illustrates an example matrix processor architecture for performing arithmetic operations in accordance with various embodiments of the present disclosure. System 200 includes logic circuits 232, 234, cache/buffer 224, data formatter 210, weight formatter 212, data input matrix 206, weight input matrix 208, matrix processor 240, output array 226, post-processing unit 228, and control logic 250. including. Matrix processor 240 includes an arithmetic logic unit (Arithmetic Logic unit). ALU), registers, and, in some embodiments, an encoder (such as a Booth encoder). Logic circuit 232 may be a circuit representing N input operators and data registers. Logic circuit 234 may be a circuit that inputs M weight operands to matrix processor 240. Logic circuit 232 may be a circuit that inputs image data operands to matrix processor 240. Weight input matrix 208 and data input matrix 206 may be stored in various types of memory, including SRAM devices. Those skilled in the art will recognize that various types of operands may be input to matrix processor 240.

In operation according to certain embodiments, the system 200 reduces redundant operations within the system and performs convolution operations by implementing hardware-specific logic to perform specific mathematical operations over large sets of data and weights. To accelerate. This acceleration not only significantly times the mathematical operations within matrix processor 240, but also directly changes the way the image data and weights are acquired and input into matrix processor 240 (and the corresponding hardware configuration). This is a great result.

In embodiments, the example formatters 210, 212 of FIG. 2 are implemented as inline formatters. In certain embodiments, formatters 210, 212 are separate configurations; in other embodiments, formatters 210, 212 are integrated together and/or with one or more other configurations. Each is implemented in hardware and operates within matrix processor 240, converting matrices into vectors on operands. In other embodiments, the formatters 210, 212 are implemented in software, but this typically results in reduced speed. Data formatter 210 converts two-dimensional or three-dimensional (e.g., 3x3x3 cubic) data, including data input matrix 206, into a single vector or string represented by rows or columns, thereby Linearize or vectorize matrix 206. In particular, formatter 210 receives data input matrix 206 and prepares input data for processing by matrix processor 240. In embodiments, this sets the parameters of the data input matrix 206 according to the hardware requirements of the matrix processor 240 so that the matrix processor 240 can efficiently perform matrix multiplications as part of the convolution calculation when generating output pixels. This is achieved by mapping to the appropriate format.

As an example, assuming matrix processor 240 includes 96 rows and 96 columns, data mapped to a 96x96 format causes matrix processor 240 to utilize its full computational power, thus providing favorable efficiency. In that case, formatter 210 would need to produce an output that is 96 columns wide. Similarly, formatter 212 is required to produce an output that is 96 rows wide based on weight input matrix 208.

In embodiments, the formatter 210 uses some multiplexers or switches to fetch some or all of the data input matrix 206 and select different elements therefrom to arrange the data according to the columns of the matrix processor 240. generate. In embodiments, the selection ensures that the appropriate data from data input matrix 206 is passed to each column in a prescribed clock cycle. In embodiments, if the weights are static, they are pre-formatted offline, stored in memory, fetched only once, and processed by the matrix processor 240 in a modified vectorized format without using the formatter 212. may be supplied directly to In other embodiments, the weights may be dynamically adjusted and provided to matrix processor 240 according to various formats and fetch operations. In embodiments, matrix processor 240 allows input of columns and rows of various sizes. That is, matrix processor 240 is designed to compute N×M calculations of arbitrary size.

In other embodiments, the number of columns in matrix processor 240 is limited (eg, to N columns) such that the number of columns in data input matrix 206 (eg, X) is greater than the number of columns in matrix processor 240. (ie, X&gt;N), control logic 250 may partition data input matrix 206 into multiple sub-matrices such that each sub-matrix is computed by matrix processor 240. In such cases, each matrix processor 240 may be running in a different thread. For example, if the data input matrix 206 consists of 192x96 data points and the matrix processor has 96 columns and 96 rows (i.e., 96x96 calculations occur in one clock cycle), the control logic 250 may divide data input matrix 206 into two sub-matrices (such as a left half of data input matrix 206 and a right half of data input matrix 206). Each submatrix consists of 96x96 data points. Each individually threaded matrix processor 240 computes the output channels of the sub-matrices to be sent and inputs them into a final output array 260, which should be large enough to hold the values from all channels (i.e. 192 values). Place the results. More generally, data input matrix 206 may be divided into any number of sub-matrices and sent to different matrix processors 240, each running in a separate thread. Like the output array 226, the data input matrix 206, data formatter 210, cache/buffer 224, logic circuit 232, and post-processing unit 228 must be able to accommodate larger amounts of data as well.

In an alternative embodiment, the control logic 250 can compute the CNN between multiple matrix processors 240 by splitting the computation along the dot product. Each segment of the dot product is computed in a different matrix processor 240, and the input products are then summed together to compute an output vector, which is stored in an output array 260.

Unlike typical software implementations of formatting functions performed by a CPU or GPU for converting convolution operations into matrix multiplications by rearranging the data into an alternative format suitable for fast matrix multiplications, the various aspects of this disclosure A sophisticated hardware implementation allows data to be formatted and executed on the fly. For example, 96 pieces of data are available per cycle, and a very large number of elements of a matrix can be processed in parallel, allowing efficient mapping of data to matrix operations. In an embodiment, for 2N fetched input data, 2N²of calculation data can be acquired in a single clock cycle. This architecture not only provides a parallel and efficient synchronous process to perform numerous mathematical operations across multiple data inputs, but also effectively reduces the number of read or fetch operations used in typical processor architectures. This greatly improves processing speed.

In embodiments, to increase the efficiency of matrix processor 240 with any number of columns and rows, formatters 212, 214 reformat different shapes of input matrix data into columns and rows suitable for matrix processor 240. I can do it. In embodiments, formatting is performed dynamically to accommodate processing of matrices with different input sizes. In embodiments, the reformatted matrix containing the input channels is provided to cache/buffer 224.

Cache/buffer 224 can fetch data from data input matrix 206 1/k times, where k is the convolution kernel width, because different data is reused. For example, when a row is fetched in a particular cycle, a particular column can access all the data in that row. In some embodiments, the cache/buffer 224 may be a partial buffer that stores partial copies of data that can be reused by convolution without having to re-access the SRAM and read the data from the SRAM. .

Once matrix processor 240 completes the calculation, the result set is shifted, e.g., from the bottom row accumulator of matrix processor 240, to an output flip-flop (not shown) that effectively forms a shift register that receives the dot product, e.g. can be done. In embodiments, pulling or shifting results from rows corresponding to output channels to output array 226, eg, one per clock cycle, may be accomplished by a state machine (not shown). The state machine may perform additional operations on the output channels before transmitting the data to SRAM and/or post-processing unit 228, for example. The internal operation of matrix processor 240 is described in more detail below.

In embodiments, matrix processor 240 includes shadow registers that enable parallel processing by storing a copy of the results that are passed through matrix processor 240 to output array 226. In embodiments, moving the operation result from the output register to the shadow register involves loading the next set of values into the ALU.

Once the accumulation is complete, convolution may begin and the accumulation may be redone before all data from the previous convolution is output to output array 226. As a result, on each clock cycle, the data in matrix processor 240 moves down one row such that the last row is output to output array 226 on each cycle. In fact, this mode of operation ensures that new computations are performed on each successive cycle without interruption, independent of additional processing operations such as saving data to SRAM.

Post-processing unit 228 may be part of a hardware accelerated pooling unit, direct memory access ("DMA") that retrieves data from memory and stores data (such as weights and results) in SRAM or the like. It may include or interact with a number of devices (not shown) such as DRAM. The device may be controlled in part or in whole by control logic 250 that manages formatters 210, 212 and other configurations within system 200.

Tracks where system 200 is in the convolution to generate addresses to read data, write results, and calculate where and how to retrieve data used in subsequent steps of the convolution Auxiliary devices that perform management functions, such as sequencers, are not shown in Figure 2.

In a particular embodiment, the weight input matrix 208 is physically split to drive weights from two different sides of the matrix processor 240 such that two two-dimensional arrays each receive a portion of the data in the weight input matrix 208. split into two regions (for example, left and right). Such implementations reduce data latency by exploiting the fact that the weights are known. In embodiments, the timing of operations may be selected such that the weight-data multiplications are spread over a certain number of cycles to reduce peak power consumption. This efficient timing of operations reduces energy-consuming steps, such as reducing the number of read operations performed by the matrix processor and improving the efficiency of data movement within the matrix (e.g., between subcircuits).

In embodiments, a state machine (not shown) configured to identify redundant data may be used. Identified redundant data can be reused across columns, so there is no need to re-fetch the data. The state machine is configured to determine how and where to shift the data to be performed based on inputs related to, for example, image size, filter size, stride, number of channels, and similar parameters. It's okay.

In embodiments, the Booth encoder is shared across several elements of the multiplication architecture of matrix processor 240. The Booth encoder can be any Booth encoder known in the art, which multiplies two numbers and converts one of the two numbers, for example, from an 8-bit value to a 12-bit or multiplier logic. It may be used to encode other values that make multiplication operations easier and therefore faster. In embodiments, the Booth encoder may be applied in parallel across rows to share the same coded alternative weight values across all columns. By loading all columns with operands, we can perform multiplication across an entire row in one clock cycle. Therefore, the cost of leveraging re-encoding to share the same data (e.g., weights) across N computational elements is paid only once for each column (or row). In contrast, existing computational architectures require a Booth encoder for every single scalar and every multiplication operation.

FIG. 3 shows details of an example configuration of the matrix processor architecture shown in FIG. 2. In embodiments, matrix processor 300 may accommodate a predetermined vector length on each axis. As shown in FIG. 3, matrix processor 300 may include an array of 6×6 tiles 302 arranged in matrix format. Each tile 302 may include a matrix 320 that in turn includes subcircuits 350. As described in detail below with reference to FIG. 4, each sub-circuit 350 may be a cell capable of performing arithmetic operations. In embodiments, subcircuit 350 performs multiplication, accumulation, and shift operations simultaneously.

In embodiments, arithmetic operations are parallelized by utilizing multiple rows and columns of matrix processor 300 to generate an N×N tile output. For example, a row size of 96 and a corresponding column size of 96 facilitates the output of a mathematical calculation of 2*9216. In other embodiments, the number of rows and columns may be different. That is, there may be N rows and M columns, and an N×M tile output may be generated. For example, if the row size is 96 and the corresponding column size is 192, one clock cycle will produce 2*18,432 results.

FIG. 4 shows an exemplary multiply-add circuit implementation of the subcircuit shown in FIG. 3. As shown in FIG. 4, the multiply-add circuit 400 includes a multiplier 430, an adder 432, logic 434, 436, 438, an accumulator 424, a shadow register 428, and an output register 440. In embodiments, accumulator 424 may be implemented as an accumulation register.

In embodiments, accumulator 424 may include a set of ALUs that include registers and a shadow register 428 that may be configured to receive the output of the ALU.

In operation, multiplier 430 receives and multiplies weights 402 and data 404 to generate a product therefrom. Each product may be provided to an adder 432 that, in response to receiving the product from multiplier 430, adds the product to the current value of accumulator 424.

In embodiments, accumulator 424 generates an accumulated value that is stored in output register 440, for example. The cumulative value is the result of convolution and may correspond to the dot product of two formatted matrices, as described with reference to FIG.

In embodiments, a copy of the result in output register 440 can be provided to shadow register 428, which outputs result 450, and accumulator 424 can be accessed again to begin a new calculation. In embodiments, the multiply-add circuit 400 of FIG. 4 can perform multiplication, addition operations, and shift operations simultaneously, i.e., within a single cycle, thereby doubling the total number of operations occurring in each cycle. can do.

In embodiments, the ClearAcc signal 408 clears the contents of the accumulator 424 when, for example, the multiplier 430 performs a multiplication operation so that the accumulation operation can be redone. In embodiments, ResultEnable signal 412 is activated in response to a determination that data 404 is valid. It is understood that the accumulator 424 may accumulate and store data, accumulate and erase data, or simply erase data.

In embodiments, results are moved from output register 440 to shadow register 428 in a single clock cycle, ie, without the need for intermediate execution and save operations.

FIG. 5 illustrates an example convolution operation according to various embodiments of this disclosure. Convolution 500 includes input channel IC of input image 502, weights 532, dot product 514, output channel OC, and accumulator 540.

In embodiments, convolution operation 500 applies individual filters (ie, weights) 532 to input image 502, for example, to detect small features within input image 502. By analyzing a series of different features in different orders, macro features can be identified in the input image 502. In other embodiments, input 502 is non-image data. For example, input 502 may be non-image sensor data such as ultrasound, radar, lidar, or other sensor data. Input 502 may be a common mathematical calculation or other type of data known to those skilled in the art.

Convolution 500 may use a different set of weights 532 for each input channel IC, as each input channel IC may contain a different set of information, and each weight matrix 532 may be configured to help identify different features. can be designed. In an embodiment, convolution 500 multiplies rectangular input matrix 504 by rectangular weight matrix 532 to obtain a partial inner product. The partial dot products may then be summed by adder 546 to generate an accumulated dot product 514 (ie, an integer) representing the output pixel 514 in the output image.

In an embodiment, each pixel of output channel OC is generated by multiplier 542 and adder 544. In embodiments, the partial dot product values correspond to the entire application of weight matrix 532 to region 504 of input image 502. In other words, each weight 532 is dot multiplied by the region 504 by a multiplier 542 to produce a partial dot product, and the partial dot products are then accumulated in an accumulator 540 to produce a cumulative output representing the convolution. .

One or more input channel ICs can be used, eg, one for each color (eg, RGB). For example, each convolution may use weights 532 representing three different matrices, one for each color. Each output channel OC 512 may be generated using different filters or weights 532 representing different characteristics of the input data 502. The number of output channels may depend on the number of functions. The number of convolutions is equal to the number of output channels OC multiplied by the number of input channels IC, and each convolution may have N convolutions for each input channel IC. Those skilled in the art will recognize that the number and type of input channels may vary and may include colored and/or clear inputs.

As shown in FIG. 5, the input matrix 504 is a Kx×Ky (i.e., 3×3) matrix that can be combined with the 3×3 weight matrix 532 across the three input channels, i.e., 3×3×IC; The depths match, producing a single element, dot product 514, in the output plane. Each dot product 514 of output channel 512 is the result of a dot multiplication.

6-8 illustrate details of example convolution operations according to various embodiments of the present disclosure. Convolution 600 includes an input data matrix 602, a weight data matrix 604, an array 606, and an inner product 630. In embodiments, array 606 is a matrix processor architecture as shown in FIGS. 2 and 3.

The input data matrix 602 of FIG. 6 includes a column 610 that, in embodiments, can be obtained by linearizing an input matrix, such as the rectangular input matrix 504 shown in FIG. 5, to obtain a vectorized form of the input matrix. . Similarly, weight data matrix 604 includes rows 620, which may be a vectorized form of a weight matrix, such as rectangular weight matrix 532 of FIG. As an example, a 3x3 input matrix and three input channels can be reformatted into a vector containing 3x3x3=27 elements that can generate a 27 element sequence 610 for use in input data matrix 602. Conversely, a 3×3 weight matrix of the same three input channels can be used to generate a 27-element row 620 for use in weight data matrix 604. Those skilled in the art will recognize that the size of the input matrix and number of input channels may vary across different applications.

In an embodiment, the input channels and input weights, depicted as rectangles in FIG. is reformatted into a vector format (e.g., a vector with 96 elements) provided in , which allows dot product operations of 96×96 elements to be performed in parallel. In particular, input data 504 and input weights 532, shown in FIG. 5 as rectangles for each input channel, are reformatted into vector form.

In an embodiment, the resulting vector format, shown as input data 602 and input weights 604 (e.g., each containing 96 elements) in FIG. is provided to multiplier 240. In embodiments, the calculation of the output channels uses the same set of input data but different sets of weights (i.e., filters) to produce the same output pixel, and reading the input data produces many outputs at once. Channels can be created. As mentioned above, it is understood that the number of input channels and the number of output channels can be chosen arbitrarily.

Further, it is understood that input data matrix 602, weight data matrix 604, and array 606 may have a different number of columns and rows than shown in FIG. In particular, the shapes of input data matrix 602 and weight data matrix 604 may be formatted to correspond to columns and rows of any arbitration configuration of array 606. Additionally, in situations where the weight data matrix 604 is known, rows 620 can be generated without the use of a formatter and stored in vectorized form.

In an embodiment, dot product 630 in FIG. 6 is generated by dot-multiplying the vector corresponding to column 610 by the vector corresponding to row 620. In an embodiment, as shown in FIG. 7, the next dot product 632 may be obtained by dot-multiplying the vector corresponding to column 612 and the vector corresponding to row 620. As those skilled in the art will recognize, once all the dot products in the first row of array 606 are filled, the dot product in the second row of array 606 is equal to It can be calculated by dot multiplication with the second row, etc.

It is important to note that FIGS. 6-8 merely serve illustrative purposes and that the dot multiplications described above can be performed simultaneously to generate one-shot matrix-matrix multiplication operations.

FIG. 9 illustrates an example deconvolution operation according to various embodiments of the present disclosure. Deconvolution system 900 includes input channel IC of input image 902, weights 922, dot products 904, 906, and output channel OC. Those skilled in the art will recognize that deconvolution operation 900 is actually a mathematical transpose (nearly the inverse) of a convolution operation, such as the convolution shown in FIG. Those skilled in the art will further recognize that neural networks can be used to learn the deconvolution operation 900 by applying procedures similar to those used for conventional convolutional neural networks. For the sake of brevity, descriptions of configurations or functions similar to those of FIG. 5 are not repeated here.

In an embodiment, deconvolution operation 900 of FIG. 9 reconstructs matrix 912 by decomposing dot product 904906 using weights 922. Similar to the convolution operation, deconvolution 900 may use a different set of weights 922 for each input channel IC. In embodiments, deconvolution 900 may be advantageously applied to images to perform image deconvolution to improve robustness against artifacts, for example. Other applications may include image data analysis and restoration, etc.

FIG. 10 illustrates a process for performing arithmetic operations to accelerate a convolutional neural network according to various embodiments of the present disclosure.

Process 1000 for performing an arithmetic operation begins at step 1002 when a first set of operands that can represent rows of a data matrix is received from a first logic circuit. This first set of operands can be vectorized so that the operands are aligned with the inputs to the matrix processor. In certain embodiments, the size of the vectorized operands is directly related to the number of inputs to the matrix processor along the axis.

At step 1004, a second set of operands that may represent columns of the weight matrix are received from the second logic circuit. This second set of operands is vectorized such that the operands are aligned within the corresponding inputs to the matrix processor. In certain embodiments, the size of the vectorized operands is directly related to the number of inputs to the matrix process along different axes.

At step 1006, the first set of operands is dot multiplied with the second set of operands to obtain one or more dot products. In certain embodiments, this set operation across the set of operands is performed in a single clock cycle.

At step 1008, the image may be convolved with the filter using the dot product to generate a convolution result.

At step 1010, the convolution results are further processed to enhance the image output. This further processing may occur using non-linear functions, normalization operations, or pooling operations.

Those skilled in the art will recognize that the computing system or programming language is not critical to implementing the invention. Those skilled in the art will also recognize that many of the elements described above can be physically and/or functionally separated into submodules or combined together.

It is noted that the elements of the following claims may be arranged in various ways, including including multiple dependencies, configurations, and combinations. For example, in embodiments, the subject matter of various claims may be combined with other claims.

Those skilled in the art will understand that the foregoing examples and embodiments are illustrative and do not limit the scope of the invention. All substitutions, enhancements, equivalents, combinations, and improvements apparent to those skilled in the art from reading the specification and studying the drawings are intended to be included within the true spirit and scope of the invention.