Decoders for analog neural memory in deep learning artificial neural network

(CLAIMS OF PRIORITY) This application claims priority to U.S. Provisional Patent Application No. 62/642,884, filed March 14, 2018, entitled "Decoders for Analog Neuromorphic Memory in Artificial Neural Network," and to U.S. Provisional Patent Application No. 15/991,890, filed May 29, 2018, entitled "Decoders For Analog Neural Memory In Deep Learning Artificial Neural Network."

FIELD OF THEINVENTION Numerous embodiments of a decoder for use with vector matrix multiplication (VMM) arrays in artificial neural networks are disclosed.

Artificial neural networks mimic biological neural networks (the central nervous systems of animals, particularly the brains), which can rely on a large number of inputs and are used to estimate or approximate functions that are largely unknown. Artificial neural networks generally contain layers of interconnected "neurons" that exchange messages.

FIG. 1 illustrates an artificial neural network, where circles represent layers of inputs or neurons. The connections (called synapses) are represented by arrows and have numerical weights that can be adjusted based on experience. This allows the neural network to adapt to the inputs and to learn. Typically, a neural network includes multiple layers of inputs. There are typically one or more intermediate layers of neurons, and an output layer of neurons that provide the output of the neural network. At each level, the neurons make decisions, either individually or together, based on the data received from the synapses.

One of the main challenges in the development of artificial neural networks for high-performance information processing is the lack of suitable hardware technologies. In practice, practical neural networks rely on a very large number of synapses, allowing high connectivity between neurons, i.e., a very high degree of computational parallelism. In principle, such complexity could be realized by digital supercomputers or specialized graphic processing unit clusters. However, in addition to high costs, these approaches also suffer from poor energy efficiency compared to biological networks, which mainly perform low-precision analog calculations and therefore consume much less energy. CMOS analog circuits have been used in artificial neural networks, but most CMOS-implemented synapses are too bulky for the large number of neurons and synapses.

Applicant previously disclosed an artificial (analog) neural network utilizing one or more non-volatile memory arrays as synapses in U.S. Patent Application Serial No. 15/594,439, which is incorporated by reference. The non-volatile memory array operates as an analog neuromorphic memory. The neural network device includes a first plurality of synapses configured to receive a first plurality of inputs and generate a first plurality of outputs therefrom, and a first plurality of neurons configured to receive the first plurality of outputs. The first plurality of synapses includes a plurality of memory cells, each of which includes spaced apart source and drain regions formed in a semiconductor substrate with a channel region extending therebetween, a floating gate disposed above and insulated from a first portion of the channel region, and a non-floating gate disposed above and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a number of electrons in the floating gate. The plurality of memory cells are configured to multiply the first plurality of inputs by the stored weight values to generate a first plurality of outputs.

Each non-volatile memory cell used in an analog neuromorphic memory system must be erased and programmed to hold a very specific and precise amount of charge in its floating gate. For example, each floating gate must hold one of N different values, where N is the number of different weights that can be exhibited by each cell. Examples of N include 16, 32, and 64.

Prior art decode circuits (such as bit line decoders, word line decoders, control gate decoders, source line decoders, and erase gate decoders) are not suitable for use with VMMs in analog neuromorphic memory systems. One reason for this is that in VMM systems, the program and verify portion of the operation (which is a read operation) operates on a single selected memory cell, while the read operation operates on all memory cells in the array.

What is needed is an improved decoding circuit suitable for use with a VMM in an analog neuromorphic memory system.

A number of embodiments are disclosed for use with vector matrix multiplication (VMM) arrays in artificial neural networks.

<figref num="1">FIG. 1 illustrates an artificial neural network.</figref><figref num="2">1 is a cross-sectional view of a conventional two-gate non-volatile memory cell.</figref><figref num="3">1 is a cross-sectional view of a conventional four-gate nonvolatile memory cell.</figref><figref num="4">1 is a side cross-sectional view of a conventional three-gate nonvolatile memory cell.</figref><figref num="5">1 is a cross-sectional view of another conventional two-gate nonvolatile memory cell.</figref><figref num="6">FIG. 1 illustrates an exemplary artificial neural network with different levels utilizing a non-volatile memory array.</figref><figref num="7">FIG. 2 is a block diagram illustrating a vector multiplier matrix.</figref><figref num="8">FIG. 2 is a block diagram showing various levels of a vector multiplier matrix;</figref><figref num="9">1 illustrates one embodiment of a vector multiplier matrix.</figref><figref num="10">1 illustrates another embodiment of a vector multiplier matrix.</figref><figref num="11">1 illustrates another embodiment of a vector multiplier matrix.</figref><figref num="12">1 illustrates another embodiment of a vector multiplier matrix.</figref><figref num="13">1 illustrates another embodiment of a vector multiplier matrix.</figref><figref num="14">1 illustrates one embodiment of a bit line decoder for a vector multiplier matrix.</figref><figref num="15">13 illustrates another embodiment of a bit line decoder for a vector multiplier matrix.</figref><figref num="16">13 illustrates another embodiment of a bit line decoder for a vector multiplier matrix.</figref><figref num="17">1 shows a system for operating a vector multiplier matrix.</figref><figref num="18">2 illustrates another system for operating a vector multiplier matrix.</figref><figref num="19">2 illustrates another system for operating a vector multiplier matrix.</figref><figref num="20">1 illustrates one embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="21">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="22">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="23">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="24">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="25">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="26">1 illustrates another embodiment of a word line driver for use with a vector multiplier matrix.</figref><figref num="27">1 illustrates a source line decoder circuit for use with a vector multiplier matrix.</figref><figref num="28">A word line decoder circuit, a source line decoder circuit, and a high voltage level shifter for use with a vector multiplier matrix are shown.</figref><figref num="29">An erase gate decoder circuit, a control gate decoder circuit, a source line decoder circuit, and a high voltage level shifter are shown for use with a vector multiplier matrix.</figref><figref num="30">1 illustrates a word line decoder circuit for use with a vector multiplier matrix.</figref><figref num="31">1 illustrates a control gate decoder circuit for use with a vector multiplier matrix.</figref><figref num="32">1 illustrates another control gate decoder circuit for use with a vector multiplier matrix.</figref><figref num="33">1 illustrates another control gate decoder circuit for use with a vector multiplier matrix.</figref><figref num="34">1 shows a current-voltage circuit for controlling word lines in a vector multiplier matrix.</figref><figref num="35">13 shows another current-voltage circuit for controlling word lines in a vector multiplier matrix.</figref><figref num="36">1 shows a current-voltage circuit for controlling the control gate lines in a vector multiplier matrix.</figref><figref num="37">13 shows another current-voltage circuit for controlling the control gate lines in a vector multiplier matrix.</figref><figref num="38">13 shows another current-voltage circuit for controlling the control gate lines in a vector multiplier matrix.</figref><figref num="39">13 shows another current-voltage circuit for controlling word lines in a vector multiplier matrix.</figref><figref num="40">13 shows another current-voltage circuit for controlling word lines in a vector multiplier matrix.</figref><figref num="41">13 shows another current-voltage circuit for controlling word lines in a vector multiplier matrix.</figref><figref num="42">10 shows the operating voltages of the vector multiplier matrix of FIG.</figref><figref num="43">11 shows the operating voltages of the vector multiplier matrix of FIG.</figref><figref num="44">12 shows the operating voltages of the vector multiplier matrix of FIG.</figref><figref num="45">13 shows the operating voltages of the vector multiplier matrix of FIG.</figref>

The artificial neural network of the present invention utilizes a combination of CMOS technology and non-volatile memory arrays.

Non-volatile Memory Cell

Digital non-volatile memories are well known. For example, U.S. Pat. No. 5,029,130 ("the '130 patent") discloses an array of split-gate non-volatile memory cells, and is incorporated herein by reference for all purposes. Such a memory cell is shown in FIG. 2. Each memory cell 210 is formed in a semiconductor substrate 12 and includes a source region 14 and a drain region 16 having a channel region 18 therebetween. A floating gate 20 is formed above and insulated from (and controls the conductivity of) a first portion of the channel region 18, and is formed above a portion of the source region 16. A wordline terminal 22 (typically coupled to a wordline) has a first portion disposed above and insulated from (and controls the conductivity of) the second portion of the channel region 18, and a second portion extending above and above the floating gate 20. A floating gate 20 and a wordline terminal 22 are insulated from the substrate 12 by a gate oxide. A bitline 24 is coupled to the drain region 16.

The memory cell 210 is erased (where electrons are removed from the floating gate) by applying a high positive voltage to the wordline terminal 22, thereby causing electrons in the floating gate 20 to tunnel through the intermediate insulator from the floating gate 20 to the wordline terminal 22 by Fowler-Nordheim tunneling.

The memory cell 210 is programmed by applying a positive voltage to the wordline terminal 22 and a positive voltage to the source 16 (where electrons are applied to the floating gate). An electronic current will flow from the source 16 towards the drain 14. When the electrons reach the gap between the wordline terminal 22 and the floating gate 20, they accelerate and heat up. Some of the heated electrons are injected into the floating gate 20 through the gate oxide 26 due to electrostatic attraction from the floating gate 20.

The memory cell 210 is read by applying a positive read voltage to the drain 14 and the word line terminal 22 (turning on the channel region under the word line terminal). If the floating gate 20 is positively charged (i.e., erased of electrons and positively coupled to the drain 16), then the portion of the channel region under the floating gate 20 will also be turned on and current will flow through the channel region 18, which is sensed as an erased or "1" state. If the floating gate 20 is negatively charged (i.e., programmed with electrons), then the portion of the channel region under the floating gate 20 will be mostly or completely off and no (or only a small) current will flow through the channel region 18, which is sensed as a programmed or "0" state.

Table 1 shows typical voltage ranges that may be applied to the terminals of memory cell 210 to perform read, erase and program operations.

Table 1: Operation of the Flash Memory Cell 210 of FIG. 2<tables><img file="JP7547536B2_D0001.tif" /></tables>

Other split gate memory cell configurations are known. For example, FIG. 3 shows a four-gate memory cell 310 with a source region 14, a drain region 16, a floating gate 20 above a first portion of the channel region 18, a select gate 28 (typically coupled to a word line) above a second portion of the channel region 18, a control gate 22 above the floating gate 20, and an erase gate 30 above the source region 14. This configuration is described in U.S. Pat. No. 6,747,310, which is incorporated herein by reference for all purposes. Here, all gates, except for the floating gate 20, are non-floating gates, meaning that they are electrically connected or connectable to a voltage source. Programming is indicated by heated electrons from the channel region 18 injecting themselves into the floating gate 20. Erasing is indicated by electrons tunneling from the floating gate 20 to the erase gate 30.

Table 2 shows typical voltage ranges that may be applied to the terminals of memory cell 310 to perform read, erase and program operations.

Table 2: Operation of the Flash Memory Cell 310 of FIG.<tables><img file="JP7547536B2_D0002.tif" /></tables>

Figure 4 shows a split gate tri-gate memory cell 410. Memory cell 410 is identical to memory cell 310 of Figure 3, except that memory cell 410 does not have a separate control gate. Erase operations (erase through the erase gate) and read operations are similar to those of Figure 3, except that there is no control gate bias. Programming operations are also performed without a control gate bias, so the program voltage of the source line is higher to compensate for the lack of control gate bias.

Table 3 shows typical voltage ranges that may be applied to the terminals of memory cell 410 to perform read, erase and program operations.

Table 3: Operation of Flash Memory Cell 410 of FIG.<tables><img file="JP7547536B2_D0003.tif" /></tables>

Figure 5 illustrates a stacked gate memory cell 510. Memory cell 510 is similar to memory cell 210 of Figure 2, except that the floating gate 20 extends over the channel region 18 and the control gate 22 extends over the floating gate 20 separated by an insulating layer. Erase, programming, and read operations operate in a similar manner as described above for memory cell 210.

Table 4 shows typical voltage ranges that may be applied to the terminals of memory cell 510 to perform read, erase and program operations.

Table 4: Operation of Flash Memory Cell 510 of FIG.<tables><img file="JP7547536B2_D0004.tif" /></tables>

In order to utilize a memory array containing one of the types of non-volatile memory cells in an artificial neural network as described above, two modifications are made: First, as explained further below, the lines are configured so that each memory cell can be individually programmed, erased, and read without adversely affecting the memory state of other memory cells in the array; Second, a continuous (analog) programming of the memory cells is provided.

Specifically, the memory state (i.e., the charge on the floating gate) of each memory cell in the array can be changed from a fully erased state to a fully programmed state, independently and continuously with minimal disturbance to other memory cells. In another embodiment, the memory state (i.e., the charge on the floating gate) of each memory cell in the array can be changed from a fully programmed state to a fully erased state, and vice versa, independently and continuously with minimal disturbance to other memory cells. This means that the cell storage is analog, or can at a minimum store one of many distinct values (such as 16 or 64 different values), which allows for very precise and individual tuning of every cell in the memory array, and makes the memory array ideal for storing and applying fine-tuning to the synaptic weights of a neural network.

Neural network using non-volatile memory cell arrays

6 conceptually illustrates a non-limiting example of a neural network utilizing a non-volatile memory array. This example uses a non-volatile memory array neural net for a face recognition application, although any other suitable application can be implemented using a non-volatile memory array based neural network.

S0 is the input, which in this example is a 32x32 pixel RGB image with 5-bit precision (i.e., three 32x32 pixel arrays, one for each color R, G, and B, each pixel with 5-bit precision). The synapse CB1 going from S0 to C1 has both a set of distinct and shared weights, scans the input image with overlapping filters of 3x3 pixels (kernels), and shifts the filters by one pixel (or two or more pixels as determined by the model). Specifically, the values of the nine pixels in the 3x3 portion of the image (i.e., called filters or kernels) are provided to the synapse CB1, which multiplies these nine input values by the appropriate weights, and after summing the outputs of the multiplication, a single output value is determined and given by the first neuron of CB1 to generate one pixel of the layer of feature map C1. The 3x3 filter is then shifted one pixel to the right (i.e., add a column of 3 pixels to the right and drop a column of 3 pixels on the left) so that the 9 pixel values of this newly positioned filter are provided to synapse CB1, which multiplies them by the same weights to determine a second single output value by the associated neuron. This process continues until the 3x3 filter has scanned for all three colors and all bits across the entire 32x32 pixel image (precision value). The process is then repeated using different sets of weights to generate different feature maps of C1 until all feature maps of layer C1 have been calculated.

In C1, in this example, there are 16 feature maps, each with 30x30 pixels. Each pixel is a new feature pixel extracted from the multiplication of the input with the kernel, and therefore each feature map is a two-dimensional array, and thus in this example, the synapses CB1 constitute 16 layers of two-dimensional arrays (note that the neuron layers and arrays referred to herein are logical, not necessarily physical, relationships, i.e., the arrays are not necessarily oriented in a physical two-dimensional array). Each of the 16 feature maps is generated by one of 16 different sets of synaptic weights applied to the filter scans. The C1 feature maps can all target different aspects of the same image feature, such as boundary identification. For example, a first map (generated using a first set of weights shared by all scans used to generate this first map) can identify circular edges, a second map (generated using a second set of weights different from the first set of weights) can identify square edges, or the aspect ratio of a particular feature, etc.

An activation function P1 (pooling) is applied before going from C1 to S1 to pool values from consecutive, non-overlapping 2x2 regions in each feature map. The purpose of the pooling step is to average to nearby locations (or a max function can also be used), e.g., to reduce dependency on edge locations, and to reduce data size before going to the next stage. In S1, there are 16 15x15 feature maps (i.e., 16 different arrays of 15x15 pixels each). The synapses and associated neurons in CB2 going from S1 to C2 scan the maps in S1 with a 4x4 filter using a filter shift of 1 pixel. In C2, there are 22 12x12 feature maps. An activation function P2 (pooling) is applied before going from C2 to S2 to pool values from consecutive, non-overlapping 2x2 regions in each feature map. In S2, there are 22 6x6 feature maps. The activation function is applied at synapse CB3 going from S2 to C3, where all neurons in C3 connect to all maps in S2. There are 64 neurons in C3. Synapse CB4 going from C3 to output S3 fully connects S3 with C3. The output at S3 contains 10 neurons, where the highest output neuron determines the class. This output can indicate, for example, the identity or classification of the content of the original image.

Each level of synapses is implemented using an array or a portion of an array of non-volatile memory cells. FIG. 7 is a block diagram of a vector matrix multiplication (VMM) array that includes non-volatile memory cells and is used as a synapse between an input layer and the next layer. Specifically, the VMM 32 includes an array of non-volatile memory cells 33, an erase gate and word line gate decoder 34, a control gate decoder 35, a bit line decoder 36, and a source line decoder 37, which decode the inputs to the memory array 33. The source line decoder 37 in this example also decodes the output of the memory array. Alternatively, the bit line decoder 36 can decode the output of the memory array. The memory array serves two purposes. First, it stores the weights used by the VMM. Second, the memory array effectively multiplies the inputs by the weights stored in the memory array and adds them for each output line (source line or bit line) to generate an output, which becomes the input to the next layer or the input to the last layer. By performing multiplication and addition functions, the memory array eliminates the need for separate multiplication and addition logic circuits and is also power efficient due to in-place memory computation.

The output of the memory array is fed to a differential adder (such as a summing op-amp) 38, which sums the outputs of the memory cell array to generate a single value for the convolution. The differential adder is such that a positive input realizes the sum of positive and negative weights. The summed output value is then fed to an activation function circuit 39, which rectifies the output. The activation function may include a sigmoid, tanh, or ReLU function. The rectified output value becomes an element of the feature map as the next layer (e.g., C1 in the above description), and is then applied to the next synapse to generate the next feature map layer or the last layer. Thus, in this example, the memory array constitutes a number of synapses (receiving input from a previous layer of neurons or from an input layer such as an image database), and the summing op-amp 38 and activation function circuit 39 constitute a number of neurons.

Figure 8 is a block diagram of the various levels of VMMs. As shown in Figure 14, inputs are converted from digital to analog by a digital-to-analog converter 31 and provided to an input VMM 32a. The output generated by the input VMM 32a is provided as input to the next VMM (hidden level 1) 32b, which in turn generates an output that is provided as input to the next VMM (hidden level 2) 32b, and so on. The various layers of the VMMs 32 are implemented using convolutional neural networks. Each VMM functions as a different layer of synapses and neurons of a neural network (CNN). Each VMM can be a standalone non-volatile memory array, or the VMMs can utilize different portions of the same non-volatile memory array, or the VMMs can utilize overlapping portions of the same non-volatile memory array. The example shown in FIG. 8 includes five layers (32a, 32b, 32c, 32d, 32e): one input layer (32a), two hidden layers (32b, 32c), and two fully connected layers (32d, 32e). Those skilled in the art will appreciate that this is merely an example, and that the system may instead include more than two hidden layers and more than two fully connected layers.

Vector Matrix Multiplication (VMM) Array

FIG. 9 shows a neuron VMM 900 that is particularly suited for memory cells of the type shown in FIG. 2, used as part of synapses and neurons between an input layer and the next layer. The VMM 900 comprises a memory array 903 of non-volatile memory cells, a reference array 901, and a reference array 902. The reference arrays 901 and 902 serve to convert the current inputs flowing into terminals BLR0-3 into voltage inputs WL0-3. The illustrated reference arrays 901 and 902 are in the column direction. In general, the reference array direction is perpendicular to the input lines. In practice, the reference memory cells are diodes connected through multiplexers (multiplexer 914, which includes one multiplexer and one cascode transistor VBLR for biasing the reference bit lines), into which the current inputs flow. The reference cells are adjusted to a target reference level.

The memory array 903 serves two purposes. First, it stores the weights used by the VMM 900. Second, it effectively multiplies the inputs (current inputs provided to terminals BLR0-3, which the reference arrays 901 and 902 convert to input voltages and provide to the word lines WL0-3) with the weights stored in the memory array to produce an output that is the input to the next layer or the input to the final layer. By performing a multiplication function, the memory array eliminates the need for separate multiplication logic and is also power efficient. Here, voltage inputs are provided to the word lines and outputs appear on the bit lines during read (inference) operations. The current on the bit lines performs a function of the sum of all the currents from the memory cells connected to the bit lines.

Figure 42 shows the operating voltages of the VMM 900. The columns in the table show the voltages applied to the word line of the selected cell, the word lines of the unselected cells, the bit lines of the selected cell, the bit lines of the unselected cells, the source lines of the selected cell, and the source lines of the unselected cells. The rows show the operations of read, erase, and program.

FIG. 10 shows a neuron VMM1000 that is particularly suited for memory cells of the type shown in FIG. 2, which are used as part of synapses and neurons between an input layer and the next layer. VMM1000 comprises a memory array 1003 of non-volatile memory cells, a reference array 1001, and a reference array 1002. VMM1000 is similar to VMM900, except that in VMM1000 the word lines run vertically. There are two reference arrays 1001 (at the top, providing a reference conversion input current to the voltages of the even rows) and 1002 (at the bottom, providing a reference conversion input current to the voltages of the odd rows). Here, the input is provided to the word lines and the output appears on the source lines during a read operation. The current on the source lines performs a function of the sum of all the currents from the memory cells connected to the source lines.

Figure 43 shows the operating voltages of the VMM 1000. The columns in the table show the voltages applied to the word line of the selected cell, the word lines of the unselected cells, the bit lines of the selected cell, the bit lines of the unselected cells, the source lines of the selected cell, and the source lines of the unselected cells. The rows show the operations of read, erase, and program.

FIG. 11 shows a neuron VMM 1100 that is particularly suited for memory cells of the type shown in FIG. 3, used as part of a neuron and synapse between an input layer and the next layer. VMM 1100 comprises a memory array 1101 of non-volatile memory cells, a reference array 1102 (which provides a reference conversion input current to the input voltages of the even rows), and a reference array 1103 (which provides a reference conversion input current to the input voltages of the odd rows). VMM 1100 is similar to VMM 900, except for VMM 1110. VMM 1100 further comprises a control line 1106 coupled to the control gates of a row of memory cells, and a control line 1107 coupled to the erase gates of an adjacent row of memory cells, where the word line, control gate line, and erase gate line are in the same direction. The VMM comprises a reference bit line select transistor 1104 (mux 1114) that selectively couples to the bit line contact of a selected reference memory cell and a switch 1105 (part of mux 1114). 1114) that selectively couples the reference bit lines to the control lines 1106 for a particular selected reference memory cell, where the inputs are provided to the word lines (of the memory array 1101) and the outputs appear on the bit lines, such as bit line 1109, during a read operation. The current on the bit line is a function of the sum of all the currents from the memory cells connected to the bit line.

Figure 44 shows the operating voltages of VMM 1100. The columns in the table show the voltages of the word line of the selected cell, the word lines of the unselected cells, the bit line of the selected cell, the bit lines of the unselected cells, the control gate of the selected cell, the control gates of the unselected cells in the same sector as the selected cell, the control gates of the unselected cells in a different sector than the selected cell, the erase gate of the selected cell, the erase gates of the unselected cells, the source line of the selected cell, and the source lines of the unselected cells. The rows show the read, erase, and program operations.

Figure 12 shows a neuron VMM1200 that is particularly suited for memory cells of the type shown in Figure 3, which are used as part of synapses and neurons between an input layer and the next layer. VMM1200 is similar to VMM1100, except that in VMM1200, erase gate lines, such as erase gate line 1201, run vertically. Here, inputs are provided on word lines and outputs appear on source lines. The current in a bit line performs a function of the sum of all the currents from memory cells connected to the bit line.

Figure 45 shows the operating voltages of VMM1200. The columns in the table show the voltages of the word line of the selected cell, the word lines of the unselected cells, the bit line of the selected cell, the bit lines of the unselected cells, the control gate of the selected cell, the control gates of the unselected cells in the same sector as the selected cell, the control gates of the unselected cells in a different sector than the selected cell, the erase gate of the selected cell, the erase gates of the unselected cells, the source line of the selected cell, and the source lines of the unselected cells. The rows show the read, erase, and program operations.

FIG. 13 shows a neuron VMM 1300 that is particularly suited for memory cells of the type shown in FIG. 3, which are utilized as part of synapses and neurons between an input layer and the next layer. VMM 1300 comprises a memory array 1301 of non-volatile memory cells, and a reference array 1302 (at the top of the array). Alternatively, another reference array can be located at the bottom, similar to that of FIG. 10. In other respects, VMM 1300 is similar to VMM 1200, except that in VMM 1300, the control gate lines, such as control gate line 1303, run vertically (so that the row-wise reference array 1302 is orthogonal to the input control gate lines), and the erase gate lines, such as erase gate line 1304, run horizontally. Here, inputs are provided to the control gate lines and outputs appear on the source lines. In one embodiment, only even rows are used, and in another embodiment, only odd rows are used. The current in the source line performs a function of the sum of all the currents from the memory cells connected to the source line.

As described herein for neural networks, the flash cells are preferably configured to operate in the sub-threshold region.

The memory cells described herein are biased in weak inversion.

Ids=Io^*e^(Vg-Vth)/kVt=w^*Io^*e^(Vg)/kVtw=e^(-Vth)/kVt

On the I-V log converter using memory cells to convert input current to input voltage: Vg=k^*Vt^*log[Ids/wp^*Io]

For a memory array used as a vector matrix multiplier VMM, the output current is: Iout=wa^*Io^*e^(Vg)/kVt, that is, Iout=(wa/wp)^*Iin=W^*IinW=e^{(Vthp-Vtha)/kVt}

The word line or control gate can be used as the input of the memory cell for an input voltage.

Alternatively, the flash memory cells can be configured to operate in the linear region.

Ids=β^*(Vgs-Vth)^*Vds;β=u^*Cox^*W/LW α(Vgs-Vth)

In an IV linear converter, the input/output current can be linearly converted to an input/output voltage using memory cells operating in the linear region.

Other embodiments of the ESF vector matrix multiplier are described in U.S. Patent Application Serial No. 15/826,345, which is incorporated herein by reference. The source lines or bit lines can be used as neuron outputs (current sum outputs).

14 illustrates one embodiment of a bitline decoder circuit 1400. Bitline decoder circuit 1400 includes a column decoder 1402 and an analog neuromorphic neuron ("ANN") column decoder 1403, each of which is coupled to a VMM array 1401. The VMM array can be based on any of the VMM designs discussed above (such as VMMs 900, 1000, 1100, 1200, and 1300) or other VMM designs.

One challenge with analog neuromorphic systems is that the system must be able to program and verify (which involves read operations) and perform ANN reads where every cell in the array is selected and read. In other words, the bit line decoder must select only one bit line in some cases and all bit lines in other cases.

The bit line decoder circuit 1400 achieves this objective. The column decoder 1402 is a conventional column decoder (program and erase, or PE, decode path) that can be used to select individual bit lines such as program and program verify (sense operations). The output of the column decoder 1402 is coupled to a program/erase (PE) column driver circuit for controlling program, PE verify, and erase (not shown in FIG. 14). The ANN column decoder 1403 is a column decoder specifically designed to enable read operations on all bit lines at once. The ANN column decoder 1403 comprises an exemplary select transistor 1405 coupled to a bit line (here BL0) and an output circuit (e.g., current adder and activation function such as tanh, sigmoid, ReLU, etc.) 1406. The same set of devices is attached to each of the other bit lines. All of the select transistors such as the select transistors 1405 are coupled to a select line 1404. During an ANN read operation, select line 1404 is enabled, turning on each of the select transistors, such as select transistor 1405, and then the current from each bit line is received by an output circuit, such as circuit 1406 and output.

15 illustrates one embodiment of a bit line decoder circuit 1500. The bit line decoder circuit 1500 is coupled to a VMM array 1501. The VMM array can be based on any of the VMM designs discussed above (such as VMMs 900, 1000, 1100, 1200, and 1300) or other VMM designs.

Select transistors 1502 and 1503 are controlled by a pair of complementary control signals (V0 and VB_0) and are coupled to a bit line (BL0). Select transistors 1504 and 1505 are controlled by another pair of complementary control signals (V1 and VB_1) and are coupled to another bit line (BL1). Select transistors 1502 and 1504 are coupled to the same output for enable programming, select transistors 1503 and 1505 are coupled to the same output for inhibit programming, etc. The output lines (program and erase PE decode paths) of transistors 1502/1503/1504/1505 are such that they are coupled to PE column driver circuits for controlling program, PE verify, and erase (not shown).

The select transistor 1506 is coupled to a bit line (BL0) and to an output and activation function circuit 1507 (e.g., a current summation and activation function such as tanh, sigmoid, ReLU, etc.). The select transistor 1506 is controlled by a control line 1508.

When only BL0 is activated, control line 1508 is deasserted and signal V0 is asserted, thus reading only BL0. During an ANN read operation, control line 1508 is asserted, turning on select transistor 1506 and similar transistors, and all bit lines are read for all neuron processing.

16 illustrates one embodiment of a bit line decoder circuit 1600. The bit line decoder circuit 1600 is coupled to a VMM array 1601. The VMM array can be based on any of the VMM designs discussed above (such as VMMs 900, 1000, 1100, 1200, and 1300) or other VMM designs.

Select transistor 1601 is coupled to a bit line (BL0) and is coupled to an output and activation function circuit 1603. Select transistor 1602 is coupled to the bit line (BL0) and to a common output (PE decode path).

When only BL0 is activated, select transistor 1602 is activated and connects BL0 to the common output. During an ANN read operation, select transistor 1601 and similar transistors are turned on and all bit lines are read.

Regarding the decode of Figures 14, 15 and 16, the unselected transistors can be negatively biased to reduce the transistor leakage from affecting the memory cell performance. Alternatively, the PE decode path can be negatively biased while the array is in ANN operation. The negative bias can be -0.1V to -0.5V or more.

17 shows a VMN 1700. The VMM system 1700 includes a VMM array 1701 and a reference array 1720 (which can be based on any of the VMM designs described above, such as VMMs 900, 1000, 1100, 1200, and 1300, or other VMM designs), a low-voltage row decoder 1702, a high-voltage row decoder 1703, a reference cell low-voltage column decoder 1704 (shown for the reference array in the column direction, meaning to provide inputs to output conversions in the row direction), bit line PE drivers 1712, bit line multiplexers 1706, activation function circuits and adders 1707, control logic 1705, and analog bias circuits 1708.

As shown, the reference cell low voltage column decoder 1704 is meant to be for a column-wise reference array 1720 and provide inputs for outputting conversions row-wise. If the reference array were row-wise, then the reference decoders would need to be at the top and/or bottom of the array and provide inputs for outputting conversions column-wise.

The low voltage row decoder 1702 provides bias voltages for read and program operations and provides decode signals for the high voltage row decoder 1703. The high voltage row decoder 1703 provides high voltage bias signals for program and erase operations. The reference cell low voltage column decoder 1704 provides the decode function for the reference cells. The bit line PE driver 1712 provides the control function for the bit lines in program, verify, and erase. The bias circuit 1705 is a shared bias block that provides multiple voltages required for the various program, erase, program verify, and read operations.

FIG. 18 illustrates a VMM system 1800. The VMM system 1800 is similar to the VMM system 1700, except that the VMM system 1800 further includes a red array 1801, a bit line PE driver BLDRV 1802, a high voltage column decoder 1803, an NVR sector 1804, and a reference array 1820. The high voltage column decoder 1803 provides high voltage bias for the vertical decode lines. The red array 1802 provides array redundancy for replacing defective array portions. The NVR (Non-Volatile Register AKA Information Sector) sector 1804 is a sector that is an array sector used to store user information, device ID, passwords, security keys, trim bits, configuration bits, manufacturing information, and the like.

19 shows a VMM system 1900. The VMM system 1900 is similar to the VMM system 1800, except that the VMM system 1900 further comprises a reference system 1999. The reference system 1999 comprises a reference array 1901, a reference array low voltage row decoder 1902, a reference array high voltage row decoder 1903, and a reference array low voltage column decoder 1904. The reference system can be shared across multiple VMM systems. The VMM system further comprises an NVR sector 1905.

The reference array low voltage row decoder 1902 provides bias voltages for read and programming operations with the reference array 1901 and also provides decode signals for the reference array high voltage row decoder 1903. The reference array high voltage row decoder 1903 provides high voltage biases for programming and operations with the reference array 1901. The reference array low voltage column decoder 1904 provides the decode function for the reference array 1901. The reference array 1901 is such that it provides a reference target or cell margin for program verify (searching for edge cells).

FIG. 20 illustrates a word line driver 2000. The word line driver 2000 selects a word line (such as the exemplary word lines WL0, WL1, WL2, and WL3 shown herein) and provides a bias voltage to that word line. Each word line is attached to a select transistor, such as a select iso (isolation) transistor 2002, which is controlled by a control line 2001. The iso transistor 2002 is used to isolate high voltages, such as erase (e.g., 8-12V), from the word line decode transistors, which can be implemented with IO transistors (e.g., 1.8V, 3.3V). Now, during any operation, the control line 2001 is activated, turning on all select transistors similar to selecting the iso transistors 2002. An exemplary bias transistor 2003 (part of the word line decode circuit) selectively couples the word lines to a first bias voltage (such as 3V), and an exemplary bias transistor 2004 (part of the word line decode circuit) selectively couples the word lines to a second bias voltage (a bias between the first bias voltage, including ground, and a negative voltage bias to reduce leakage from unused memory rows). During an ANN read operation, all used word lines are selected and tied to the first bias voltage. All unused word lines are tied to the second bias voltage. During other operations, such as program operations, only one word line is selected until the other word line is tied to the second bias voltage, which may be a negative bias (e.g., -0.3 to -0.5V or higher) to reduce array leakage.

21 shows a word line driver 2100. Word line driver 2100 is similar to word line driver 2000, except that the top transistors, such as bias transistor 2103, may be individually coupled to bias voltages, rather than all such transistors being tied together as in word line driver 2000. This allows all the word lines to have different independent voltages simultaneously.

Figure 22 illustrates a word line driver 2200. Word line driver 2200 is similar to word line driver 2100, except that bias transistors 2103 and 2104 are coupled to a decoder circuit 2201 and an inverter 2202. Thus, Figure 22 illustrates a decode sub-circuit 2203 within word line driver 2200.

23 shows a word line driver 2300. Word line driver 2300 is similar to word line driver 2100, except that bias transistors 2103 and 2104 are coupled to the output of stage 2302 of shift register 2301. Shift register 1301 allows for independent control of each column, including the serial shifting of data (serial clocking the register), allowing one or more rows to be enabled simultaneously in response to shifting data patterns.

FIG. 24 shows a word line driver 2400. Word line driver 2400 is similar to word line driver 2000, except that each select transistor is further coupled to a capacitor, such as capacitor 2403. Capacitor 2403 can provide a precharge or bias to the word line at the start of operation, enabled by transistor 2401 to sample the voltage on line 2440. Capacitor 2403 acts to sample and hold (S/H) the input voltage for each word line. Transistor 2401 is off during ANN operation (array current adder and activation function) of the VMM array, meaning that the voltage on the S/H capacitor acts as a (floating) voltage source for the word line. Alternatively, capacitor 2403 can be provided by the word line capacitance from the memory array.

FIG. 25 shows a word line driver 2500. Word line driver 2500 is similar to the word line drivers described above, except that bias transistors 2501 and 2502 are connected to switches 2503 and 2504, respectively. Switch 2503 receives the output of opa (operational amplifier) 2505 and switch 2504, which provides a reference input to the negative input of opa 2505, thereby essentially providing the voltage stored by capacitor 2403 due to the closed loop action provided by opa 2505, transistor 2501, switches 2503 and 2504. In this way, when switches 2503 and 2504 are closed, the voltage at input 2506 is superimposed on capacitor 2403 by transistor 2501. Alternatively, capacitor 2403 can be provided by the word line capacitance from the memory array.

Figure 26 shows a word line driver. Word line driver 2600 is similar to the word line drivers previously described, except for the addition of amplifier 2601 which acts as a voltage buffer for the voltage on capacitor 2604 to drive a voltage onto word line WL0, meaning that the voltage on the S/H capacitor acts as a (floating) voltage source for the word line. This is to avoid, for example, word line to word line coupling from affecting the capacitor voltage.

FIG. 27 illustrates a high voltage source line decoder circuit 2700. The high voltage source line decoder circuit includes transistors 2701, 2702, and 2703 configured as shown. Transistor 2703 is used to deselect the source line to a low voltage. Transistor 2702 is used to drive a high voltage to the source line of the array, and transistor 2701 is used to monitor the voltage on the source line. Transistors 2702, 2701 and driver circuitry (e.g., opa) are configured in a closed loop fashion (force/sense) to maintain PVT (process, voltage, temperature) and varying current load conditions. The SLE (driving source line node) and SLB (monitored source line node) may be at one end of the source line. Alternatively, SLE may be at one end of the source line and SLN at the other end.

FIG. 28 shows a VMM high voltage decode circuit comprising a word line decoder circuit 2801, a source line decoder circuit 2804 and a high voltage level shifter 2808 suitable for use with memory cells of the type shown in FIG.

Word line decoder circuit 2801 includes a PMOS select transistor 2802 (controlled by signal HVO_B) and an NMOS select transistor 2803 (controlled by signal HVO_B), configured as shown.

Source line decoder circuit 2804 includes an NMOS monitor transistor 2805 (controlled by signal HVO), a drive transistor 2806 (controlled by signal HVO), and a transistor 2807 (controlled by signal HVO_B) configured as shown.

A high voltage level shifter 2808 receives a valid signal EN and outputs a high voltage signal HV and its complement HVO_B.

FIG. 29 shows a VMM high voltage decode circuit comprising an erase gate decoder circuit 2901, a control gate decoder circuit 2904, a source line decoder circuit 2907 and a high voltage level shifter 2911 suitable for use with memory cells of the type shown in FIG.

The erase gate decoder circuit 2901 and the control gate decoder circuit 2904 use the same design as the word line decoder circuit 2801 of FIG.

The source line decoder circuit 2907 uses the same design as the source line decoder circuit 2804 of FIG.

High voltage level shifter 2911 uses the same design as high voltage level shifter 2808 in FIG.

Figure 30 shows a word line decoder 300 for exemplary word lines WL0, WL1, WL2, and WL3. Exemplary word line WL0 is coupled to pull-up transistor 3001 and pull-down transistor 3002. When pull-up transistor 3001 is activated, WL0 is enabled. When pull-down transistor 3002 is activated, WL0 is disabled. The function of Figure 30 is similar to Figure 21 without the isolation transistors.

31 shows a control gate decoder 3100 for exemplary control gate lines CG0, CG1, CG2, and CG3. Exemplary control gate line CG0 is coupled to pull-up transistor 3101 and pull-down transistor 3102. When pull-up transistor 3101 is activated, CG0 is enabled. When pull-down transistor 3102 is activated, CG0 is disabled. The select and deselect functions of FIG. 31 are similar to the select and select functions of FIG. 30 for the control gates.

32 shows a control gate decoder 3200 for exemplary control gate lines CG0, CG1, CG2, and CG3. Control gate decoder 3200 is similar to control gate decoder 3100, except that control gate decoder 3200 includes capacitors, such as capacitor 3203, coupled to each control gate line. These sample and hold (S/H) capacitors can provide a precharge bias to each control gate line prior to operation, meaning that the voltage on the S/H capacitor acts as a (floating) voltage source for the control gate line. The S/H capacitors can be provided by control gate capacitance from a memory cell.

33 illustrates an exemplary control gate decoder 3300 for control gate lines CG0, CG1, CG2, and CG3. Control gate decoder 3300 is similar to control gate decoder 3200, except that control gate decoder 3300 further comprises a buffer 3301 (such as an opa).

34 shows a current-to-voltage circuit 3400. The circuit comprises a diode-connected reference cell circuit 3450 and a sample-and-hold circuit 3460 configured. The circuit 3450 comprises an input current source 3401, an NMOS transistor 3402, a cascoding bias transistor 3403, and a reference memory cell 3404. The sample-and-hold circuit consists of a switch 3405 and an S/H capacitor 3406. The memory 3404 is biased in a diode-connected configuration with a bias on the bit line to convert an input current to a voltage, such as to supply a word line.

35 shows a current-to-voltage circuit 3500 which is similar to current-to-voltage circuit 3400, but with the addition of an amplifier 3501 after the S/H capacitor. The current-to-voltage circuit 3500 includes a configured diode-connected reference cell circuit 3550, a sample and hold circuit 3470, and an amplifier stage 3562.

36 shows a current-voltage circuit 3600, which is of the same design as the current-voltage circuit 3400 of the control gate in a diode-connected configuration. The current-voltage circuit 3600 comprises a diode-connected reference cell circuit 3650 and a sample-and-hold circuit 3660 configured.

FIG. 37 shows a current-voltage circuit 3700 in which a buffer 3790 is placed between a reference circuit 3750 and an S/H circuit 3760 .

Figure 38 shows a current-voltage circuit 3800 similar to Figure 35 with the control gates connected in a diode-connected configuration. The current-voltage circuit 3800 includes a diode-connected reference cell circuit 3550 configured, a sample and hold circuit 3870, and an amplifier stage 3862.

Figure 39 shows a current-voltage circuit 3900 similar to Figure 34 applied to the memory cell of Figure 2. The current-voltage circuit 3900 comprises a diode-connected reference cell circuit 3950 and a sample-and-hold circuit 3960 configured.

FIG. 40 shows a current-voltage circuit 4000 similar to FIG. 37 applied to the memory cell of FIG. 2, in which a buffer 4090 is placed between a reference circuit 4050 and an S/H circuit 4060.

Figure 41 shows a current-voltage circuit 4100 similar to Figure 38 applied to the memory cell of Figure 2. The current-voltage circuit 4100 comprises a diode-connected reference cell circuit 4150 configured, a sample-and-hold circuit 4170, and an amplifier stage 4162.

It should be noted that, as used herein, both the terms "over" and "on" are inclusive of "directly" (without any intermediate material, element, or space disposed between them) and "indirectly on" (with an intermediate material, element, or space disposed between them). Similarly, the term "adjacent" includes "directly adjacent" (without any intermediate material, element, or space disposed between them) and "indirectly adjacent" (with an intermediate material, element, or space disposed between them), "attached" includes "directly attached" (without any intermediate material, element, or space disposed between them) and "indirectly attached to" (with an intermediate material, element, or space disposed between them), and "electrically coupled" includes "directly electrically coupled" (without any intermediate material, element, or space disposed between them) and "indirectly electrically coupled to" (with an intermediate material, element, or space disposed between them) that electrically couples the elements together. For example, forming an element "above a substrate" can include forming the element directly on the substrate, without any intermediate materials/elements between them, and forming the element indirectly on the substrate, with one or more intermediate materials/elements between them.