Neural network circuits providing early integration before analog-to-digital conversion

Embodiments of the invention relate to neural network circuits, and more particularly to circuits adapted to provide initial integration prior to analog-to-digital conversion.

According to embodiments of the invention, a neural network circuit is provided. A first synapsis array is configured to output a plurality of analog voltages. A second synapsis array has a plurality of input wires. A network is operatively coupled to the first synapsis array and the second synapsis array. At least one comparator is operatively coupled to the first synapsis array. The at least one comparator compares the plurality of analog voltages to a predetermined threshold to generate a vector of bits with each bit indicating whether one of the plurality of analog voltages exceeds a predetermined threshold. , and is adapted to transmit a vector of bits over a network. At least one analog-to-digital converter is operatively coupled to the first synapsis array and the network. At least one analog-to-digital converter is configured to convert the plurality of analog voltages to a vector of digital values and to transmit the vector of digital values over the network. At least one modulator is operatively coupled to the second synapsis array and network. The at least one modulator is configured to receive a vector of bits from the network and process the vector of bits when a corresponding bit of the vector of bits indicates that a predetermined threshold has been exceeded. , processing the vector of bits, receiving the vector of digital values from the network, and processing the vector of digital values, including pulsing each of the plurality of input wires of the second synapsis array. a vector of digital values, the method comprising: providing a pulse to each of the plurality of input wires of the second synapsis array, each pulse having a duration proportional to a corresponding digital value of the vector of digital values; is configured to perform the following processing. Receiving the vector of bits and/or processing the vector of bits occurs in parallel with converting the plurality of analog voltages and/or transmitting the vector of digital values.

According to embodiments of the invention, a method of operating a neural network circuit and a computer program product for operating a neural network circuit are provided. A plurality of analog voltages output from the first synapsis array are compared to predetermined thresholds to generate a vector of bits. Each bit of the vector of bits indicates whether one of the plurality of analog voltages exceeds a predetermined threshold. A vector of bits is processed. Processing the vector of bits provides a pulse to each of the plurality of input wires of the second synapsis array when a corresponding bit of the vector of bits indicates that a predetermined threshold has been exceeded. Including. In parallel to processing the vector of bits, the plurality of analog voltages are converted to a vector of digital values. Following processing the vector of bits, each of the plurality of input wires of the second synapsis array is provided with a pulse each having a duration proportional to a corresponding digital value of the vector of digital values. Ru.

<figref num="1">FIG. 2 illustrates an example non-volatile memory-based crossbar array, or crossbar memory, according to embodiments of the present invention.</figref><figref num="2">FIG. 3 is a diagram illustrating an example synapsis within a neural network according to an embodiment of the invention.</figref><figref num="3">FIG. 3 illustrates an exemplary array of neural cores according to embodiments of the invention.</figref><figref num="4">FIG. 2 illustrates an example neural network according to an embodiment of the invention.</figref><figref num="5">1 illustrates an exemplary multi-core neural network according to embodiments of the invention; FIG.</figref><figref num="6A">FIG. 3 is a timeline diagram illustrating an example integration according to an embodiment of the invention.</figref><figref num="6B">FIG. 3 is a timeline diagram illustrating an example integration according to an embodiment of the invention.</figref><figref num="7">FIG. 3 illustrates a method of operating a neural network according to an embodiment of the invention.</figref><figref num="8">FIG. 2 illustrates a computing node according to an embodiment of the invention.</figref>

An artificial neural network (ANN) is a distributed computing system consisting of a number of neurons connected to each other through connection points called synapses. Each synapsis encodes the strength of the connection between the output of one neuron and the input of another. The output of each neuron is determined by the aggregate inputs received from other neurons connected to that neuron. Therefore, the output of a given neuron is based on the outputs of connected neurons from previous layers and the strength of the connections determined by the synaptic weights. ANNs are trained to solve a particular problem (eg, pattern recognition) by adjusting synaptic weights such that a particular class of inputs yields a desired output.

ANNs may be implemented on various types of hardware, including crossbar arrays, also known as crosspoint arrays or crosswire arrays. A basic crossbar array configuration includes a set of conductive row wires and a set of conductive column wires formed to intersect the set of conductive row wires. Crossovers between two sets of wires are separated by crosspoint devices. Crosspoint devices serve as weighted connections between neurons in the ANN.

In various embodiments, a non-volatile memory based crossbar array or crossbar memory is provided. A plurality of junctions are formed by row lines intersecting column lines. A resistive memory element, such as a non-volatile memory, is in series with a selector at each junction coupling between one of the column lines and one of the row lines. The selector may be a volatile switch or transistor of various types known in the art. A variety of resistive memory elements may be used as described herein, including memristors, phase change memories, conductive-bridging RAM, and spin-transfer torque RAM. It will be recognized that it is suitable for

A fixed number of synapses may be provided on a core, and then multiple cores may be connected to yield a complete neural network. In such embodiments, interconnections between cores are provided to convey the output of neurons on one core to another core, eg, via a packet-switched or circuit-switched network. In packet-switched networks, greater flexibility of interconnection pays for the power and speed costs due to the need to transmit, read, and act upon address bits. It may be realized. In circuit switched networks, address bits are not required, so flexibility and ease of reconfiguration must be achieved through other means.

In various exemplary networks, multiple cores are arranged in an array on a chip. In such embodiments, the relative positions of the cores may be referenced by four directions (north, south, east, west). The data carried by the neural signals may be encoded into the pulse duration carried by each wire using digital voltage levels suitable for buffering or other forms of digital signal restoration.

One approach to routing is to use a digital network-on-chip to rapidly route packets to any other core, and the output of each core paired with a digital-to-analog converter at the input edge of each core. Our goal is to provide analog-to-digital converters at the edge.

In a variety of computing applications, including the implementation of deep neural networks (DNNs) using analog resistive memory elements, the data is used to drive pulse-modulated data at the edge of the next array. There is often a need to perform analog-to-digital conversion (ADC) at the bottom edge of one array prior to transmitting data. In such implementations, the large size of each ADC leads to the need to time multiplex the use of shared ADC resources. This, in turn, significantly expands the total operating time required to complete the final time division multiplexed use of each shared ADC resource. Since ADC circuits are area inefficient, shared ADC resources are used to perform ADC operations, transmit data, and prepare new data to modulate input pulses in the next array. and waiting for the longest possible pulse in the next array takes a considerable amount of time.

There remains a need in the art for a method to more quickly perform pulse modulated integration in subsequent arrays. Accordingly, various embodiments of the present invention enable transmission of the most significant bit (MSB) for initial integration prior to ADC (Analog-to-Digital Conversion) for neural network computing.

Referring to FIG. 1, an exemplary non-volatile memory-based crossbar array, or crossbar memory, is shown. A plurality of junctions 101 are formed by row lines 102 intersecting column lines 103. A resistive memory element 104, such as a non-volatile memory, is in series with a selector 105 at each junction 101 coupling between one of the row lines 102 and one of the column lines 103. The selector may be a volatile switch or transistor of various types known in the art.

It will be appreciated that a variety of resistive memory elements are suitable for use as described herein, including memristors, phase change memories, conductive bridging RAMs, and spin injection RAMs.

Referring to FIG. 2, an example synapsis within a neural network is shown. Multiple inputs x from node 201₁...x_nThe weight corresponding to w_ijis multiplied. weight Σx_iw_ijThe sum of the values is<math num="1"><img file="JP7398552B2_D0001.tif" /></math>is given to the function f(·) at node 202 to obtain . It will be appreciated that neural networks include multiple such connections between layers, and that this is exemplary only.

Referring now to FIG. 3, an exemplary array of neural cores according to an embodiment of the present invention is shown. Array 300 includes multiple cores 301. The cores in array 300 are connected to each other by lines 302, as described further below. In this example, the array is two-dimensional. However, it will be appreciated that the invention may be applied to one-dimensional or three-dimensional arrays of cores. Core 301 includes a non-volatile memory array 311 that implements synapsis as described above. Core 301 includes a west side and a south side, each of which may serve as an input while the other serves as an output. It will be appreciated that the west/south nomenclature is employed merely to facilitate relative positional reference and is not intended to limit the direction of input and output.

In various exemplary embodiments, the west side includes support circuitry 312 that is dedicated to that entire side of the core 301, shared circuitry 313 that is dedicated to a subset of rows, and row-by-row dedicated to individual rows. circuit 314. In various embodiments, the south side similarly includes support circuitry 315 that is dedicated to that entire side of core 301, shared circuitry 316 that is dedicated to a subset of columns, and per-column circuitry that is dedicated to individual columns. circuit 317.

Referring to FIG. 4, an exemplary neural network is shown. In this example, multiple input nodes 401 are connected to each other with multiple intermediate nodes 402. Intermediate node 402 is then interconnected with output node 403. It will be appreciated that this simple feedforward network is presented for illustrative purposes only, and that the invention is applicable regardless of the particular neural network configuration.

Referring to FIG. 5, an example multi-core neural network is shown. In this example, two cores are shown, which may be implemented as described in the previous paragraph. Input row 501 (which may correspond to input node 401, for example) provides an output voltage (V_N=Σx_iw_ij) is passed through the crossbar array 510 to provide the A transfer function is applied resulting in an activation y that is applied to one of the input rows 504. In this example, y=f(V_N), where f is an arbitrary transfer function. Regardless of the transfer function chosen, one comparator is required with the analog voltage inducing exactly 1/2 of the complete integration time, as indicated by threshold 503.

In embodiments of the invention, a single bit per neuron is transferred from a first synapsis array (e.g., crossbar 510) to the next synapsis array (e.g., crossbar 520) within a given neural network. sent to. If this bit is 1 (high) then a pulse of 1/2 maximum duration is immediately modulated onto the next array. This integration performs precise ADC operations, transmits N bits per neuron from one array to the next, and in parallel with the process of performing any necessary squashing operations in the next array. It is possible to do so.

In various embodiments, the analog comparator threshold used to calculate the first transmitted MSB is such that this level is just below and just above 1/2 of the largest modulated pulse produced. designed to accurately accommodate transitions between Therefore, the idea is independent of the squashing function applied to the N bits of data. When N bits of data per neuron are finally transmitted and squashed, the second array only needs to apply the lowest N-1 bits to modulate the pulse. The same exact total pulse modulated duration is applied to the second array, but overall about 1/2 of the maximum duration is saved by the earlier efficient transmission of the MSB information.

Referring to FIG. 6, a timeline for integration according to an embodiment of the invention is provided. In both FIG. 6A and FIG. 6B, the time series runs from T=0 to T=t. At T=0, any previous integration is finished leaving the capacitor charged and V_Nso that the final value of is ready for ADC conversion.

FIG. 6A shows the operation of a set of shared ADC resources without initial integration. A first ADC conversion of N bits/column is performed at 601 by the source core. The results are transmitted and/or crushed to subsequent cores at 602. At this stage, the transmitted value corresponds to N bits/ready string. At 603, these first received N bits/sequences are prepared for integration at the destination core. Next, at 604, integration of the first available excitation may begin. The process is repeated for all columns as they become ready. Therefore, a final ADC conversion is performed at 605. These last bits are transmitted and/or crushed at 606. The associated received bits are prepared at the destination core at 607 to enable integration of the final excitation at 608. Integration of all received excitations is completed at T=t.

FIG. 6B illustrates the operation of a set of shared ADC resources with initial integration according to an embodiment of the invention. In this case, the most significant bit is measured at 611 and then transmitted at 612 for each column. Next, integration of the first half of all excitations may proceed at 613 on the destination core. In parallel with the integration at the destination core, an initial ADC conversion of N bits/column is performed at 614 by the source core. The results are sent and/or crushed to the destination core at 615. At this stage, the transmitted value corresponds to N bits/ready string. At 616, the received N bits/sequences are prepared at the destination core. Integration of the first available excitation beyond the most significant bit may then begin at 617. The process is repeated for all columns as they become ready. Therefore, a final ADC conversion is performed at 618. The resulting bits are transmitted and/or squashed at 619. The received bits are prepared at the destination core at 620 to enable integration of the final excitation at 621. The initial integration of the most significant bits (at 613) allows the remaining integrations (at 617 and 621) to be completed well before T=t. The overall time savings is shown at 622.

Referring to FIG. 7, a method of operating a neural network according to an embodiment of the invention is illustrated. At 701, a plurality of analog voltages output from a first synapsis array are compared to a predetermined threshold to generate a vector of bits. Each bit of the vector indicates whether one of the plurality of analog voltages exceeds a predetermined threshold. At 702, the vector of bits is processed. Processing the vector of bits provides a pulse to each of the plurality of input wires of the second synapsis array when a corresponding bit of the vector of bits indicates that a predetermined threshold has been exceeded. Including. At 703, in parallel with processing the vector of bits, the plurality of analog voltages are converted to a vector of digital values. At 704, subsequent to processing the vector of bits, a pulse is applied to each of the plurality of input wires of the second synapsis array, each having a duration proportional to a corresponding digital value of the vector of digital values. , supplied.

In various embodiments, a set of circuits is provided for converting a vector of analog voltages at the edges of a first array to a vector of durations at the edges of a second array. A comparator compares each analog voltage against a common analog threshold, resulting in a most significant amplitude bit for each analog voltage, resulting in a vector of most significant amplitude bits. A transmission system conveys a vector of most significant amplitude bits from an edge of the first array to an edge of the second array. A circuit injects a vector of constant duration pulse signals into the second array in view of the value of the most significant amplitude bit. A plurality of A/D converters convert the analog voltages at the edges of the first array into a vector of high resolution residual digital data representations containing a vector of previously calculated most significant amplitude bits. A transmission system conveys a vector of high resolution residual digital data from an edge of the first array to an edge of the second array. A circuit introduces a vector of variable duration pulse signals in view of the values of the high resolution residual digital data vector. The operations performed on the most significant amplitude bits (comparing, conveying the vector of most significant amplitude bits, and constant duration signal injection into the second array) are the same as the operations of the A/D converter (conversion in parallel with transmitting a vector of high-resolution residual digital data, and varying duration signal injection into the second array.

In some embodiments, the analog voltage represents a signed quantity with the most significant amplitude bit turned on only for positive analog voltages that exceed a common threshold. In some embodiments, the analog voltage represents a signed quantity and two comparators (each with a common analog threshold) are used.

In some embodiments, the analog voltage is stored on a capacitor.

Referring now to FIG. 8, a schematic diagram of an embodiment of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. In any case, computing node 10 may be implemented as or perform any of the functions indicated above, or a combination thereof.

At computing node 10 there is a computer system/server 12 that is operable in a number of other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments and/or configurations that may be suitable for use with computer system/server 12 include personal computer systems, server computer systems, clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe including, but not limited to, computer systems and distributed cloud computing environments including any of the aforementioned systems or devices, and the like.

Computer system/server 12 may be described in the general context of computer system executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system/server 12 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in FIG. 8, computer system/server 12 in computing node 10 is shown in the form of a general purpose computing device. Components of computer system/server 12 include one or more processors or processing units 16, system memory 28, and bus 18 that couples various system components, including system memory 28, to processor 16. Good, but not limited to these.

Bus 18 includes several busses, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor buses or local buses, using any of a variety of bus architectures. Represents one or more of any of the type bus structures. By way of example, and not as a limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association ( VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe) bus, and Advanced Microcontroller Bus Architecture (AMBA).

Computer system/server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system/server 12 and includes both volatile and nonvolatile media, removable and non-removable media. Including both media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32. Computer system/server 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. Merely by way of example, storage system 34 may read from and write to non-removable, non-volatile magnetic media (not shown and commonly referred to as a "hard drive"). It is possible to be provided. Although not shown, magnetic disk drives for reading from and writing to removable, nonvolatile magnetic disks (e.g., "floppy disks"), as well as CD-ROMs, DVDs, etc. - An optical disk drive may be provided for reading from and writing to removable, non-volatile optical disks such as ROM or other optical media. In such cases, each medium may be connected to bus 18 by one or more data medium interfaces. As further shown and explained below, memory 28 includes at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention. That's fine.

By way of example, and not as a limitation, a program/utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data. may be stored in memory 28. An operating system, one or more application programs, other program modules, and/or program data may each include an implementation of a networking environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

Computer system/server 12 may also include a keyboard, pointing device, display 24, etc., or one or more devices that allow a user to interact with computer system/server 12. / communicate with one or more external devices 14, such as any device (e.g., network card, modem, etc.) that enables server 12 to communicate with one or more other computing devices; You can. Such communication may occur via an input/output (I/O) interface 22. Additionally, computer system/server 12 may be connected to one or more networks, such as a local area network (LAN), a general purpose wide area network (WAN), and/or a public network (e.g., the Internet). , can communicate via network adapter 20. As shown, network adapter 20 communicates with other components of computer system/server 12 via bus 18. Although not shown, it should be understood that other hardware or software components, or combinations thereof, may also be used in conjunction with computer system/server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems, among others.

The invention may be implemented as a system, method, and/or computer program product. A computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the invention.

A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Not limited. A non-exhaustive list of more specific examples of computer readable storage media is as follows: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable Read-only memory (EPROM or flash memory), Static Random Access Memory (SRAM), Portable Compact Disk Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Floppy - Mechanically encoded devices such as disks, punched cards or raised structures in grooves on which instructions are recorded, and any suitable combinations of the above. As used herein, a computer-readable storage medium includes radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light passing through a fiber optic cable), It should not be interpreted as a temporary signal itself, such as an electrical signal transmitted via wiring.

The computer readable program instructions described herein may be transferred from a computer readable storage medium to a respective computing/processing device or over a network, such as the Internet, a local area network, a wide area network, or a wireless network. Alternatively, it may be downloaded to an external computer or external storage device via a combination thereof. The network may include copper transmission cables, transmission fiber optics, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or combinations thereof. A computer readable program such that a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and is stored on a computer readable storage medium within the respective computing/processing device. Transfer instructions.

Computer-readable program instructions for carrying out the operations of the present invention may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state configuration data, or Smalltalk(R) instructions. written in any combination of one or more programming languages, including object-oriented programming languages such as C++ or similar, and traditional procedural programming languages such as the "C" programming language or similar programming languages; It may be source code or object code. Computer-readable program instructions may be executed in whole or in part on your computer, or as a standalone software package, or in part on your computer. , and may be executed partially or entirely on a remote computer or server. In a scenario that runs entirely on a remote computer or server, the remote computer is a computer that connects to the user's computer over any type of network, including a local area network (LAN) or a wide area network (WAN). or the connection may be made to an external computer (eg, over the Internet using an Internet service provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), performs aspects of the invention. Computer readable program instructions may be executed by utilizing state information of the computer readable program instructions to customize electronic circuitry.

Aspects of the invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart diagrams and/or block diagrams, as well as combinations of blocks in the flowchart diagrams and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions are implemented by a processor of the computer or other programmable data processing device such that those instructions are specified in one or more blocks of flowcharts and/or block diagrams. / may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to create a machine to create the means for performing the operations. These computer-readable program instructions also include instructions for a computer-readable storage medium on which the instructions are stored to perform aspects of the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams. The information may be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner to provide an article of manufacture containing the information.

Computer-readable program instructions also indicate that instructions executed on a computer, other programmable apparatus, or other device perform the functions/operations specified in one or more blocks of a flowchart and/or block diagram. a sequence of operations loaded onto a computer, other programmable data processing equipment, or other device, and performed on the computer, other programmable equipment, or other device to produce a computer-implemented process to It may also be something that causes a step to be executed.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. . In addition, each block in the block diagrams and/or flowchart diagrams, and combinations of blocks in the block diagrams and/or flowchart diagrams, perform a designated function or operation, or implement a combination of dedicated hardware and computer instructions. Note also that it can be implemented by a dedicated hardware-based system.

The descriptions of various embodiments of the invention have been presented for purposes of illustration and are not intended to be exhaustive or limited to the disclosed embodiments. Many variations and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is used to best explain the principles of the embodiments, a particular application, or technical improvements over technology found in the marketplace, or to explain the embodiments disclosed herein in other words. have been chosen to enable one of ordinary skill in the art to understand them.