Distributed storage system, distributed storage method and distributed facility

Distributed multi-mode storage management

DISTRIBUTED MULTIMODE STORAGE MANAGEMENT

The present invention relates to information storage.

Numerous electronic technologies (for example, digital computers, calculators, audio devices, video equipment, telephone systems, and/or the like) have increased productivity and reduced costs in most areas of business, science, education, and entertainment. These electronic systems perform operations involving information storage systems. The speed and ease of information storage operations can have a significant impact on the overall performance of the information storage system. However, attempts to store conventional information usually involve an inverse relationship between speed and manageable complexity.

Information storage systems involve operations that can belong to one of two categories. One category relates to storage operations associated with user-initiated activities. The other category involves management and maintenance activities initiated by the system. The speed and ease of these operations are related to the type of address space used to store information.

The traditional attempt to use the physical addressing space is theoretically regarded as operating at a very fast speed, but the actual management and maintenance operations in the conventional physical addressing space are extremely complicated and have not been implemented in practice. The management and maintenance of the conventional logical address space is considered to involve lower complexity than the physical address space. However, the conventional logical address space operation is not as fast as the physical address space.

Although conventional storage systems can operate at a level that may have previously been considered acceptable, these storage systems are increasingly insufficient to meet the requirements and long-term needs for improved applications and platforms. Conventional attempts to achieve both increased speed and manageable complexity to enable improved system development have not been successful.

The description can include efficient and effective multi-mode storage devices that implement multiple different types of address spaces for different storage space activities. The multi-mode selective basic exposure storage device can realize the selective exposure of the underlying aspect of the storage device. In one embodiment, the distributed storage system includes: a plurality of appliances, a distributed multi-mode storage management coordinator, and a communication mechanism that transfers messages between multiple multi-mode storage management systems and includes distributed multi-mode storage management information . The first of the multiple appliances may include: multiple storage devices (Solid-State Drive (SSD)), having a first storage partition and a second storage partition, the first storage partition including the first A type of interface and a first information storage area configured to store the first type of information. The second storage partition includes a selective underlying exposure (SUE) interface and a second information storage area that stores the second type of information. Area, where the SUE interface exposes the characteristics of the second information storage area. The distributed multi-mode storage management coordinator can include multiple multi-mode storage management systems that guide the transfer of information to multiple storage devices, where the multiple multi-mode storage management systems include selected basic characteristics via the SUE interface and the second partition. A multi-mode selective basic exposure (SUE) management system that guides archive activities in the second partition.

Messages can expose selected basic traits across multiple devices. The metadata storage space and the user data storage space can be uniformly expanded across multiple appliances. The SUE mapping is maintained across multiple systems and the representative geometric configuration of the SUE address space mapping matches the representative geometric configuration of the basic physical address space. Redundancy can be controlled at the level of the multi-mode storage management system. Multiple appliances are arranged in the framework, and the components of the framework can be linearly adjusted across multiple systems. The distributed multi-mode storage management coordinator operates across multiple storage devices based on selective basic exposure levels. The failure of a single unit will not shut down the distributed storage system, even if the node includes more than one storage device. Free space accounting can occur at the level of selected basic trait blocks.

In one embodiment, the distributed storage method includes: setting a plurality of appliances in a redundant array of independent disks (RAID) configuration, wherein at least one of the plurality of appliances includes a plurality of storage devices; and Managing information storage in multiple storage devices includes mapping the address space of the first type to the SUE address space corresponding to one of the characteristics of the basic storage device. The management may include coordinated management of information storage in multiple storage devices across a RAID configuration. The mapping data can be logically distributed redundantly across multiple storage devices. The coordination includes transmitting a message between a first one of the plurality of appliances and a second one of the plurality of appliances. Free space account processing can occur at the level of the multi-mode storage management system. Reclamation is managed by the main block owner who triggers distributed block selection and the selected block is distributed to another block. The non-primary block owner performs entity scanning and effective judgment search.

In an exemplary implementation, a distributed facility includes: multiple appliances that store information in stripes; a distributed multi-mode storage management coordinator; and a communication mechanism in a distributed multi-mode storage management coordinator system Send messages to the second of multiple appliances. The first of the plurality of appliances includes: a plurality of storage devices having a first storage partition and a second storage partition, the first storage partition including a first type of interface and a device configured to store the first type of information The first information storage area and the second storage partition include an SUE interface and a second information storage area storing a second type of information, wherein the SUE interface exposes the characteristics of the second information storage area. The distributed multi-mode storage management coordinator can include multiple multi-mode storage management systems that guide the transmission of information to multiple storage devices, where the distributed multi-mode storage management coordinator includes a multi-mode basic exposure management system and a multi-mode basic exposure management The system guides the file activities of the second partition through the SUE interface and the selected basic characteristics of the second partition. Distributed facilities may further include physically striping metadata and user data across multiple systems. Redundancy can be controlled at the level of the distributed multi-mode storage management coordinator at a higher level than the storage device. The failure unit is a storage node and the failure of a single node will not shut down the distributed facility, where the storage node includes one of multiple appliances.

All the above-mentioned features and the following description are provided to assist the understanding of this disclosure. That is, the content of the disclosure should not be interpreted as being limited to the "example" embodiments described in this article, and may be embodied in different forms. The following embodiments are only examples for fully exposing the present invention, and are described to provide the concept of the present invention to those skilled in the art. Therefore, if there are many ways to implement the components of the present disclosure, it should be understood that the present disclosure can be implemented by any of the methods or the like.

If it is mentioned in this disclosure that any configuration includes/includes a specific component or any itinerary includes/includes a specific operation, this situation means that it may further include other components or other operations. That is, the terms used herein are only intended to describe specific example embodiments, and are not intended to limit the scope of the present disclosure. In addition, the described examples provided to assist the understanding of the present disclosure include supplementary embodiments thereof.

The terms used herein have meanings that those familiar with the art will generally understand. Common terms should be interpreted in the context of this disclosure to have the same meaning. In addition, the terms used in this article should not be interpreted in an excessively ideal or formal sense, unless explicitly defined as such in this article.

Describe efficient and effective multi-mode storage methods that can use multiple different types of address spaces and address space activities. In some embodiments, the multi-mode selective underlying exposure (SUE) storage device realizes selective exposure of some basic characteristics of the storage device without exposing other basic characteristics. Multi-mode storage and SUE methods can improve performance while limiting complexity to manageable categories.

In some exemplary implementations, the basic characteristics of the physical address space can be selectively exposed. The entire storage hierarchy method can be implemented, and the basic characteristics of one hierarchy can be selectively exposed to another hierarchy. Selective exposure can occur through address space configuration and mapping between address spaces. The selectively exposed basic traits can more efficiently and effectively implement various activities that are performed at a hierarchy level different from the hierarchical level with the exposed basic traits. Various activities can include storage management operations. It should be understood that the multi-mode storage and SUE method may include multiple configurations and implementations.

Hereinafter, example embodiments of the present disclosure will be described below with reference to the accompanying drawings.

I. Multi-mode storage device

FIG. 1 is a block diagram illustrating an exemplary storage device 100 including a selective base exposure (SUE) storage partition 101 according to an embodiment. The SUE storage partition 101 may include a selective basic exposure (SUE) interface 102 and a basic storage area 103.

The basic storage area 103 can store information, and the SUE interface 102 can selectively combine the characteristics of the basic storage area 103 itself (for example, characteristics, features, functions, or the like) (for example, related to size, representative geometric structure, management function) , Write operations, erase operations, or the like related to physical characteristics) are exposed to external components or storage system hierarchy (not shown in the figure). Exposure may be associated with the characteristics of the information (user data and metadata) stored in the basic storage area 103. The SUE storage partition 101 may expose a part of the basic characteristics (for example, characteristics, features, functions, or the like).

In an exemplary implementation that exposes a part of the basic traits, compared to a system that does not selectively expose a part of the basic traits, it can perform more efficiently (for example, faster, less bandwidth, and less power consumption). Activities associated with the exposed characteristics (for example, free space management, recovery and conditioning for free space use, over provisioning, trim operations, power cycling, or the like). Activities can be performed with less complexity than methods that expose more or all basic traits.

In some embodiments, the choice of which part of the basic trait is exposed may be determined based on comparison or a balance of speed and complexity. It should be understood that the SUE storage partition 101 may be included in a single-mode storage device with a single partition, or the SUE storage partition 101 may be included in a multi-mode storage device with multiple partitions.

FIG. 2 is a block diagram illustrating an exemplary multi-mode storage device 220 according to one embodiment. The storage device 220 may include a first partition 230 and a second partition 240.

It should be understood that multiple modes and corresponding partitions can be associated with or based on multiple things. Various things can include different exposures of the basic storage area, different address spaces (for example, logical, virtual, or physical), different storage management modes (for example, internal management and external management), and different basic storage information (for example, metadata and usage Information) and so on. Internal management and external management may include storage device management system components and operations (eg, flash management system (FMS), solid-state device management system, or the like). The partitions and corresponding components can also be of different types.

The partitions and corresponding interfaces in the multi-mode storage device 220 may be associated with different types of address spaces (for example, logical address spaces and selective base exposure (SUE) address spaces). More than one partition and corresponding interface in the multi-mode storage device 220 can also be associated with the same type of address space (for example, more than one partition and the corresponding interface in the multi-mode storage device 220 can be associated with the SUE address space Associated).

The first partition 230 may include a first type of interface 231 and a basic storage area 233. The second partition 240 may include a second type of interface 241 and a basic storage area 243. In some embodiments, the first partition 230 may be a first type of address space partition (for example, a logical address space partition), and the second partition 240 may be a second type of address space partition (For example, SUE address space partition and virtual address space partition). It should be understood that the partition can be the SUE storage partition.

FIG. 3 is a block diagram illustrating another exemplary multi-mode storage device 350 according to an embodiment. The storage device 350 may include a first partition 370 and a second partition 380.

In some embodiments, the first partition 370 may be a first type of address space partition, and the second partition 380 may be an SUE address space partition. The first partition area 370 may include a first type interface 371 and a basic storage area 373. The second partition area 380 may include the SUE interface 381 and the basic storage area 383. It should be understood that some activities such as the first partition-related activities 372 (for example, FMS) can be performed internally for one partition (for example, in the storage device 350), and may be performed externally for another partition (in the figure) Not shown) implementation.

Different types of information can be stored in different partitions. In some embodiments, there may be two types of information, for example, metadata and user data. User data can mainly be generated by user applications, and metadata can mainly be auxiliary information associated with user data (for example, the location of the file in the storage system hierarchy, the size of the content in the file, access time, modification Time, user identity (ID) or the like). The first flash memory management system can focus on managing metadata. Metadata can be used to manage the storage of user data.

It should be understood that the storage system can guide or implement operations associated with user-initiated activities in a different way than system operations associated with management or maintenance activities. For example, from the user's point of view, user-initiated reads or writes can be directed to a specific address or location, and from the system's point of view, system operations can be directed to physical blocks and page.

It should also be understood that the storage device 350 may include a variety of configurations and implementations. In some embodiments, the storage device 350 may be a solid-state device. The storage device 350 may include a flash memory device (for example, a NAND type flash memory device, a NOR type flash memory device, or the like).

FIG. 4 illustrates a block diagram of an exemplary multimode solid state drive (MM-SSD) 400 according to one embodiment. The multi-mode solid state drive (SSD) 400 may be an exemplary implementation of a multi-mode storage device. The multi-mode solid state drive 400 can include a logical address space partition 410, a logical interface 411 that can include flash translation logic (FTL) 413, a basic physical address space 412, and a SUE address space partition Area 420, SUE interface 421, and basic physical address space 423.

The logical address space partition 410 can receive and store logically addressed system data (e.g., metadata), and the SUE address space partition 420 can receive user data addressed according to the underlying exposed address space (e.g., application Program data). User data can be stored in a basic physical address space 423 that can include flash memory storage components (for example, different types of floating thyristors). The flash memory storage components can be configured into a variety of configurations and granularities. For example, the flash memory storage device may be configured as a plurality of dies, and the die 470 among the plurality of dies may include a block 473, a block 479, and pages within the block.

In some embodiments, the SUE interface 421 may expose the characteristics of the underlying physical address space 423. The selective nature of the basic physical address space 423 can be exposed by coordinating the user data addressing and the basic operations of the MM-SSD 400. Coordination may be associated with the exposure of the management operation of the address space 423 of the underlying entity. Basic entity storage management characteristics can include multiple basic entity address blocks that are managed together (for example, in a single operation, as a single management unit, in a block set, in a frequency band, and in response to a single management command) (For example, 471, 472, 473, and 474) groups.

FIG. 5 is a block diagram illustrating an exemplary process of translating address space information into logical address space information according to an embodiment. The SUE address block 503 may include information associated with various management and maintenance operations (for example, 505, 507, and 508). The physical address space 502 may include multiple dies (511, 512, 513, 514, 521, 522, 523, 524, 531, 532, 533, 534, 541, 542, 543, and 544). Each die may include multiple physical address blocks (for example, 515 and 519), and each physical address block may include multiple physical address pages.

The physical address space 502 can access address storage locations based on physical blocks and physical pages. The SUE interface 501 can receive the SUE address block 503 information, and can translate or reconfigure the received information into a configuration compatible with the physical address space 502. The SUE address block 503 information may correspond to the information involved in the entity management operation.

In some embodiments, management and maintenance operations can be directed to physical blocks in the physical address space 502 (for example, physical blocks 515, 519, and 539). The management operation can be directed to the physical address space or the physical level management unit. The entity-level management unit may include management of multiple addresses, pages, blocks, etc. that are substantially simultaneously managed (for example, in response to management operations or commands). For example, the erase operation can be directed to a physical block from each die (similar to block 515 shown in black).

Since the SUE address block 503 is configured to match the physical block, each piece of information (eg, 505, 507, and 508) for each corresponding physical block can be included in the SUE address block 503. In some exemplary implementations, the SUE interface 501 can receive SUE address block 503 information, can identify the information 505, 507, and 508 as corresponding to physical blocks 515, 517, and 528, respectively, and can perform corresponding management accordingly And maintenance operations. In some embodiments, the erasure management operation can be performed on the information in multiple physical blocks, and the writing operation can be performed on the information in the page.

The geometric structure of the two address spaces can also be different. In some embodiments, the logical address space may be single-dimensional (for example, how the logical block address (logical block address; LBA) offset is aligned), and the physical address space may be multi-dimensional, including: Various characteristics of error correction code (ECC), physical page, physical block, physical die, etc. (including some or a subset of them). The SUE address space can be one-dimensional or a limited or reduced number of dimensions. In some exemplary implementations of the SUE address space, the dimensions of the basic physical address space can be abstracted into a single dimension or a reduced number of dimensions. The selected dimensions (e.g., blocks and pages) associated with the management activities of the underlying physical address space (e.g., recycling/abandonment project collection, power cycling, or the like) can be abstracted into the SUE address space, and the basic Other characteristics or activities of the physical address space (for example, ECC) may not be abstracted into the SUE address space.

It should be understood that the selective exposure of basic characteristics may include coordination performed by other components (not shown in the figure) in the entire system before delivering user data to the MM-SSD 400 instead of the MM-SSD 400. In some embodiments, the MM-SSD 400 can be coupled to management components that operate at different levels of the entire system hierarchy.

Figure 6 is a block diagram illustrating a system 600 according to one embodiment. The system 600 may include multiple MM-SSDs (for example, 620, 630, 640, and 650) that are communicatively coupled to the multi-mode storage management system 610.

It should be understood that some activities (for example, some storage management operations and flash memory management system operations) can be controlled by the multi-mode storage management system 610, and other activities (for example, other storage management operations and flash memory management system operations) can be controlled by MM-SSD 620, 630, 640 and 650 control. In some embodiments, MM-SSD 620, 630, 640, and 650 may include controllers 621, 631, 641, and 651 that can control or guide some activities of MM-SSD 620, 630, 630, and 650, respectively. The pattern storage management system 610 may include a controller 611 that can control or guide some activities of the MM-SSD 620, 630, 640, and 650.

In some exemplary implementations, the controllers 621, 631, 641, and 651 can respectively control or guide the activities of the first partition in the MM-SSD 620, 630, 640, and 650, and the controller 611 can control or guide The activity of the second partition in MM-SSD 620, 630, 640 and 650. The controller 611 can control the activities in the MM-SSD 620, 630, 640, and 650 via the selective basic exposure interface.

In some embodiments, the system 600 may include multiple volumes (for example, 671, 672, and 673). In some exemplary implementations, the system 600 may include user space, and the user space may be mapped to multiple volumes and the storage space may be presented to the user as multiple volumes. It should be understood that the volume can have different sizes. It should also be understood that SUE addressable units of different sizes can be associated with multiple volumes.

Figure 7 is a block diagram illustrating a system 700 according to one embodiment. The system 700 may include a multi-mode SSD (MM-SSD) 750, and the multi-mode SSD (MM-SSD) 750 is communicatively coupled to the multi-mode storage management system 720 included in the appliance 710. The appliance 710 may be any of various computer/electronic appliances (devices). It should be understood that other multi-mode SSDs may be coupled to the multi-mode storage management system 720. The system 700 can manage the storage of metadata 730 and user data 740.

The multi-mode storage management system 720 may include a controller 745. The controller 745 may include a flash memory management system 741 (for user data) and a SUE mapper 742.

The multi-mode SSD 750 may include a logical address space partition 770 and an SUE address space partition 780. The logical address space partition 770 may include a physical address space 777 and a controller 775 that may include a flash memory management system 771 (for metadata). The flash memory management system 771 may include a logical interface 772, and the logical interface may include FTL 773. The physical address space 777 may include NAND flash memory. The SUE address space partition 780 may include the SUE interface 782 and the physical address space 787 which may include NAND flash memory.

The metadata 730 information can be received in the logical address block 791, transferred in the logical address block 792, and delivered from the multi-mode management system 720 to the logical address space partition 770. It should be understood that the logical address blocks 791 and 792 can be the same (for example, the logical address block 791 can be unchanged and simply transferred to the logical address space partition 770).

The logical interface 772 can translate the logical block address (LBA) associated with the metadata into a physical address block 793 associated with the physical address space 777. The FMS 771 can guide the storage management and maintenance operations associated with the physical address space 777. Metadata can be stored in the NAND flash memory in the physical address space 777.

The user data in the logical address block 797 can be transferred to the FMS 741. Since the basic features and characteristics of the physical address space 787 are exposed through the SUE interface 782, the FMS 741 can guide the flash memory management and maintenance operations associated with the basic features and characteristics of the physical address space 787. The SUE mapper 742 can map the logical address block 797 to the SUE address block 798.

The SUE address block 798 can be translated by the selective basic exposure (SUE) interface 782 into a physical address block 799 associated with the NAND flash memory device contained in the physical address space 787 (for example, similar to the figure 517 and 519 in 5). It should be understood that the logical address block may have a different size from the SUE address block, and the SUE address block may have a different size from the physical address block.

Compared with the conventional method, the above implementation of various activities in the hierarchy promotes more efficient and convenient management. Conventional methods have limited flexibility when dealing with activities that affect multiple layers. Some conventional methods must perform activities at multiple levels, resulting in a rapidly increasing adverse impact on overall performance (eg, log-on-log, FMS at the driver level, and FMS at the system level) .

For example, in a disk array (redundant array of independent disks) storage system, there are several items that need to be managed together (for example, data storage and corresponding parity storage), and several items are in the upper storage hierarchy The impact occurs at both the level (for example, the disk array system management level) and the lower storage level (for example, the storage drive level). The life cycle of the information can be different for each level (for example, the user may want to overwrite the information, but the raid system may still need the information for parity recalculation), resulting in the driver level writing for the user Import the "new" data, but the system level still retains the "old" information for the RAID system. This situation produces a write amplification factor without the ability to perform trimming.

FIG. 8 is a flowchart describing a method of driving multimode selective underlying exposure (MM-SUE) according to an embodiment.

Compared with the direct overwrite system without over-provisioning of disk drives (for example, HDD), the drive with 7% over-provisioning (for example, SSD) is 15 times more difficult to perform. In addition, compared to the system without over-provisioning For the system, the drive with 7% over-provisioning (for example, SSD) is also 15 times more difficult to function, and the total is 225 (15×15) times more difficult. Multi-mode storage devices that allow the FMS to be moved up to the upper storage hierarchy promote reduction (for example, 7% over-provisioned systems work within 15 times the difficulty compared to non-over-provisioned systems, and 28% over-provisioning Compared with the system without over-provisioning, the system works in the range of only 3 times the difficulty), which leads to the reduction of the write amplification factor. In some exemplary implementations, the selected base address blocks and pages used to guide management operations from the upper storage hierarchy can be coordinated with or matched with the base entity level. This situation allows the life cycle of information at the user level to become different from the life cycle of information at the system level, but from a management point of view, the two life cycles can be aligned to be the same (for example, the life cycle at the user level) The duration of the cycle and the duration of the life cycle at the system level may correspond to the use and erasure of user space).

In operation 810, the first part of the device may be configured or designated as the first area for storing the first type of information. In some embodiments, the first region may be a metadata region, and the first type of information may be metadata. The error correction code (ECC) size can be changed.

In operation 820, a first type of interface operation may be performed based on the first address space information. In some exemplary implementations, the first area may be a metadata area, and the first type of information may be metadata. In some embodiments, the first type of interface can be a logical address space interface, and can perform operations based on logically addressed information. Logical interface operations can include flash translation logic (FTL), including: receiving metadata and logical addresses; and translating the address blocks visible at the system-level configuration into the address area at the physical-level configuration piece.

In operation 830, the second part of the device is configured or designated as a second area for storing the second type of information. In some embodiments, the second area may be a user data area, and the second type of information may be user data. The SUE address space can abstract or remove the complexity associated with the physical address space, while still exposing the relationship or correspondence with the basic physical address space configuration. In some exemplary implementations, the physical space dimension can be abstracted into the SUE address page dimension and the SUE address block dimension. The physical address space can be abstracted by the SUE address.

In operation 840, a second type of interface operation may be performed based on the second address space information, where the second type of interface may selectively expose basic characteristics. The second address space information may be selective basic exposure (SUE) address space information, where the SUE address space information may correspond to the basic characteristics. The basic characteristics may include the representative geometric structure or dimension of the physical address space geometric structure. The SUE interface may expose dimensions associated with basic system management operations (for example, free space management, recycling and adjustment for free space purposes). The over-provisioning percentage in the metadata area may be different from the over-provisioning percentage in the user data area.

Figure 9 is a block diagram illustrating an exemplary multi-mode SSD 920 according to one embodiment. In FIG. 9, the multi-mode SSD 920 is in contrast to conventional attempts at the logically addressed SSD 910 and the physically addressed SSD 930.

The logically addressed SSD 910 may include a logical interface 911, an FTL 912, and a logical address space 913. The physically addressed SSD 930 may include a physical interface 931 and a physical address space 932. The multi-mode SSD 920 may include a logical interface 921, an FTL 922, a logical address space 923, an SUE interface 924, and a physical address space 925.

The multi-mode SSD 920 promotes the convenience and selective exposure of the basic characteristics of the drive. Unlike conventional methods that are insufficiently exposed or have excessively high complexity, the multi-mode SSD 920 allows appropriate exposure without undue complexity.

However, conventional SSDs do not actually have a good linear address space. To be precise, they usually have a clustered controller containing flash memory chips. The flash memory chips have dies, and the dies are configured Instead, it operates in blocks consisting of pages with data to be stored in groups or strings of transistors. The physically addressed SSD 930 attempts to expose all the basic physical address characteristics of the storage media, thereby allowing operations that are considered extremely fast operations (for example, compared to the logically addressed SSD 910), but will produce extremely complex methods. The logically addressed SSD 910 has a solution that hides all or almost all the basic details of the characteristics of the storage medium and is regarded as a space that is a single linear plane mapping space. However, the data is finally stored in a space with many hidden basic details. Slow down the system in the physical area (for example, compared to the physically addressed SSD 930).

The multi-mode SSD 920 promotes the convenience and flexible configuration and implementation of FMS operations. The multi-mode SSD 920 can perform FMS operations in the logical address space 923 mainly in the internal controller of the multi-mode SSD 920 (for example, the controller 775 in FIG. 7), while the FMS operations in the SUE address space 925 can mainly be performed in multiple modes. The mode is executed at the system level in a controller external to the SSD 920 (for example, the controller 745 of FIG. 7).

The ability to split or divide the FMS operation of the multi-mode SSD 920 is different from the following FMS operation methods, that is, the SSD 910 and the FMS operation method of the SSD 930 that do not allow the split or division of the FMS operation. The FMS operation of the logically addressed SSD 910 is mainly performed in the internal controller of the logically addressed SSD 910, while the FMS operation of the physically addressed SSD 930 is mainly performed in the controller external to the physically addressed SSD 930. Executed at the system level.

In some embodiments, the multi-mode SSD 920 can selectively expose some basic address space characteristics, and the logically addressed SSD 910 and the physically addressed SSD 930 do not promote the selective exposure of some basic address space characteristics. And it does not expose other characteristics. In some embodiments, exposing the basic traits to an external FMS (eg, multimodal storage management system 720) may involve mapping selected exposures of the basic traits.

II. Selective Basic Exposure (SUE) Mapping

Another embodiment of the present invention implements a selective base exposure (SUE) mapping scheme to generate a mapping from a logical address space for user data in a storage system to an SUE address space. The SUE mapping scheme can selectively expose the salient features of the underlying physical storage medium so as to allow certain storage medium management functions to run across multiple storage devices at the system level instead of at the individual storage device level.

For example, some embodiments can selectively expose the characteristics of the user's address space across multiple NAND flash non-volatile memory devices in the storage appliance. The SUE pages and blocks of the SUE mapping scheme can be aligned with corresponding physical pages and blocks that are collectively managed as a unit in each physical NAND flash non-volatile memory device. In the SUE mapping scheme, individual dies in physical NAND flash non-volatile memory devices are not distinguished, but they are still indirectly reflected in the SUE block size.

The correlation between the physical pages and blocks of the storage device and the SUE pages and blocks of the SUE mapping scheme allows the coordination and implementation of certain NAND flash across all NAND flash non-volatile memory devices in the storage system at the system level. Flash memory management functions, such as erasing, programming, recycling (collection of discarded items), free space management or the like. The system-level implementation of certain storage media management functions can provide advantageous efficiencies in the deployment of storage resources.

Referring again to FIG. 3, a multi-mode storage device (for example, 350, 400, 620), such as a NAND flash non-volatile memory device, can be implemented in conjunction with the SUE mapping scheme described in this disclosure. For example, in some embodiments, the multi-mode storage device may be a solid state drive (SSD) based on NAND flash memory. In some embodiments, the multi-mode storage device may conform to a standardized physical appearance size, such as a standard disk drive appearance size or a standard memory card appearance size.

Referring again to FIG. 5, as described above, the multi-mode storage device may include multiple dies 511, 512, 513, 514, 521, 522, 523, 524, 531, 532 with multiple NAND flash memory cells. , 533, 534, 541, 542, 543, and 544, or memory chips. The NAND flash memory cell on each die can be subdivided into multiple discrete physical blocks of the memory cell, such as physical blocks 515, 517, 519, 528, and 539.

The erasing and management of free space can be performed with respect to the blocks of the memory cell on one or more discrete groups of the dies on the multi-mode storage device. For example, a multi-mode storage device can include one hundred and twenty-eight dies, and can be erased with respect to one block from each of the one hundred and twenty-eight dies as a group or unit. Manage free space. Alternatively, a multi-mode storage device may include one hundred and twenty-eight dies, and may be erased relative to one block from a set of dies as a group (for example, a group of thirty-two dies) In addition and management of free space.

FIG. 10 is a block diagram depicting an exemplary SUE block for storage in the user area of a multi-mode storage device and a page corresponding to the SUE according to an embodiment of the present invention.

Referring now to FIG. 10, a sequence of physical blocks 1012, 1014, 1016, and 1018 forming the SUE block 1010 is shown. Each physical block 1012, 1014, 1016, and 1018 of the memory cell is further divided into a plurality of discrete physical pages (for example, 1021, 1022, 1023, and 1024) of the memory cell. The SUE page 1030 includes corresponding physical pages 1032, 1034, 1036, and 1038 from each of the physical blocks 1012, 1014, 1016, and 1018 in the corresponding SUE block 1010.

In some embodiments of the present invention, the SUE configuration of the memory cell in a multi-mode storage device (for example, 350, 400, or 620) can be generated. Regarding programming and erasing, SUE pages and SUE blocks can be organized with respect to each group of die collectively managed as a unit on the storage device. A SUE block can be defined as a memory cell that includes a physical block. The physical blocks are each selected from each die included in the crystal particle set of the multi-mode storage device, and these physical blocks can be used as a unit. Land erasure and management. The SUE page can be defined as discrete sections or segments including SUE blocks, and these discrete sections or segments can be programmed together.

For example, in some embodiments, a multi-mode storage device can include one hundred and twenty-eight dies, and can collectively erase and manage the freedom on the individual physical blocks included in each die space. The corresponding SUE block 1010 can be defined as a memory cell with one hundred and twenty-eight physical blocks, each of which is selected from one hundred and twenty-eight dies of the multi-mode storage device. The corresponding SUE page 1030 can be defined as including one hundred twenty-eight sections or segments corresponding to one hundred and twenty-eight physical blocks, respectively.

In another embodiment, the multi-mode storage device may include one hundred and twenty-eight dies, and (for example) can collectively erase and manage the free space on the respective physical blocks of thirty-two dies at the same time . The corresponding SUE block 1010 can be defined as a memory cell with thirty-two physical blocks, each of which is selected from thirty-two dies of the multi-mode storage device. In this situation, the corresponding SUE page 1030 can be defined as including thirty-two sections or segments corresponding to thirty-two physical blocks, respectively.

In yet another embodiment, the multi-mode storage device may include one hundred and twenty-eight dies divided into four planes, and can manage the free space on the blocks of the respective planes included in each die. The corresponding SUE block 1010 can be defined as a memory cell including one hundred twenty-eight blocks with respective planes included in the memory device. In this situation, the corresponding SUE page 1030 can be defined as including one hundred twenty-eight sections or segments corresponding to the blocks of the respective planes.

FIG. 11 is a block diagram depicting an exemplary SUE block and a corresponding SUE page of a user storage space used for storage in a user area of a multi-mode storage device according to an embodiment of the present invention.

Referring now to FIG. 11, an exemplary SUE block (S block) 1110 of the user storage space is shown as a block diagram. In some embodiments, the SUE block 1110 can be regarded as similar to a virtual block (V block). The SUE block 1110 may be the basic unit of memory media management operations at the level of individual storage devices. The SUE block can be composed of multiple SUE pages (S pages). In some exemplary implementations, the SUE page may be considered similar to a virtual page (V page). For example, the SUE block 1110 depicted in FIG. 11 includes four SUE pages (S pages) 1121, 1122, 1123, and 1124.

As depicted in Figure 5, the physical memory unit cells allocated to the SUE page (S page) in the SUE block (S block) can span a single multi-mode storage device (eg, 350, 400, or 620) A plurality of dies in are located in corresponding physical pages and physical blocks. Alternative embodiments may include partitioning into blocks of any number of pages based on the relationship between the size of the physical erase block and the size of the programmable physical page of the multi-mode storage device.

FIG. 12 is a block diagram depicting an exemplary SUE meta page for storage in a user area of a multi-mode storage device and a corresponding SUE page according to an embodiment of the present invention.

Referring now to FIG. 12, an exemplary meta page (MPAGE) 1210 is shown as a block diagram. The meta page 1210 may be composed of multiple SUE pages (S pages) that span multiple storage devices in the storage system. For example, the meta page 1210 depicted in FIG. 12 includes five SUE pages (S pages) 1211, 1212, 1213, 1214, and 1215. Alternative embodiments may include meta pages that are divided into any number of SUE pages based on the number of individual multi-mode storage devices in the storage system and the number of dies collectively managed as a unit in each of the multi-mode storage devices .

The physical memory cell assigned to each SUE page can be located in an individual multi-mode storage device (for example, 350, 400, or 620). The memory unit cells assigned to the various SUE pages 1211, 1212, 1213, 1214, and 1215 forming the meta page 1210 may be located in multiple storage devices (for example, 620, 630, 640) associated with the storage system (for example, storage appliance). And 650).

Therefore, although the size or width of the SUE pages 1121, 1122, 1123, and 1124 may correspond to the number of dies commonly managed as a unit in each multi-mode storage device, the size or width of the meta page 1210 may correspond to The number of multi-mode storage devices in the storage system.

FIG. 13 is a block diagram depicting an exemplary SUE metablock and corresponding SUE metapage for storage in a user area of a multi-mode storage device according to an embodiment of the present invention.

Referring to FIG. 13, an exemplary metablock (M block) 1310 is shown as a block diagram. The meta block (M block) 1310 may be composed of a plurality of meta pages 1311, 1312, 1313, and 1314. Like the meta page (M page) 1210, the physical memory cell assigned to the meta block (M block) 1310 can be located in multiple storage devices associated with the storage system. That is, the meta-block (M-block) 1310 may include a separate block selected from each die, and each die is included in a corresponding subset of the following die, that is, each included in the storage system A corresponding subset of dies that are collectively managed as a unit in a multi-mode storage device (for example, 620, 630, 640, and 650). Therefore, the size of the metablock 1310 can correspond to the number of dies commonly managed in each multi-mode storage device (eg, 350, 400, or 620) and the multi-mode storage device included in the storage system (eg, 620, 630, 640, and 650).

14 is a block diagram depicting another exemplary SUE metablock and corresponding SUE block for storage in the user area of the multi-mode storage device according to an embodiment of the present invention.

Referring to FIG. 14, an exemplary meta block (M block) 1410 may be represented as a block diagram composed of a plurality of SUE blocks (S blocks) 1110. The metablock 1410 may be a respective SUE block 1411 selected from each subset of the die jointly managed in each multi-mode storage device (for example, 620, 630, 640, and 650) included in the storage system , 1412, 1413, 1414, and 1415. Similarly, the meta page 1210 may be a corresponding SUE page selected from each of the corresponding SUE blocks 1411, 1412, 1413, 1414, and 1415 in the meta block 1410 (for example, 1211, 1212, 1213, 1214 And the gathering of 1215).

In some embodiments of the present invention, certain memory media management functions (for example, erasing, programming, recycling (collection of discarded items), free space management, or the like) can be performed at the metablock level. That is, these memory media management functions can be coordinated at the storage system level rather than the individual storage device level.

In order to achieve the desired system-level memory management, the logical address space dedicated to user data can be mapped to the SUE address space. User data (for example) is generated by applications and virtual machine (virtual machine; VM) operating systems. Addressing. Therefore, the user area of the multi-mode storage device included in the storage system can be addressed through the basic exposed interface. System-level memory mapping and management results in a lower write amplification factor, which allows for reduced storage area deployment, resulting in cost savings.

15 is a block diagram depicting an exemplary SUE mapping scheme that can be implemented by a multi-mode storage system to provide address mapping from logical addresses to SUE addresses according to an embodiment of the present invention.

Referring to FIG. 15, an SUE mapping scheme 1500 that can be implemented by a storage system (for example, the multi-mode storage management system 610 of FIG. 6) to provide an address mapping from a logical address to an SUE address in an embodiment of the present invention is illustrated. The SUE mapping scheme 1500 can correlate the logical address space with the SUE address space. The SUE address space can reveal the salient characteristics of the underlying physical storage media. The SUE address space can be used to address the aggregate physical storage space of multiple storage devices included in the storage system.

The user data 1502 can be received as input, for example from a host application and a virtual machine operating system. The host user data can be organized into storage units (for example, , The logically addressed block or logical block). Each logical block of the received user data can be addressed by a logical block address (LBA). For example, in some embodiments, the input logic block addressing may be associated with the Small Computer System Interface (SCSI) standard promulgated by the American National Standards Institute (ANSI).

The logically addressed blocks of the user data 1502 can be combined into SUE addressable units or hybrid mapping system (HMS) mapping blocks (HMB). In some embodiments, an integer number of logical blocks can be grouped to form an SUE addressable unit. For example, in Figure 15, eight logical blocks are combined to form each SUE addressable unit. In alternative embodiments, any integer or fractional number of logical blocks can be combined to form an SUE addressable unit.

In some embodiments, the SUE addressable unit may have the smallest mapping granularity for the storage system. In various embodiments, the SUE addressable unit size may include 4K bytes, 8K bytes, or any other suitable information size or information chunk.

In some embodiments, the storage system may include a set of volumes, and each volume may include a set of SUE addressable units, and each addressable unit may include a set of logical units. Different volumes can use different SUE addressable unit sizes. It should be understood that a volume can have several characteristics. Volumes can correspond to: applications, a single user-level file system, logical drives, namespaces (for example, a collection of adjacent logical addresses associated with a given namespace), logical unit numbers (Logical Unit Number; LUN) etc.

In the example implementation depicted: logically addressed blocks 1531, 1532, 1533, 1534, 1535, 1536, 1537 addressed by logical block addresses 1521, 1522, 1523, 1524, 1525, 1526, 1527, and 1528 And 1538 are combined into SUE addressable unit 1503; logically addressed blocks 1551, 1552, 1553, 1554, 1555, 1556 addressed by logical block addresses 1541, 1542, 1543, 1544, 1555, 1546, 1547, and 1548 , 1557, and 1558 are combined into SUE addressable unit 1504; and logically addressed blocks 1581, 1582, 1583, 1584, 1585 addressed by logical block addresses 1571, 1572, 1573, 1574, 1575, 1576, 1577, and 1578 , 1586, 1587, and 1588 are combined into the SUE addressable unit 1505. Logic blocks can span addressable units. There can be multiple blocks for each addressable unit.

A data compression algorithm may be performed on the user data 1502 in the SUE addressable units (for example, 1503, 1504, and 1505) as appropriate to generate compressed SUE addressable units (for example, 1507, 1508, and 1509). A header section (for example, 1511, 1512, and 1513) may be generated corresponding to each compressed SUE addressable unit (for example, 1507, 1508, and 1509). For example, the header section can contain information for recycling and data recovery activities.

The compressed SUE addressable unit and header section can be placed in the storage device transfer block or SSD transfer block (SSD transfer block; STB) 1515 and 1517. In the depicted example, header sections 1511, 1512, and 1513 and corresponding compressed SUE addressable units 1507, 1508, and 1509 are included in STBs 1515 and 1517. In some embodiments, a closed logical block of compressed SUE addressable units and user data 1502 may be allowed to transmit blocks across two or more storage devices.

An integer number of storage device transfer blocks can be aligned to SUE pages 1591, 1592, 1593, and 1594 for transfer to a multi-mode storage device. In some embodiments, a closed logical block of compressed SUE addressable units and user data 1502 may be allowed to span two or more than two SUE pages 1591, 1592, 1593, and 1594.

In some embodiments, an error check such as an error correction code (ECC) may not be performed with respect to the user data 1502 at the system level, but an error check must be performed by the individual multi-mode storage device.

Metadata associated with user data can be stored in a logically addressed system area of a multi-modal storage device (eg, 350, 400, or 620). For example, in some embodiments, a partition of a memory cell addressed by logical block addressing in a multi-mode storage device may store a mapping table, which may map the SUE addressable unit to the SUE address The mapping table in the space. For example, the mapping table can store indicators, and each indicator can point to an individual SUE addressable unit. Therefore, the mapping of the SUE addressable unit and the offset corresponding to the logical block and the SUE addressable unit can be used to determine the storage location of the logical block of the user data in the SUE address space.

In some embodiments, information can be stored in a volume (for example, 671, 672, and 673). There can be multiple volumes or namespaces, and different volumes or namespaces can be associated with SUE addressable units of different sizes. It should also be understood that volumes or namespaces of different sizes can be associated with SUE addressable units of the same size.

FIG. 16 is a schematic diagram depicting an exemplary storage system that can implement the SUE mapping scheme of FIG. 15. As illustrated in FIG. 16, an exemplary storage system 1602 may include a processor 1604, a memory 1606, a network interface 1608, an input/output (I/O) device 1610, a display device 1612, and a bus 1614 And multiple non-volatile memory devices 1616. Various components of the storage device 1602 can be coupled by a local data link 1618. In various embodiments, the local data link 1618 includes, for example, address bus, data bus, serial bus, and parallel. Bus bars or any combination of said bus bars.

The processor 1604 may include a digital processor suitable for any general or special application of controlling the storage system 1602. The memory 1606 may include any digital memory device suitable for storing data and instructions for the processor 1604 to access.

The network interface 1608 may include any network suitable for connecting the storage system 1602 to a communication network (for example, a local area network (LAN) or an internet protocol (IP) network) by means of communication. Road connection interface. The network interface 1608 can implement storage network connection standards, such as the Internet Small Computer System Interface (iSCSI) protocol.

The input/output device 1610 may include any suitable device for sending digital information to the storage system 1602 or receiving digital information from the storage system 1602. The display device 1612 may include any suitable device for displaying text or a graphical user interface (GUI).

The bus 1614 may include, for example, a peripheral component interconnect express (PCIe) bus or any other suitable high-speed serial expansion bus used for communication in a storage system. Bus 1614 can use standard NVM Express (Non-Volatile Memory Express; NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCI) to access storage in storage system 1602 A device, such as a non-volatile memory device 1616. The non-volatile memory device 1616 may include, for example, a NAND flash memory-based solid state drive (SSD) or any other suitable non-volatile memory device.

In an alternative embodiment, a general computing device may implement the functions of the SUE mapping scheme 1500 of FIG. 15. For example, general computing devices may include servers, workstations, personal computers, or the like.

Program code (such as source code, object code, or executable code) stored on a computer-readable medium such as non-volatile memory device 1616 can be loaded into working memory or operating memory (e.g., memory It is executed by the processor 1604 in order to perform the functions of the SUE mapping scheme 1500 in FIG. 15. In alternative embodiments, executable instructions can be stored in firmware, or functions can be executed by dedicated hardware.

FIG. 17 is a flowchart showing an exemplary method of mapping a logical address space to an SUE address space according to an embodiment of the present invention. Referring now to FIG. 17, an exemplary process flow may be executed by, for example, the storage system 1602 in FIG. The address space is an embodiment of the SUE mapping scheme in order to address the aggregate physical storage space of a plurality of storage devices included in the storage system.

The process starts at operation 1702, where, for example, user data is received from a host application or a virtual machine operating system. The received user data can be organized in logical blocks and addressed by logical block addresses. The logical block may correspond to the smallest addressable memory unit size associated with the native host file system, database, or the like.

In operation 1704, as described above, logical blocks may be combined into SUE addressable units. For example, an integer number of logical blocks can be grouped to form each SUE addressable unit. In operation 1706, a data compression algorithm may be selectively performed on the user data in the SUE addressable unit, as explained above. (The components shown in dotted lines in Figure 17 are optional items.)

In operation 1708, a header section may be generated and added to each SUE addressable unit. The header section can be generated to include: for example, information used in recycling and data recovery activities, as described above. In operation 1710, the compressed SUE addressable unit and header section may be placed in the storage device transmission block, as explained above.

In operation 1712, as further explained above, an integer number of storage device transfer blocks can be linked and aligned to the SUE page, and in operation 1714, the storage device transfer block corresponding to the SUE page can be transferred to the multi-mode The storage device can be stored in the user area. In operation 1716, metadata about the user data in the SUE page may be sent to the multi-modal storage device to be stored in the system area, as described above.

III. Multi-mode storage management system

Another embodiment of the present invention is shown in FIG. 18, which illustrates the use of the SUE addressing scheme in order to allow the storage system to store the logic in the storage system (such as the storage system 1602 of FIG. 16) and the exemplary multi-mode storage of the SUE storage space The management system 1802 performs addressing. The multi-mode storage management system 1802 may include an SUE storage manager 1804, a logical storage manager 1806, a recycling manager 1808, and a storage array manager 1810.

The SUE storage manager 1804 can provide user data storage mapping, reading and writing functions. The SUE storage manager 1804 can use the SUE address mapping scheme to map user data to the user area of the storage system. The SUE storage manager 1804 can access the user data stored in the user area to the storage device of the storage system through the SUE interface.

The SUE mapping solution can distribute the logical block address to physical address mapping function between the storage system and the storage device. In other words, the SUE mapping scheme can combine storage system level mapping or virtualization from logical block addresses to SUE addresses and storage device level mapping or translation from SUE addresses to physical addresses.

The SUE mapping scheme can expose certain physical characteristics or representative geometric structures of the storage device to the storage system, so that certain non-volatile memory management functions related to user data can span multiple storage devices at the storage system level instead of at the storage system level. Executed at the level of individual storage devices. The redistribution of user data management tasks from the individual storage device level to the storage system level can generate high system efficiency, including reducing the write amplification factor, thereby allowing reduced resource deployment and lower costs.

The logical storage manager 1806 can provide system data storage mapping, reading, and writing functions. The logical storage manager 1806 may use a logical address mapping scheme, such as logical block addressing (LBA), to map system data to the system area of the storage device. The logical storage manager 1806 can access the system data stored in the system area to the storage device through a logical interface.

Therefore, in some embodiments, the memory space of an associated storage device or multiple associated storage devices can be subdivided or partitioned into separate storage areas or address spaces, including logically addressed System area and user area addressed by SUE. The storage device may include two host interfaces: a logical host interface, which provides access to the logically addressed system area; and an SUE host interface, which provides access to the user area of the SUE address. The non-volatile memory management function of the system data can be executed by the individual storage device controller.

The recycling manager 1808 can provide non-volatile memory management of user data at the storage system level, including free space management and recycling or discarded item collection functions. Therefore, individual storage devices included in the storage system may not perform local recycling (collection of discarded items) of user data. The recovery manager 1808 can implement various free space management and recovery methods. In some embodiments, the recovery manager 1808 can also execute the novel free space management and recovery methods described in this disclosure.

The storage array manager 1810 or the redundant array of independent disks (RAID) manager can provide storage management of user data for an array of multiple storage devices included in the storage system, including data recovery functions. Therefore, individual storage devices included in the storage system may not perform the die-level RAID function for user data. The storage array manager 1810 can implement various storage management and data recovery methods. In some embodiments, the storage array manager 1810 can also execute the novel storage management and data recovery methods described in this disclosure.

19, another exemplary multi-mode storage management system 1902 is described, which uses an SUE addressing scheme to allow the storage system to address the logic and SUE storage space in the storage system (such as the storage system 1602 of FIG. 16). The multi-mode storage management system 1902 may include data alignment unit (DAU) 1904, SUE storage access manager 1906, data compression manager 1908, volume mapping engine 1910, buffer manager 1912, metablock The manager 1914, the reclamation manager 1916, the storage array manager 1918, and the logical storage access manager 2020.

The data alignment unit (DAU) 1904 can receive logically addressed media access commands, such as read, write, and unmap commands, from a small computer system interface (SCSI) target. The command is based on a linear addressing scheme in which the memory block is indicated by an integer index, using logical block addressing (LBA) and SCSI memory location abstraction standards. In logical block addressing, a single integer base address is used to identify the beginning of each logical block of data, and each linear base address is uniquely associated with a single logical block. Therefore, logical block addressing hides or obscures specific details or features of the storage device from the operating system, file system, device driver, and host application.

During the write operation, the data alignment unit 1904 can combine the logical blocks of the data received from the SCSI target into SUE mapping blocks. For example, in some embodiments, an integer number of logical blocks may be grouped to form an SUE mapping block. The data compression manager 1908 can selectively execute a data compression algorithm on the user data in the SUE mapping block.

During the read operation, the data alignment unit 1904 can receive a read command from the SCSI target and pass the read request to the SUE storage access manager 1906. The data alignment unit 1904 can receive the requested user data from the SUE storage access manager 1906 and transfer the requested user data to the SCSI target.

The SUE storage access manager 1906 can provide user data reading functions and user data writing functions. During the write operation, the SUE storage access manager 1906 may generate a header section for each SUE mapping block. The header section may contain information used in recycling and data recovery activities, for example. The SUE storage access manager 1906 may place the compressed SUE mapping block together with the corresponding header section in the storage device transmission block. In some embodiments, the compressed SUE mapping block and the closed logical block of user data are allowed to transmit blocks across two or more storage devices.

The SUE storage access manager 1906 can further align an integer number of storage device transmission blocks to the SUE page for transmission to the storage device. The SUE storage access manager 1906 may transfer the storage device transfer block corresponding to the SUE page to the write buffer.

In some embodiments, the compressed SUE mapping block and the closed logical block of user data are allowed to span two or more SUE pages. Each SUE page can correspond to an individual storage device of the storage system. In the SUE mapping scheme, the SUE page can be a stylized basic unit or a basic unit of write operations.

During the read operation, the SUE storage access manager 1906 can determine the location of the requested user data, and request to read the requested user data from the associated storage device to the read buffer. The SUE storage access manager 1906 can send user data from the read buffer to the data alignment unit 1904.

As a sub-function or as a supplementary function of the SUE addressing scheme, the data compression manager 1908 can execute a compression algorithm on user data. The data compression function performed by the data compression manager 1908 can help offset the inherent system factors that cause write amplification.

The volume mapping engine 1910 can coordinate the SUE address mapping function. The volume mapping engine 1910 can maintain a user area mapping table that records the current location of user data. The user area mapping table may include mapping information that correlates the logical block address of the stored user data with the SUE address. The user area mapping table can be stored in the logically addressed system area of the associated storage device.

During the write operation, the volume mapping engine 1910 can update the user area mapping table based on the new or revised SUE address location of the written user data received from the SUE storage access manager 1906 .

During the read operation, the volume mapping engine 1910 can look up the SUE address location of the requested user data in the user area mapping table based on the requested logical block address, and store the access management to the SUE The device 1906 provides the SUE address location.

The volume mapping engine 1910 can organize user data into SUE pages, SUE blocks, meta pages, and meta blocks. The SUE block can be mapped to several physical blocks on individual storage devices. In some embodiments, multiple physical blocks mapped to the same SUE block may be located on separate dies of the storage device. All physical blocks mapped to the same SUE block can be erased and managed as a unit at the storage device level. Therefore, the SUE block can correspond to a group of physical blocks that are collectively managed on each die regarding recycling and free space management. Equivalently, individual physical block groups on the die corresponding to the SUE block can be managed as a storage medium unit.

Each SUE block may include several SUE pages, and each SUE page may be aligned to a physical page of a respective physical block mapped to the SUE block. The corresponding SUE page of each SUE block located across all storage devices included in the storage system can be mapped to a meta page. Similarly, corresponding SUE blocks located across all storage devices included in the storage system can be mapped to meta blocks.

Storage media management functions (for example, reclamation and free space management) can be performed at the level of the multi-mode storage management system 1902 relative to the metablocks of user data. Therefore, the storage medium management function can be performed at the level of the multi-mode storage management system 1902 with respect to the corresponding physical block group jointly managed in each storage device included in the storage system.

Programmable operations and read operations can be performed relative to the meta page of user data. Therefore, the programming operation and the reading operation can be performed with respect to the corresponding physical page group jointly managed in each non-volatile memory device included in the storage system.

Therefore, the storage devices included in the storage system can be virtualized by exposing the significant organization or representative geometric structure of the physical storage area to the multi-mode storage management system 1902. The group of physical blocks jointly managed on the individual dies included in a single storage device can be presented as an SUE block to the multi-mode storage management system 1902, and is positioned across all storage devices included in the storage system The corresponding physical block groups jointly managed on the respective dies can be presented as metablocks to the multi-mode storage management system 1902.

Similarly, a group of physical pages that are commonly programmed on individual dies included in a single storage device can be presented as an SUE page to the multi-modal storage management system 1902, and across all storage included in the storage system The collectively programmed group of physical pages on the individual dies positioned by the device can be presented to the multi-modal storage management system 1902 as meta pages.

The buffer manager 1912 can manage the pool of read and write buffers. During the write operation, the buffer manager 1912 can transfer the storage device received from the SUE storage access manager 1906 before the user data is separately sent to the individual storage device as an SUE page by means of the storage array manager 1918. The blocks are accumulated in the write buffer until approximately the entire meta page of user data has been accumulated.

During the read operation, the buffer manager 1912 can provide a read buffer to support the read cache function. The SUE page of the user data received from the storage array manager 1918 as a storage device transfer block can be stored in the read buffer until it is forwarded to the SUE storage access manager 1906.

The metablock manager 1914 can track the current status of individual metablocks defined in the user area of the storage device, such as erased, active, closed, recycled, or erased. The current state can be stored in the metablock information table, and the metablock information table can be stored in memory and backed up in the system area of the storage device. The metablock manager 1914 can also maintain a corresponding list of metablocks currently in a specific state, such as the erased list, the recovered list, and the erased list. The metablock manager 1914 can select specific metablocks for submission to the SUE storage access manager 1906 for recycling activities.

The reclamation manager 1916 can serve the reclamation request from the metablock manager 1914 to recover valid user data from the specified metablock and relocate the valid user data to other metablocks. The recovery manager 1916 may request the following actions, that is, erase and recover the physical memory cell corresponding to the specified metablock to provide free space in the user area of the storage device.

The storage array manager 1918 can provide the user area of the storage device to the SUE interface, and provide the system area of the storage device to the logical interface. The storage array manager 1918 may provide data protection functions, such as RAID striping and parity checking. For example, in some embodiments, the storage device transfer block may be used as a RAID element, and the RAID stripe may include the storage device transfer block located across all SUE pages in the meta page. Therefore, if a single storage device in the storage system fails, the storage array manager 1918 can use the reverse parity calculation to recover data from the failed storage device.

The logical storage access manager 1920 can use logical addressing methods to provide system data reading and writing functions. The logical storage access manager 1920 can store and retrieve metadata about user data, including user area mapping table, metablock information table, volume table, storage system files, log files, and the like.

Regarding the user data stored in the user area, individual non-volatile memory devices can be responsible for certain memory media management functions (for example, read retry, failed physical block mapping, error correction code (ECC), import (Advanced) incremental step pulse programming (incremental step pulse programming; ISPP) or the like). Regarding the system data stored in the system area, individual non-volatile memory devices can be responsible for all memory media management functions (for example, recycling, wear leveling, read and write cache, read retry, Failure physical block mapping, error correction code (ECC), advanced incremental step pulse programming (ISPP) or the like).

Referring now to FIG. 20, another exemplary multi-mode storage management system 2002 or hybrid mapping system (HMS) that uses the SUE addressing scheme to allow the storage system to address the logic and SUE storage space in the storage device is described.

The multi-mode storage management system 2002 can serve as a global flash translation layer (GFTL) responsible for the management of non-volatile memory media in the user area distributed across multiple storage devices included in the storage system. ). The multi-mode storage management system 2002 can perform non-volatile memory media access functions, address mapping functions, recycling and wear averaging functions. The address mapping function is used to map host application logic address space elements to SUE locations In the address space data structure, the SUE address space data structure is aligned to a physical non-volatile memory location.

The multi-mode storage management system 2002 can include a data alignment unit (DAU) 2004, a user area access manager (UAAM) 2006, a user area mapping engine (UAME) 2008, a buffer manager (BM) 2010, System Area Access Manager (SAAM) 2012, Metablock Manager (MBM) 2014, Metablock Information Manager (MBI) 2016, Storage Device Control Manager (SCM) 2018 , Storage Device Access Manager (SAM) 2020, Global State Manager (GSM) 2022, and Global Error Manager (GEM) 2024. The multi-mode storage management system 2002 may be communicatively connected to the system state manager 2026, the system log record and statistics manager 2028, the target device 2030, and a plurality of nonvolatile memory (NVM) devices 2032.

The global error manager 2024 can manage global errors that occur in the storage system including the multi-mode storage management system 2002. The system state manager 2026 may manage the state (eg, operating conditions) of the multi-modal storage management system 2002. The system log record and statistics manager 2028 can provide system logs/statistics based on the operation of the storage system including the multi-mode storage management system 2002. The non-volatile memory device 2032 may include any type of non-volatile memory widely used. The target device 2030 may be another memory device, which is a target for read/write operations.

The data alignment unit (DAU) 2004 can receive logically addressed media access commands from the target device 2030, for example, read, write, and unmap commands. The data alignment unit 2004 may receive a logical block addressing (LBA) buffer list as input. During the write operation, the data alignment unit 2004 may combine the logical data blocks received from the target device 2030 into an SUE mapping block or a hybrid mapping block (HMB). For example, in some embodiments, an integer number of logical blocks may be grouped to form an SUE mapping block.

The data alignment unit 2004 can combine both the alignment and misaligned user data traffic arriving from the target device 2030, thereby performing read/modify/write operations on the misaligned write traffic, so as to Align the data to the unit used to map the logical address to the physical address (SUE mapping block). The data alignment unit 2004 can place the user data in the SUE mapping block alignment buffer list. In various embodiments, the SUE mapping block may contain a fixed amount of data, such as 4 KB, 8 KB, 16 KB, or the like.

During the read operation, the data alignment unit 2004 can receive a read command from the target device 2030 and pass the read request to the user area access manager 2006. The data alignment unit 2004 can receive the requested user data from the user area access manager 2006 and transfer the requested user data to the target device 2030.

FIG. 21 is a schematic diagram illustrating a user area access manager (UAAM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. Referring to Figure 21, the user area access manager (UAAM) 2006 can include a read manager (read manager; RM) 2102, a write manager (write manager; WM) 2104, a data compression manager (DC) 2106, Data Decompression Manager (DD) 2108, Reclamation Manager (RC) 2110, Free Space Account Manager (FSA) 2112, Flow Control Manager (FC) 2114, and Service Quality Manager (QoS) 2116.

The read manager (RM) 2102 can receive the read request from the data alignment unit 2004 and serve the read request. The reading manager 2102 may request related mapping information from the user area mapping engine (UAME) 2008. The read manager 2102 may notify the storage device access manager 2020 of the read request. During the read operation, the read manager 2102 may request the buffer manager 2010 to release the user data in the read buffer. The read manager 2102 may notify the data decompression manager 2108 of the decompression request regarding the read request user data.

The write manager (WM) 2104 can receive the write request from the data alignment unit 2004. During the write operation, the write manager 2104 may generate a metadata header on the SUE mapping block stream, and generate mapping information about the SUE address of the user data for the user area mapping engine 2008. The write manager 2104 can notify the data compression manager 2106 of the compression request of the scheduled user data compression command, and notify the storage device access manager 2020 of the write request. The write manager 2104 can request the buffer manager 2010 to release the user data written in the buffer. When the current metablock in which data is written has become full, the write manager 2104 may request the metablock manager (MBM) 2014 to open a new active metablock.

The data compression manager (DC) 2106 can receive compression requests from the write manager 2104 and serve the compression requests. In some embodiments, the data compression manager 2106 may implement a data compression algorithm on the user data in the SUE mapping block. In some other embodiments, the data compression manager 2106 may schedule data compression tasks to an external compression unit (not shown in the figure).

The data decompression manager (DD) 2108 can receive the decompression request from the read manager 2102 and serve the decompression request. In some embodiments, the data decompression manager 2108 may implement a data decompression algorithm on the user data in the SUE mapping block. In some other embodiments, the data decompression manager 2108 may schedule data decompression tasks to an external decompression unit (not shown in the figure).

The recycling manager (RC) 2110 can receive a recycling request from the metablock manager 2014, and the service recycling request can recover free space by recovering valid data from the specified metablock. The reclamation manager 2110 may request related mapping information from the user area mapping engine 2008, and notify the read manager 2102 of the read request for the specified metablock. The recycling manager 2110 can parse the metadata header of the SUE mapping block in the data stream read by the attached storage device, and notify the write manager 2104 of the writing of all valid data remaining in the specified metablock ask. The recovery manager 2110 may also serve a request from the storage device control manager (SCM) 2018 to recover part of the metablock data.

The free space account manager (FSA) 2112 can receive the mapping information from the write manager 2104 during the write operation, and generate free space information about the obsolete user data stored in the metablock. The free space account manager 2112 can notify the free space information to the metablock information manager 2016 to update the corresponding metablock information table entry.

The flow control manager (FC) 2114 can monitor system resources, such as read/write buffers, compression buffers, storage buses, and other queue depths or the like. If the system-level resource deployment falls below the preset limit, the flow control manager 2114 may reset the throttling-down level in the service quality manager 2116. In some embodiments, system administrator commands can be used to establish the required level of resource deployment. The flow control manager 2114 can provide system administrators with statistical data for interface-level throttling.

The quality of service manager (QoS) 2116 may define a quality of service policy based on the system resource deployment level and delay measurement. The service quality manager 2116 can implement multiple queues to serve different service quality strategy pools. Regarding the delay-based strategy, the service quality manager 2116 may implement a time stamp on the queue entry. The service quality manager 2116 can monitor various queue parameters and select requests to ensure that policies are not violated. Upon request by the traffic control manager 2114, the service quality manager 2116 can throttle traffic on the deployment-based policy queue.

22 is a schematic diagram illustrating a user area mapping engine (UAME) that can be implemented by a multi-modal storage management system according to an embodiment of the present invention. Referring to Figure 22, the user area mapping engine (UAME) 2008 can include a volume manager (VM) 2202, a map page read manager (MPRM) 2204, and a map page write manager (Map page write manager; MPWM) 2206 and map page cache manager (MPCM) 2208.

The Volume Manager (VM) 2202 can provide services for generating, destroying, and managing volumes and handle multiple deployment strategies. The volume manager 2202 can maintain relevant information in the volume table that can be stored in the memory and backed up in the system area, and can provide access services to the entries in the volume table. The volume manager 2202 can use the system area access manager 2012 to back up and restore the volume table.

When the map page cache manager 2208 detects a map page miss, the map page read manager (MPRM) 2204 can receive and serve the missing map page provided by the map page cache manager 2208. Request. The mapped page write manager (MPWM) 2206 can receive and service requests from the mapped page cache manager 2208 for the recall of the mapped page.

The Mapping Page Cache Manager (MPCM) 2208 can serve the mapping entry information request provided by the read manager 2102 and the recycling manager 2110, and the mapping entry update provided by the write manager 2104. When it is detected that the mapping page is missing, the mapping page cache manager 2208 may request the mapping page reading manager 2204 for the missing mapping page. The mapped page cache manager 2208 may request the mapped page write manager 2206 to recall the mapped page.

The Buffer Manager (BM) 2010 can manage the pool of read and write buffers. During the write operation, the buffer manager 2010 can allocate and release storage device transfer blocks to accumulate the user data received from the write manager 2104 in the write buffer. When approximately the entire meta page of the user data has been accumulated, the buffer manager 2010 can receive a request from the write manager 2104 for releasing the user data written in the buffer, and transfer the user data to the storage device for access Manager 2020.

During the read operation, the buffer manager 2010 can allocate and release the storage device transfer block in the read buffer to support the read cache function. The SUE page of the user data received from the storage device access manager 2020 as the storage device transfer block can be initially stored in the read buffer. The buffer manager 2010 can receive a request for releasing user data in the read buffer from the read manager 2102, and the buffer manager 2010 can transfer the storage device transfer block to the read manager 2102.

The system area access manager (SAAM) 2012 can serve requests for accessing system data stored in the system area of the storage device included in the storage system. The system area access manager 2012 can receive and serve requests from the volume manager 2202 and the metablock information manager 2016 for backing up and restoring the volume table and the metablock information table, respectively. The system area access manager 2012 can receive and service requests from the mapped page write manager 2206, the mapped page read manager 2204, and the mapped page cache manager 2208 to access the user area mapping table.

FIG. 23 is a schematic diagram illustrating a metablock manager (MBM) that can be implemented by a multi-modal storage management system according to an embodiment of the present invention. Referring to FIG. 23, the metablock manager 2014 may include a reclamation metablock picker (RCMBP) 2302 and a metablock state manager (MBSM) 2304.

The Recycle Metablock Picker (RCMBP) 2302 can monitor parameters related to user area metablocks, such as erasure count, outdated data level, residence time, and the like. Based on the monitored parameters, the recycling metablock selector 2302 can select metablocks for recycling or discarded item collection. The reclaimed metablock selector 2302 can implement various loss averaging strategies. For example, the recycled metablock selector 2302 may try to maintain the metablock erasure count within a better value range, and try to separate relatively dynamic (hot) data and relatively static (cold) data in a single metablock. .

The Metablock State Manager (MBSM) 2304 can track the current state of the metablock in the user area, for example, active, closed, erased, erased, recycled, or discarded item collection. The metablock state manager 2304 can change metablocks through various states by updating the metablock information table. The metablock state manager 2304 can also maintain various lists of metablocks in a specific state, for example, a list of erased metablocks, a list of recovered metablocks, and a list of erased metablocks. The metablock status manager 2304 can monitor the list of erased metablocks to determine individual metablocks that are ready for recycling (collection of discarded items).

Metablock Information Manager (MBI) 2016 can maintain a metablock information table. The metablock information manager 2016 can provide access services to entries in the metablock information table for other modules. The metablock information manager 2016 can send a request to back up and restore the metablock information table to the system area access manager 2012.

24 is a schematic diagram illustrating a storage device control manager (SCM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. Referring to Figure 24, storage device control manager (SCM) 2018 or solid state device (SSD) control manager (SCM) may include storage device logging and statistics manager (storage device logging and statistics manager; SLS) 2402, S block S-Block erase engine (SBEE) 2404 and storage device error manager (SEM) 2406.

The storage device log record and statistics manager (SLS) 2402 can maintain a log of storage device access history.

The S-block erasing engine (SBEE) 2404 can receive erasing requests from the recovery manager 2110 and manage the erasing process by means of the meta-block manager 2014. The S block erasing engine 2404 can send the S block erasing request to the storage device access manager 2020.

The storage device error manager (SEM) 2406 may send a request for reclaiming part of the metablock data to the reclaim manager 2110.

25 is a schematic diagram illustrating a storage device access manager (SAM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. Referring to FIG. 25, the storage device access manager (SAM) 2020 may include a logical access manager (SLA) 2502, a RAID manager (RAID) 2504, and a read lookup engine (RLE) 2506 And storage initialization manager (storage initialization manager; SI) 2508.

The read search engine 2506 can provide/manage read operation search. The storage initialization manager 2508 can manage the initialization operations of the storage system.

The logical access manager (SLA) 2502 or SSD logical access manager can provide access services for system data in the system area of the storage device. The logical access manager 2502 can use various logical block addresses to address system data in the system area of the storage device. The logical access manager 2502 can use standard fast NVM (NVMe) or non-volatile memory host controller interface specification (NVMHCI), and can send commands to access storage devices or solid state drives included in the storage system ( SSD).

The RAID manager (RAID) 2504 can provide storage management of user data for an array of multiple storage devices included in the storage system, including data recovery functions. Therefore, individual storage devices included in the storage system may not perform die-level RAID functions for user data. The RAID manager 2504 can implement various storage management and data recovery methods. In some embodiments, the RAID manager 2504 can also perform the novel storage management and data recovery methods described in this disclosure.

The RAID manager 2504 can provide the user area of the storage device to the SUE interface, and provide the system area of the storage device to the logical interface. The RAID manager 2504 can provide data protection functions such as RAID striping and parity checking. For example, in some embodiments, the storage device transfer block may be used as a RAID element, and the RAID stripe may include the storage device transfer block located across all SUE pages in the meta page. Therefore, if a single storage device in the storage system fails, the RAID manager 2504 can use reverse parity calculation to recover data from the failed storage device.

FIG. 26 is a schematic diagram illustrating a global state manager (GSM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. Referring to FIG. 26, the global state manager (GSM) 2022 may include a power fail manager (PFM) 2602 and an error and crash manager (PFCM) 2604.

The power failure manager 2602 can manage problems associated with the power failure of the multi-mode storage management system 2002. The error and corruption manager 2604 can manage the error/corruption problems that occur in the multi-mode storage management system 2002.

The functions of the multi-mode storage management systems 1802, 1902, and 2002 of FIG. 18, FIG. 19, and FIG. 20 can be implemented by the storage system 1602 of FIG. 16. In alternative embodiments, the functions of the multi-mode storage management systems 1802, 1902, and 2002 can be implemented by normal computing devices or by dedicated hardware.

The presented multi-modal approach includes multiple features and characteristics that promote effective and efficient storage of information. The features and characteristics can improve many different qualities associated with performance. In some embodiments, the flexibility of the described segmentation method allows for relatively fast and manageable complexity. A relatively large amount of user data can be stored in the SUE address space for extremely fast storage and management of user data. A relatively small amount of metadata can be stored in the logically addressed area, allowing the storage system to take full advantage of the abstraction nature of metadata for reduced complexity.

In addition, increasing the flexibility of over-provisioning in a relatively small metadata area has a much larger percentage of over-provisioning impact, which helps speed up metadata storage operations and reduce compensation speed, that is, it will be attributed in other ways to The speed of occurrence decreases as the complexity increases. Compared with other storage systems, this situation allows a better overall allocation and better impact of over-provisioned resources. Flexibility can also promote improved life cycle retention by allowing different storage areas of a block to be reassigned or re-allocated between two partitions. The nature (e.g., type) of the data stored in the area may indicate that one type of data is written/erased less than another type of data in the area (e.g., most metadata is compared to There is no major change in user data), and physical blocks in one partition can be reassigned to another partition to balance the wear and tear on a specific area. Flexibility also allows power cycling improvements by moving power cycling responsibilities up to the system level.

Some parts of the detailed description are presented based on the operation strokes, logical blocks, processing, and other symbolic representations of data bits in the computer memory. These descriptions and expressions are generally used by those who are familiar with data processing technology to effectively transfer the subject of their work to other people who are familiar with the technology. A procedure, logic block or process is here and is usually conceived as a self-consistent sequence of steps or instructions that lead to a desired result. The steps include the physical manipulation of the physical quantity. Usually (but not necessarily), these physical quantities take the form of electrical, magnetic, optical, or quantum signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system. It has been proved that it is convenient to refer to these signals as bits, values, elements, symbols, characters, items, numbers, or the like, mainly for common usage reasons.

However, it should be borne in mind that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to such quantities. Unless specifically stated, as is obvious from the following discussion, it should be understood that throughout this application, the use of terms such as "processing", "arithmetic", "calculation", "determination", "display" or the like refers to manipulation And the actions and strokes of computer systems or similar processing devices (for example, electrical, optical, or quantum computing devices) that transform data expressed as physical (for example, electronic) quantities. The term refers to the manipulation or transformation of physical quantities in components of a computer system (for example, registers, memory, other such information storage, transmission, or display devices) into other physical quantities that are similarly expressed as physical quantities in other components The movement and itinerary of the data processing device.

IV. Distributed

It should be understood that the presented multi-modal approach can be implemented in a distributed architecture. Distributed architecture can contain various attributes. The distributed architecture can be adjusted linearly according to the number of appliances or systems without being limited to a single thread or having a single bottleneck. Compared with the non-distributed architecture, the distributed architecture can be implemented with little or no degradation. This is because the multi-mode storage management system can continuously operate at the SUE address block level across multiple storage nodes.

Figure 27 is a block diagram illustrating an exemplary appliance 2701 coupled to a network 2705 according to one embodiment. It should be understood that the network 2705 can be coupled to other devices or appliances (not shown in the figure).

In an exemplary implementation, the appliance 2701 may be similar to the system 600. The appliance or system 2701 may include a multi-mode storage management system 2710 and storage devices 2720, 2730, 2740, and 2750.

The multi-mode storage management system 2710 may include a processor 2713, memories 2714, 2715, 2717, and 2718, a bus 2719, and ports 2711 and 2712. The bus 2719 can be a PCI bus, and the ports 2711 and 2712 can be dual 25 G Ethernet ports. The processor 2713 can be a 4 or 6-core Intel Xeon, and the memory 2714, 2715, 2717, and 2718 can be 64 GB of RAM. In one embodiment, there may be no need for PCIe bridge chips (which may result in reduced cost and power consumption compared to methods using PCIe bridge chips).

The storage devices 2720, 2730, 2740, and 2750 may be multi-mode SSDs. It should be understood that there may be multiple characteristics of a distributed system (for example, 4 devices or multi-mode SSDs can operate at a power envelope of 200 W, which can make full use of over-provisioning due to cost, and there may be a single PCIe adapter. Hardware (HW)). In an exemplary implementation, the PCIe bridge chip can be eliminated, thereby saving cost and power consumption.

Figure 28 is a block diagram illustrating a distributed system 2800 according to one embodiment. The distributed system 2800 may include a network 2890 and appliances 2810, 2820, 2830, 2840, 2850, 2860, 2870, and 2880. Each of the appliances 2810, 2820, 2830, 2840, 2850, 2860, 2870, and 2880 may include a multi-mode storage management system and multiple storage devices (for example, 2891, 2892, 2893, and 2894). In an exemplary implementation, each of appliances 2810, 2820, 2830, 2840, 2850, 2860, 2870, and 2880 may be similar to appliance 2701.

Figure 29 is a block diagram illustrating a distributed system 2900 according to one embodiment. The distributed system 2900 may include multiple storage nodes 2911, 2912, 2913, and 2914, which include a global namespace 2920. Multiple storage nodes 2911, 2912, 2913, and 2914 can be coupled to an external connectivity mechanism 2930, which in turn can be coupled to include FTP client 2941, ISCI component 2942, NFS component 2943, CIFS component 2944, and Web component Multiple external components for 2945. Multiple storage nodes 2911, 2912, 2913, and 2914 can also be coupled via a node-to-node connectivity mechanism 2950. The multiple storage nodes 2911, 2912, 2913, and 2914 can store information related to multiple external components including the FTP client 2941, ISCI component 2942, NFS component 2943, CIFS component 2944, and Web component 2945.

It should be understood that the selective basic exposure method described above with respect to a single appliance or system can be extended to be utilized by multiple distributed appliances or systems. A logically addressed block can be associated with a selective base exposure (SUE) addressable block or unit. The SUE addressable unit can be compressed and associated with the header, and then can be configured in a storage device transfer block (STB). The associations may be similar to those depicted in FIG. 15.

However, in the distributed approach, the distributed SUE metablock (DSUE metablock) can contain information associated with multiple physical blocks, where only one of the multiple physical blocks can be located in a single storage In the device, there may be the first one of the storage devices associated with the first appliance or node among multiple physical blocks and the storage device associated with the second appliance or node among the multiple physical blocks The second one in. In an exemplary implementation, the DSUE metablock may include information stored in a physical flash memory block by each SSD, and the information storage may span all drives and all nodes.

During normal operation, there are primary block owners and as many redundant block owners as needed. The concepts of primary block owner and redundant block owner can be used in multiple design elements. The main block owner can be the node responsible for the life cycle of a given SUE metablock and the processing of free space accounts. The redundant block owner may be the node responsible for processing the redundant copy of the information for the free space account of a given SUE metablock. The redundant block owner can be promoted or changed to the new main block owner after the node of the previous main block owner fails. The main mapping owner may be the owner of the mapping information for a given logical address. The owner of the secondary mapping may be the owner of the redundant mapping information.

Figure 30 is a flowchart of a block life cycle according to an embodiment.

Block 3010 is a hot block of one of the blocks to which incoming host data is written. Block 3020 is one of multiple normal blocks that are inactive in the system where the host data resides. In an exemplary implementation, most host data resides in normal blocks and normal blocks constitute most of the blocks in the system.

The block 3030 is one of a plurality of sequence lists of blocks selected for recycling (for example, as part of the collection of discarded items) by the block selection mechanism. These blocks can be selected based on having the most amount of free space. The block 3040 is a recovery block involved in the process of moving valid data from the selected block to the cold block. The cold block 3080 is one of the blocks to which the valid data from the reclaimed block is written. The erased block 3050 is a block in the sequence table of multiple blocks that are erased after valid data is transmitted to the cold block 3080.

In one embodiment, the space allocation for a given open block can be allocated by a single thread on a single appliance or system. In an exemplary implementation, for a single block, each node may have one block designated as a hot block, and each node may have one block designated as a cold block. The main block owner can delegate the allocation to the block user and the allocator used for a given block can be the block user. The distribution itself can be single-threaded and the processing of messages can be parallelized. If the main block owner fails, the block is considered closed and no further writes to the block will occur until recycling (for example, collection of discarded items).

In one embodiment, the concept of the main block owner (the block that manages the life cycle of the block and the block handled by the free space account) can be separated from the space allocator. Therefore, the metablocks can be made smaller and larger. Metablocks can still span multiple drives and appliances or nodes, which are only divided into smaller blocks. In an exemplary implementation, instead of 2K blocks, there are 2K*N' blocks (where N'=N is rounded to the nearest power of 2 or 1<<log2(N-1)+1). A node can perform allocation from its "own" hot block and cold block. The data itself can still be spread across multiple nodes. The parity data is written to the node itself. If the redundancy level is greater than 1, then additional redundancy calculations on individual nodes can be performed.

Figure 31 is a block diagram illustrating a metablock of a distributed system according to one embodiment. The Distributed Selective Basic Exposure (DSUE) metablock can be implemented in a 4-node system, and the original SUE metablock definition can be subdivided into 4 new blocks forming the DSUE metablock.

For example, there may be 4 nodes 3110, 3120, 3130, and 3140. The node 3110 may include SSDs 3111, 3112, 3113, and 3114. The node 3120 may include SSDs 3121, 3122, 3123, and 3124. The node 3130 may include SSDs 3131, 3132, 3133, and 3134. The node 3140 may include SSDs 3141, 3142, 3143, and 3144. The original SUE metablock 3151 can be subdivided into 4 new blocks (3152, 3153, 3154, and 3155) forming the DSUE metablock. When nodes are added or deleted such that the power of 2 is crossed, the new block generated will adopt the new size. Block selection will be based solely on percentages.

The mapping data can be logically distributed redundantly across multiple appliances or systems. Each appliance or system can redundantly manage a certain part of the mapping data. The appliance or system can calculate the primary and redundant owners of the mapped data. Hashing can be used to calculate the primary and redundant owners of the mapped data. For reading, you can query these appliances or systems to find mapping information. For writing, these appliances or systems can record (journal) the mapping update and finally write the mapping data.

The multi-mode approach can be implemented in multiple RAID versions. RAID calculations can use Reed Solomon (RS) erasure codes for data protection. The RS encoding calculation can be performed by the block owner. In an exemplary implementation, a failure backup can be used (this is because the loss of the main appliance or system does not lead to data loss). If more than one symbol is to be corrected (for example, RAID 6), redundant calculations can be used.

In one embodiment, the failed unit may be regarded as a single storage node and the failure of a single storage node may not shut down the appliance or the system. The distributed architecture can also handle the blast radius problem. The burst radius involves how many appliances or systems are affected by the failure of a single component (for example, in data centers, facilities, networks). The burst radius in the data center can be considered similar to an enterprise high availability (HA) problem. Even if failure can be handled through replication and other techniques, the concept of influence among components is still important relative to usability issues. In the distributed appliance or system approach, the workload can be shared across N appliances or systems to allow M failures, thereby obtaining NM capacity. Usually on less customized hardware, redundancy is considered to be at a higher or architectural level rather than a lower level.

In a distributed appliance or system approach, free space management can be more complicated than a non-distributed approach. In one embodiment, free space account processing may occur at the SUE address block level. Physical blocks can be allocated between appliances or systems. Each appliance or system can track the free space to track the "F" block. In one embodiment, free space tracking or account processing can be expressed by the following relationship: F free space blocks for each appliance or system are equal to N physical blocks multiplied by M redundant blocks divided by P appliances or systems. For a particular configuration, the more exact number of appliances or systems that track free space for a given block can be calculated by any appliance or system in the storage array.

Distributed appliances or system methods can also support various recycling activities (for example, collection of waste items). Recycling can be initiated by the main block owner, but managed by the block user. Recycling can start after the block is allocated to the cold band for users of a given block.

Block users can trigger distributed block selection. Blocks can be assigned block selection scores based on various criteria (for example, valid space, invalid space). Each appliance or system can calculate a block selection score for a block within its range (that is, a block it is tracking free space to track). The block can be selected based on the block selection score, and the instruction of the selected block can be returned to the main block owner who then performs another overall block selection for recycling. In one example, the block with the best block selection score can be returned to the main block owner, and the overall best block can be selected for recycling or discarded item collection. The indication of the selected block can be passed to the appliance or system, which then starts its recycling of the physical fragments of the super block (which can be similar to a non-distributed multi-mode storage management system). Each appliance or system can perform a physical scan, search to determine whether it is valid, send an allocation request to the main block owner, send the data to the appropriate node, and send the mapping update to the appropriate node. Each appliance or system itself can throttle between the host and the recovered data.

In one embodiment, there may be two selected blocks. One block can be in the recycling process and one block can be ready for the recycling process. Each node will schedule the order of the selected blocks for recycling, but will allocate space in its own cold block. There may be optimizations that can be used to maintain the data in both the request and the response sequentially. There may be other possible filtering conditions, such as data obsolescence, volume/logical clustering, snapshot clustering, etc.

In the write process, the write command can reach any appliance or system. The appliance or system can become an orchestrator for commands throughout the entire flow of the write process. Consistent hashing can be used to identify the appropriate node for a given piece of data. In an exemplary implementation, the hashing solution has an improved property that only 1/n index keys are moved for adding or deleting objects (where n is the number of slots in the hash table).

Figure 32 is a diagram for adding and removing consistent hashing in nodes according to one embodiment. Physical free space account processing can be evenly distributed across multiple appliances or systems using redundant copies on different random systems via consistent hashing. The mapping of the logical address space can be evenly distributed across multiple appliances or systems using redundant copies on different random systems via consistent hashing.

Figure 33 is a flowchart of an exemplary write process 3300 according to one embodiment. In block 3310, a write instruction can be received. Write instructions can come from the host and the appliance can compress data. In block 3320, a write allocation process can be performed. In block 3330, a write data transfer process can be executed. In block 3340, a write map update process can be performed. In block 3350, the process of writing to the free space account can be executed. In block 3370, the write completion process can be executed.

FIG. 34 is a flowchart of an exemplary write allocation process 3400 according to one embodiment. In block 3410, the first appliance may request an allocation in the hot block that is turned on. In block 3420, the second appliance may perform the distribution stroke. In block 3430, the first appliance may receive distribution information from the second appliance.

FIG. 35 is a block diagram illustrating an exemplary implementation of a distributed system that implements a write allocation stroke operation according to an embodiment.

The distributed system 3500 may include a network 3590 and a distributed array 3599. The distributed array 3599 may include appliances 3510, 3520, 3530, 3540, 3550, 3560, 3570, and 3580. Each of the appliances 3510, 3520, 3530, 3540, 3550, 3560, 3570, and 3580 may include a multi-mode storage management system and multiple storage devices (for example, 3591, 3592, 3593, and 3594). In an exemplary implementation, each of appliances 3510, 3520, 3530, 3540, 3550, 3560, 3570, and 3580 may be similar to appliance 2701.

In step 1, the host can write 100 KB of data to the appliance 3510. In step 2, the appliance 3510 can compress the data. In step 3, the appliance 3510 may request an allocation in the hot block. In step 4, the appliance 3530 can perform the distribution and in step 5, the distribution response can be sent back to the appliance 3510. In step 6, the appliance 3510 may receive the distribution response.

FIG. 36 is a flowchart of an exemplary write data transfer process 3600 according to an embodiment. In block 3610, the data can be sent to the allocated appliance. In block 3620, the allocated appliance can write data. In block 3630, data can be sent for parity calculation.

FIG. 37 is a block diagram of an exemplary implementation of a distributed system that implements a write data transfer process operation according to an embodiment. In step 7, the appliance 3510 may send the data to the allocated appliance. In step 8, the instruments 3520, 3530, and 3540 can write data. In step 9, the appliance 3510 can send data to the appliance 3570 for parity calculation, and in step 10, the appliance 3570 can write the parity information.

FIG. 38 is a flowchart of an exemplary write map update process 3800 according to one embodiment. In block 3810, the mapping update can be sent to the primary and redundant owners. In block 3820, the primary and redundant owners can journal the mapping update and can write the mapping later. In block 3830, the primary and redundant owners can send responses. In block 3840, responses can be accumulated.

FIG. 39 is a block diagram of an exemplary system implementation of a write map update stroke operation according to an embodiment. In step 11 of Figure 39, the appliance 3510 may send the mapping update to the primary and redundant owners. In step 12, the appliances 3540, 3560, and 3570 can write mapping updates, and in block 13, they can send responses. In step 14, the appliance 3510 may accumulate responses.

FIG. 40 is a flowchart of an exemplary write free space account process 4000 according to one embodiment. In block 4010, free space updates can be sent. In block 4020, the appliance can update the free space. In block 4030, the appliance can send a response. In block 4040, responses can be accumulated.

Figure 41 is a block diagram of an exemplary system that implements the operation of writing a free space account processing program according to one embodiment. In step 15, the appliance 3510 may send a free space update. In step 16, the appliances 3520, 3560, and 3570 can update the free space, and in step 17, they can send a response. In step 18, the appliance 3510 may accumulate responses.

FIG. 42 is a flowchart of an exemplary write completion process 4200 according to one embodiment. In block 4210, the appliance that writes the data can send an instruction indicating that the data is safely written. In block 4220, responses can be accumulated. In block 4230, the appliance can send an instruction indicating that the parity is securely written. In block 4240, the command completion instruction can be forwarded.

FIG. 43 is a block diagram of an exemplary system for implementing a write completion stroke operation according to an embodiment. In step 19, the tools 3520, 3530, and 3540 that write the data can forward instructions for safe storage of the written data. In step 20, the appliance 3510 can accumulate responses, and in step 21, the appliance 3570 that stores the location can receive the parity safety instruction. In step 22, the command completion instruction can be forwarded to other storage nodes, other devices, or any external system via the network 3590.

FIG. 44 is a block diagram of an exemplary recovery initialization process 4400 according to one embodiment. When the block becomes ready for recycling, it can be guided to the free block list. Depending on how many hot and cold blocks are currently active and the number of blocks in the free block list, the main block owner at the front end of the erase queue can promote itself to a hot block or Cold block. If a block becomes a cold block, it can trigger a recovery process.

FIG. 45 is a flowchart of an exemplary recovery process 4500 according to one embodiment. In step 4510, the block can be promoted to a cold block, and block selection can be requested when necessary. In step 4520, block selection of potential blocks can be performed for recycling and the result can be forwarded. In step 4530, block selection and selection can be performed. In step 4540, the selected block may be broadcast. In step 4550, recycling can start and a mapping lookup can be issued. In step 4560, the mapping response can be processed. In step 4570, the write stroke can be recovered.

Figures 46 to 51 are block diagrams illustrating an exemplary recycling implementation according to one embodiment. Physical free space account processing can be distributed evenly across the system using redundant copies on different random systems via hashing. Regarding the mapping, the logical address space can be evenly distributed across the system by hashing using redundant copies on different random systems. In one example, block 3520 BB may be the next in the list of free blocks and may decide to become a cold block.

FIG. 46 is a block diagram showing the broadcast from the appliance 3520 that the block has become a cold block in step 51 and the block selection is requested if necessary. When the recovery is likely to be in progress, the allocation request generated by the promotion to a cold block can start extremely quickly, because the selected block will already exist in the queue.

FIG. 47 is a block diagram showing that other appliances perform block selection in step 52 and send the result back to the appliance 3520. When recycling is not running and block selection is necessary, the system or appliance (including the requesting system or appliance) can perform block selection. In step 53, the appliance 3520 can select a block.

FIG. 48 is a block diagram illustrating an exemplary broadcast of the block selected in step 54.

FIG. 49 is a block diagram illustrating that the appliance 3540 sends a mapping search to the appliance 3510 in step 55 based on the header that determines whether the data is valid. It should be understood that multiple appliances can be recycled and the appliance 3540 is shown for simplicity. The appliance 3540 may start to issue a mapping lookup (using a hidden header (eg, header 1511 of FIG. 15) to determine the first header) at the beginning of the selected block.

FIG. 50 is a block diagram illustrating step 57 of the appliance 3540 processing the mapping response from the appliance 3510. If the data is invalid, the itinerary can move to the next header (assuming it is valid). It is possible that data can overflow to the appliance 3550. If the data is valid, the second part of the data can be extracted from the appliance 3550 (the sequence is not shown).

Fig. 51 is a block diagram illustrating the recovery write process. Except that the original location information is sent as part of the mapping update to allow obsolete updates to be discarded, the steps for reclaiming the write stroke can be similar to the host write operation. In step 71, the appliance 3520 may perform an allocation, and in step 72, the appliance 3540 may request an allocation in the cold block. In step 73, the appliance 3520 may send a distribution response, and in step 74, the appliance 3540 may receive distribution information.

The presented multi-modal approach includes multiple features and characteristics that promote effective and efficient storage of information. The features and characteristics can improve many different qualities associated with performance. In one embodiment, the flexibility of the described segmentation method allows for relatively fast and manageable complexity. A relatively large amount of user data can be stored in the SUE address space for extremely fast storage and management of user data. On the other hand, a relatively small amount of metadata can be stored in a logically addressed area, allowing the system to take full advantage of the abstraction nature of metadata for reduced complexity.

In addition, increasing the flexibility of over-provisioning in a relatively small metadata area produces a much larger percentage of the impact of over-provisioning, which helps speed up metadata storage operations and reduce compensation speed, that is, it will be attributed in other ways As the complexity increases, the speed decreases. Compared with other storage systems, this situation allows a better overall allocation and better impact of over-provisioned resources. Flexibility can also facilitate improved life cycle retention by allowing different storage areas of a block to be reassigned or re-allocated between two partitions. The nature (e.g., type) of the data stored in the area may indicate that one type of data is written/erased less frequently than another type of data in the area (e.g., most metadata is compared to user data No major changes), and the physical blocks in one partition can be reassigned to another partition to balance the wear and tear on a specific area. Flexibility also allows power cycling improvements by moving power cycling responsibilities up to the system level.

In one embodiment, the multi-mode selective basic exposure storage device or method may include multiple network attached storage (NAS) devices (for example, storage 100, 300, 400, 600, 700) In distributed systems or shared storage systems (for example, NAS systems or clusters). Multi-mode selective basic exposure NAS devices can provide data storage services for other devices on the network (for example, clients, servers, etc.). Storage services can include the storage, management, and maintenance of extremely large data sets (often referred to as "big data"). In an exemplary implementation, the multi-mode selective basic exposure NAS device can be included in the online transaction processing (OLTP) system of the general database for storing and maintaining transaction data, or can be a part of the online transaction processing system. part. A database management system (DBMS) or a distributed DBMS can split a database across a cluster of computer systems or storage devices in OLTP.

Ability to flexibly reconfigure or move different types of data and/or storage management operations to different parts of the system (for example, the Internet, network connection "cloud", data center, network, etc.) to achieve a variety of greater capabilities. The multi-mode selective basic exposure storage method realizes the utilization of applications that involve splitting different types of data and address access. In one embodiment, the type of information (such as "user data") that is considered to be big (big or large) can be stored in a part of the system (eg, cloud, data center, or network) and be Other types of data that are considered small (such as "metadata") can be handled in another part of the system (for example, the cloud, data center, or network). For example, in the past, user data and metadata were usually not split or accessed in different ways.

In the case of a multi-mode selective basic exposure system, the "big data" that is often accessed can be better suited for storing this type of information (for example, physical addressing) or closer to being able to better handle all the information. The storage area handling of the processing capacity of the information (thereby reducing the delay and the impact on network communication). "Small data" or metadata can be stored in a smaller storage area that is logically accessed with reduced complexity.

Some parts of the detailed description are presented based on the operating procedures, logical blocks, strokes, and other symbolic representations of the data bits in the computer memory. These descriptions and expressions are generally used by those who are familiar with data processing technology to effectively transfer the subject of their work to other people who are familiar with the technology. The program, logic block or itinerary is here and is usually conceived as a self-consistent sequence of steps or instructions leading to a desired result. The steps include the physical manipulation of the physical quantity. Usually (but not necessarily), these physical quantities take the form of electrical, magnetic, optical, or quantum signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system. It has been proved that it is convenient to refer to these signals as bits, values, elements, symbols, characters, items, numbers, or the like, mainly for common usage reasons.

However, it should be borne in mind that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to such quantities. Unless specifically stated, as is obvious from the following discussion, it should be understood that throughout this application, the use of terms such as "processing", "arithmetic", "calculation", "determination", "display" or the like refers to manipulation And the actions and strokes of computer systems or similar processing devices (for example, electrical, optical, or quantum computing devices) that transform data expressed as physical (for example, electronic) quantities. The term refers to the manipulation or transformation of physical quantities in components of a computer system (for example, registers, memory, other such information storage, transmission, or display devices) into other physical quantities that are similarly expressed as physical quantities in other components The movement and itinerary of the data processing device.

The configuration shown in each of the conceptual diagrams should be understood only from a conceptual point of view. In order to assist the understanding of this disclosure, the form, structure, and size of each component shown in each conceptual diagram have been exaggerated or reduced. The configuration actually implemented may have a physical shape different from the shape depicted in each conceptual diagram. Each concept diagram is not intended to limit the physical shape of the components.

The device configuration shown in each block diagram is provided to assist in understanding the content of this disclosure. Each block can be formed by smaller blocks according to its function. Alternatively, multiple blocks may form larger blocks according to functions. That is, the spirit or scope of the content of the disclosure is not limited to the configuration shown in the block diagram.

In the above, the present disclosure has been described based on some example embodiments. However, the purpose of the present disclosure can be achieved by means different from the above-mentioned example embodiments that include the subject matter of the present disclosure according to the nature of the technology. Therefore, it should be understood that the above example embodiments are illustrated in descriptive and illustrative views rather than restrictive views. That is, the spirit or scope that includes the subject matter of the disclosure and can reach the goal of the disclosure should be included in the spirit or scope of the inventive concept of the disclosure.

Therefore, the modified or changed technical concept that does not deviate from the scope or spirit of the content of the disclosure should be included in the scope of the patent application below. The scope of the disclosure is not limited to the above example embodiments.

<p>none</p>

The accompanying drawings incorporated in and forming a part of this specification are included for illustrative description of the principles of the present invention, and are not intended to limit the present invention to the specific implementation described therein. The drawings are not drawn to scale unless specifically indicated otherwise. FIG. 1 is a block diagram illustrating an exemplary storage device having an SUE storage partition according to an embodiment. FIG. 2 is a block diagram illustrating an exemplary multi-mode storage device according to an embodiment. FIG. 3 is a block diagram illustrating another exemplary multi-mode storage device according to an embodiment. FIG. 4 is a block diagram illustrating an exemplary multimode solid state drive (MM-SSD) according to an embodiment. FIG. 5 is a block diagram illustrating an exemplary process of translating address space information into logical address space information according to an embodiment. Figure 6 is a block diagram illustrating an exemplary system according to one embodiment. Figure 7 is a block diagram illustrating a system according to one embodiment. FIG. 8 is a flowchart describing a method of driving a multi-mode base exposure according to an embodiment. FIG. 9 is a block diagram illustrating an exemplary multi-mode SSD device. FIG. 10 is a block diagram depicting an exemplary SUE block and a corresponding SUE page for storage in the user area of a multi-mode storage device according to an embodiment of the present invention. FIG. 11 is a block diagram depicting an exemplary SUE block and a corresponding SUE page of a user storage space used for storage in a user area of a multi-mode storage device according to an embodiment of the present invention. FIG. 12 is a block diagram depicting an exemplary SUE metapage and a corresponding SUE page for storage in the user area of the multi-mode storage device according to an embodiment of the present invention. FIG. 13 is a block diagram depicting an exemplary SUE metablock and corresponding SUE meta page for storage in a user area of a multi-mode storage device according to an embodiment of the present invention. 14 is a block diagram depicting another exemplary SUE metablock and corresponding SUE block for storage in the user area of the multi-mode storage device according to an embodiment of the present invention. 15 is a block diagram depicting an exemplary SUE mapping scheme that can be implemented by a multi-mode storage system to provide address mapping from logical addresses to SUE addresses according to an embodiment of the present invention. FIG. 16 is a schematic diagram depicting an exemplary storage system that can implement the SUE mapping scheme of FIG. 15. FIG. 17 is a flowchart showing an exemplary method for mapping a logical address space to an SUE address space according to an embodiment of the present invention. 18 is a schematic diagram illustrating an exemplary multi-mode storage management system using an SUE addressing scheme to allow the storage system to address the address logic and SUE storage space in the storage device according to an embodiment of the present invention. 19 is a schematic diagram illustrating another exemplary multi-mode storage management system that uses an SUE addressing scheme to allow the storage system to address the address logic and SUE storage space in the storage device according to an embodiment of the present invention. 20 is a schematic diagram illustrating another exemplary multi-mode storage management system that uses an SUE addressing scheme to allow the storage system to address the address logic and SUE storage space in the storage device according to an embodiment of the present invention. FIG. 21 is a schematic diagram illustrating a user area access manager (UAAM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. FIG. 22 is a schematic diagram illustrating a user area mapping engine (UAME) that can be implemented by a multi-modal storage management system according to an embodiment of the present invention. FIG. 23 is a schematic diagram illustrating a metablock manager (MBM) that can be implemented by a multi-modal storage management system according to an embodiment of the present invention. 24 is a schematic diagram illustrating a storage device control manager (SCM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. 25 is a diagram illustrating a storage device access manager that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. device access manager; SAM) schematic diagram. FIG. 26 is a schematic diagram illustrating a global state manager (GSM) that can be implemented by a multi-mode storage management system according to an embodiment of the present invention. Figure 27 is a block diagram of an exemplary appliance coupled to a network according to one embodiment. Figure 28 is a block diagram of a distributed system according to one embodiment. Figure 29 is a block diagram of a distributed system according to one embodiment. Figure 30 is a flowchart of a block life cycle according to an embodiment. Figure 31 is a block diagram of a distributed system metablock according to one embodiment. Fig. 32 is a block diagram of consistent hash utilization when adding and removing nodes according to one embodiment. Figure 33 is a flowchart of an exemplary write process according to one embodiment. Figure 34 is a flowchart of an exemplary write allocation process according to one embodiment. FIG. 35 is a block diagram of an exemplary implementation of a system for implementing a write allocation stroke operation according to an embodiment. Figure 36 is a flowchart of an exemplary write allocation process according to one embodiment. FIG. 37 is a block diagram of an exemplary implementation of a system for implementing a write allocation stroke operation according to an embodiment. FIG. 38 is a flowchart of an exemplary write map update process according to an embodiment. FIG. 39 is a block diagram of an exemplary implementation of a system for implementing a write map update run operation according to an embodiment. FIG. 40 is a flowchart of an exemplary process of writing a free space account according to an embodiment. FIG. 41 is a block diagram of an exemplary implementation of a system for implementing a process operation of writing a free space account according to an embodiment. FIG. 42 is a flowchart of an exemplary write completion process according to an embodiment. FIG. 43 is a block diagram of an exemplary implementation of a system for implementing a write completion stroke operation according to an embodiment. FIG. 44 is a block diagram of an exemplary recovery initialization process according to an embodiment. Figure 45 is a flowchart of an exemplary recovery process according to one embodiment. FIG. 46 is a block diagram of a broadcast from an appliance showing that a block has become a cold block and a request for block selection is necessary. FIG. 47 is a block diagram showing other appliances that perform block selection and send the result back to the appliance. Fig. 48 is a block diagram illustrating an exemplary broadcast of selected blocks. FIG. 49 is a block diagram illustrating an appliance that sends a mapping search to the appliance based on a header that determines whether the data is valid. Figure 50 is a block diagram illustrating an appliance that handles a mapped response from another appliance. FIG. 51 is a block diagram illustrating a write-recovery process according to an embodiment.

<bio-deposit></bio-deposit>

<sequence-list-text></sequence-list-text>