Varying effective resolution by screen location by changing active color sample count within multiple render targets

[Cross-reference of related applications]

This application is filed on the same date as this application, the entire contents of which are incorporated herein by reference, "METHOD FOR EFFICIENT CONSTRUCTION OF HIGH RESOLUTION DISPLAY BUFFERS". In connection with US Patent Application No. 14 / 246,064 (Agent Document No. SCEA13055US00) pending simultaneous pending by the same applicant by Tobias Berghoff, named.

This application was filed on April 5, 2014, and the entire contents of which are incorporated herein by reference, "GRAPHICS PROCESSING ENHANCEMENT BY TRACKING OBJECT AND / OR PRIMITIVE IDENTIFIERS (graphics by object tracking and / or primitive identifier). Related to US Patent Application No. 14 / 246,067 (Agent Document No. SCEA13056US00) pending simultaneous pending by the same applicant by Tobias Berghoff, entitled "Improvement of Processing".

This application was filed on April 5, 2014, and the entire contents of which are incorporated herein by reference, "GRADIENT ADJUSTMENT FOR TEXTURE MAPPING TO NON-ORTHONORMAL GRID (texture mapping slope for non-orthonormal lattices). Coordination) by Mark Evan Cerny, in connection with US Patent Application No. 14 / 246,068 (Agent Document No. SCEA13057US00) pending simultaneous pending by the same applicant.

This application was filed on April 5, 2014, and the entire contents of which are incorporated herein by reference, "VARYING EFFECTIVE RESOLUTION BY SCREEN LOCATION BY ALTERING RASTERIZATION PARAMETERS" Related to US Patent Application No. 14 / 246,063 (Agent Document No. SCEA13059US00), simultaneously pending by the same applicant, by Mark Evan Cerny, entitled "Position-Variable Effective Resolution".

This application was filed on April 5, 2014, and the entire contents of which are incorporated herein by reference, "VARYING EFFECTIVE RESOLUTION BY SCREEN LOCATION IN GRAPHICS PROCESSING BY APPROXIMATING PROJECTION OF VERTICES ONTO CURVED VIEWPORT". US Patent Application No. 14 / 246,066 (agent document number) co-pending by the same applicant by Mark Evan Cerny, entitled "Effective resolution that varies with screen position in graphic processing by projecting vertices close to the port" It is related to the SCEA13060US00).

This application was filed on April 5, 2014, and the entire contents of which are incorporated herein by reference, "GRADIENT ADJUSTMENT FOR TEXTURE MAPPING FOR MULTIPLE RENDER TARGETS WITH RESOLUTION THAT VARIES BY SCREEN LOCATION (depending on screen position). Related to US Patent Application No. 14 / 246,062 (Agent Document No. SCEA13061US00), co-pending by the same applicant, by Mark Evan Cerny, entitled "Multiple Rendering Texture Mapping Tilt Adjustment for Targets of Different Resolutions". ..

[Technical field]

Aspects of the present disclosure relate to computer graphics. In particular, the present disclosure relates to changing the resolution depending on the position of the screen.

Graphics processing usually requires the coordination of two processors, a central processing unit (CPU) and a graphics processing unit (GPU). A GPU is a dedicated electronic circuit designed to accelerate the generation of an image in a framebuffer intended for output to a display. GPUs are used in embedded systems, mobile phones, personal computers, tablet computers, portable gaming devices, workstations, and game consoles. GPUs are typically designed to be efficient when working with computer graphics. GPUs often have a highly parallel processing architecture, which makes the GPU more efficient than general-purpose CPUs for algorithms in which large blocks of data are processed in parallel.

The CPU may send GPU instructions, commonly referred to as drawing commands, which render, for example, a specific texture that has changed for the previous frame in the image to implement a specific graphics processing task. Command the GPU. These drawing commands may be linked by a CPU with a graphics application programming interface (API) to issue graphics rendering commands that correspond to the state of the virtual environment of a particular application.

To render textures for a particular program, the GPU may perform a series of processing tasks in a "graphics pipeline" to transform the visuals in the virtual environment into images that could be rendered on the display. Good. A typical graphics pipeline is a rendering or shading operation on a virtual object in virtual space, transformation and rasterization of the virtual object in the scene to produce pixel data suitable for display, and subject. It may include performing additional rendering tasks on the pixels (or fragments) before outputting the rendered image to the display.

Virtual objects in an image are often described in virtual space by shapes known as primitives that together shape the objects in the virtual scene. For example, an object in a rendered 3D virtual world may be decomposed into a series of separate triangular primitives with vertices defined by coordinates in 3D space, thereby making these polygons the surface of the object. To configure. Each polygon may have an associated index that may be used by the graphics processing system to distinguish a given polygon from other polygons. Similarly, each vertex may have an associated index that may be used to distinguish a given vertex from other vertices. The graphics pipeline may perform some operations on these primitives to generate visuals about the virtual scene and convert this data into a two-dimensional format suitable for pixel reproduction on the display. The term graphics primitive information (or simply "primitive information") as used herein is used to refer to data that represents a graphics primitive. Such data includes, but is not limited to, vertex information (eg, data representing vertex position or vertex index) and polygon information, such as polygon index, and information relating a particular vertex to a particular polygon.

As part of the graphics pipeline, GPUs may perform rendering tasks by implementing programs commonly known as shaders. A typical graphics pipeline is a vertex shader, which may manipulate certain properties of a primitive on a per-vertical basis, a vertex shader, as well as a pixel shader (also known as a "fragment shader"). It may include a pixel shader that operates downstream from the vertex shader in the graphics pipeline and may manipulate a value pixel by pixel before sending the pixel data to the display. Fragment shaders may manipulate the values associated with applying textures to primitives. Pipelines are likewise geometric shaders that use the output of vertex shaders to generate a new set of primitives, as well as compute that may be implemented by the GPU to perform some other common computational task. Other shaders at various stages in the pipeline, such as shaders (CS), may be included.

<p> Graphical display devices with a wide field of view (FOV) have been developed. Such devices include head-mounted display (HMD) devices. In an HMD device, a small display device is worn on the user's head. The display device has display optics in front of one eye (monocular HMD) or both eyes (binocular HMD). HMD devices typically include sensors that may detect the orientation of the device and change the scene shown by the display optics as the user's head moves. As usual, most stages of rendering a scene for a large FOV display are performed by planar rendering, where all parts of the scene have the same number of pixels per unit area.</p><p> To provide a realistic experience, it is desirable that the graphics presented by the wide FOV display device be rendered in high quality and efficiently.</p><p> It is in this context that this disclosure emerges.</p>

<p> The teachings of the present invention may be readily understood by considering the following detailed description with the accompanying drawings.</p>

<figref num="1A-1B">FIG. 6 is a schematic showing certain parameters of a wide field of view (FOV) display.</figref><figref num="1C">FIG. 5 shows different solid angles for different parts of a wide FOV display.</figref><figref num="2A-2C">It is a figure which shows the example of the relative importance of the pixel in the different area of the display of a different wide FOV by the aspect of this disclosure.</figref><figref num="2D">It is a figure which shows the example of the different pixel resolution for the different area of the screen of a certain FOV display by the aspect of this disclosure.</figref><figref num="3A">It is a block diagram of the graphics processing system according to the aspect of this disclosure.</figref><figref num="3B">It is a block diagram of the graphics processing pipeline by the aspect of this disclosure.</figref><figref num="4A">It is a figure which shows typically the example which changes the effective resolution by the position of a screen by changing the active color sample count in a plurality of render targets by the aspect of this disclosure.</figref><figref num="4B">It is a figure which shows typically the example which changes the effective resolution by the position of a screen by changing the active color sample count in a plurality of render targets by the aspect of this disclosure.</figref><figref num="4C">It is a figure which shows typically the example which changes the effective resolution by the position of a screen by changing the active color sample count in a plurality of render targets by the aspect of this disclosure.</figref><figref num="4D">It is a schematic diagram which shows the example of the metadata composition which implements the pixel active sample count which fluctuates by the position of the screen by the aspect of this disclosure.</figref><figref num="4E">FIG. 5 is a schematic diagram showing an alternative example of a metadata configuration that implements pixel active sample counts that vary with screen position according to aspects of the present disclosure.</figref>

Although the detailed description below includes many specific details for illustration purposes, one of ordinary skill in the art will recognize that many modifications and modifications to the following details are within the scope of the present invention. Accordingly, exemplary embodiments of the invention described below are described without losing generality to the claimed invention and without imposing restrictions on the claimed invention.

Preface Figures 1A-1C show previously unrecognized problems with large FOV displays. Figure 1A shows a 90 ° FOV display and Figure 1B shows a 114 ° FOV display. In traditional large FOV displays, 3D geometry is rendered using a planar projection onto the view plane. However, rendering geometry on a high FOV view plane has proven to be very inefficient. As seen in FIG. 1C, the edge region 112 and the central region 114 of the view plane 101 are the same area but show very different solid angles, as seen by the observer 103. As a result, pixels near the edges of the screen retain much more meaningless information than pixels near the center. Traditionally when rendering a scene, these areas have the same number of pixels and the same amount of time is spent rendering areas of the same size.

Figures 2A-2C show the relative importance of different parts of a large FOV display in two dimensions for different sized fields of view. FIG. 2A shows the variation in solid angle for each square of the planar checkerboard perpendicular to the view direction when the checkerboard is at an angle of 114 °. In other words, Figure 2A shows the inefficiency of traditional plane projection rendering for 114 ° FOV displays. Figure 2B represents the same information about a 90 ° FOV display. In such a plane projection rendering, the projection compresses the tile 202 in the image 201 at the edges and the tile 203 at the corners to a smaller solid angle compared to the tile 204 at the center. Due to this compression, and because each tile in image 201 has the same number of pixels in screen space, there is almost four times the inefficiency factor to render the edge tile 202 compared to the center tile 204. To do. This means that the conventional rendering of the edge tile 202 requires approximately four times as much processing per unit solid angle as that of the center tile 204. In the case of square tile 203, the inefficiency factor is almost eight times. When averaged over all images 201, the inefficiency factor is approximately 2.5 times.

The inefficiency depends on the size of the FOV. For example, in the 90 ° FOV display shown in FIG. 2B, the inefficiency factor is approximately double when rendering edge tile 202, approximately triple when rendering square tile 203, and when rendering image 201. It is almost 1.7 times.

Another way to see this situation is shown in Figure 2C, where the screen 102 was divided into rectangles of approximately equal "importance" in terms of the number of pixels per stretched unit solid angle. Each rectangle makes about the same contribution to the final image seen through the display. It may be possible to see how plane projection distorts the importance of the edge rectangle 202 and the corner rectangle 203. In practice, the square rectangle 203 is a central rectangle due to display optics that may choose to increase the visual pixel density (expressed as the number of pixels per solid angle) towards the center of the display. May make a small contribution to.

Based on previous observations, image 210 for a wide FOV display is smaller in edge regions 212, 214, 216, 218 than in central region 215, and also in edge regions 212, 214, 216, as shown in FIG. 2D. , 218 to have a smaller pixel density in the angular regions 211, 213, 217, and 219. Traditional graphical images on the screen of a wide FOV display to have the same effect as varying pixel density across screens without the need to significantly modify the underlying graphical image data or data format or data processing. Rendering is equally advantageous. According to aspects of the present disclosure, these advantages may be obtained in the graphics pipeline by varying the number of active color samples for which pixel shaders are called for different areas of the screen of a large FOV display device. There is.

To implement this, some graphics pipelines use metadata that specifies the number of active color samples per pixel in different areas of the screen. Metadata is related to the screen, not the image. Pixel shader execution is performed only on the active color sample in the graphics pipeline, rather than changing the image data. For example, in image data, there may be four color samples per pixel. For the full resolution area of the screen, the metadata may specify an active count of 4, in which case the pixel shader will be called for all four color samples. In the 3/4 resolution area, the active count can be 3, in which case the pixel shader will be called for 3 out of 4 (eg, the first 3) color samples. In the 1/2 resolution area, the active count will be 2, and the pixel shader will be called for two of the color samples. In the 1/4 resolution area, the active count will be 1, and the pixel shader will be called for only one of the four color samples.

Systems and Devices Aspects of the present disclosure include a graphics processing system configured to implement graphics processing with variable pixel sample resolution. As an example and not as a limitation, FIG. 3A shows a block diagram of a computer system 300 that may be used to implement graphics processing according to aspects of the present disclosure. According to aspects of the present disclosure, the system 300 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable game device, a workstation, a game console, or the like.

The system 300 may generally include a central processing unit (CPU) 302, a graphics processor unit (GPU) 304, and a memory 308 accessible by both the CPU and GPU. The CPU 302 and GPU 304 may each include one or more processor cores, such as a single core, two cores, four cores, eight cores, or more. The memory 308 may be in the form of an integrated circuit that provides addressable memory, such as RAM, DRAM, and the like. Memory 308 may include graphics memory 328 that stores graphics resources and may temporarily store graphics buffer 305 of data for the graphics rendering pipeline. The graphics buffer 305 is, for example, a vertex buffer that stores vertex parameter values, an index buffer that holds vertex indexes, a depth buffer that stores depth values of graphics content (eg, Z buffer), a stencil buffer, and completion that is sent to the display. It may include a frame buffer for storing frames, and other buffers. In the example shown in FIG. 3A, the graphics memory 328 is shown as part of the main memory. In an alternative embodiment, the graphics memory may be a separate component, perhaps integrated into the GPU 304.

As an example, and not as a limitation, the CPU 302 and GPU 304 may use the data bus 309 to access memory 308. In some cases, it may be useful for the system 300 to include two or more different buses. Memory 308 may contain data that may be accessed by CPU 302 and GPU 304. The GPU 304 may include a plurality of compute units configured to perform graphics processing tasks in parallel. Each compute unit may include its own dedicated local memory store, such as a local data share.

The CPU may be configured to execute CPU code 303c, which may include graphics, a compiler, and a graphics API. The graphics API may be configured to issue drawing commands to a program implemented by the GPU. CPU code 303C may implement physics simulation and other functions as well. The GPU 304 may be configured to operate as discussed above. In particular, the GPU may execute GPU code 303G, which may implement shaders such as the compute shader CS, vertex shader VS, and pixel shader PS discussed above. To facilitate the passage of data between the compute shader CS and the vertex shader VS, the system may include one or more buffers 305, which may include framebuffer FB. GPU code 303G may also optionally implement other types of shaders (not shown) such as pixel shaders or geometric shaders. Each compute unit may include its own dedicated local memory store, such as a local data share. The GPU 304 may include a texture unit 306 configured to perform certain operations to apply the texture to the primitive as part of the graphics pipeline.

According to aspects of the disclosure, CPU code 303c and GPU code 303g and other elements of system 300 specify an active sample configuration for a particular region of display device 316 within multiple regions of the display device. Is configured to be received by the rasterization stage of the graphics pipeline. The rasterization stage receives pixel data for one or more pixels in a particular area. Pixel data specifies the same sample count (the number of color samples for each pixel) across all surfaces. The active sample count is less than or equal to the color sample count, and the color sample count is two or more. For each pixel in a particular area, the rasterization stage calls the pixel shader PS only for the active sample. The metadata MD specifies different active sample configurations for regions with different pixel sample resolutions (the number of pixel samples per unit area of the display). Thus, the pixel sample resolution can vary for different regions of the display device 316, and the graphics processing load simply counts the active sample count for the low resolution regions of the display compared to the high resolution regions. It may be reduced by reducing it.

In some embodiments, the metadata MD comprises a mask of the active sample for each region. GPU304 performs a logical AND between the mask and the sample covered by the primitive, and the pixel shader determines the active sample for the primitive called for it.

In an alternative embodiment, the metadata MD specifies an active sample count for a particular region of display device 316 among multiple regions of the display device. The active sample count is less than or equal to the color sample count, and the color sample count is two or more. For each pixel in a particular region, the rasterization stage calls the pixel shader PS only for a certain number of color samples of pixels equal to the active sample count, and usually sequential color samples are defined consistently. Start with the first color sample in the sample order.

In some embodiments, the CPU code 303c, the GPU code 303g, and the texture unit 306 are to implement some texture mapping operation to implement a modification to the texture mapping operation along with a screen position-dependent variable pixel resolution. It may be further configured. For example, pixel shader PS and texture unit 306 generate one or more texture coordinate UVs for pixel position XY for a primitive to provide a set of coordinates for one or more texture mapping operations, and texture coordinate UVs (perhaps). The gradient value Gr may be calculated from (including corrections to deal with various sample densities across the screen) and the texture may be configured to determine the level of detail (LOD) to which the primitive is applied.

As an example, and not as a limitation, certain components of the GPU, such as certain types of shaders or texture units 306, are application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or system on a chip. It may be implemented as dedicated hardware such as a chip (SoC or SOC).

As used herein and, as is generally understood by those of skill in the art, application specific integrated circuits (ASICs) are customized for specific use rather than intended for general purpose use. It is an integrated circuit to be used.

As used herein and generally understood by those skilled in the art, field programmable gate arrays (FPGAs) were designed to be configured by a customer or designer after manufacture-thus, " It is a "field programmable" integrated circuit. FPGA configurations are typically specified using a hardware description language (HDL) similar to the language used for ASICs.

As used herein and generally understood by those skilled in the art, system-on-a-chip or system-on-chip (SoC or SOC) is all computers or other electronic systems. It is an integrated circuit (IC) that integrates the components of the system on a single chip. The SoC or SOC may include digital signals, analog signals, and mixed signals, and often radio frequency functions, all on a single chip substrate. A typical application is in the area of embedded systems.

A typical SoC includes the following hardware components: One or more processor cores (for example, a microcontroller, microprocessor, or digital signal processor (DSP) core). Memory blocks, such as read-only memory (ROM), random access memory (RAM), electronically erasable programmable read-only memory (EEPROM), and flash memory. Timing sources such as oscillators or phase-locked loops. Peripherals such as counter timers, real-time timers, or power-on reset generators. External interfaces, such as industry standards such as Universal Serial Bus (USB), Firewire, Ethernet®, Universal Asynchronous Receiver / Transmitter (USART), Serial Peripheral Interface (SPI) Bus. An analog interface that includes an analog-to-digital converter (ADC) and a digital-to-analog converter (DAC).

Voltage regulators and power management circuits These components are connected by exclusive or industry standard buses. Direct memory access (DMA) controllers route data directly between external interfaces and memory, bypassing processor cores, thereby increasing SoC data throughput.

A typical SoC has both the hardware components described above and the executable instructions (eg, software or firmware) that control the processor core (s), peripherals, and interfaces. Including.

According to aspects of the present disclosure, some or all of the functionality of the shader or texture unit 306 may instead be implemented by properly configured software instructions executed by a software programmable general purpose computer processor. These instructions may be embodied in a computer-readable medium, such as memory 308 or storage device 315.

System 300 may also include well-known support function 310, which may also communicate with other components of the system, for example, by bus 309. Such support functions may include, but are not limited to, input / output (I / O) elements 311, power supply (P / S) 312, clock (CLK) 313, and cache 314. In addition to cache 314, GPU 304 may include its own GPU cache 314G, which may be configured so that programs running on GPU 304 can read or write through GPU cache 314G.

System 300 may include display device 316 to present the rendered graphics 317 to the user. In an alternative embodiment, the display device 316 is a separate component that works with system 300. The display device 316 may display a flat panel display, a head-mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other visible text, numbers, graphical symbols, or images. It may be in the form of a device. In a particularly useful embodiment, the display device 316 is a large field of view (FOV) device with a curved screen. The display device 316 displays the rendered graphic image 317 processed according to the various techniques described herein.

System 300 may optionally include mass storage devices 315 such as disk drives, CD-ROM drives, flash memories, tape drives, etc. for storing programs and / or data. System 300 may also optionally include user interface unit 318 to facilitate interaction between system 300 and the user. The user interface unit 318 may include a keyboard, mouse, joystick, light pen, game controller, or other device that may be used with a graphical user interface (GUI). System 300 may also include network interface 320, which allows the device to communicate with other devices through network 322. The network 322 may be, for example, a local area network (LAN), a wide area network such as the Internet, a personal area network such as a Bluetooth® network, or another type of network. These components may be hardware, software, or firmware, or any combination of two or more of them.

Graphics Pipeline According to aspects of the present disclosure, the system 300 is configured to implement a predetermined portion of the graphics rendering pipeline. FIG. 3B shows an example of a graphics rendering pipeline 330 according to aspects of the present disclosure.

Rendering pipeline 330 renders graphics as an image depicting a scene with 2D or preferably 3D geometry within virtual space (sometimes referred to herein as "world space"). It may be configured in. The early stages of the pipeline also include operations performed in virtual space before the scene is rasterized and transformed into screen space as a set of discrete pixels suitable for output on display device 316. Good. Throughout the pipeline, the various resources contained in the graphics memory 328 may be utilized in the pipeline stage, and the inputs and outputs to the stage are contained in the graphics memory before the final value of the image is determined. It may be temporarily stored in the buffer.

The rendering pipeline may act on input data 332, which may contain one or more virtual objects defined by a set of vertices, one or more virtual objects being set up in virtual space and a scene. It has a geometry defined with respect to the coordinates within. The early stages of the pipeline may include those broadly classified as vertex processing stage 334 in FIG. 3B, which may also include various calculations to process the vertices of an object in virtual space. This may include a vertex shading calculation 336, which includes the position values (eg, XY and Z depth values), color values, lighting values, texture coordinates, etc. of the vertices in the scene. Various parameter values may be manipulated. Preferably, the vertex shading calculation 336 is performed by one or more programmable vertex shaders. The vertex processing stage may optionally include additional vertex processing calculations such as tessellation and geometry shader calculations 338, which may optionally be used to generate new vertices and new geometries in virtual space. Good. At the end of the stage called Vertex Processing 334, at this stage of the pipeline, the scene is defined by a set of vertices, each with a set of vertex parameter values 339.

Pipeline 330 may then proceed to rasterization processing stage 340, which involves transforming scene geometry into scene space and discrete pixels, i.e. a set of pixels. Virtual space geometry may be transformed into screen space geometry through operations that may essentially calculate the projection of objects and vertices from virtual space onto the observation window (or "viewpoint") of the scene. The vertices may be defined as a set of primitives.

The rasterization process stage 340 shown in FIG. 3B may include a primitive assembly operation 342 that may set up primitives defined by each set of vertices in the scene. Each vertex may be specified by an index, and each primitive may be specified for these vertex indexes, which may be stored in the index buffer in graphics memory 328. Primitives may preferably include at least triangles, each defined by three vertices, but may also include point primitives, line primitives, and other polygonal shapes. During the primitive assembly operation 342, a primitive may be arbitrarily selected. For example, a primitive whose index indicates a certain winding order may be considered backwards or may be selected from the scene.

After the primitive has been assembled, the rasterization processing stage may include a scan transformation operation 344, which samples the primitive at each pixel for further processing once the sample is covered by the primitive. Fragments (sometimes called pixels) may be generated from primitives. Optionally, multiple samples of each pixel are taken without primitives during scan transformation operation 344, and those samples may be used for antialiasing. In certain embodiments, different pixels may be sampled differently. For example, some edge pixels may include a lower sampling density than the center pixel to optimize some form of rendering for some type of display device 316, such as a head-mounted display (HMD). Fragments (or "pixels") generated from a primitive during scan transformation 344 may have parameter values that may be interpolated from the vertex parameter values 339 of the vertices of the pixel-generating primitive to the position of the pixel. Good. The rasterization stage 340 may include 346 parameters interpolation operations to calculate these interpolated fragment parameter values 349, which the interpolated fragment parameter values 349 are for further processing in later stages of the pipeline. It may be used as an input for.

According to aspects of the present disclosure, certain operations are performed between the primitive assembly 342 and the scan transform 344 to address that different subsections of the screen have different pixel resolutions. In certain embodiments, once the position of the screen with respect to the vertices of the primitive is known, coarse rasterization 343 is performed and all pre-defined screen subsections with which the primitives overlap (coarse rasterization herein). May find (sometimes called tiles). For each subsection in which the primitives overlap, a subsection-dependent metadata MD that allows the effective resolution to be modified for that subsection, such as the active sample count or other parameters, is received. Scan Transform 344 and subsequent processing stages generate final pixel values by performing pixel processing only on a specified number of active samples for one or more related subsections.

The graphics pipeline 330 further manipulates the interpolated parameter value 349, including the additional pixel processing operations shown overall in 350 in Figure 3B, and how fragments contribute to the final pixel value for display 316. Further calculations may be performed to determine. Some of these pixel processing tasks may include pixel shading calculations 352 that may be used to further manipulate the interpolated parameter value 349 of the fragment. Pixel shading calculations may be performed by programmable pixel shaders, and pixel shader calls 348 may be initiated based on primitive sampling during rasterization stage 340. As mentioned earlier, the pixel shader call 348 is also initiated based on the metadata MD that specifies the active sample count for each pixel in a particular area of display device 316 to which the primitive is rendered. You may. For each pixel in a particular area, the pixel shader call 348 only occurs for a certain number of color samples of pixels equal to the active sample count.

FIG. 4A shows an example of how the metadata MD can be configured to specify different active color samples for different regions 401 of the display screen 316. In some embodiments, each region may correspond to a fixed size portion of the display. In other embodiments, each region may correspond to a variable size portion of the display. In a further embodiment, the metadata MD may define each region 401 by a range of pixels in the vertical and horizontal directions. In yet further embodiments, the metadata MD may define each region by a coarse rasterized tile of some size, eg, 32 pixels x 32 pixels. Metadata associated with a particular region contains information that specifies the active color sample count for that region. As an example and not as a limitation, the metadata may be stored in the form of a table in memory 308 and / or graphics memory 328.

As an example, and not as a limitation, each pixel 403 in each area 401 of the display screen 316 may be specified to have eight depth samples and four color samples. Metadata for any particular region may specify an active color sample count of 1, 2, 3, or 4 for that region, depending on the desired resolution of that region. In the example shown in FIG. 4A, the central region of screen 316 is desired to have full resolution, so the metadata MD specifies an active color sample count of 4 for these regions.

In these embodiments, much of the processing within the graphics pipeline 330 occurs as usual. For example, other parts of the rasterization process 340, such as primitive assembly 342 and scan conversion 344 and parameter interpolation 346, will be implemented as before. The screen will have a single pixel format, eg, color sample count, depth sample count, position, and other parameters. A new feature of this aspect of the disclosure is that for pixel shader calls 348 in each region 401, the metadata MD specifies the active color sample count and the pixel shaders are called as many times as the active color sample count.

In some embodiments, the pixel shader call 348 is unrolled by default and a depth sample is always written. If the call is always unrolled, the pixel shading calculation 352 supersamples the pixels by being called for each active color sample, as opposed to multisampling the pixels.

For example, if the metadata MD specifies 4 color samples per pixel for the central region 401C and 2 color samples per pixel within the edge region 401E, it will be in the central region 401C compared to the edge region 401E. It is equivalent to doubling the resolution in the horizontal and vertical directions.

Figure 4B shows an example of a single sample per pixel image, and a 2 pixel x 2 pixel quad 406 shows four color samples that completely describe the four pixels. Depth samples are indistinguishable from color samples in this respect in the examples. For triangle 405, this one quad is a pixel for pixel shading calculation 352 as a single fragment with three covered samples, because three of the four pixel positions are covered by the triangle. Passed to shader PS. The pixel shader PS shades four color samples and stores three of the four color samples in the framebuffer FB.

In Figure 4C, in contrast, there are four color samples per pixel, so the 16 samples shown correspond only to 2x2 pixels, while in the example of Figure 4B it is 4x4 pixels. Met. This figure shows that when the active pixel is unrolled as a pixel shader call occurs for each covered sample, three fragments are generated and each fragment contains one covered sample.

The active sample count can fluctuate by selectively disabling the samples specified by the metadata MD. For example, if the upper right and lower left of each pixel are rendered inactive, then only one active sample position is covered, so only the center of the three fragments shown in Figure 4C is passed to the pixel shader PS. In some embodiments, the metadata is fixed for the optics and FOV of display 316. An example of such a metadata structure is shown schematically in Figure 4D. FIG. 4D shows an example of how the metadata MD may be configured to specify different active pixel samples (or active color samples) for different subsections 401 of the display screen 316. In the example shown in FIG. 4D, the central subsection of screen 316 is desired to have full resolution, and the subsections far from the center have progressively lower resolutions. As an example, and not as a limitation, each pixel 403 in each area 401 of the display screen 316 is specified to have a fixed number of depth and color samples, eg, 8 depth samples and 4 color samples. there is a possibility. Metadata for any particular region may specify an active color sample count of 1, 2, 3, or 4 for that region, depending on the desired resolution of that region.

In an alternative embodiment, the metadata may vary to implement fove rendering for eye tracking. In such an embodiment, the system 300Includes hardware to track the user's gaze, that is, where the user's eyes are looking, and associate this information with the corresponding screen position the user is looking at. An example of such hardware could include a digital camera that is in a known position with respect to the screen of display device 316 and is oriented in the user's overall direction. The digital camera may be part of the user interface 318 or a separate component. CPU code 303C may include image analysis software, which analyzes the image from the camera and (a) whether the user is in the image and (b) whether the user is facing the camera. Please, (c) whether the user is facing the screen, (d) whether the user's eyes are observable, (e) the orientation of the pupil of the user's eyes with respect to the user's head, and (f) with respect to the camera. Determine the orientation of the user's head. From the known position and orientation of the camera with respect to the screen, the orientation of the pupil of the user's eyes with respect to the user's head, and the orientation of the user's head with respect to the camera, the image analysis software can determine whether the user is looking at the screen. If so, it may also determine screen space coordinates for portion 401 of the screen that the user is viewing. CPU code 303C then passes these screen coordinates to GPU code 303G, which may determine one or more subsections containing part 401. Since the GPU code can then modify the metadata MD accordingly, the pixel resolution will be highest in one or more subsections, including part 401, as shown in Figure 4E, part 401. Gradually lower in subsections far from.

Seeing Figure 3B again, pixel shading calculation 352 outputs values to one or more buffers 305 in graphics memory 328, sometimes referred to as render targets, or multiple render targets (MRTs) if there are more than one. It may be output. MRT allows pixel shaders to optionally output to two or more render targets that are two or more render targets, each with the same screen dimensions, but probably with different pixel formats. .. The render target format limitation is that any one render target can only accept up to four independent output values (channels), and that the formats of these four channels are tightly coupled with each other. Often means. MRT allows a single pixel shader to output more values in a mixture of different formats. The format of the render target is that the render target stores multiple values for screen spatial pixels, but for various performance reasons, the format of the render target has become more specialized in recent hardware generations, depending on the texture unit. It is "texture-like" in that it sometimes (but not always) requires something called a "resolve" that reforms the data before it fits into the read.

Pixel processing 350 may reach render output operation 356, which may include what is commonly known as raster operation (ROP). The rasterization operation (ROP) is only performed once for each render target in multiple render targets (MRTs) and multiple times for pixels. During output operation 356, the final pixel value 359 may be determined within the framebuffer, which optionally includes fusion fragments, applied stencils, depth testing, and processing tasks per sample. Good. The final pixel value of 359 includes the collected output for all active render targets (MRTs). The GPU 304 uses the final pixel value of 359 to configure the finished frame 360, which may optionally be displayed in real time on the pixels of the display device 316.

The output operation 350 may also include a texture mapping operation 354, which to some extent has one or more shaders (eg, pixel shader PS, compute shader CS, vertex shader VS, or other). It may be carried out by the type shader) and to some extent by the texture unit 306. The shader calculation 352 includes calculating the texture coordinate UV from the screen space coordinate XY, transmitting the texture coordinate to the texture operation 354, and receiving the texture data TX. The texture coordinate UV can be calculated in any way from the screen space coordinate XY, but is usually calculated from the interpolated input value, and sometimes from the result of the previous texture operation. The gradient Gr is often calculated directly from the texture coordinate quad by the texture unit 306 (texture computing hardware unit), but optionally, it is explicitly calculated by the pixel shader calculation 352 and may rely on the texture unit 306. It is possible that the default calculation will be performed without being passed to the texture calculation 354.

The texture operation 356 generally includes the following stages that may be performed by some combination of the pixel shader PS and the texture unit 306. First, one or more texture coordinate UVs for pixel position XY are generated and used to provide a coordinate set for each texture mapping operation. The gradient value Gr is then calculated from the texture coordinate UV (possibly with a correction for the non-orthogonality of the sample position) and used to determine the level of detail (LOD) to which the texture applies the primitive.

A further aspect of the present disclosure comprises a graphics processing method, wherein the method receives metadata specifying an active sample configuration for a particular area of the display device among multiple areas of the display device. Specifying the same number of color samples for each pixel for one or more pixels in a particular area, receiving pixel data, and for each pixel in a particular area, with an active sample by active sample configuration Includes calling the pixel shader only for the color samples specified to be.

A further aspect of the present disclosure comprises a graphics processing method in which different areas of the screen of a display device have different pixel resolutions.

Another further aspect is a computer-readable medium, the computer-readable medium having computer-executable instructions embodied within the computer-readable medium, and the computer-executable instructions, when executed, in the following manner: Implement one or both.

A further embodiment is an electromagnetic or other signal carrier computer readable instruction that performs one or both of the following methods.

A further embodiment is a computer program product that is downloadable from a communication network and / or stored on a computer-readable and / or microprocessor executable medium, and is a program code instruction that implements one or both of the above methods. It is a computer program product characterized by containing.

Another further aspect is a graphics processing system configured to implement one or both of the previous methods.

The above is a complete description of preferred embodiments of the invention, but various alternatives, modifications, and equivalents can be used. Therefore, the scope of the invention should not be determined with reference to the above description, but instead with reference to the scope of the appended claims, along with the full scope of its equivalents. The optional features described herein (favorable or unfavorable) may be combined with any other features described herein (favorable or unfavorable). In the appended claims, [indefinite article "A" or "An"] refers to one or more quantities of items following an article, unless otherwise stated explicitly. The scope of the appended claims is the means plus function limitation unless the means plus function limitation is explicitly listed in a given patent claim using the phrase "means for". Is not interpreted as containing.