The following is derivative of extended conversations and tutoring by Martin Peres ( irc_nick : mupuf ).
This article is an introductory discussion of nvidia gpus with respect to nouveau. Nouveau is the open-source driver which runs the Nvidia cards on your linux pc. A schematic breakdown of the nvidia architecture can be understood through this figure.
The first important repository is the envytools (https://github.com/pathscale/envytools), this contains the most valuable set of tools and documentation you ll find for/on nvidia cards.
To have a look at how the X Window System ( X11 - current major version ) GUI framework works, let's say all the rendering is done by sending commands to the x-server. Applications connect to the x-server through a socket which allocates a context for their windows and issues commands like draw a line etc. For 2D rendering, the x-server has the EXA interface. The EXA can be accelerated by graphics drivers and the acceleration is done by DDX modules. DDX is device dependent X, the part of X that interacts with the hardware. It is a set of loadable modules that drive the hardware and are found in a separate repository than the main x-server. The ddx then either sends commands to the card directly (pre-drm) or sends them to the kernel which will then send them to the card (drm). DRM (Direct Rendering Manager) is a component of the Direct Rendering Infrastructure which is a system to provide efficient video acceleration. It consists of two in-kernel modules, a drm driver and one which has hardware specific support for video acceleration. This pair of drivers allow a userspace client to access the video hardware directly.
The x-server runs as a root and hence can communicate with the GPU directly. Nouveau is the open-source driver for nvidia gpus and the ddx for it is called xf86-video-nouveau (xf86 = xfree 86). When the ddx communicates with the kernel, it goes through the libdrm that translates some calls to IOCTLs (they are input/output system control calls which are used for device-specific operations which cannot be expressed by regular system calls). Libdrm provides a userspace interface to kernel DRM services. There aren't many drm modules, the main ones are radeon, intel and nouveau. We have 3 nouveau specific components i.e. nouveau ddx, libdrm_nouveau and nouveau drm.
Till now we have been discussing the method for 2D rendering. To discuss 3D, it is important to note that applications do no connect to the x-server through xlib anymore instead they connect to libgl, which is a part of mesa. Mesa is a software library for 3D computer graphics that provides a generic OpenGL implementation for rendering three-dimensional graphics on multiple platforms. A flowchart representation for better understanding can be found here.
Mesa is basically a giant compiler which takes opengl commands as input and gives some shader code + memory management as output. If we look at mesa classics, we ll see intel, radeon, nouveau_vieux, swrast these drivers get opengl commands then compile everything, execute some runtime code and generate shader codes.
A few years ago, people started working on a new compiler-like architecture which is now known as Gallium. Gallium is a set of intermediate representations and sort of a generic ISA. The state trackers expose an interface to applications and translate things to the gallium ISA. When a new graphic card arrives on the market, all they have to do is compile the ISA to the ISA of their card. Most drivers are using the infrastructure and all the active ones are using gallium. Only the intel driver is still using the classic MESA infrastructure. Nouveau has several mesa drivers.
nouveau_vieux = tnt2 -> geforce 4
geforce 5, 6, 7 is using the just-released
nv30 driver
geforce 8 --> 400 (Fermi) is using nv50
geforce 400 --> the latest card is using
the nvc0 driver
That marks an end to one part of our introduction, now we move on to codenames and vocabulary. We understand the nvidia chipset's code names by an example.
'geforce gtx 460'
geforce : Basic distinction being geforce and quadro. Quadro stands for professional cards and geforce are manufactured for the average user. Mobile quadro is just a geforce.
gtx: It gives a basic idea of the speed and performance of the card. It is not very important.
400: It tells us the chipset architecture of the card. 400 series are mostly based on Fermi architecture.
60: It gives us the speed index.
The register 0 of the card stores some information on the chipset. Let us take an example of a gtx460 which stores the value 0c4100a1. The c4 is actual internal name of the chipset at nvidia so it belongs to the nvc4 family and that is the only main thing that matters. Chipset families are a group of chipsets which behave mostly in the same manner. The first real family was nv30 then came the nv40, nv50, now the nvc0 (Fermi) and nve0 (Kepler). The following page provides a basic distinction of family based on the code names: http://nouveau.freedesktop.org/wiki/CodeNames
It is time we started talking about engines. When the host computer send the commands to the gpu, these commands go through the PFIFO (Packet First In First Out queing discipline). The PFIFO decodes these instructions and routes them to the right engines. Some of the major engines are described below:
PGRAPH = 3D engine (opengl/directx cast in silicon :P )
PDISPLAY = It manages the CRTCs (CRT Controllers) and thus the screens themselves.
There are video-decoding engines like the PMPEG or PVDEC, PVP, VUC,VP. PCOUNTER is particularly of our interest and should be read about.
Now once we discuss the different ISAs that are to be found in the GPUs, go through the documentation would be relatively easy.
So, there is the shader ISA that keeps changing with every new family. This ISA has become so generic that people started using it to do general computing GPGPU/CUDA/OpenCL. We also have the HWSQ which is a simple ISA that can only assign values to registers. HWSQ was introduced on nv20 and retired in nvc0. The new ISA that got introduced is Fµc, it used to stand for Fermi microcode as Fermi requires microcodes written in fµc to operate properly but it was actually introduced in nv98 for doing general cryptography operations (PCRYPT). PDAEMON was the first useful user and it got introduced in nva3. It functions as RTOS (Real-Time Operating System) in the card that can do about anything.
In the beginning, we had discussed that each application is allocated a context before drawing anything as the GPU can handle many applications at one time. Stop a process, switch to a new one , finish it and get back to the original one through the use of channels. Up to nv30, this context switching was done by the host computer and the GPU would only issue an interrupt request. In nv40, it got better and the GPU manages the context switching on her own. So now there is another microcode in the card that handles context switching and it is referred to as ctxprogs. These ctxprogs have been used up to Fermi where they were replaced by fµc. Nowadays all engines are becoming software which is good and bad for us. Good because it gives us more control over the hardware and bad because the code is becoming more complex.
The idea behind microcodes is that they are all over the card and are being increasingly used in the card. Almost all engines of the Kepler family are using fµc which is very good.
https://github.com/pathscale/envytools/blob/master/hwdocs/nvbars.txt will give you a basic start on how to interact with the card. If you use mmap bar0 you can access all the registers. As we modify the registers by mapping them to our memory it is called Memory Mapped Input/Output (MMIO) as opposed to Programmed Input/Output (PIO).
So, let's talk about the tools:
nvapeek = Reads a register from a card
nvalist = lists the nvidia gpus connected
nvapeek -c X Y Z :
x = card number (by default, it is 0);
Y = base address to be read;
Z = how many bytes should be read
nvapeek e114 10 will read registers e114, e118, e11c, e120.
nvapoke = write to a register
nvapoke -c X Y Z:
Y = address of the reg
Z = value to be poked
nvagetbios > vbios.rom : will get your vbios from PRAMIN
add "-s prom" and it will read it from PROM (the EEPROM containing the vbios)
nvawatch = read a reg in a tight loop and display the value every time it changes
nvbios= decodes partially your vbios
lookup is a nice tool
rnndb is the register database it is the database for lookup
lookup -a NVxx reg value
will tell you the name of the reg and what the value means
to reverse engineer, we often use MMIO traces (recording every access the proprietary driver aka ‘the blob’ does to the PCIE port)
these traces can be decoded with demmio which is basically lookup but for every line of the multi-million line trace you may generate with a mmiotrace.
This article is an introductory discussion of nvidia gpus with respect to nouveau. Nouveau is the open-source driver which runs the Nvidia cards on your linux pc. A schematic breakdown of the nvidia architecture can be understood through this figure.
The first important repository is the envytools (https://github.com/pathscale/envytools), this contains the most valuable set of tools and documentation you ll find for/on nvidia cards.
To have a look at how the X Window System ( X11 - current major version ) GUI framework works, let's say all the rendering is done by sending commands to the x-server. Applications connect to the x-server through a socket which allocates a context for their windows and issues commands like draw a line etc. For 2D rendering, the x-server has the EXA interface. The EXA can be accelerated by graphics drivers and the acceleration is done by DDX modules. DDX is device dependent X, the part of X that interacts with the hardware. It is a set of loadable modules that drive the hardware and are found in a separate repository than the main x-server. The ddx then either sends commands to the card directly (pre-drm) or sends them to the kernel which will then send them to the card (drm). DRM (Direct Rendering Manager) is a component of the Direct Rendering Infrastructure which is a system to provide efficient video acceleration. It consists of two in-kernel modules, a drm driver and one which has hardware specific support for video acceleration. This pair of drivers allow a userspace client to access the video hardware directly.
The x-server runs as a root and hence can communicate with the GPU directly. Nouveau is the open-source driver for nvidia gpus and the ddx for it is called xf86-video-nouveau (xf86 = xfree 86). When the ddx communicates with the kernel, it goes through the libdrm that translates some calls to IOCTLs (they are input/output system control calls which are used for device-specific operations which cannot be expressed by regular system calls). Libdrm provides a userspace interface to kernel DRM services. There aren't many drm modules, the main ones are radeon, intel and nouveau. We have 3 nouveau specific components i.e. nouveau ddx, libdrm_nouveau and nouveau drm.
Till now we have been discussing the method for 2D rendering. To discuss 3D, it is important to note that applications do no connect to the x-server through xlib anymore instead they connect to libgl, which is a part of mesa. Mesa is a software library for 3D computer graphics that provides a generic OpenGL implementation for rendering three-dimensional graphics on multiple platforms. A flowchart representation for better understanding can be found here.
Mesa is basically a giant compiler which takes opengl commands as input and gives some shader code + memory management as output. If we look at mesa classics, we ll see intel, radeon, nouveau_vieux, swrast these drivers get opengl commands then compile everything, execute some runtime code and generate shader codes.
A few years ago, people started working on a new compiler-like architecture which is now known as Gallium. Gallium is a set of intermediate representations and sort of a generic ISA. The state trackers expose an interface to applications and translate things to the gallium ISA. When a new graphic card arrives on the market, all they have to do is compile the ISA to the ISA of their card. Most drivers are using the infrastructure and all the active ones are using gallium. Only the intel driver is still using the classic MESA infrastructure. Nouveau has several mesa drivers.
nouveau_vieux = tnt2 -> geforce 4
geforce 5, 6, 7 is using the just-released
nv30 driver
geforce 8 --> 400 (Fermi) is using nv50
geforce 400 --> the latest card is using
the nvc0 driver
That marks an end to one part of our introduction, now we move on to codenames and vocabulary. We understand the nvidia chipset's code names by an example.
'geforce gtx 460'
geforce : Basic distinction being geforce and quadro. Quadro stands for professional cards and geforce are manufactured for the average user. Mobile quadro is just a geforce.
gtx: It gives a basic idea of the speed and performance of the card. It is not very important.
400: It tells us the chipset architecture of the card. 400 series are mostly based on Fermi architecture.
60: It gives us the speed index.
The register 0 of the card stores some information on the chipset. Let us take an example of a gtx460 which stores the value 0c4100a1. The c4 is actual internal name of the chipset at nvidia so it belongs to the nvc4 family and that is the only main thing that matters. Chipset families are a group of chipsets which behave mostly in the same manner. The first real family was nv30 then came the nv40, nv50, now the nvc0 (Fermi) and nve0 (Kepler). The following page provides a basic distinction of family based on the code names: http://nouveau.freedesktop.org/wiki/CodeNames
It is time we started talking about engines. When the host computer send the commands to the gpu, these commands go through the PFIFO (Packet First In First Out queing discipline). The PFIFO decodes these instructions and routes them to the right engines. Some of the major engines are described below:
PGRAPH = 3D engine (opengl/directx cast in silicon :P )
PDISPLAY = It manages the CRTCs (CRT Controllers) and thus the screens themselves.
There are video-decoding engines like the PMPEG or PVDEC, PVP, VUC,VP. PCOUNTER is particularly of our interest and should be read about.
Now once we discuss the different ISAs that are to be found in the GPUs, go through the documentation would be relatively easy.
So, there is the shader ISA that keeps changing with every new family. This ISA has become so generic that people started using it to do general computing GPGPU/CUDA/OpenCL. We also have the HWSQ which is a simple ISA that can only assign values to registers. HWSQ was introduced on nv20 and retired in nvc0. The new ISA that got introduced is Fµc, it used to stand for Fermi microcode as Fermi requires microcodes written in fµc to operate properly but it was actually introduced in nv98 for doing general cryptography operations (PCRYPT). PDAEMON was the first useful user and it got introduced in nva3. It functions as RTOS (Real-Time Operating System) in the card that can do about anything.
In the beginning, we had discussed that each application is allocated a context before drawing anything as the GPU can handle many applications at one time. Stop a process, switch to a new one , finish it and get back to the original one through the use of channels. Up to nv30, this context switching was done by the host computer and the GPU would only issue an interrupt request. In nv40, it got better and the GPU manages the context switching on her own. So now there is another microcode in the card that handles context switching and it is referred to as ctxprogs. These ctxprogs have been used up to Fermi where they were replaced by fµc. Nowadays all engines are becoming software which is good and bad for us. Good because it gives us more control over the hardware and bad because the code is becoming more complex.
The idea behind microcodes is that they are all over the card and are being increasingly used in the card. Almost all engines of the Kepler family are using fµc which is very good.
https://github.com/pathscale/envytools/blob/master/hwdocs/nvbars.txt will give you a basic start on how to interact with the card. If you use mmap bar0 you can access all the registers. As we modify the registers by mapping them to our memory it is called Memory Mapped Input/Output (MMIO) as opposed to Programmed Input/Output (PIO).
So, let's talk about the tools:
nvapeek = Reads a register from a card
nvalist = lists the nvidia gpus connected
nvapeek -c X Y Z :
x = card number (by default, it is 0);
Y = base address to be read;
Z = how many bytes should be read
nvapeek e114 10 will read registers e114, e118, e11c, e120.
nvapoke = write to a register
nvapoke -c X Y Z:
Y = address of the reg
Z = value to be poked
nvagetbios > vbios.rom : will get your vbios from PRAMIN
add "-s prom" and it will read it from PROM (the EEPROM containing the vbios)
nvawatch = read a reg in a tight loop and display the value every time it changes
nvbios= decodes partially your vbios
lookup is a nice tool
rnndb is the register database it is the database for lookup
lookup -a NVxx reg value
will tell you the name of the reg and what the value means
to reverse engineer, we often use MMIO traces (recording every access the proprietary driver aka ‘the blob’ does to the PCIE port)
these traces can be decoded with demmio which is basically lookup but for every line of the multi-million line trace you may generate with a mmiotrace.
Comments
Post a Comment