Fundamentals of Graphics Programming

This page is an overview of some of the fundamentals in Graphics Programming. While there are some code samples, this section doesn't go through the specific C++ Code and OpenGL API calls used to access the Render Pipeline, and is just meant to be a conceptual overview of the project. To see the actual code, please checkout the project on GitHub (CSC4900 Graphics Engine Building )

Linear Algebra

There are a lot of technical elements to graphics programming, especially when it comes to 3D Math. Before getting into the details, ensure that you have a basic understanding of linear algebra, and working with vectors, matrices, and matrix transformation. You don't need to know exactly how the underlying math works, but just know the basics of how to use it. I'm going to assume for the sake of space that you have a basic understanding of these topics, though I will go over some topics in future sections.

Programming

You will need a good grasp of programming fundamentals. While I don't go into the specifics of OpenGL code here, the engine is built in C++, and any sample code in this section uses C-like syntax. Having a good grasp of C++ and the fundamentals of computer memory and memory management will be useful in better understanding the sections below.


When rendering a 3D object, we create what are known as meshes. Meshes are a collection of vertices (3D positions) that we wrap into triangles.

Here's a representation of a square mesh, also known as a quad:

In this example, the square is made up of 4 vertices and 2 triangles. For convenience, I labled the vertices A,B,C,D.
Let's say that C is the origin of our graph such that:

A = (0,1,0), B = (1,1,0), C = (0,0,0) and D = (1,0,0)

There are 2 ways of representing these meshes in memory. The first is relatvely simple, just store an array of vertices, with every 3 vertices making up a triangle.


    vec3[] vertices = { 
        A, B, D,    // The first triangle
        A, D, C     // The second triangle
    }
        

However, you'll notice that using this method, we end up storing some vertices twice, and when you start creating larger 3D meshes, this problem get's exponentially worse. It would be better to store each vertex only once. The second way of storing a mesh solves this problem.
In the second method, we use 2 arrays. The first array is a list of all the vertices in the mesh, the second is a list of indices that represent the triangles of the array.


    vec3[] vertices = { 
        A, B, C, D  // A list of the vertices with no duplicates
    }
    
    int[] triangles = {
        1, 2, 4,    // The first triangle
        1, 4, 3     // The second triangle
    }
        

One thing to note when creating 3D meshes, the direction in which you draw the triangles is important. All triangles must be wrapped in a clockwise direction, otherwise they will render inside out. The direction of the triangle is based on the order you store it in.

So, { A, D, B } is backwards since it is wrapped counter-clockwise, instead, you want to store it as { A, B, D }.

Also, as long as they are wrapped clockwise, the order doesn't matter, so { A, B, D } is equivilant to { D, A, B } since they are wrapped in the same directions.


If you've taken a course in Linear Algebra, you've probably heard of matrix transformation. If you are like me, however, you probably wondered how these transformations could be useful. In graphics programming matrix transformations are absolutely essential to everything we do. In this section, I am not going to go over how matrices or matrix transformations work. Instead, I will simply explain how we use them in when writing code. If you'd like to see a bit more of the math behind it, check out Learn OpenGL. He goes over the basics of transformations and how they work. For a more in-depth look into matrix math, there are plenty of resources and tutorials on the internet.

There are 3 core matrices used in rendering: the model matrix, view matrix, and projection matrix. Each of these matrices transforms points from one space to another, allowing graphics programmers to easily swap between different spaces. The model matrix transforms from local space to world space (Local Space refers to the mesh objects vertices origin. World Space refers to what the Graphics Programmer defines as the scenes actual space, based around the origin). The view matrix converts from world space to view space (View space refers to where the space about the camera's position in the world). The projection matrix converts the view space into clip space (Clip space is what the camera can actually see in the world). Finally, after all that is done, one more autmatic transformation is performed to convert clip space into actual screen space.
Here's a good diagram of this

By using these transformations, we can now create 3D meshes in their own "model" space, and easily convert each of it's vertices to world space. We can also take each of these vertices, and then convert them into coordinates that fit within the viewport of the users device. With these transformations, we now have the mathematical tools that we need in order to turn a 3D object into a 2D image on a screen.


In order to render objects using the GPU, we first have to get the mesh data to the GPU. In OpenGL, we do this using buffers. A GPU buffer is simply an array of bytes stored in the GPU memory for use in shader programs. Buffers can be used to store all sorts of data, but generally, we use it for storing vertex information. This vertex info is going to be stored in an array of values on the GPU. The most basic thing you need to store is just the position data of each vertex.

In the buffer below, you simply have a an array of floats, where every 3 floats represents a vertex.

However, when rendering a mesh, you often need more than just the vertex positions but sometimes want to store other information as well. For this reason, when we store vertices on the GPU, we also have the ability to store other data with each vertex. This data could be a color, a texture position (also known as a UV), a normal value (important for lighting), or any other data that can be represented as primitve data types with static size.

Here's an example of the array when storing more data:


When creating these arrays, it's important to keep track of the stride of your vertex data. The stride is the distance from the start of one vertex to the start of the next vertex. Since you might store different types of data (like floats or ints), the stride is simply the count of bytes your vertex takes up. In the example above, the stride is 9 * sizeof(float) or 36 bytes (since a float is 4 bytes).

Now the question is, how do we access and use this vertex data? In the next section we will cover how to create shaders for running programs on the GPU itself.


Now that we have our Mesh, and we've moved it to the GPU, we have to actually render it using the GPU. Shaders are programs stored on the GPU that can be run in parallel for large sets of data. This is what makes the GPU so efficient for rendering items on the screen. We could have thousands of vertices on the screen at a any given time. If we were to render each of these vertices one at it time, it would take forever to render even a single frame, let alone 60 frames each second. The GPU, however, has potentially thousands of cores that can all run in parallel. However, while it has a lot of cores, each of the cores is significantly smaller than a typical CPU core. As a result, programs run on the GPU are limited in size, and should primarily be used for performin mathmatical operations with as few branches as possible.

There are six stages to the render pipeline that all run in a linear sequence:

Vertex Shader -> Shape Assembly -> Geometry Shader -> Tests and Blending -> Fragment Shader -> Rasterization

For most graphics programmers, we only really care about the Vertex Shader and the Fragment Shader stages of the pipeline, everything else is done automatically for us (though occasionally we may modify the geometry shader, but I won't be going over that).
Before we get into the specifics of the Vertex and Fragment Shader, lets go over a few general concepts in he shader world: input, output, and uniforms.

Inputs and Outputs: Every stage of the shader process has a series of inputs and outputs. These inputs are the outputs created after the previous stage of the render pipeline completed. In general, you can think of the render pipeline as being a conveyer with a bunch of machines that moves the data along from one machine to another. Each of these machines takes in the data and transform it, until at the very end you have an image that can be drawn to the screen. So, the input of one machine is the output of another.

Uniforms: When writing shaders, there is often some data that will be the same for every vertex in a mesh. These are called uniforms. Uniforms can be used for all sorts of things. For example, you may want to color your whole object a single color. Rather than store a color value on every vertex, you can store one uniform color, and use that on each vertex in the shader. Uniforms are used all over the place, because often an object, while made up of many vertices, has object-wide properties that we don't want to store in every vertex.

Vertex Shader

The Vertex Shader is the first stage of the render pipeline and run over every vertex in your mesh. Since the vertex shader is the first stage of the pipeline it's input is provided from the CPU in the form of a vertex buffer. The primary output of the vertex shader is the screen-view position of the vertex (the normalized (x,y,z) position in relation to the OpenGL window also known as normalized device coordinates).
To achieve this output, we use the the matrix transformations above. However, all of these Matrices are used for transformations are kept on the CPU. This is where Uniforms come in handy. Before running the vertex shader, I make sure to create a Uniform for each of the matrices (model, view, and projection). Then, I simply multiply the vertex position by the matrices in the proper order.
Here's the Vertex Shader used for rendering a mesh written in GLSL 3:


    #version 330 core
    layout (location = 0) in vec3 pos;  //Vertex Position Input
    layout (location = 1) in vec2 uv0;  //Vertex Texture Position Attribute


    out vec2 texCoord;                  //Output Texture Coordinate to fragment shader

    uniform mat4 model;                 //The mesh model Matrix
    uniform mat4 view;                  //The scene view matrix
    uniform mat4 projection;            //The scene projection matrix


    void main()
    {
        //Set the texure Texture Coordinate output
        texCoord = vec2(uv0.x, uv0.y);
        
        //Set the default gl_position output 
        gl_Position = projection * view * model  * vec4(pos, 1.0f);     
    }
                


As you can see, the shader looks a lot like a simple C program. The key difference is the variables declared at the top are inputs and outputs. The 'pos' and 'uv0' are the vertex attribute input given. Inputs are defined using the 'in' keyword. The 'layout' keyword is used to tell the shader at what position in the vertex the input is located at. After the input comes the use defined output, which in this case is the texture coordinate (the 2D position in the texture that the vertex should display). There are also a few default ouputs, like gl_position is a that don't have to be declared. Finally, the Uniforms are declared, which in this case are the 3 transformation matrices we will be using to get the screen position of the vertex (gl_position).

After finishing in the Vertex Shader, the outputs are passed into the fragment shader.

Fragment Shader

The Fragment shader is one of the last stages of the render pipeline. The fragment shader is run for every fragment on the OpenGL window. For simplicity, you can think of these fragments as being the individual pixels on the screen (though sometimes they are different sizes). The primary output of the fragment shader is the color that the current fragment should be rendered as using an RGBA (red,green,blue,alpha) color encoding. We represent this color as a 4D Vector, where each position is a value between 0 and 1.
Here's the Fragment shader associated with the Vertex Shader above:


    #version 330 core

    in vec2 texCoord;               //The texture coordinate given to use by the vertex shader

    out vec4 FragColor;             //The output color that the fragment will be drawn with

    uniform vec4 ourColor;          //A Uniform that we use to modify the color of the entire object
    uniform sampler2D ourTexture;   //The texture to draw on the surface of the object

    void main()
    {
        //Set the color output to the texture color multiplied by object color
        FragColor = texture(ourTexture, texCoord) * ourColor;  
    }
                


This time, you'll notice I didn't define my inputs using a location. Instead, I just have to make sure my input name matches the output name of the vertex shader. The compiler will automatically assign the outputs of the vertex shader to the inputs of the fragment shader. I have only 1 output in the fragment shader, and thats the fragment Color I want to draw to the screen.

Using these 2 Shader's, we can create a Shader Program and compile it on the GPU. Once we do this, OpenGL give's us a Shader ID, which we can use to tell the GPU which shader program we want to use when rendering our objects.