The geometry shader is unique in contrast to the other shader types in that it processes a whole primitive (triangle, line, or point) at once and can actually change the amount of data in the OpenGL pipeline programmatically. A vertex shader processes one vertex at a time; it cannot access any other vertex’s information and is strictly one-in, one-out. That is, it cannot generate new vertices, and it cannot stop the vertex from being processed further by OpenGL. The tessellation shaders operate on patches and can set tessellation factors, but have little further control over how patches are tessellated, and cannot produce disjoint primitives. Likewise, the fragment shader processes a single fragment at a time, cannot access any data owned by another fragment, cannot create new fragments, and can only destroy fragments by discarding them. On the other hand, a geometry shader has access to all of the vertices in a primitive (up to six with the primitive modes GL_TRIANGLES_ADJACENCY andGL_TRIANGLE_STRIP_ADJACENCY), can change the type of a primitive, and can even create and destroy primitives.
Geometry shaders are an optional part of the OpenGL pipeline. When no geometry shader is present, the outputs from the vertex or tessellation evaluation shader are interpolated across the primitive being rendered and are fed directly to the fragment shader. When a geometry shader is present, however, the outputs of the vertex or tessellation evaluation shader become the inputs to the geometry shader, and the outputs of the geometry shader are what are interpolated and fed to the fragment shader. The geometry shader can further process the output of the vertex or tessellation evaluation shader, and if it is generating new primitives (this is called amplification), it can apply different transformations to each primitive as it creates them.
The Pass-Through Geometry Shader
As explained back in Chapter 3, “Following the Pipeline,” the simplest geometry shader that allows you to render anything is the pass-through shader, which is shown in Listing 8.16.
Listing 8.16. Source code for a simple geometry shader
#version 430 core
layout (triangles) in;
layout (triangle_strip) out;
layout (max_vertices = 3) out;
void main(void)
{
int i;
for (i = 0; i < gl_in.length(); i++)
{
gl_Position = gl_in[i].gl_Position;
EmitVertex();
}
EndPrimitive();
}
This is a very simple pass-through geometry shader, which sends its input to its output without modifying it. It looks similar to a vertex shader, but there are a few extra differences to cover. Going over the shader a few lines at a time makes everything clear. The first few lines simply set up the version number (430) of the shader just like in any other shader. The next couple of lines are the first geometry shader-specific parts. They are shown again in Listing 8.17.
Listing 8.17. Geometry shader layout qualifiers
#version 430 core layout (triangles) in; layout (triangle_strip) out; layout (max_vertices = 3) out;
These set the input and output primitive modes using a layout qualifier. In this particular shader we’re using triangles for the input and triangle_strip for the output. Other primitive types, along with the layout qualifier, are covered later. For the geometry shader’s output, not only do we specify the primitive type, but the maximum number of vertices expected to be generated by the shader (through themax_vertices qualifier). This shader produces individual triangles (generated as very short triangle strips), so we specified 3 here.
Next is our main() function, which is again similar to what might be seen in a vertex or fragment shader. The shader contains a loop, and the loop runs a number of times determined by the length of the built-in array, gl_in. This is another geometry shader-specific variable. Because the geometry shader has access to all of the vertices of the input primitive, the input has to be declared as an array. All of the built-in variables that are written by the vertex shader (such as gl_Position) are placed into a structure, and an array of these structures is presented to the geometry shader in a variable calledgl_in.
The length of the gl_in[] array is determined by the input primitive mode, and because in this particular shader, triangles are the input primitive mode, the size of gl_in[] is three. The inner loop is given again in Listing 8.18.
Listing 8.18. Iterating over the elements of gl_in[]
for (i = 0; i < gl_in.length(); i++)
{
gl_Position = gl_in[i].gl_Position;
EmitVertex();
}
Inside our loop, we’re generating vertices by simply copying the elements of gl_in[] to the geometry shader’s output. A geometry shader’s outputs are similar to the vertex shader’s outputs. Here, we’re writing to gl_Position, just as we would in a vertex shader. When we’re done setting up all of the new vertex’s attributes, we call EmitVertex(). This is a built-in function, specific to geometry shaders that tells the shader that we’re done with our work for this vertex and that it should store all that information away and prepare to start setting up the next vertex.
Finally, after the loop has finished executing, there’s a call to another special, geometry shader-only function, EndPrimitive(). EndPrimitive() tells the shader that we’re done producing vertices for the current primitive and to move on to the next one. We specified triangle_strip as the output for our shader, and so if we continue to call EmitVertex() more than three times, OpenGL continues adding triangles to the triangle strip. If we need our geometry shader to generate separate, individual triangles or multiple, unconnected triangle strips (remember, geometry shaders can create new or amplify geometry), we could call EndPrimitive() between each one to mark their boundaries. If you don’t callEndPrimitive() somewhere in your shader, the primitive is automatically ended when the shader ends.
Using Geometry Shaders in an Application
Geometry shaders, like the other shader types, are created by calling the glCreateShader() function and using GL_GEOMETRY_SHADER as the shader type, as follows:
glCreateShader(GL_GEOMETRY_SHADER);
Once the shader has been created, it is used like any other shader object. You give OpenGL your shader source code by calling glShaderSource(), compile the shader using the glCompileShader()function, and attach it to a program object by calling the glAttachShader() function. Then the program is linked as normal using the glLinkProgram() function. Now that you have a program object with a geometry shader linked into it, when you draw geometry using a function like glDrawArrays(), the vertex shader will run once per vertex, the geometry shader will run once per primitive (point, line, or triangle), and the fragment will run once per fragment. The primitives received by a geometry shader must match what it is expecting based in its own input primitive mode. When tessellation is not active, the primitive mode you use in your drawing commands must match the input primitive mode of the geometry shader. For example, if the geometry shader’s input primitive mode is points, then you may only use GL_POINTS when you call glDrawArrays(). If the geometry shader’s input primitive mode is triangles, then you may use GL_TRIANGLES, GL_TRIANGLE_STRIP, or GL_TRIANGLE_FAN in yourglDrawArrays() call. A complete list of the geometry shader input primitive modes and the allowed geometry types is given in Table 8.1.
Table 8.1. Allowed Draw Modes for Geometry Shader Input Modes
| Geometry Shader Input Mode | Allowed Draw Modes |
points |
GL_POINTS |
lines |
GL_LINES, GL_LINE_LOOP, GL_LINE_STRIP |
triangles |
GL_TRIANGLES, GL_TRIANGLE_FAN, GL_TRIANGLE_STRIP |
lines_adjacency |
GL_LINES_ADJACENCY |
triangles_adjacency |
GL_TRIANGLES_ADJACENCY |
When tessellation is active, the mode you use in your drawing commands should always beGL_PATCHES, and OpenGL will convert the patches into points, lines, or triangles during the tessellation process. In this case, the input primitive mode of the geometry shader should match the tessellation primitive mode. The input primitive type is specified in the body of the geometry shader using a layout qualifier. The general form of the input layout qualifier is
layout (primitive_type) in;
This specifies that primitive_type is the input primitive type that the geometry shader is expected to handle, and primitive_type must be one of the supported primitive modes: points, lines, triangles, lines_adjacency, or triangles_adjacency. The geometry shader runs once per primitive. This means that it’ll run once per point for GL_POINTS; once per line for GL_LINES, GL_LINE_STRIP, and GL_LINE_LOOP; and once per triangle for GL_TRIANGLES, GL_TRIANGLE_STRIP, and GL_TRIANGLE_FAN. The inputs to the geometry shader are presented in arrays containing all of the vertices making up the input primitive. The predefined inputs are stored in a built-in array called gl_in[], which is an array of structures defined in Listing 8.19.
Listing 8.19. The definition of gl_in[]
in gl_PerVertex
{
vec4 gl_Position;
float gl_PointSize;
float gl_ClipDistance[];
} gl_in[];
The members of this structure are the built-in variables that are written in the vertex shader:gl_Position, gl_PointSize, and gl_ClipDistance[]. You should recognize this structure from its declaration as an output block in the vertex shader described earlier in this chapter. These variables appear as global variables in the vertex shader because the block doesn’t have an instance name there, but their values end up in the gl_in[] array of block instances when they appear in the geometry shader. Other variables written by the vertex shader also become arrays in the geometry shader. In the case of individual varyings, outputs in the vertex shader are declared as normal, and the inputs to the geometry shader have a similar declaration, except that they are arrays. Consider a vertex shader that defines outputs as
out vec4 color; out vec3 normal;
The corresponding input to the geometry shader would be
in vec4 color[]; in vec3 normal[];
Notice that both the color and normal varyings have become arrays in the geometry shader. If you have a large amount of data to pass from the vertex to the geometry shader, it can be convenient to wrap per-vertex information passed from the vertex shader to the geometry shader into an interface block. In this case, your vertex shader will have a definition like this:
out VertexData
{
vec4 color;
vec3 normal;
} vertex;
And the corresponding input to the geometry shader would look like this:
in VertexData
{
vec4 color;
vec3 normal;
// More per-vertex attributes can be inserted here
} vertex[];
With this declaration, you’ll be able to access the per-vertex data in the geometry shader usingvertex[n].color and so on. The length of the input arrays in the geometry shader depends on the type of primitives that it will process. For example, points are formed from a single vertex, and so the arrays will only contain a single element, whereas triangles are formed from three vertices, and so the arrays will be three elements long. If you’re writing a geometry shader that’s designed specifically to process a particular primitive type, you can explicitly size your input arrays, which provides a small amount of additional compile-time error checking. Otherwise, you can let your arrays be automatically sized by the input primitive type layout qualifier. A complete mapping of the input primitive modes and the resulting size of the input arrays is shown in Table 8.2.
Table 8.2. Sizes of Input Arrays to Geometry Shaders
| Input Primitive Type | Size of Input Arrays |
points |
1 |
lines |
2 |
triangles |
3 |
lines_adjacency |
4 |
triangles_adjacency |
6 |
You also need to specify the primitive type that will be generated by the geometry shader. Again, this is determined using a layout qualifier, like so:
layout (primitive_type) out;
This is similar to the input primitive type layout qualifier, the only difference being that you are declaring the output of the shader using the out keyword. The allowable output primitive types from the geometry shader are points, line_strip, and triangle_strip. Notice that geometry shaders only support outputting the strip primitive types (not counting points—obviously, there is no such thing as a point strip).
There is one final layout qualifier that must be used to configure the geometry shader. Because a geometry shader is capable of producing a variable amount of data per vertex, OpenGL must be told how much space to allocate for all that data by specifying the maximum number of vertices that the geometry shader is expected to produce. To do this, use the following layout qualifier:
layout (max_vertices = n) out;
This sets the maximum number of vertices that the geometry shader may produce to n. Because OpenGL may allocate buffer space to store intermediate results for each vertex, this should be the smallest number possible that still allows your application to run correctly. For example, if you are planning to take points and produce one line at a time, then you can safely set this to two. This gives the shader hardware the best opportunity to run fast. If you are going to heavily tessellate the incoming geometry, you might want to set this to a much higher number, although this may cost you some performance. The upper limit on the number of vertices that a geometry shader can produce depends on your OpenGL implementation. It is guaranteed to be at least 256, but the absolute maximum can be found by calling glGetIntegerv() with the GL_MAX_GEOMETRY_OUTPUT_VERTICES parameter.
You can also declare more than one layout qualifier with a single statement by separating them with a comma, like so:
layout (triangle_strip, max_vertices = n) out;
With these layout qualifiers, a boilerplate #version declaration, and an empty main() function, you should be able to produce a geometry shader that compiles and links but does absolutely nothing. In fact, it will discard any geometry you send it, and nothing will be drawn by your application. We need to introduce two important functions: EmitVertex() and EndPrimitive(). If you don’t call these, nothing will be drawn.
EmitVertex() tells the geometry shader that you’ve finished filling in all of the information for this vertex. Setting up the vertex works much like the vertex shader. You need to write into the built-in variable gl_Position. This sets the clip-space coordinates of the vertex that is produced by the geometry shader, just like in a vertex shader. Any other attributes that you want to pass from the geometry shader to the fragment shader can be declared in an interface block or as global variables in the geometry shader. Whenever you call EmitVertex, the geometry shader stores the values currently in all of its output variables and uses them to generate a new vertex. You can call EmitVertex() as many times as you like in a geometry shader, until you reach the limit you specified in your max_verticeslayout qualifier. Each time, you put new values into your output variables to generate a new vertex.
An important thing to note about EmitVertex() is that it makes the values of any of your output variables (such as gl_Position) undefined. So, for example, if you want to emit a triangle with a single color, you need to write that color with every one of your vertices; otherwise, you will end up with undefined results.
EmitPrimitive() indicates that you have finished appending vertices to the end of the primitive. Don’t forget, geometry shaders only support the strip primitive types (line_strip and triangle_strip). If your output primitive type is triangle_strip and you call EmitVertex() more than three times, the geometry shader will produce multiple triangles in a strip. Likewise, if your output primitive type is line_strip and you call EmitVertex() more than twice, you’ll get multiple lines. In the geometry shader, EndPrimitive() refers to the strip. This means that if you want to draw individual lines or triangles, you have to call EndPrimitive() after every two or three vertices. You can also draw multiple strips by calling EmitVertex() many times between multiple calls to EndPrimitive().
One final thing to note about calling EmitVertex() and EndPrimitive() in the geometry shader is that if you haven’t produced enough vertices to produce a single primitive (e.g., you’re generatingtriangle_strip outputs and you call EndPrimitive() after two vertices), nothing is produced for that primitive, and the vertices you’ve already produced are simply thrown away.
Discarding Geometry in the Geometry Shader
The geometry shader in your program runs once per primitive. What you do with that primitive is entirely up to you. The two functions EmitVertex() and EndPrimitive() allow you to programmatically append new vertices to your triangle or line strip and to start new strips. You can call them as many times as you want (until you reach the maximum defined by your implementation). You’re also allowed to not call them at all. This allows you to clip geometry away and discard primitives. If your geometry shader runs and you never call EmitVertex() for that particular primitive, nothing will be drawn. To illustrate this, we can implement a custom backface culling routine that culls geometry as if it were viewed from an arbitrary point in space. This is implemented in the gsculling example.
First, we set up our shader version and declare our geometry shader to accept triangles and to produce triangle strips. Backface culling doesn’t really make a lot of sense for lines or points. We also define a uniform that will hold our custom viewpoint in world space. This is shown in Listing 8.20.
Listing 8.20. Configuring the custom culling geometry shader
#version 330 // Input is triangles, output is triangle strip. Because we're going // to do a 1 in 1 out shader producing a single triangle output for // each one input, max_vertices can be 3 here. layout (triangles) in; layout (triangle_strip, max_vertices=3) out; // Uniform variables that will hold our custom viewpoint and // model-view matrix uniform vec3 viewpoint; uniform mav4 mv_matrix;
Now inside our main() function, we need to find the face normal for the triangle. This is simply the cross products of any two vectors in the plane of the triangle—we can use the triangle edges for this. Listing 8.21 shows how this is done.
Listing 8.21. Finding a face normal in a geometry shader
// Calculate two vectors in the plane of the input triangle vec3 ab = gl_in[1].gl_Position.xyz - gl_in[0].gl_Position.xyz; vec3 ac = gl_in[2].gl_Position.xyz - gl_in[0].gl_Position.xyz; vec3 normal = normalize(cross(ab, ac));
Now that we have the normal, we can determine whether it faces toward or away from our user-defined viewpoint. To do this, we need to transform the normal into the same coordinate space as the viewpoint, which is world space. Assuming we have the model-view matrix in a uniform, simply multiply the normal by this matrix. To be more accurate, we should multiply the vector by the inverse of the transpose of the upper-left 3 × 3 submatrix of the model-view matrix. This is known as the normal matrix, and you’re free to implement this and put it in its own uniform if you like. However, if your model-view matrix only contains translation, uniform scale (no shear), and rotation, you can use it directly. Don’t forget, the normal is a three-element vector, and the model-view matrix is a 4 × 4 matrix. We need to extend the normal to a four-element vector before we can multiply the two. We can then take the dot product of the resulting vector with the vector from the viewpoint to any point on the triangle.
If the sign of the dot product is negative, that means that the normal is facing away from the viewer and the triangle should be culled. If it is positive, the triangle’s normal is pointing toward the viewer, and we should pass the triangle on. The code to transform the face normal, perform the dot product, and test the sign of the result is shown in Listing 8.22.
Listing 8.22. Conditionally emitting geometry in a geometry shader
// Calculate the transformed face normal and the view direction vector
vec3 transformed_normal = (vec4(normal, 0.0) * mv_matrix).xyz;
vec3 vt = normalize(gl_in[0].gl_Position.xyz - viewpoint);
// Take the dot product of the normal with the view direction
float d = dot(vt, normal);
// Emit a primitive only if the sign of the dot product is positive
if (d > 0.0)
{
for (int i = 0; i < 3; i++)
{
gl_Position = gl_in[i].gl_Position;
EmitVertex();
}
EndPrimitive();
}
Generating Geometry in the Geometry Shader
Just as you are not required to call EmitVertex() or EndPrimitive() at all if you don’t want to produce any output from the geometry shader, it is also possible to call EmitVertex() andEndPrimitive() as many times as you need to produce new geometry. That is, until you reach the maximum number of output vertices that you declared at the start of your geometry shader. This functionality can be used for things like making multiple copies of the input or breaking the input into smaller pieces. This is the subject of the next example, which is the gstessellate sample in the book’s accompanying source code. The input to our shader is a tetrahedron centered around the origin. Each face of the tetrahedron is made from a single triangle. We tessellate incoming triangles by producing new vertices halfway along each edge and then moving all of the resulting vertices so that they are variable distances from the origin. This transforms our tetrahedron into a spiked shape.
Because the geometry shader operates in object space (remember, the tetrahedron’s vertices are centered around the origin), we need to do no coordinate transforms in the vertex shader and, instead, do the transforms in the geometry shader after we’ve generated the new vertices. To do this, we need a simple, pass-through vertex shader. Listing 8.25 shows a simple pass-through vertex shader.
Listing 8.25. Pass-through vertex shader
#version 330
in vec4 position;
void main(void)
{
gl_Position = position;
}
This shader only passes the vertex position to the geometry shader. If you have other attributes associated with the vertices such as texture coordinates or normals, you need to pass them through the vertex shader to the geometry shader as well.
As in the previous example, we accept triangles as input to the geometry shader and produce a triangle strip. We break the strip after every triangle so that we can produce separate, independent triangles. In this example, we produce four output triangles for every input triangle. We need to declare our maximum output vertex count as 12—four triangles times three vertices. We also need to declare a uniform matrix to store the model-view transformation matrix in the geometry shader because we do that transform after generating vertices. Listing 8.26 shows this code.
Listing 8.26. Setting up the “tessellator” geometry shader
#version 430 core layout (triangles) in; layout (triangle_strip, max_vertices = 12) out; // A uniform to store the model-view-projection matrix uniform mat4 mvp;
First, let’s copy the incoming vertex coordinates into a local variable. Then, given the original, incoming vertices, we find the midpoint of each edge by taking their average. In this case, however, rather than simply dividing by two, we multiply by a scale factor, which will allow us to alter the spikiness of the resulting object. Code to do this is shown in Listing 8.27.
Listing 8.27. Generating new vertices in a geometry shader
// Copy the incoming vertex positions into some local variables vec3 a = gl_in[0].gl_Position.xyz; vec3 b = gl_in[1].gl_Position.xyz; vec3 c = gl_in[2].gl_Position.xyz; // Find a scaled version of their midpoints vec3 d = (a + b) * stretch; vec3 e = (b + c) * stretch; vec3 f = (c + a) * stretch; // Now, scale the original vertices by an inverse of the midpoint // scale a *= (2.0 - stretch); b *= (2.0 - stretch); c *= (2.0 - stretch);
Because we are going to generate several triangles using almost identical code, we can put that code into a function (shown in Listing 8.28) and call it from our main tessellation function.
Listing 8.28. Emitting a single triangle from a geometry shader
void make_face(vec3 a, vec3 b, vec3 c)
{
vec3 face_normal = normalize(cross(c - a, c - b));
vec4 face_color = vec4(1.0, 0.2, 0.4, 1.0) * (mat3(mvMatrix) * face_normal
gl_Position = mvpMatrix * vec4(a, 1.0);
color = face_color;
EmitVertex();
gl_Position = mvpMatrix * vec4(b, 1.0);
color = face_color;
EmitVertex();
gl_Position = mvpMatrix * vec4(c, 1.0);
color = face_color;
EmitVertex();
EndPrimitive();
}
Notice that the make_face function calculates a face color based on the face’s normal in addition to emitting the positions of its vertices. Now, we simply call make_face four times from our main function, which is shown in Listing 8.29.
Listing 8.29. Using a function to produce faces in a geometry shader
make_face(a, d, f); make_face(d, b, e); make_face(e, c, f); make_face(d, e, f);