The methods we have used so far for calculating the color at a surface point are pretty limited. This is mostly related to the fact that our base model color, described by the diffuse reflection coefficient $k_m$, was either a constant across the surface, or was interpolated to the fragments from the color specified at the vertices of each triangle.

Our main goal for today is to retrieve a better value for $k_m$, using a concept called texturing. This will allow us to look up a $k_m$ value at each surface point, using some reference image. This essentially samples the image and "pastes" the image onto our 3d surface. This concept can also be applied to create other effects, like "bumping" the surface.

The main idea of texturing: sampling an image to paste it onto our meshes.

The main idea of texturing is to sample the pixels in an image to determine the properties (usually, a color) of a point on our surface. To distinguish the screen pixels (in our destination image) from this texturing image, the pixels in this texture image are called texels. We need the following ingredients to texture our surfaces:

Ingredient #1: We need an image to sample and paste, like the image at the top-right of these notes. We'll also need to write the data in this image to the GPU.

Ingredient #2: We need to associate our surface points with a "location" in this image, so we can look up the color. This is done using texture coordinates, which are 2d coordinates (since they are defined with respect to the reference image). We will denote these texture coordinates as $(s, t)$ and we'll assume that these coordinates are either provided to us, or can be inferred analytically. The combination of these texture coordinates $(s, t)$ and the element (usually a color) at this coordinate is called a texture element, or texel for short.

The image below shows how a point on Spot's eye, which has texture coordinates $(s, t)$ is mapped to the surface with coordinates $(x, y, z)$. The $s$ coordinate is the distance from the point to the left side of the image and the $t$ coordinate is the distance from either the top or bottom side of the image - it depends on how the image is represented. A question you might have is: how do we get the $(s, t)$ coordinates for a point on the surface?

Ingredient #3: We need to determine how we will look up the texel values in the image. The texture coordinates on the surface will almost never align exactly with the center of a texel. Furthermore, we need to consider the relative size of a pixel (associated with our fragment) and the texels in the texture image. We'll revisit this concept later, and WebGL will take care of this for us, but we need to tell it what to do. For now, let's assume we always look up the texel whose center is nearest to the $(s, t)$ texture coordinates we provide for the lookup.

The special case of a sphere.

As we have seen with our graphics programs, it's usually a good idea to break up our task into smaller pieces. Let's leave ingredient #2 aside for now and assume that we can calculate texture coordinates analytically. To do so, we'll assume our model is a sphere, just like we did in Lecture 2.

The surface of a sphere can be parametrized by two variables, which are actually angles. Let's assume the sphere is centered on the origin, $\vec{c} = (0, 0, 0)$. Consider the vector from the origin to a point on the sphere, denoted by $\vec{p}$. The first angle is the angle between the z-axis and $\vec{p}$ which is called the polar angle, $\phi$. The second angle is called the azimuthal angle, denoted by $\theta$, and measures the angle from the x-axis to the projection of $\vec{p}$ onto the x-y plane. You can think of this like the angle around the equator.

The range of $\phi$ goes from the north pole to the south pole, which is 180 degrees, or $\pi$ radians: $0 \le \phi \le \pi$. The angle $\theta$ wraps around the entire equator, so $0 \le \theta \le 2\pi$ (360 degrees). For any given $(\theta, \phi)$, we can calculate the (x,y, z) coordinates on the surface of the sphere as:

$$ x = R \sin\phi\cos\theta, \quad y = R\sin\phi\sin\theta, \quad z = R\cos\phi. $$

We can also go the other way around, i.e. from $(x, y, z)$ to $(\theta, \phi)$ using:

$$ \theta = \arctan\left(\frac{y}{x}\right) \in [0, 2\pi], \quad \phi = \arccos\left(\frac{z}{R}\right) \in [0, \pi]. $$

Note that the range of the GLSL atan function is $[-\pi, \pi]$ and the range of acos is $[0, \pi]$. We can almost use this to look up a texture coordinate in an image. We need to apply scaling factors that take into account the range of our angles, as well as the width of the image. When we do it with WebGL however, it knows the width and height of our texture image, so we just need to get $(s, t) \in [0, 1]^2$:

$$ s_{\mathrm{sphere}} = \frac{1}{2\pi}\left(\arctan\left(\frac{y}{x}\right) + \pi\right) \in [0, 1], \quad t_{\mathrm{sphere}} = \frac{1}{\pi} \arccos\left(\frac{z}{R}\right) \in [0, 1]. $$

Doing it with WebGL.

Before we can look up texels in our shader programs, we need to write the image data to the GPU. Let's first assume we have the image of Spot loaded into our HTML document using an img tag:

<img id="spot-texture" src="spot.png" hidden/>

The hidden property means that it won't be rendered in the HTML document - but we'll be able to retrieve it on the JavaScript side using the id. Here is how to set up a texture for WebGL using this image (assume we have a WebGLRenderingContext called gl and a WebGLShaderProgram called program):

// retrieve the image
let image = document.getElementById("spot-texture");

// create the texture and activate it
let texture = gl.createTexture();
gl.activeTexture(gl.TEXTURE0); // we are using texture index 0 <-- make a note of this!
gl.bindTexture(gl.TEXTURE_2D, texture);

// define the texture to be that of the requested image
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, 1); // may or may not need depending on y-axis of texture image
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, image);

Remember, we also need to tell WebGL how to do the lookup for texels (ingredient #3). For both minification and magnification (more about this soon), we'll specify that we want WebGL to use the nearest filter (gl.NEAREST):

gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);

Now, on the GLSL side, assuming we calculated the s and t (float) values, we can use a sampler2D to "sample" our texture object. This will actually be a uniform, and we'll need to tell WebGL what to use for this uniform (just like we did last class for scalars, vectors, matrices). This is where the texture "index" noted above is important. If we want to bind this sampler2D to our texture at gl.TEXTURE0, we need to set the value of this uniform to be 0. If we activated and wanted to use gl.TEXTURE8, this would be 8.

So for a shader with the following declaration (in GLSL):

uniform sampler2D tex_Image;

the corresponding JavaScript to bind our texture in unit 0 with this sampler2D would be:

gl.uniform1i(gl.getUniformLocation(program, 'tex_Image'), 0); // pass the N in gl.TEXTUREN when activating the texture unit

Finally, to perform the texel lookup, we can use the following within the main() of our shader:

vec3 km = texture2D(tex_Image, vec2(s, t)).rgb;

The texture2D function in GLSL returns a vec4, so we extract the first three components of the result with .rgb (does the same as .xyz).

More general models: storing an $(s, t)$ value at each vertex.

For more general surfaces represented by a mesh, we don't have an analytic way of getting the $(s, t)$ values. Furthermore, we cannot possibly store every single (s, t) value for points on the surface - we're not even storing every $(x, y, z)$ value of the surface points! Just like we're only storing the $(x, y, z)$ coordinates of each vertex, we'll only store the $(s, t)$ values at each vertex. Again, we will (actually, WebGL will) use interpolation to figure out the $(s, t)$ coordinates at a fragment within each triangle.

So what we'll do is pass some $(s, t)$ to our vertex shader and send it (using a varying) to be interpolated and, subsequently, input to the fragment shader. Next, the fragment shader will look up which color (or other value) to use in the lighting model.

Just like we have been writing vertex coordinates, normals, colors to the GPU, we'll now write texture coordinates. Assume that we have a 1d array in our mesh called texcoords with a stride of 2 to store the $(s, t)$ values at each vertex.

// enable the attribute in the program
let a_TexCoord = gl.getAttribLocation(program, "a_TexCoord");
gl.enableVertexAttribArray(a_TexCoord);

// create a buffer for the texture coordinates and write to it
let texcoordBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, texcoordBuffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(mesh.texcoords), gl.STATIC_DRAW);

// associate the data in the texcoordBuffer with the a_TexCoord attribute whenever we want to use them
gl.bindBuffer(gl.ARRAY_BUFFER, texcoordBuffer); // redundant here, but safe!
gl.vertexAttribPointer(a_TexCoord, 2, gl.FLOAT, 0, false, false);

Note that we are using 2 instead of 3 since there are only two values per vertex: $s$ and $t$.

A note about .obj files.

The Wavefront (.obj) files we have been using are a common format for computer graphics models. The first character of every line represents what the remaining values on that line correspond to:

• v: $(x, y, z)$ coordinates of a vertex,
• vt: $(s, t)$ texture coordinates at a vertex,
• vn: $(n_x, n_y, n_z)$ normal at a vertex,
• f: an $n$-sided polygon represented by a list of vertex indices. This can include the indices of v/vt/vn, depending on the number of forward slashes in each vertex description. The number of vertices $n$ is equal to the number of spaces on this line (including the first one after the f).

Revisiting ingredient #3: minification and magnification.

Let's re-interpret our texturing process at the level of a texel. In other words, we are mapping our texel colors to the final pixel (which is associated with the fragments obtained from the rasterization process). At a certain view, the size of the texel will be exactly the size of the pixel, though this rarely happens. When the surface we are painting is very far away, each pixel we are processing can cover many texel values (size of pixel > size of texel), a phenomenon known as minification. When we are closer to the surface, there may be many texels that map to a single pixel (size of pixel < size of texel), which is known as magnification.

 
Minification (left) versus Magnification (right) (Interactive Computer Graphics, Angel & Schreiner, 2012)

In the demo below, try to zoom into the chess board (by scrolling the mouse). As we get closer, the effects of magnification start to become apparent, and we can see the "blockiness" (aliasing) in the final image. This is because we were initially using the nearest sample when retrieving a texel (gl.NEAREST). If you change the dropdown to LINEAR, the magnification filter will be changed to gl.LINEAR which uses a weighted average of the surrounding texels. This is more expensive but will smooth out the blocky artifacts in the image.

The difference between "nearest" and "linear" filters in terms of the magnification problem is also demonstrated in the image below:

Now try scrolling away from the chess board, and notice that the colors don't look as patterned. If you rotate (click and drag) you should also start to see some "flickering" effects. We are now seeing the effects of minification: the texel lookup has many texels to pick from for a single pixel and the one it picks appear somewhat arbitrary. Now change the dropdown to MIPMAP. The chess board should appear to have the correct pattern again, regardless of the distance or how you rotate the square. A mipmap is a minification filter in which a sequence of images is first generated by halving the width and height of the image at each level. Texels will then be looked up at the appropriate level, which can again, use either nearest or linear filters. One disadvantage of mipmaps, however, is that the resulting rendering can look a bit blurry.

A mipmap can be generated for the current bound texture (assuming it is bound to gl.TEXTURE_2D) in WebGL using:

gl.generateMipmap(gl.TEXTURE_2D);

Note that since the width and height are halved at each level of the mipmap, the original image width and height should be a power of 2 (e.g. 512 x 1024). For more information on the texture parameters that can be set, please see this WebGL documentation.

The level of the mipmap can be determined by sampling the nearby pixels and determining if there is a large change in texel value. This works because the GPU actually processes 2x2 batches of pixels (a quad) so it can use the texels retrieved by sampling these 4 pixels.

Other types of texturing.

Another method for determining the color at a surface point is called procedural texturing, which consists of defining an explicit function (procedure) to describe the relationship between the surface coordinates and the color. This can work using either the 3d surface coordinates or the 2d parametric description of the surface. Note that we already did a form of procedural texturing! Recall Lab 3 when we assigned a kmFunction for BB-8.

Texturing isn't restricted to looking up the color of the surface. We can look up other items that might go into our lighting model. For example, we might want to use an image to look up the normal vector at a point on the surface, or we might want to displace the surface by some amount defined in an image:

• normal mapping (a type of bump map): each RGB value in the texture image corresponds to the normal vector $\vec{n} = (n_x, n_y, n_z)$ to be applied in the lighting model.
• height mapping (a type of bump map): each RGB value in the texture image (a grayscale image) corresponds to the height perturbation of the surface, from which the normal vector can be recomputed by looking at nearby texels and calculating the gradient. The geometry of the model does not change.
• displacement mapping: each RGB value in the texture image corresponds to a displacement of the surface, which actually modifies the geometry. This modification can be implemented in the vertex shader and thus requires a high-resolution model, which can be expensive.

It's also possible to look up the specular coefficient (the shininess exponenent $p$ in the Phong Reflection Model) which can produce fingerprint-like effects.

Other texturing methods are also possible such as environment maps (looking up the background image assuming the scene is enclosed in a cube or sphere) and projective texturing, whereby an input image (with known camera orientation and perspective settings) is pasted into our model.

© Philip Claude Caplan, 2023 (Last updated: 2023-11-07)