Project 5: Austin Eng

.gitignore

@@ -152,3 +152,5 @@ local.properties
 
 # TeXlipse plugin
 .texlipse
+
+node_modules

README.md

@@ -3,28 +3,75 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Austini Eng
+* Tested on: **Google Chrome 54.0.2840.59** on
+  Linux Mint 18, i5-5200U @ 2.20GHz 7.7GB, GTX 940m
 
-### Live Online
+## [Live Demo](http://austineng.github.io/WebGL-Deferred-Shading)
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+[![](img/recording.gif)](http://austineng.github.io/WebGL-Deferred-Shading)
 
-### Demo Video/GIF
+## Deferred Shading
 
-[![](img/video.png)](TODO)
+In deferred shading, we pack geometry information into a `G-Buffer`. This allows separate between rasterization and lighting computation. Below, we have visualized separate colors, normals, positions, and depth attributes. Once these have been stored in a buffer, we can render the scene once for each light and add all the results together.
 
-### (TODO: Your README)
+![](img/colors.png)
+![](img/normals.png)
+![](img/positions.png)
+![](img/depth.png)
+![](img/composite.png)
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+## Bloom
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+![](img/bloom.png)
 
+To compute bloom, we run a shader that just keeps the highlights, then blur that result in two directions, and then add it back to the original position.
 
-### Credits
+Doing the blur in x and y separately is much more efficient. If we did the blur all at once, each thread running the fragment shader would need to look at 81 adjacent pixels. If we do them separately, each looks at just 9 pixels, twice. Each iteration stores the weighted average of the 9 influencing pixels based on [gaussian weights](http://dev.theomader.com/gaussian-kernel-calculator/).
+
+Adding this effect is still pretty slow. Even a small amount of bloom adds about 30ms of time per frame.
+
+
+## Scissor Optimization
+
+![](img/scissor.png)
+
+We can optimize the naive implementation by only rendering the portion of the screen that overlaps a given light. Above is visualized the bounding boxes around the lights.
+
+## G-Buffer Optimization
+
+We can reduce memory overhead by optimizing the attributes stored in the G-Buffer. Less memory access should theoretically increase performance of our shaders. This was done in two ways. Instead of storing the geometery's surface normal and normal map separately, these were combined into just one attribute. Furthermore, only the x and y components were stored. We can compute the correct z component later.
+
+We can also store just the rgb components of the color because the scene has no transparent objects. Decreasing the number of components stored, we can pack everything we need into just two buffers instead of the original four.
+
+[X, Y, Z, NX], [R, G, B, NY]
+
+
+## Tiled Rendering
+
+If there are many lights in the scene, the naive implementation must make many redundant memory accesses. We render the scene once for every light, so if we have N lights, a fragment is rendered potentially N times. This is N separate fetches to access the memory stored in our G-Buffer. Performance would be signficantly better if we knew ahead of time which tiles influenced the fragment and computed the contribution of all of these in one pass.
+
+Tiled deferred shading breaks the scene into tiles (here, I used 32x32) and computes the lights which could influence each tile before shading. Then, the fragment shader only needs to be run once to add up the contributions of all the lights. Below is a visualization of the number of lights affecting each tile.
+
+![](img/tiles.png)
+
+Unfortunately, WebGL does not have compute shaders. To accomplish tiled deferred shading, I packed all the light attributes (position, color, radius) into two RGBAs: [X, Y, Z, r, R, G, B, _]. These were stored in a texture. Then a shader was run over a viewport of size (# tiles) x (# lights / 4). To create a mask denoting whether a light influenced a given tile. The size of the texture output by this shader was (# tiles) x (# lights / 4) because 4 flags were packed into each pixel. Each thread therefore computes 4 tile/light intersections.
+
+The resulting texture is then bound and passed to the fragment shader. Each thread can now compute the index of the tile it its contained in, lookup exactly which lights it is influenced by, andd then perform all of the necessary lighting calculations.
+
+
+## Performance Analysis
+
+GBuffers | None | Scissor | Tiled
+---------|------|---------|-------
+4        | 120  |   90    | 28
+3        |  -   |   64    | 28
+2        |  -   |   40    | 27
+*Time (ms) per frame with different optimizations and number of G-Buffers. Tests were done using 200 lights*
+
+Adding tiled deferred shading significantly improved performance. Its impacts signficantly outweigh the performance gains earned by reducing the number of G-Buffers. This makes sense because optimizing the G-Buffer really only removes one or two memory accesses per pixel. On the other hand, tiled deferred shading allows us to decrease memory access by factor which scales with the number of lights. With 200 lights, this is an enormous improvement.
+
+## Credits
 
 * [Three.js](https://github.com/mrdoob/three.js) by [@mrdoob](https://github.com/mrdoob) and contributors
 * [stats.js](https://github.com/mrdoob/stats.js) by [@mrdoob](https://github.com/mrdoob) and contributors

dev.js

@@ -0,0 +1,16 @@
+var express =  require('express')
+var app = express()
+var http = require('http')
+
+app.set('port', process.env.PORT || 3000);
+app.use(express.static(__dirname));
+
+var livereload = require('livereload');
+var server = livereload.createServer({
+    exts: ['js', 'png', 'glsl']
+});
+server.watch(__dirname);
+
+http.createServer(app).listen(app.get('port'), function(){
+  console.log("Express server listening on port " + app.get('port'));
+});

glsl/copy.frag.glsl

@@ -10,11 +10,23 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
 void main() {
-    // TODO: copy values into gl_FragData[0], [1], etc.
-    // You can use the GLSL texture2D function to access the textures using
-    // the UV in v_uv.
+    vec3 norm = applyNormalMap(v_normal, vec3(texture2D(u_normap, v_uv)));
+    vec3 col = vec3(texture2D(u_colmap, v_uv));
 
-    // this gives you the idea
     // gl_FragData[0] = vec4( v_position, 1.0 );
+    // gl_FragData[1] = vec4( v_normal, 0.0 );
+    // gl_FragData[2] = texture2D(u_colmap, v_uv);
+    // gl_FragData[3] = texture2D(u_normap, v_uv);
+
+    gl_FragData[0] = vec4(v_position, norm.x);
+    gl_FragData[1] = vec4(col, norm.y);
 }

glsl/debug.frag.glsl

@@ -0,0 +1,9 @@
+#version 100
+precision highp float;
+precision highp int;
+
+uniform vec4 u_color;
+
+void main() {
+    gl_FragColor = u_color;
+}

glsl/deferred/ambient.frag.glsl

@@ -3,25 +3,29 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 2
 
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
+const vec3 AMBIENT_COLOR = vec3(0.05, 0.07, 0.12);
+
 void main() {
-    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    // vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
-    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    // vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    // vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
+
+    // Extract needed properties from the g-buffers into local variables
+    vec3 colmap = gb1.rgb;
 
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    gl_FragColor = vec4(AMBIENT_COLOR * colmap, 1.0);
 }

glsl/deferred/blinnphong-pointlight-tiled.frag.glsl

@@ -0,0 +1,102 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 2
+
+uniform float u_nlights;
+uniform float u_ntiles;
+uniform vec2 u_tilesize;
+uniform vec2 u_resolution;
+uniform vec3 u_cameraPos;
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+uniform sampler2D u_lightbuffer;
+uniform sampler2D u_tilebuffer;
+uniform int u_debug;
+
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    // vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    // vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+
+    // If nothing was rendered to this pixel, set alpha to 0 so that the
+    // postprocessing step can render the sky color.
+    if (depth == 1.0) {
+        gl_FragColor = vec4(0, 0, 0, 0);
+        return;
+    }
+
+    // Extract needed properties from the g-buffers into local variables
+    // These definitions are suggested for starting out, but you will probably want to change them.
+    // vec3 pos = gb0.xyz;     // World-space position
+    // vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    // vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+    // vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    // vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
+    // vec3 nor = gb1.xyz;
+
+    vec3 pos = gb0.xyz;
+    vec3 colmap = gb1.rgb;
+    vec3 nor = vec3(gb0[3], gb1[3], 0.0);
+    nor.z = (1.0 - nor.x*nor.x - nor.y*nor.y);
+    nor.z *= sign(dot(nor, u_cameraPos - pos));
+    nor = normalize(nor);
+
+    vec2 tileDim = ceil(u_resolution / u_tilesize);
+    vec2 t_xy = floor(v_uv * u_resolution / u_tilesize);
+    float t_uv = (t_xy.y * tileDim.x + t_xy.x) / u_ntiles + 0.1 / u_ntiles;
+
+    vec3 cumul = vec3(0.0);
+    float count = 0.0;
+    for (int i = 0; i < 200; i += 4) {
+        if (i >= int(u_nlights)) break;
+        float l_uv = float(i) / u_nlights;
+        vec4 mask = texture2D(u_tilebuffer, vec2(t_uv, l_uv));
+
+        for (int j = 0; j < 4; ++j) {
+            l_uv = float(i + j) / u_nlights + 0.1 / u_nlights; // slight offset
+            if (mask[j] > 0.0) {
+                count += mask[j];
+
+                vec4 v1 = texture2D(u_lightbuffer, vec2(l_uv, 0.0));
+                vec4 v2 = texture2D(u_lightbuffer, vec2(l_uv, 0.5));
+                vec3 l_col = v2.rgb;
+                float l_rad = v1[3];
+                vec3 l_pos = v1.xyz;
+
+                float dist = distance(l_pos, pos);
+                if (dist > l_rad) {
+                    continue;
+                }
+
+                vec3 L = normalize(l_pos - pos);
+                float lambert = max(dot(L, nor), 0.0);
+                vec3 V = normalize(u_cameraPos - pos);
+                vec3 H = normalize(L + V);
+                float specular = float(lambert > 0.0) * pow(max(dot(H, nor), 0.0), 10.0);
+                cumul += l_col * pow(1.0 - dist / l_rad, 2.0) *
+                (
+                    colmap * lambert + 
+                    vec3(1,1,1) * specular
+                );
+            }
+        }
+    }
+    if (u_debug > 0) cumul = vec3(count / u_nlights);
+
+
+    gl_FragColor = vec4(cumul, 1.0);
+}

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -2,8 +2,9 @@
 precision highp float;
 precision highp int;
 
-#define NUM_GBUFFERS 4
+#define NUM_GBUFFERS 2
 
+uniform vec3 u_cameraPos;
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
@@ -23,10 +24,9 @@ vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
-    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
-    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    // vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    // vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
 
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
@@ -35,5 +35,37 @@ void main() {
         return;
     }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    // Extract needed properties from the g-buffers into local variables
+    // These definitions are suggested for starting out, but you will probably want to change them.
+    // vec3 pos = gb0.xyz;     // World-space position
+    // vec3 geomnor = gb1.xyz;  // Normals of the geometry as defined, without normal mapping
+    // vec3 colmap = gb2.rgb;  // The color map - unlit "albedo" (surface color)
+    // vec3 normap = gb3.xyz;  // The raw normal map (normals relative to the surface they're on)
+    // vec3 nor = applyNormalMap (geomnor, normap);     // The true normals as we want to light them - with the normal map applied to the geometry normals (applyNormalMap above)
+    // vec3 nor = gb1.xyz;
+    vec3 pos = gb0.xyz;
+    vec3 colmap = gb1.rgb;
+    vec3 nor = vec3(gb0[3], gb1[3], 0.0);
+    nor.z = (1.0 - nor.x*nor.x - nor.y*nor.y);
+    nor.z *= sign(dot(nor, u_cameraPos - pos));
+    nor = normalize(nor);
+
+    float distance = distance(u_lightPos, pos);
+    if (distance  > u_lightRad) {
+        return;
+    }
+
+    vec3 L = normalize(u_lightPos - pos);
+    float lambert = max(dot(L, nor), 0.0);
+    vec3 V = normalize(u_cameraPos - pos);
+    vec3 H = normalize(L + V);
+    float specular = float(lambert > 0.0) * pow(max(dot(H, nor), 0.0), 10.0);
+
+    gl_FragColor = vec4(
+        u_lightCol * pow(1.0 - distance / u_lightRad, 2.0) *
+        (
+            colmap * lambert + 
+            vec3(1,1,1) * specular
+        )
+        , 1);
 }