Project 5: David Liao

README.md

@@ -3,26 +3,108 @@ WebGL Deferred Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* David Liao
+* Tested on: Google Chrome 54.0.2840.71 (64-bit) on
+  Windows 10, i7-6700K @ 4.00GHz 16GB, Radeon R9 Nano 4096MB
 
 ### Live Online
 
-[![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
+[Try me live!](http://159.203.88.91/WebGL-Deferred-Shading)
 
 ### Demo Video/GIF
 
-[![](img/video.png)](TODO)
+![](img/small.gif)
 
-### (TODO: Your README)
+## Deferred Shading
+In deferred shading, the lighting step is postponed to a later step. Instead, in the first pass through, all geometry data is accumulated into texture buffers known as G-buffers. This is then later on used in more complex lighting operations. The benefit of doing this is that all the fragments that are in the G-buffers will ultimately be used in a single lighting pass (the depth-test has already been applied to these fragments). This ensures that we dont run multiple lighting passes over objects that might never make it to a pixel.
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+## Shaders
+### Default
+The default lighting is simply a product of the attenuation and the color from the texture. The attenuation is the falloff of the light, which in our case is linear.
+![](img/blinnphong.PNG)
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+### Blinn-Phong
+Blinn-phong is the standard baseline shader. https://www.wikiwand.com/en/Blinn%E2%80%93Phong_shading_model
 
+![](img/deferred-1478498011215.png)
+
+### Ramp-Shading
+Ramp shading is a non-photorealistic shader that "bands" together colors of similar value and rounds them to the nearest band. In the demo, you can tune the number of bands that are used. I did an experiment where I included ramp shading as a deferred shader and compared it to the post-processing version. Instead of banding the colors themselves, in this shader, I banded the lambertian and specular coefficients and then applied it to the colors. As a result, we have a more smoother transition between color bands. The main performance difference is negligible–the deferred shader computes the bands on the fly within that same shader.
+![](img/deferred-1478638975696.png)
+
+## Post Process Effects
+
+### Ramp-Shading (Post-process)
+In this version of the Ramp-shading, we are directly looking at the final result of the fragments and band the color values independently through their components.
+![](img/deferred-1478638967271.png)
+
+### Edge Highlights
+Edge highlights are performed in two ways. One through a single-pass approximated sobel filter and one through a two-pass sobel filter. The latter method uses the standard horizontal and vertical sobel operators shown above. The latter combines the two as a component-wise sum. The rationale behind doing this combination is that the two passes are additively blended together and so we leverage the associativity and commutativity of the two sobel operators to combine them. This is not entirely accurate since the blending isn't necessarily factored in, but the end result looks nice and runs faster (by one shader load and execution). The single pass is on the left, and the two-pass is on the right.
+
+<img src="img/deferred-1478639131697.png" width="400" height="300"/> <img src="img/deferred-1478639137171.png" width="400" height="300"/>
+
+
+### Depth of Field Blur
+Similar to edge highlights, DoF is also applying a filter over the final fragments. There is a slider that adjusts the focus of the DoF.  Due to limitations of glsl/WebGL, I couldn't make the kernel size adjustable (no dynamic array allocation). The algorithm used to determine blurring rate is taking a focal value (which ranges from 0 to 1) and the depth value and using a quadratic falloff value as a standard deviation for a gaussian kernel. The reason we use a falloff value raised to a power is to widen the depth range that is within focus. We also needed to scale the value to have it map to reasonable standard deviation values.
+![](img/blur.gif)
+## Optimizations
+### Scissor Test
+The scissor test is an optimization that discards fragments that fall outside of a rectangular region. This acceleration leverages light attenuation to allow for early termination when performing shading per light. Any region outside of the scissor rectangle doesn't need any light processing.
+
+| Scissor Test           | B-P w/o Scissor | B-P w/ Scissor |
+|------------------------|-----------------|----------------|
+| Milliseconds per frame | 6               | 6             |
+
+<img src="img/deferred-1478639472121.png" width="400" height="300"/>
+
+### G-Buffer Packing
+The baseline implementation uses 4 buffers for positions, normals, color maps, and normal maps. We can precompute the normals by applying the normal map in the copy pass. This way the memory bandwith is much less throughout the pipeline. 
+
+| G-Buffer Optimization  | B-P w/o Opt | B-P w/ Opt |
+|------------------------|-------------|------------|
+| Milliseconds per frame | 6           | 6         |
+| Bandwith in MB         | 11          | 11         |
+
+Though it seems like the optimization didn't do much :|.
+## Performance Factors
+### Deferred pipeline
+
+|Deferred Shader vs Time/Frame| Default | Blinn-Phong | Ramp Shading |
+|------------------------|---------|-------------|--------------|
+| Milliseconds per frame | 6      | 6          | 6           |
+
+### Post Process pipeline
+
+|Post Process vs Time/Frame| Baseline | Ramp Shading (Post) | Edge Highlights (One) | Edge Highlights (Two) |
+|------------------------|----------|---------------------|-----------------------|-----------------------|
+| Milliseconds per frame | 6       | 6                  |7                    | 7                   |
+
+### Depth of Field Blur
+
+|Kernel Size vs Time/Frame| 3  | 5  | 10 | 20 | 30 | 40 |
+|------------------------|----|----|----|----|----|----|
+| Milliseconds per frame | 6 | 6 | 6 | 13 | 34 | 90 |
+
+It seems that after 10, the quadratic nature of generating gaussian kernels on the fly seems to take its toll. As a result, I think a kernel of size 10 works best visually and performance-wise.
+
+## Kernel Experiments
+Here were some odd images that I got by using different 3x3 kernels without thresholding–you need to threshold the value obtained from the convolution in order to get the highlighted edges. 
+
+| -1 | -1 | -1 |
+|----|----|----|
+| -1 | 8  | -1 |
+| -1 | -1 | -1 |
+
+<img src="img/deferred-1478498035688.png" width="400" height="300"/><img src="img/deferred-1478498028576.png" width="400" height="300"/>
+
+Here is the standard 2-pass sobel filter without thresholding.
+
+| -1 | -2 | -1 |
+|----|----|----|
+| 0  | 0  | 0  |
+| 1  | 2  | 1  |
+
+<img src="img/deferred-1478497944786.png" width="400" height="300"/>
 
 ### Credits
 

glsl/copy.frag.glsl

@@ -10,11 +10,30 @@ varying vec3 v_position;
 varying vec3 v_normal;
 varying vec2 v_uv;
 
+#define OPTIMIZATION 0
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
 void main() {
     // TODO: copy values into gl_FragData[0], [1], etc.
     // You can use the GLSL texture2D function to access the textures using
     // the UV in v_uv.
 
     // this gives you the idea
-    // gl_FragData[0] = vec4( v_position, 1.0 );
+    #if OPTIMIZATION == 0
+    gl_FragData[0] = vec4(v_position, 1.0);
+    gl_FragData[1] = vec4(v_normal, 1.0);
+    gl_FragData[2] = texture2D(u_colmap, v_uv);
+    gl_FragData[3] = texture2D(u_normap, v_uv);
+    #elif OPTIMIZATION == 1
+    gl_FragData[0] = vec4(v_position, 1.0);
+    gl_FragData[1] = vec4(applyNormalMap(v_normal, texture2D(u_normap, v_uv).rgb), 1.0);
+    gl_FragData[2] = texture2D(u_colmap, v_uv);
+    #endif
 }

glsl/deferred/ambient.frag.glsl

@@ -10,18 +10,20 @@ uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
+const float ambientCoeff = 0.1;
+
 void main() {
     vec4 gb0 = texture2D(u_gbufs[0], v_uv);
     vec4 gb1 = texture2D(u_gbufs[1], v_uv);
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
     // TODO: Extract needed properties from the g-buffers into local variables
-
+    vec3 colmap = gb2.rgb;
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0); // set alpha to 0
         return;
     }
 
-    gl_FragColor = vec4(0.1, 0.1, 0.1, 1);  // TODO: replace this
+    gl_FragColor = vec4(ambientCoeff * colmap, 1);
 }

glsl/deferred/blinnphong-pointlight.frag.glsl

@@ -7,11 +7,16 @@ precision highp int;
 uniform vec3 u_lightCol;
 uniform vec3 u_lightPos;
 uniform float u_lightRad;
+uniform vec3 u_viewPos;
+uniform vec3 u_viewDir;
+
 uniform sampler2D u_gbufs[NUM_GBUFFERS];
 uniform sampler2D u_depth;
 
 varying vec2 v_uv;
 
+#define OPTIMIZATION 0
+
 vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
     normap = normap * 2.0 - 1.0;
     vec3 up = normalize(vec3(0.001, 1, 0.001));
@@ -26,14 +31,34 @@ void main() {
     vec4 gb2 = texture2D(u_gbufs[2], v_uv);
     vec4 gb3 = texture2D(u_gbufs[3], v_uv);
     float depth = texture2D(u_depth, v_uv).x;
-    // TODO: Extract needed properties from the g-buffers into local variables
-
+    #if OPTIMIZATION == 0
+    vec3 pos = gb0.xyz;
+    vec3 colmap = gb2.rgb;
+    vec3 nor = applyNormalMap(gb1.xyz, gb3.xyz);
+    #elif OPTIMIZATION == 1
+    vec3 pos = gb0.xyz;
+    vec3 nor = gb1.xyz;
+    vec3 colmap = gb2.rgb;
+    #endif
     // If nothing was rendered to this pixel, set alpha to 0 so that the
     // postprocessing step can render the sky color.
     if (depth == 1.0) {
         gl_FragColor = vec4(0, 0, 0, 0);
         return;
     }
+    vec3 D = u_lightPos - pos;
+    vec3 lightDir = normalize(D);
+    float lambertian = max(dot(lightDir, nor), 0.0);
+
+    float specular = 0.0;
+    float attenuation = max(u_lightRad - length(D), 0.0) / u_lightRad;
+    if (lambertian > 0.0) {
+      vec3 viewDir = normalize(u_viewPos-pos);
+      vec3 halfDir = normalize(lightDir + viewDir);
+      float specAngle = max(dot(halfDir, nor), 0.0);
+      specular = pow(specAngle, u_lightRad);
+    }
 
-    gl_FragColor = vec4(0, 0, 1, 1);  // TODO: perform lighting calculations
+    vec3 color = (colmap * lambertian * u_lightCol + specular * u_lightCol) * attenuation;
+    gl_FragColor = vec4(color, 1);
 }

glsl/deferred/debug.frag.glsl

@@ -38,15 +38,15 @@ void main() {
     if (u_debug == 0) {
         gl_FragColor = vec4(vec3(depth), 1.0);
     } else if (u_debug == 1) {
-        // gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
+        gl_FragColor = vec4(abs(pos) * 0.1, 1.0);
     } else if (u_debug == 2) {
-        // gl_FragColor = vec4(abs(geomnor), 1.0);
+        gl_FragColor = vec4(abs(geomnor), 1.0);
     } else if (u_debug == 3) {
-        // gl_FragColor = vec4(colmap, 1.0);
+        gl_FragColor = vec4(colmap, 1.0);
     } else if (u_debug == 4) {
-        // gl_FragColor = vec4(normap, 1.0);
+        gl_FragColor = vec4(normap, 1.0);
     } else if (u_debug == 5) {
-        // gl_FragColor = vec4(abs(nor), 1.0);
+        gl_FragColor = vec4(abs(nor), 1.0);
     } else {
         gl_FragColor = vec4(1, 0, 1, 1);
     }

glsl/deferred/default.frag.glsl

@@ -0,0 +1,45 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 4
+
+uniform vec3 u_lightCol;
+uniform vec3 u_lightPos;
+uniform float u_lightRad;
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    // TODO: Extract needed properties from the g-buffers into local variables
+    vec3 pos = gb0.xyz;
+    vec3 colmap = gb2.rgb;
+
+    // If nothing was rendered to this pixel, set alpha to 0 so that the
+    // postprocessing step can render the sky color.
+    if (depth == 1.0) {
+        gl_FragColor = vec4(0, 0, 0, 0);
+        return;
+    }
+    vec3 D = u_lightPos - pos;
+
+    float attenuation = max(u_lightRad - length(D), 0.0);
+
+    vec3 color = colmap * attenuation;
+    gl_FragColor = vec4(color, 1);
+}

glsl/deferred/edge.frag.glsl

@@ -0,0 +1,56 @@
+#version 100
+precision highp float;
+precision highp int;
+
+#define NUM_GBUFFERS 4
+
+uniform vec3 u_lightCol;
+uniform vec3 u_lightPos;
+uniform float u_lightRad;
+uniform sampler2D u_gbufs[NUM_GBUFFERS];
+uniform sampler2D u_depth;
+uniform vec2 u_pixSize;
+uniform float u_kernel[9];
+
+
+
+varying vec2 v_uv;
+
+vec3 applyNormalMap(vec3 geomnor, vec3 normap) {
+    normap = normap * 2.0 - 1.0;
+    vec3 up = normalize(vec3(0.001, 1, 0.001));
+    vec3 surftan = normalize(cross(geomnor, up));
+    vec3 surfbinor = cross(geomnor, surftan);
+    return normap.y * surftan + normap.x * surfbinor + normap.z * geomnor;
+}
+
+void main() {
+    vec4 gb0 = texture2D(u_gbufs[0], v_uv);
+    vec4 gb1 = texture2D(u_gbufs[1], v_uv);
+    vec4 gb2 = texture2D(u_gbufs[2], v_uv);
+    vec4 gb3 = texture2D(u_gbufs[3], v_uv);
+    float depth = texture2D(u_depth, v_uv).x;
+    vec3 pos = gb0.xyz;
+    vec3 colmap = gb2.rgb;
+
+    if (depth == 1.0) {
+        gl_FragColor = vec4(0, 0, 0, 0);
+        return;
+    }
+
+    vec4 color = vec4(0.0, 0.0, 0.0, 1.0);
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(-1, -1)) * u_kernel[0];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(0, -1)) * u_kernel[1];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, -1)) * u_kernel[2];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(-1, 0)) * u_kernel[3];
+    // color += texture2D(u_gbufs[2], v_uv) * u_kernel[4];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 0)) * u_kernel[5];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(-1, 1)) * u_kernel[6];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(0, 1)) * u_kernel[7];
+    // color += texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 1)) * u_kernel[8];
+    // color = texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(-1, -1)) - texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 1))
+    //       + texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(0, -1)) - texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 1))
+    //       + texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, -1)) - texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 1))
+    //       + texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(-1, -1)) - texture2D(u_gbufs[2], v_uv + u_pixSize * vec2(1, 1));
+    // gl_FragColor = color;
+}

glsl/red.frag.glsl → glsl/deferred/red.frag.glsl

@@ -2,6 +2,8 @@
 precision highp float;
 precision highp int;
 
+uniform float u_bounds[4];
+
 void main() {
     gl_FragColor = vec4(1, 0, 0, 1);
 }