numgl

Image processing with WebGL.

87
5
JavaScript

NumGL

  • Apply image processing algorithms using WebGL.
  • Supports pictures, videos, webcam streams and user defined arrays.
  • Everything is sent to the fragment shader as 2D textures.
  • Let fragment shaders handle the hard work.
  • Blog post.

Docs

Store everything as a texture


  • The store_* functions return a storageId, which is passed into the processing functions.
window.onload = function() {
	// Send the picture to WebGL as texture. To do this with video/webcam...
	// ... change store_picture to store_video / store_webcam.
	var storageId = numgl.store_picture("imgElementID");
	numgl.show_texture(storageId);
}

Greyscaling images


numgl.grey(storageId);

Fiddle with it:


Convolution kernel


// [-1,-1,-1,0,0,0,1,1,1] is a 3x3 kernel, where:
// - 1st row is [-1,-1,-1]; 2nd row is [0,0,0]; 3rd row is [1,1,1].
numgl.convolution(storageId, [-1,-1,-1,0,0,0,1,1,1]);

// Other kernels are supported.

JS Fiddle with it:


Thresholding


// 80 is the thresholding value.
numgl.threshold(storageId, 80);

JS Fiddle:


Example - Combine convolution with threshold.


  • The image processing functions return the GLSL variable that will hold the final pixel value.
  • This return variable can be used to chain call other image processing functions.
window.onload = function() {
	var imageId = numgl.store_picture("image");

	numgl.show_canvas(imageId);
	// Convolution followed by threshold
	var convResult = numgl.convolution(imageId,[-1,-1,-1,0,0,0,1,1,1]);
	numgl.threshold(convResult,10);
	numgl.do_it();
}

JS Fiddle:


Other options


  • Enable / disable fps for video and webcam
// If this is not set, the fps are not shown.
numgl.set_fps_element(fpsElementId);

  • Store JS arrays as RGBA textures
// Draws the following 2x2 RGBA texture: 
// [white, black,
// black, white]

// numgl.store_array()'s last 2 args are width and height.
var arrayId = numgl.store_array([255,255,255,255,
								0,0,0,255,
								0,0,0,255,
								255,255,255,255], 
								2, 2);

// show_texture() calls do_it() internally.
numgl.show_texture(arrayId);

  • Read pixels from canvas
// If no width and height are specified, read_canvas() will read the whole canvas.
console.log(numgl.read_canvas().toString());

// Using the above example the result would be: "255,255,255,255,0,0,0,255,0,0,0,255,255,255,255,255"

  • Flip texture - arrays aren’t flipped, but images and videos are by default (texture and clipspace coordinates are different). JS Fiddle example
// Called before numgl.do_it()
numg.textures[storageId].flipTexture = true/false
// Paint the screen blue.
numgl.fragColor = "vec4(0, 0, 1, 1);"
  • See the generated GLSL code.
// Fragment shader code. Vertex shader code function is vs_code().
console.log(numgl.fs_code("pretty"));

How does it work

  • Video and webcam use requestAnimationFrame so the video/webcam frames are drawn multiple times. Pictures are only drawn once.

  • Javascript functions generate GLSL code, and when the user calls numgl.do_it() that GLSL code is compiled and executed. eg:

window.onload = function() {
	// "image" is the <img> tag ID.
	var imageId = numgl.store_picture("image");

	numgl.grey(imageId);
	// Console log the GLSL code generated by the numgl.grey() call:
	console.log(numgl.fs_code("pretty"));
}

The resulting GLSL code: (uTexture0 is the stored texture - it can be an array, picture or video frame)

precision highp float;
uniform vec2 uResolution;
uniform sampler2D uTexture0;
uniform vec2 uTextureSize0;
varying vec2 vTextCoords;

void main(void) {
	float float_0 = 0.2126 * texture2D(uTexture0, vTextCoords).r;
	float float_1 = 0.7152 * texture2D(uTexture0, vTextCoords).g;
	float float_2 = 0.0722 * texture2D(uTexture0, vTextCoords).b;
	float float_3 = float_0 + float_2 + float_1;
	vec4 vec4_0 = vec4(float_3,float_3,float_3,1);

	gl_FragColor = vec4_0;
 }
  • The final fragment color (gl_FragColor) is set by these GLSL code creating functions, and can be checked or set manually using numgl.fragColor. JS Fiddle example.

Cross-origin / Same-origin resources


  • For local images / video, you’ll need to set up a local server. It’s easy, just open up your console, go to your project’s folder and type in python3 -m http.server 8000.

  • Images and video from other websites require the inline HTML attribute crossorigin="anonymous" - see the img tag here, for example.


Webcams on Chrome


  • Webcams only work on https sites if you’re using Chrome. On Firefox, webcams should work fine (for now).