numgl

Image processing with WebGL.

JavaScript

NumGL

Apply image processing algorithms using WebGL.
Supports pictures, videos, webcam streams and user defined arrays.
Everything is sent to the fragment shader as 2D textures.
Let fragment shaders handle the hard work.
Blog post.

Docs

Store everything as a texture

The store_* functions return a storageId, which is passed into the processing functions.

window.onload = function() {
	// Send the picture to WebGL as texture. To do this with video/webcam...
	// ... change store_picture to store_video / store_webcam.
	var storageId = numgl.store_picture("imgElementID");
	numgl.show_texture(storageId);
}

Greyscaling images

numgl.grey(storageId);

Fiddle with it:

Convolution kernel

// [-1,-1,-1,0,0,0,1,1,1] is a 3x3 kernel, where:
// - 1st row is [-1,-1,-1]; 2nd row is [0,0,0]; 3rd row is [1,1,1].
numgl.convolution(storageId, [-1,-1,-1,0,0,0,1,1,1]);

// Other kernels are supported.

JS Fiddle with it:

Thresholding

// 80 is the thresholding value.
numgl.threshold(storageId, 80);

JS Fiddle:

Example - Combine convolution with threshold.

The image processing functions return the GLSL variable that will hold the final pixel value.
This return variable can be used to chain call other image processing functions.

window.onload = function() {
	var imageId = numgl.store_picture("image");

	numgl.show_canvas(imageId);
	// Convolution followed by threshold
	var convResult = numgl.convolution(imageId,[-1,-1,-1,0,0,0,1,1,1]);
	numgl.threshold(convResult,10);
	numgl.do_it();
}

JS Fiddle:

Image convolution + threshold

Other options

Enable / disable fps for video and webcam

// If this is not set, the fps are not shown.
numgl.set_fps_element(fpsElementId);

Store JS arrays as RGBA textures

// Draws the following 2x2 RGBA texture: 
// [white, black,
// black, white]

// numgl.store_array()'s last 2 args are width and height.
var arrayId = numgl.store_array([255,255,255,255,
								0,0,0,255,
								0,0,0,255,
								255,255,255,255], 
								2, 2);

// show_texture() calls do_it() internally.
numgl.show_texture(arrayId);

Read pixels from canvas

// If no width and height are specified, read_canvas() will read the whole canvas.
console.log(numgl.read_canvas().toString());

// Using the above example the result would be: "255,255,255,255,0,0,0,255,0,0,0,255,255,255,255,255"

Flip texture - arrays aren’t flipped, but images and videos are by default (texture and clipspace coordinates are different). JS Fiddle example

// Called before numgl.do_it()
numg.textures[storageId].flipTexture = true/false

Manually define fl_FragCoord. JS Fiddle example

// Paint the screen blue.
numgl.fragColor = "vec4(0, 0, 1, 1);"

See the generated GLSL code.

// Fragment shader code. Vertex shader code function is vs_code().
console.log(numgl.fs_code("pretty"));

How does it work

Video and webcam use requestAnimationFrame so the video/webcam frames are drawn multiple times. Pictures are only drawn once.
Javascript functions generate GLSL code, and when the user calls numgl.do_it() that GLSL code is compiled and executed. eg:

window.onload = function() {
	// "image" is the <img> tag ID.
	var imageId = numgl.store_picture("image");

	numgl.grey(imageId);
	// Console log the GLSL code generated by the numgl.grey() call:
	console.log(numgl.fs_code("pretty"));
}

The resulting GLSL code: (uTexture0 is the stored texture - it can be an array, picture or video frame)

precision highp float;
uniform vec2 uResolution;
uniform sampler2D uTexture0;
uniform vec2 uTextureSize0;
varying vec2 vTextCoords;

void main(void) {
	float float_0 = 0.2126 * texture2D(uTexture0, vTextCoords).r;
	float float_1 = 0.7152 * texture2D(uTexture0, vTextCoords).g;
	float float_2 = 0.0722 * texture2D(uTexture0, vTextCoords).b;
	float float_3 = float_0 + float_2 + float_1;
	vec4 vec4_0 = vec4(float_3,float_3,float_3,1);

	gl_FragColor = vec4_0;
 }

The final fragment color (gl_FragColor) is set by these GLSL code creating functions, and can be checked or set manually using numgl.fragColor. JS Fiddle example.

Cross-origin / Same-origin resources

For local images / video, you’ll need to set up a local server. It’s easy, just open up your console, go to your project’s folder and type in python3 -m http.server 8000.
Images and video from other websites require the inline HTML attribute crossorigin="anonymous" - see the img tag here, for example.

Webcams on Chrome

Webcams only work on https sites if you’re using Chrome. On Firefox, webcams should work fine (for now).