How can I dynamically place text on top of images (in speech bubbles) using HTML5 Canvas?

Question

I'm making a Twine-based comic using HTML5 Canvas elements instead of images. The Canvas elements contain an image with a textbox on it. Later, I'll align this textbox with the speech bubble so I can alter the speech bubble contents dynamically using a custom <<panel>> widget. However, when I use the widget, nothing shows up.

WIDGET CODE

<<widget "panel">>
	/* convenience variables! */
	<<set _imgurl to $args[0]>>
	<<set _balloontext to $args[1]>>
	
	/* if there is just an image URL arg and no balloon text arg...*/
	<<if _imgurl && !_balloontext>>
		[img[$img + _imgurl]
	<</if>>
	
	/* if there is both... */
	<<if _imgurl && _balloontext>>
	  	<<script>>
			$(output).wiki("This text appears right now.");
			
			//create image but do not add it to page
			var img = document.createElement("img");
			
			$(output).wiki("This text does not appear!! Why?");
			
			var canvas = document.createElement("canvas");
			var imgUrl = State.variables[img]+State.temporary[imgurl];
			
			img.src = imgUrl;
			
			//add canvas to DOM
			var story = document.getElementById("story");
			story.appendChild(canvas);

			ctx = canvas.getContext('2d');
			canvas.width = img.width();
			canvas.height = img.height();
			ctx.drawImage(img, 0, 0);
			ctx.font = "36pt Verdana";

			//redraw image
			ctx.clearRect(0,0,canvas.width,canvas.height);
			ctx.drawImage(img, 0, 0);

			//refill text
			ctx.fillStyle = "red";
			ctx.fillText(State.temporary[balloontext],40,80);
	  	<</script>>
	<</if>>
<</widget>>

PASSAGE CODE

<<panel "panel0.png" "Hello!">>

(Note: $img is a global variable referring to the path I'm using for my images. The following Passage input creates an image successfully:)

[img[$img + "rosequartz.png"][Link]]

 

Edit:

Thanks to the answers below, this code worked:

window.panel = function(imgUrl, balloonText, link) {
	//if given both image url and balloon text
	if (imgUrl != null && balloonText != null) {	
		/*create image and canvas*/
		var img = document.createElement('img');
		var canvas = document.createElement('canvas');

		img.onload = function () {
			/*add canvas and image to DOM*/
			var story = document.getElementById("story");
			story.appendChild(img);
			story.appendChild(canvas);

			/*My image is 2560 x 960 pixels*/
			canvas.width = img.naturalWidth/2;
			canvas.height = img.naturalHeight/2;

			var ctx = canvas.getContext('2d');
			ctx.strokeStyle="#FF0000";
			ctx.strokeRect(0,0,canvas.width,canvas.height);
			ctx.drawImage(img, 0, 0, canvas.width, canvas.height);

			/*fill text*/
			ctx.font = "36pt Comic Sans MS";
			ctx.textBaseline = "top";
			ctx.fillStyle = "black";
			ctx.fillText(balloonText,620,100);
			$(story.content).wiki(canvas.height);
		}
		img.src = State.variables.img + imgUrl;
	}
	//if given only an image url
	else if (imgUrl != null && balloonText == null) {
		$(story.content).wiki("[img[$img + _imgurl]]");
	}
}

<<run panel("panel0.png", "Hello World!");>>

However, this caused major performance issues (scrolling lags a LOT with my trackpad) so I'll try using SVGs instead.

Answer 1

A. There are a number of syntax errors in your example:

1. The [img[imagename]] markup in your widget is missing a trailing close square bracket.

[img[$img + _imgurl]]

2. You need to use a String based key to access the properties of the State.variables and the State.temporary objects.

var imgUrl = State.variables['img'] + State.temporary['imgurl'];
...
ctx.fillText(State.temporary['balloontext'],40,80);

3. You need to define your ctx variable, using an operator like var, before you can use it.

var ctx = canvas.getContext('2d');

4. The width and height attributes of the img element are not functions.

canvas.width = img.width;
canvas.height = img.height;

B. There is a timing issue in your example.
The width and height attributes of the img element won't have a non-zero value until the referenced image is visualised, this means your canvas element has a width & height of zero. I believe that this same timing issues (and the lack of image visualisation) may also be the reason why the drawImage() function calls are not working.

I tested this theory by also adding code like the following to the relevant sections of your example...

story.appendChild(img);

canvas.width = 300;
canvas.height = 300;

I then uses the following jQuery within the Developer Console after the Passage was rendered, and the now visualised img element was successfully drawn on to your canvas element.

$('#story>canvas')[0].getContext('2d').drawImage($('#story>img')[0], 0, 0);

C. Widget vs Custom Macro.
The majority of the code within your widget is Javascript, because of that I suggest using the Macro API to create a Custom Macro instead.

Comments

2. You need to use a String based key to access the properties of the State.variables and the State.temporary objects.

var imgUrl = State.variables['img'] + State.temporary['imgurl'];
...
ctx.fillText(State.temporary['balloontext'],40,80);

While it would work that way, it makes more sense to just access the data using dot notation:

var imgUrl = State.variables.img + State.temporary.imgurl;
...
ctx.fillText(State.temporary.balloontext,40,80);

C. Widget vs Custom Macro.
The majority of the code within your widget is Javascript, because of that I suggest using the Macro API to create a Custom Macro instead.

A macro is unnecessary there.  Just make it a JavaScript function, and then call it using "<<run FunctionName(Parameters)>>".

---

Thanks for this very thorough answer. I had completely missed all those syntax errors! I fixed them.

The canvas and its text display properly now. I also added a border. However, the image still doesn't show up, and putting the canvas rendering functions in :passageend or :passagedisplay didn't alleviate the timing issue.

I also had to specify the width and height manually instead of relying on img.width and img.height, or else the canvas would not show up due to the invalid size.

<<script>>
		/*create image but do not add it to page*/
		var img = document.createElement('img');
		
		var canvas = document.createElement('canvas');
		var imgUrl = State.variables.img+State.temporary.imgurl;
		
		img.src = imgUrl;
		img.id = "image";
		/*add canvas and image to DOM*/
		var story = document.getElementById("story");
		story.appendChild(img);
		story.appendChild(canvas);
		/*My image is 2560 x 960 pixels, but my CSS scales it to 100% of #story*/
		canvas.width = 2560;
		canvas.height = 960;
		
		$(document).one(':passageend', function (story) {
			var ctx = canvas.getContext('2d');
			ctx.strokeStyle="#FF0000";
			ctx.strokeRect(0,0,canvas.width,canvas.height);
			var x = document.getElementById("image");
			ctx.drawImage(x, 0, 0);
			ctx.font = "36pt Comic Sans MS";
			/*fill text*/
			ctx.fillStyle = "yellow";
			ctx.fillText(State.temporary.balloontext,0,80);
    		$(story.content).wiki(canvas.height);
		});
<</script>>

 

---

The image doesn't load immediately, and the JavaScript doesn't wait for it to load by default, so you're probably trying to work with an image that isn't there yet.

You'll probably want to do something like this:

<<script>>
	/*create image but do not add it to page*/
	var img = document.createElement('img');
	var canvas = document.createElement('canvas');
	var imgUrl = State.variables.img+State.temporary.imgurl;

	img.id = "image";
	img.onload = function () {
		/*add canvas and image to DOM*/
		var story = document.getElementById("story");
		story.appendChild(this);
		story.appendChild(canvas);
		/*My image is 2560 x 960 pixels, but my CSS scales it to 100% of #story*/
		canvas.width = 2560;
		canvas.height = 960;
		
		$(document).one(':passageend', function (story) {
			var ctx = canvas.getContext('2d');
			ctx.strokeStyle="#FF0000";
			ctx.strokeRect(0,0,canvas.width,canvas.height);
			var x = document.getElementById("image");
			ctx.drawImage(x, 0, 0);
			ctx.font = "36pt Comic Sans MS";
			/*fill text*/
			ctx.fillStyle = "yellow";
			ctx.fillText(State.temporary.balloontext,0,80);
    			$(story.content).wiki(canvas.height);
		});
	}
	img.onerror = alert('Error: Unable to load image.');
	img.src = imgUrl;
<</script>>

(Note: I didn't test that, but I think that will work.  "this" should be equivalent to "img" within the function that img.onload calls.)

You can change the "img.onerror" to do whatever you want if the image fails to load.

See also:

https://stackoverflow.com/questions/1977871/check-if-an-image-is-loaded-no-errors-in-javascript

---

A macro is unnecessary there.  Just make it a JavaScript function, and then call it using "<<run FunctionName(Parameters)>>".

While placing the bulk of the Javascript code within a Javascript function does help abstract that code and makes it more re-usable, it doesn't address the main issue of using a TwineScript based widget to execute what is essentially a Javascript-only solution and thus adding an unnecessary overhead (of using TwineScript to access/execute Javascript) to the process. Said overhead lessen (disappears?) if the widget is a custom macro.

---

@greyelf, I was talking about changing the entire widget into a JavaScript function, not merely part of it.  The first "it" in my sentence referred to the widget.

---

@kryptot7
Change the Javascript in your last example to the following and the Image & Text will appear in the canvas.

	/*create image but do not add it to page*/
	var img = document.createElement('img');
	var canvas = document.createElement('canvas');
	var imgUrl = State.variables.img + State.temporary.imgurl;

	img.onload = function () {
		/* Add the canvas to the DOM. */
		var story = document.getElementById("story");
		story.appendChild(canvas);

		/* Adjust the canvas's size to fit the whole image. */
		canvas.width = this.width;
		canvas.height = this.height;
		
		var ctx = canvas.getContext('2d');

		/* Add a border. */
		ctx.strokeStyle = "#FF0000";
		ctx.strokeRect(0, 0, canvas.width, canvas.height);

		/* Draw the image. */
		ctx.drawImage(this, 0, 0);

		/* Show the balloon text. */
		ctx.font = "36pt Comic Sans MS";
		ctx.fillStyle = "yellow";
		ctx.fillText(State.temporary.balloontext, 0, 80);

   		/*$(story.content).wiki(canvas.height);*/
	};
	img.onerror = function () {
		alert('Error: Unable to load image.');
	};
	img.src = imgUrl;

 

Answer 2

The problem you're having is related to the fact that the <<script>> content (at least when ran from Twine2 itself) gets stripped of newlines. So this line in particular ...

			//create image but do not add it to page

... comments out everything that follows, including all the other lines. Try using comments like these instead:

			/* create image but do not add it to page */

... and then you can start debugging all the other problems greyelf pointed out. :)

Comments

[...] the <<script>> content (at least when ran from Twine2 itself) gets stripped of newlines.

Newlines would only be modified if the <<script>> macro invocation was placed within the business-end of one of the nobr features.  If that did happen to be the case, then using line comments would be a bad idea, yes.

---

I was using a "nobr" tag on my widget passage, so this was part of the problem. Changing the line comments to this format

/* comment */

 fixed the issue of my code not executing.