AR interactive reader exploration

Development Blogs

For this project I plan on making an interactive reader that will allow the user to adjust “values” in a text and have those values change the information on the screen… but do that on a physical piece of paper.

This idea is an extension of Bret Victor’s Explorable Explorations project (found here:!/ExplorableExplanations). Using Victor’s API we can potentially read a piece of paper using a phone camera and apply the logic in AR.


There are two key parts of this that I’ll need to dive into, firstly Explorable Explorations is written in HTML and JavaScript… of witch I know neither. So I’ll need to find a way to convert their project into something more comfortable for my skills as a programmer. For this, I plan on making a Unity Android app that will interface with Victor’s project and do the necessary functions.

Secondly, I need to get an AR setup in unity to be able to read the paper. I’ll have to look into API’s and methods of reading text from a camera, then positioning them in the game world based on the text’s location. Which is a huge task on it’s own.

For this project, I have about 12 weeks to work on this, so I’ll need to budget my time carefully. I’d like to have Explorable Explorations working with Unity by week 4, this will leave me 8 weeks to get the AR portion of the project done.

Ideas on how to overcome these issues

So I have a few ideas on how to deal with the issues noted above. For the HTML & JavaScript part, I’m hoping that once I do a deep dive into Victor’s code I’ll find a way to re-rout the JavaScript functionality into C# so I can function with Unity. Using WebGL is another option as it already functions in the web. The third option is to write a plugin that’ll display an HTML text file into Unity, then using that to function.

For AR it’ll be a bit trickier. Since I don’t have the resources to write a text reader on my own (and doing so would be overkill for this project anyways) I’ll have to create a cheat for our purposes. However, if I simply create a syntax that’ll tell the program where what words are, and that they have a variable attached to them, then we can overlay the unity text over them. The issue with this is that we’ll have to create a way to easily import and export these variables to make this application usable to a wide market, as well as making a syntax that’s dynamic enough to be used in multiple situations but not distracting to the user.

For a first draft I’m thinking something like this [ Variable ][#]. Unity will look for the square brackets, then grab the # after it to tell the code what variable it is. This is something that I’ll have to tackle more in the future and is likely to change.

Another option for this is to have a barcode over every variable, however this will defeat the purpose as it’ll make the text unreadable without a camera. We could potentially put a small QR code after the variables that the program looks for, but that’s another can of worms.

The third is that I could use a image to text API (Such as OCR, however that would be taking some of the work away from my hands but could be useful if crunched for time.

Quick Notes

Points of exploration: Making a plugin to get HTML & JavaScript code to work with Unity | Augmented Reality

Unknowns: Augmented Reality theory | JavaScript | HTML

External factors: Explorable Explorations| OCR (potentially)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s