For this project I plan on making an interactive reader that will allow the user to adjust “values” in a text and have those values change the information on the screen… but do that on a physical piece of paper.
This idea is an extension of Bret Victor’s Explorable Explorations project (found here: http://worrydream.com/#!/ExplorableExplanations). Using Victor’s API we can potentially read a piece of paper using a phone camera and apply the logic in AR.
Secondly, I need to get an AR setup in unity to be able to read the paper. I’ll have to look into API’s and methods of reading text from a camera, then positioning them in the game world based on the text’s location. Which is a huge task on it’s own.
For this project, I have about 12 weeks to work on this, so I’ll need to budget my time carefully. I’d like to have Explorable Explorations working with Unity by week 4, this will leave me 8 weeks to get the AR portion of the project done.
Ideas on how to overcome these issues
For AR it’ll be a bit trickier. Since I don’t have the resources to write a text reader on my own (and doing so would be overkill for this project anyways) I’ll have to create a cheat for our purposes. However, if I simply create a syntax that’ll tell the program where what words are, and that they have a variable attached to them, then we can overlay the unity text over them. The issue with this is that we’ll have to create a way to easily import and export these variables to make this application usable to a wide market, as well as making a syntax that’s dynamic enough to be used in multiple situations but not distracting to the user.
For a first draft I’m thinking something like this [ Variable ][#]. Unity will look for the square brackets, then grab the # after it to tell the code what variable it is. This is something that I’ll have to tackle more in the future and is likely to change.
Another option for this is to have a barcode over every variable, however this will defeat the purpose as it’ll make the text unreadable without a camera. We could potentially put a small QR code after the variables that the program looks for, but that’s another can of worms.
The third is that I could use a image to text API (Such as OCR https://ironsoftware.com/csharp/ocr/), however that would be taking some of the work away from my hands but could be useful if crunched for time.
External factors: Explorable Explorations| OCR (potentially)