My posts tend to be 'off the cuff' - meaning I'm just writing out in 'one go' about stuff I'm currently thinking about. Not really a lot of pre-planning (in most cases, save for tutorials). Though I do go back and add bits, correct grammar errors, and put in links, pictures, etc. So apologies if you were expecting highly formalized PR or Marketing spiel. ;) (Yes, I know. You weren't!)

Enhanced eBooks, AR books, ePUB, glTF, webGL, publisher support? - a brief summary of what I found when trying to get a handle on enhancing the book experience

What exactly is an enhanced eBook? Or even an enhanced (non-electronic) book?

Those were the questions I had when I was initially trying to understand how to improve the book experience. My thought process started back when I was interning with a Marketing professor - I'd gone back to school, you see, to get a Communications degree, after years in science.

The professor was working on media engagement mechanisms. Long story short, she was finding that the more the reader had to interact with the media, the more they remembered it.  I'd seen other research like this, with interactive games as the subject, years earlier - with experimental evidence. Experimental evidence is the pinnacle of proof in science. It's not hypothesis, not simply correlation, it's a causal connection. In other words, it's not just a hunch that A is related somehow to B, it shows that A causes B.  So I wasn't surprised that she was finding similar results with correlation studies.

But it did convince me that to enhance books, they needed to be more interactive.

Now, I'd seen attempts at interaction before - growing up - from pop-up books, to cut-out and make something books, to audio books, to even "make your own adventure" books where you made choices and then flipped to that section (versus some other section). It was only a matter of time before someone tried to create interaction with computers.

That brings us to eBooks. Now, for a good introduction to what is enhanced and what isn't in books, try this link.  It also includes why you don't see enhancements a great deal - meaning, the major eBook publishers don't support it. And as for Google Books, which does support such things in its ePUB. They still aren't open to small, indie publishers after closing Google Books to new publishers back in 2015*. Apparently, big publishing houses can still publish, but they aren't really doing much with enhanced books either - at least from my perspective. You might know different, so if you do, please comment and share what you know.

 *Update January 14, 2017: Apparently, Google Books is taking publishers again, including indie self-publishers, from select countries. No guarantees, but it seems they want to test out the new system. Google Books does support some parts of ePub 3, embedding video, gifs, audio...but not interactivity (no scripting), etc.

Also, if you want to learn more about scientific visualization in documents, and why it is important and useful, try this paper.

So, while I originally thought about animated books (meaning, I would embed video, audio and 2D and 3D animations in my books with interactivity, hopefully), when I tried to find out if Barnes and Noble, Amazon or Google supported that - I found the doors closed, shuttered and nailed up with boards. (see note above about Google Books taking down a few nails...)

Then I began working with a VR standards group, and it got me thinking about VR and then AR as a way to enhance books. When I saw Metaio and Vuforia demos, I realized that yes, this was a good mechanism.

Was AR a perfect mechanism? Hell, no! 


There are so many technical hurdles with AR books that it will make your head spin. And while there are AR books out there, from the National Geographic book (a Vuforia based one that was one of their earliest adopters) to Devar books (Devarbook.com) with their childrens' books using AR (Vuforia, I believe), and toys and marketing signage...there are even museum exhibits now...but overall, book-wise, it's still pretty sparse.

Some links to National Geographics continuing work with AR.
https://video.nationalgeographic.com/video/augmented-reality-sci
and
https://www.youtube.com/watch?v=lOe5HWXdomk
and
https://www.nationalgeographic.com/magazine/2017/03/explore-pokemon-augmented-reality/
and
https://www.youtube.com/watch?v=D0ojxzS1fCw&list=PL07GAekWOvTzyK42MXKgdlLHvVM0xSxY9&index=5

Why? Easy. Technical hurdles, of course, but also artistic ones - making 2D and 3D interactive artwork for an AR book is similar to making games. It's a lot more than most illustrators are used to - or prepared for.

It would be much easier to simply be able to embed animations in some standard eBook format - though even that takes a lot more work than still images. But I looked carefully at that first.

So what do I find?


ePUB can be embedded with animations, even though finding a distribution channel would be tricky since neither Kindle nor Nook support it, and Google has shut out indies for the time being. I won't speak to Apple, because I don't know - though I'm told Apple and Kobo both have such support.

So right now, Apple and Kobo seem the best options if you want to do enhanced eBooks (non-AR).

You might want to just publish on PDF if you simply want your document to have animations and other enhancements. You can embed 3D content in PDFs, but for book authors, that is probably not a great option for selling your work.  Still, it's an option.

You might want to wait for more support for glTF embedding. Microsoft Office will be supporting 3D interactive content embedding using glTF, which is a Kronos back OpenGL based transmission format. In other words, a standard way to pass heavy 3D stuff around in a non-proprietary format that any app could (potentially) read. Kronos sees strong support growing for glTF. See link.

Here's an announcement regarding Microsoft's joining of the glTF effort.
https://blogs.windows.com/buildingapps/2016/10/28/bringing-3d-to-everyone-through-open-standards/

and, perhaps more importantly to embedding 3D interactive content, a post from the Office team on how they will support embedding 3D models into documents.

https://answers.microsoft.com/en-us/msoffice/forum/msoffice_officeinsider-mso_win10/new-3d-in-office-now-available-to-office-insiders/cac671a4-56de-4cbd-9d48-ba7efdf0eea2?page=2&auth=1

So what's the answer? AR? Well, I'm pursuing that, of course, but it's a long road and not an easy content creation path to follow.  Waiting for Nook, Kindle and Google to save us?  Well, for many, that is very likely - though you can enhance your books using Apple or Kobo.

To be honest, I don't know the answer. There are companies like Adobe Mixamo, trying to give you a library of 3D assets to play with and use - which Adobe wants you to use in Flash, of course. You're not limited to Flash. In fact, apparently you can export to other formats so you can import your rigs into game engines, or if you want to have a longer pipeline, to glTF - so that you could embed in WebGL pages.

Interesting article/tutorial here on doing just that (from Mixamo to Blender to WebGL).

But to me, that isn't original content, nor typically fit for animations in fiction. In fact, it just seems like a 3D version of the 2D selection of stock images that Office would/will let you use to 'spice' up your document.

Nor am I ready to say just put your book online as a webpages with WebGL content, or as a PDF or as a DOCX file. Books may end up that way, but the first requires having a server backend and the latter two aren't the currently preferred eBook formats.

 I can't give you any real answers yet. I'm still working on them myself. I just wanted to sort of walk through what is currently going on in the enhanced book experience. I'm sure there is more than this. I'm sure I've missed important points. So feel free to comment and let me know your thoughts.

AR is a good enhancement, but so is video and potentially VR, and even just embedded animations. There is progress on giving book 'illustrators' new tools for creating interactive content - some easier (like Mixamo) than others (Unity, Blender, Photoshop, Gimp, Painter...Flash...others?) AR providers are also trying to create tools, ARKit, ARCore, Vuforia, others.

But nothing is really a full pipeline yet, nor easy enough to make it usable to most book writers or book illustrators (perhaps we will see a new breed of book animators?)

Even my work, which is focused on AR for books, is a bit cumbersome and doesn't allow for an easy pipeline nor even an integrated one (i.e. I'm not writing a book app like Nook or Kindle, just a way to do AR within Unity, with 2D/3D assets created in a tool like Blender, Maya, Painter, etc.)

Note: Exciting stuff! I just discovered this tutorial!!!
https://www.awn.com/animationworld/tutorial-take-sketches-finished-3d-characters-iclone

What it tells me is that you can start with a sketch and map that onto a character, then modify and rig and animate the character. It uses tools like CrazyTalk 8 and iClone 6 from Reallusion. I find the idea pretty exciting. I had thought about mapping a sketch to a base body/face model myself using OpenCV but put it on the TODO list for later. If someone else has done it, and for a reasonable price, then I'm cool with that!

In fact, looking at the Reallusion website, I'd say this is the secret behind a lot of the cool animated paintings you've seen in commercials, websites and even in AR (particularly a museum AR, done by a Canadian design house, where I saw the paintings 'come to life'.  Looks like they might not have 'rolled their own' and this was, at least in part, a big portion of their magical pipeline. :)  (The image from their website shows an animation of various characters, including this one from the famous Ingres' painting, I believe. In it she is rocking her head back and forth.)

It may be quite helpful in trying to streamline a pipeline that goes from an illustrators sketches to a 3D animated scene as the illustration. I haven't gone through everything yet so I can't tell if CrazyTalk Animator is different than CrazyTalk, etc. but these tools are definitely on my radar now.

The only thing I can say is that interactivity in books will increase engagement, research has shown that. And, that interactivity does not mean 'games' though many have taken it that way. It just means being able to touch the object, move it, look at it from different directions, have it react to your presence...and so on.

You don't need AR or VR for that. You  just need to be able to affect some change to the illustration (like have it giggle or move or say something to you.)

I'm going to continue to look at embedding animations in documents (ePUB, PDF, DOCX) and in webpages (WebGL), because I think that will help me in refining creating apps that enhance books - or even in forcing a merging between the two. Google may have barred indies from their books platform, but that still leaves the whole Google Play world open.

And I think I'm going to take a second look at Kobo and see how far they will support enhanced books.

I'm hoping to find a nice blend between AR, VR and embedded animation. Or at least, a set of workflows that are optimal.

What do you think?

Comments

Post a Comment

Popular posts from this blog

Getting started with Unity's new Shader Graph Node-based Shader Creator/Editor (tutorial 6 - Getting Glow/Bloom Effect wihout Post-Processing by Inverting Fresnel...Sort Of...)

Image
Introduction (to the Tutorial Series): I will be writing about my own experiences using Unity's new beta Shader Graph, part of its upcoming 2018 release (also in beta).  The Shader Graph lets you create a variety of Unity shaders using nodes - not requiring you to write code. I will be writing about this journey over multiple posts, usually spaced about a week apart. Each post will be a short tutorial on how to use various node types to create different shader effects. And will include brief discussions on types of shaders and their uses, and later, how the Shader Graph compares with code-based shaders. I will try not to get overly technical, but will try to give you an idea of the complexity involved in shaders, from lighting to vertex and fragment manipulation. Given the nature of beta software, expect the Shader Graph (and later tutorials) to vary from earlier ones that you can now find online. Even within my tutorials there will be changes if/when the beta evolves - in...
Read more

Getting started with Unity's new Shader Graph Node-based Shader Creator/Editor (tutorial 5 - Exploring Fresnel/Color Rim and Update on Vertex Displacement Attempts)

Image
Introduction (to the Tutorial Series): I will be writing about my own experiences using Unity's new beta Shader Graph, part of its upcoming 2018 release (also in beta).  The Shader Graph lets you create a variety of Unity shaders using nodes - not requiring you to write code. I will be writing about this journey over multiple posts, usually spaced about a week apart. Each post will be a short tutorial on how to use various node types to create different shader effects. And will include brief discussions on types of shaders and their uses, and later, how the Shader Graph compares with code-based shaders. I will try not to get overly technical, but will try to give you an idea of the complexity involved in shaders, from lighting to vertex and fragment manipulation. Given the nature of beta software, expect the Shader Graph (and later tutorials) to vary from earlier ones that you can now find online. Even within my tutorials there will be changes if/when the beta evolves - incl...
Read more

Getting started with Unity's new Shader Graph Node-based Shader Creator/Editor (tutorial 2 - tiling, offsets, blending, subgraphs and custom channel blending)

Image
Introduction (to the Tutorial Series): I will be writing about my own experiences using Unity's new beta Shader Graph, part of its upcoming 2018 release (also in beta).  The Shader Graph lets you create a variety of Unity shaders using nodes - not requiring you to write code. I will be writing about this journey over multiple posts, usually spaced about a week apart. Each post will be a short tutorial on how to use various node types to create different shader effects. And will include brief discussions on types of shaders and their uses, and later, how the Shader Graph compares with code-based shaders. I will try not to get overly technical, but will try to give you an idea of the complexity involved in shaders, from lighting to vertex and fragment manipulation. Given the nature of beta software, expect the Shader Graph (and later tutorials) to vary from earlier ones that you can now find online. Even within my tutorials there will be changes if/when the beta evolves - incl...
Read more