AR Storytelling

Interactive Storytelling: Step Into the Magic

Project:

Transforming traditional narratives into interactive, physical-digital experiences

My Contributions:

Led UX for an AR storytelling game designed for children aged 5–8, blending physical tokens with interactive 3D animations.
Conducted user research and usability testing with kids and parents to inform design decisions and validate experience flow.
Designed and prototyped immersive scenes using Blender, Mixamo, Procreate, and Unity AR.
Created story-driven interactions that matched children's developmental needs, encouraging creativity, movement, and emotional engagement.
Synthesized behavioral insights to drive iterative improvements in UI, animation cues, and instructional clarity.

Role: UX Researcher | UX Designer

Duration: 8 Months

Background & Inspiration

AR Snake and Ladder - Project created in April 2023.

My earlier AR experiments, such as an AR-enabled Snake and Ladder game using tablets, revealed how virtual elements layered onto physical gameplay could evoke powerful emotional responses, including surprise and even fear when a snake appeared on the board.

PageCraft - A storytelling and crafting platform

The research paper called PageCraft (which enables children to replicate and build stories), made me curious if this technology can drive creativity and learning in young children.

How can augmented reality storytelling experiences be designed to effectively engage children aged 5–8 in learning, while aligning with their cognitive, emotional stages?

Market Research

The market research indicates a robust presence of augmented reality (AR) products for children, which is promising for the industry. Among these are AR storybooks and animation storytelling products that incorporate voice interactions. However, these products generally do not allow children the autonomy to control the narrative's progression, limiting their interactive experience.

Literature Review

Child-friendly UI: Big touch targets, bright visuals, and friendly audio cues let early readers explore intuitively without needing advanced reading or fine motor skills.
Story-driven design: Embed learning tasks in a playful narrative so kids unlock new story elements by completing challenges, reinforcing education through fun.
Multi-sensory AR: Combine 3D visuals, sound effects, and interactive elements so children see, hear and “touch” learning concepts in their real environment, making abstract ideas concrete.
Intuitive AR interactions: Use the device camera as a “magic window” — kids naturally discover digital content by pointing and moving, eliminating complex instructions or menus.
Immediate feedback & rewards: Offer instant, positive responses (animations, sounds, or stickers) for each success, keeping young learners motivated with frequent mini-wins.
Safety & comfort: Ensure comfortable play (lightweight devices, easy posture) and keep sessions short to prevent fatigue, making the experience age-appropriate and safe.
Collaborative play: Include co-play options (friends, siblings, or guided parent involvement), reflecting that children learn best in social, guided environments.
Research-driven iteration: Test early and often with kids and educators to fine-tune pacing, difficulty and clarity, ensuring the design truly resonates with the target audience.
Learning alignment: Map game actions and story outcomes to core educational goals (counting, reading, science concepts) so narrative progress equals real learning progress.
Technical robustness: Prioritize stable AR tracking and graceful error handling so technology enhances the experience (instead of distracting), keeping focus on the story and learning.

Concept & Story Selection

To ensure relatability and immersive narrative:

I defined selection criteria: familiar tale, magical realism, limited characters, single protagonist and for ages 5-8.
Jack and the Beanstalk was selected for its narrative simplicity, dramatic structure, and interactive potential.

Experience Design Strategy

I broke the story into five chapters, each focused on a single interaction.
Key activities included:

Storyboarding to design interaction points, visuals, and core animations
Prototyping using Blender (3D models + animations), Mixamo (character animations), Procreate (background sketches), and ChatGPT (age-adapted narrative)
Physical tokens acted as triggers for 3D characters and scenes in AR
Developed prototype using Unity AR.

Story Chapter Illustration

Token Illustration

Blender Animation

Storyboarding

Unity Prototype

Rendered 3D

Sneak Peak:
Chapter 4: Giant's House

In the prototype, Jack skips meeting the giantess and goes straight into the giant’s room to retrieve the Gold coins and Hen, simplifying the story by reducing characters.

Pilot Testing

To validate desirability and appropriateness, I conducted online interviews and demo sessions with 2 mothers of kids aged 5–8.

To validate user flow, I tested the AR storytelling game with two Industrial Design Students.

Two six-year-olds interacted with the prototype using a tablet and physical board. I observed behaviors and engagement patterns.

Key Insights

Parents:

High potential to engage kids who dislike reading
Suggested improvements:
- Improve consistency between illustrations and 3D characters
- Include instruction guide
Both moms expressed interest in buying the product for their children.

Industrial Design Students:

Demonstrated creativity (e.g., making the cow drink water).
Observed challenges with token placement and spatial context.
Highlighted need for better visual feedback and scene continuity.

6 Year Olds :

Overlong text disrupted engagement.
Kids created their own play paths (e.g., chopping & regrowing beanstalk).
Children attempted to interact with AR elements using real-world gestures (e.g., reaching into tablet screen).
Instructions and scene transitions were unclear, affecting flow.

Design Iterations

The constructive feedback received from preliminary testing resulted in iterations.

01. Addition of Instructions

a. There are 4 tokens.
b. Always drag the token on the story.
c. Do not cover the token with hand.
d. Read the story and follow along.

02. Dividing the text

03. Added beanstalk interactions.

04. Revised the giant illustration

05. Addition of giant following Jack

06. Adding feedbacks and clues.

The exchange was followed by the 'coin' sound and an animating ring as a feedback.

The animating ring was added to guide the user to sow the beans in the particular region.

Usability Testing with 7 Children (Ages 5–8)

The final prototype was tested with 7 participants. I used a mix of observation and semi-structured guidance to collect insights.

Key Observations

The feedback were compiled based on the observations from the participant.

Instructions

5- and 7-year-olds were visibly delighted by AR token transformations. "Can I try Moo?"
Some younger (especially aged 5) children appeared nervous or needed help with instructions.

Story Chapter 1: Introduction

The P1 (7M) and P5 (7F) explored and went beyond the tasks. As mentioned in the observation -

One child tried to push the cow off the table.
The other tried to feed his finger to the animating cow and acted as if the cow bit them.

Story Chapter 2: Exchanging the beans

P1 (7M) made some funny remarks on the selling of the cow. Upon completing the task, P5 (7M) said “Yes” as a sign of achievement.

Expressed joy when completing tasks successfully.

The animation did not work for P4 (5M) initially. When it finally worked, they did not express any emotions.

65% of participants read the text on the screen.

Story Chapter 3: Growing of Beanstalk

The P2 (5M), P4 (5M), P5 (7M), and P6 (7F) were amazed to see the animation of the beanstalk growing from the ground.

Except for P7 (8M), all participants saw the beanstalk from all around in an awe.

P1 (7M) had intuitively picked up Jack's token since he thought Jack would be planting the beans in the ground. - Very different from what I had anticipated.

Story Chapter 4: Getting the Golden hen

P1 (7M), P4 (5M), P6 (7F), and P7 (8M) started interaction by placing their token on the gold coins rather than the Jack illustration. P2 (5M) placed the Jack-token directly on the 3d Giant’s face and went to get the hen directly.
P4 (7M) was scared when he heard the sound after placing the Jack-token.

All the participants enjoyed Jack being chased by the giant.

Story Chapter 5: Chopping the beanstalk

P1 (7M) intuitively took the Jack token to chop the beanstalk.
P1 (7M) and P2 (5M) enjoyed it when they saw Jack’s dance. P4 danced with Jack.
P6 (7F) felt accomplished when she chopped the beanstalk; she repeated multiple times, “I chopped it off.”

General Observations

The P1 (7M), P5 (7M) and P6 (7F) were more curious and active than others; they asked questions, talked about their experience, and played freely.

P1 (7M) and P6 (7F) had difficulty reading the story. However, they read most of it by themselves.

The P2 (5M) and P4 (5M) were younger and smaller than the rest participants. They had trouble coordinating the token movement and seeing the tablet simultaneously; hence, they missed the initial interactions.

It was also observed that many a times, the AR interaction would spoilt the user experience, however children played regardless.

Results

Engagement: The revised prototype generated strong engagement, with most children actively exploring and discussing the AR content.
Learning & Interaction: The project showcased that an interactive AR narrative could not only foster enjoyment but also enhance engagement in reading, resulting in improved vocabulary and cognitive skills through story-driven learning.
Iterative Learning: Continuous feedback from diverse user groups (parents, young children, design students) was invaluable for refining usability and ensuring a balanced, accessible experience.

Learnings

Designing for Children as End Users:
In this project, my target users were children, which required me to dive deep into child development principles to tailor an engaging experience. I discovered that different age groups responded uniquely to the storytelling narrative, revealing the importance of age-specific design considerations.

Conducting User Research with Children:
To ensure effective research with children, it’s crucial to first build trust and make them comfortable. Before beginning any usability tests, I spent 15-30 minutes engaging in casual conversations, making sure the environment felt familiar and non-intimidating. This required patience and adapting my language to their age level, helping them understand the study’s purpose.

Designing for Augmented Reality (AR):
Despite not having a technical background, I learned Unity and developed the entire AR experience. Designing in AR introduces a new layer of complexity by adding a third dimension, where considerations of character distance and size become crucial. This project gave me the opportunity to explore the possibilities of 3D digital spaces.

Design Balance:

Integrating physical play with digital storytelling introduces unique challenges—maintaining narrative coherence while supporting intuitive interactions is crucial.

User-Centric Iterations:

Iterative testing highlighted the importance of clear instructions and reducing cognitive load, especially for younger users.