CFC Prototyping: Second Prototype & User Testing Report
Today we conducted user testing on our second (or revised) prototypes from our previous rapid prototypes created at CFC. For my current iteration, I created a new prototype. My last prototype for CFC combined Unity and virtual reality to test separate perspectives from a mushroom, a human, and a tree in a shared environment. This first prototype focused on the experience of being in a first person perspective of other species, in addition to testing out the affordances and limitations of virtual reality. In my new prototype, I chose to move away from virtual reality and instead focus my attention on storytelling, simple interactions, and the flow between scenes and different perspectives (both perspective changes in species, as well as first and third person camera angles).
In this prototype, I am intending to tell the story of a Sequoia tree's life, and what species are important for helping this tree through its journey. Below I will provide screenshots and briefly discuss the context and interactions of each scene:
Scene 1: Pine Cone (Third Person Perspective)
In this scene, the user is introduced to a pine cone in an open space. The user can drag their mouse to orbit around the pine cone, and use the mouse wheel to zoom into the pine cone. There is an invisible trigger area close to one end of the pine cone. When the user's camera hits this trigger, a ring of particles appears. The user must then click in the ring to go to the next scene.
Scene 2: Chickaree's Perspective (First Person Perspective)
In this scene, the user is placed in a forest. They are looking from a Chickaree (Douglas Squirrel)'s perspective. The text prompts the user to find a pine cone. I chose this story because Squirrel's dislodge seeds from the pine cones, (hopefully) dropping a few seeds into the soil to launch a new Sequoia's chance for life. The user can use the mouse to look, and WASD keys to move around the forest. The camera sensitivity is quick, the position translation is fast, and the camera bobs up and down when moving. These first person parameters were adjusted in attempts to visualize the perspective from a small, fast moving animal. When the user comes upon a pine cone, they trigger a new scene to load.
Scene 3: Seedling (Third Person Perspective)
Similar to the first scene, this scene has a seedling in an open space. The user has the same interactions as the first scene (they can drag the mouse around to orbit and zoom in). When they get close to the mushrooms near the bottom roots, the ring of particles appear. The user can click the ring to transfer to the next scene.
Scene 4: Mushroom's Perspective (First Person Perspective)
In this scene, the user experiences a mushroom's perspective. This scene is visualizing a mushroom's attempt to connect with other trees in the forest, in order to provide nutrients and a communication method to its fellow tree partner. In the user's camera, the top of the mushroom is slightly apparent, but the rest of the scene is pitch black, with a white raycast target in the middle of the screen. The user is stationary (WASD keys are disabled), but they can rotate their view and click into space in attempts to hit other trees (which are visualized through particle effects). When the user hits 5 trees, they are transported to the next scene.
Scene 5: Sequoia Tree (Third Person Perspective)
Similar to the prior third person perspective scenes, in this scene the user is in an open space with a Sequoia tree. When close to a broken branch midway up the trunk, the particle ring appears. When the particle ring is clicked, the user is transported into a new scene.
Scene 6: Sequoia Tree's Perspective (First Person Perspective)
In this scene, the user experiences a tree's perspective. They are placed in the mid trunk of a Sequoia tree that is currently experiencing a forest fire. Contrary to the popular belief that fires are death sentences for forests, Sequoia trees actually benefit from forest fires (when these fires are under control). The fires kill the competing plants and animals around them. In particular, the plants around them are competing for nutrients and sunlight. The fire also helps Sequoia trees by drying their pine cones out, causing them to drop and open up, releasing their seeds. The user is once again stationary (WASD keys are disabled) but they can move their camera perspective, which is clamped on both the x and y axes. This was done to keep the camera within the tree trunk. The user once again has a raycast target, and is asked to click pine cones in the tree. When they click a pine cone, the pine cone drops into the fire. Once the player hits 5 pine cones, they are transported to the next scene.
Scene 7: Old Growth Tree (Third Person Perspective)
Once again, the visuals and interactions in this scene are similar to prior third person perspective scenes. The user is in an open space with a bare old growth Sequoia tree. At this point, there is no ring to press.
While I did not get to the other scenes in my storyboard, I think they are worth mentioning as part of my storyboard and narrative process. From the old growth tree scene, I intended to progress forward in the narrative by creating a scene that visualizes a tree's perspective and its interactions with CO2. From there, I would move to a scene from a bat's perspective using a tree as its habitat, and end the story with a scene from a tree's perspective as it is cut down. I wanted to include these because I am interested in visualizing a "circle of life" theme, yet also critiquing our notion of nature and trees as a symbol of a cyclical process by mentioning that 95% of Sequoia trees have been eradicated from Earth. As I move forward in my prototypes and iterative design process, I hope to begin testing these other vignettes.
Part 1: Self Testing
For my self-testing session, I played through my prototype and jotted down my insights as I went through the experience. I think its important as an artist/designer to play test your own experience in order to see what aspects you are proud of, as well as areas that can be improved or should be changed. Here are my notes below:
- I am happy with the particle effects and skybox visuals in the third person scenes. I should maybe make the trigger for the ring of particles bigger, because right now you have to be just in the right spot to find the particle ring.
- The chickaree movement is really fast. I am unsure whether to make it slower for users, or to try and keep it closer to the perspective of a squirrel's sense of movement and time. Maybe in my next iterations I should play with the surrounding environment's time scales rather than changing the player's movements and camera sensitivity.
-The pine cone is kind of difficult to find without knowing exactly where it is. Perhaps I should add more pine cones to the scene.
- The mushroom scene is visually pleasing once I have found the trees to click on, but the lack of visuals before the trees are visualized is somewhat frustrating. I should consider making a cue or something visible for a frame of reference of where the camera is rotated, as well as a visual for when the user clicks.
- In the fire scene, I really like the fire below, but I'm concerned that people will automatically think the fire is a negative aspect and misunderstand the reason for dropping the pine cones. I want to make this information more clear. The depth of field is also probably too much, although I personally like the blurred background that forces the user to notice the details up close. The pine cones are also difficult to find if you don't know where to look. The mouse movement is a bit sensitive to aim, but I find dropping the pine cones to be a satisfying interaction.
- I should delete the particle effect on the old tree until it leads to an actual playable scene. The particle effect also looks weaker in this scene- I'm not sure if that is based on the particle system, the color of the tree, or the post processing effects.
Part 2: Observation of others user Testing
In this section, I will discuss what I observed during the user testing. My process for user testing was one-to-one testing, and an open discussion afterwards. I decided to refrain from explaining or intervening during the session unless the user asked me for help or was obviously in need of assistance after a considerable amount of time (I estimate that I waited about 45 seconds or more before I would help those that were lost but would not ask for help). When I was asked for assistance or clarification, I would help by explaining and pointing to directions. I avoided taking over the controls at all cost, unless my explanation of what they needed to do seemed to not register with the user and they were completely stuck. In my notes below, I have also placed descriptions of each user's prior experience with games next to each user, based on if they told me they do/don't play games (many users who do not play games felt they needed to share that before the experience). I think this information is important for evaluating how my game performed on different audiences.
User 1 (Experienced game player) :
-Zoom in third person perspective was not intuitive.
-The user went a different way in the Chickaree scene.
-The user came across floating trees I was unaware of.
-Fungi scene was not intuitive for this player.
User 2 (Beginner game player) :
-Zoom in function in third person perspective was not noticed.
-Rotating around the objects was difficult for this user, the rotation was fast.
-This user wanted to confirm every movement they made (I assume they were trying to not break the game).
User 3 (Beginner game player) :
- The use of WASD keys was new to this user. Not intuitive at first, but once they experienced them they were quick to learn.
- Difficult time clicking pine cones in the forest fire scene.
- They asked me what they were supposed to do in each scene.
- The mushroom game was difficult for this user.
User 4 (Experience with games was not specified) :
- This user rotated around and zoomed in very quickly in the third person perspectives. They passed by the triggered particles continuously.
- Very difficult time aiming for the pine cones. Apparent frustration with this mechanic and vignette.
- Rapid clicking in mushroom scene.
User 5 (Experienced game player):
- In the chickaree scene. the player noted that the movement was really fast. The user noted that this scene made them feel dizzy.
- Mushroom scene was confusing for this player. They were unclear of their objectives.
- It was not as difficult for this user to aim for pine cones as other user testing sessions.
User(s) 6 (Experience with games was not specified):
- Use of WASD keys was not immediately intuitive. They quickly caught on to these controls for movement.
- This user needed assistance in finding the trees in the mushroom scene. They also needed assistance finding pine cones in the fire scene.
User 7 (Experienced game player):
- This user only needed assistance in the mushroom scene. In this scene, they eventually asked what they were supposed to do. The user was adept at all of the controls and goals for the other scenes.
User 8 (Experienced game player):
- This user did not like the chickaree perspective.
- They did not have difficulties with the controls, but they noted that aiming for the pine cones was too difficult.
User 9 (Experienced game player):
- No difficulty with controls in any of the scenes. Their greatest difficulty was finding the pine cones in the fire scene and finding the trees in the mushroom scene.
- They chose to not find the pine cone immediately in the chickaree scene. They claimed they wanted to run around and experience this perspective longer.
Part 3: Discussion with users of their experience during user testing
In our discussions, the user and I discussed what worked (what aspects of the game were satisfying or made sense?), what needed improvement (what area of the game was confusing or frustrating; did the controls make sense; did the story make sense?) and potential ideas for the next iteration (what areas would you change; do you have any ideas for the next steps of this prototype?). These discussions tended to primarily revolve around controls, mechanics, visuals and storytelling techniques.
- Mushroom level was confusing. The user did not understand what they were supposed to do, or what their controls were.
-The tree on fire scene feels like you are on drugs. The camera is lagging a lot, it is difficult to aim, and the blur is too much.
- The experience was very pretty, but the control options were confusing, because "I am switching from third person to first person to mobile characters to immobile characters."
-Perhaps indicating the controls for each scene would make it more intuitive, especially for those that don't play games. It depends on what audience the experience is targeting.
- It would be nice if each scene displayed the controls. WASD is not familiar to them, and they didn't know when to use which controls.
- Being in the experience is very pretty and it is beautiful, but because the experience is trying to represent nature closely, nothing stands out as something to do ( the user's example of elements that stand out included glowing or rotating objects, or something visually striking that grabs your attention).
- This user said they love becoming the other species. The prototype was very beautiful, and they really liked the experience of changing perspectives.
- The user was curious about the process of creating these scenes. The user was interested in the use of photogrammetry assets, because they had experience and an interest in photogrammetry.
- The user recommended having haptics to expand the experience. They encouraged the use of haptics to help guide players where to go, and also in adding an extra touch to the experience of being other species.
- The user recommended including the controls in each scene, because it was confusing as to what controls they could use, due to the change in interactions and perspectives in each scene.
- The mouse was too sensitive and too fast. This user noted they had a really difficult time finding where to click and what to click.
- This user said the story was confusing. They said that maybe it was because the text was too small at the bottom of the screen and went away too quickly. The text disappearing also made the user not understand what perspective they were in, the context for this perspective, and the interactions they were supposed to employ. They did not gather from the experience that this prototype was about a tree's life.
- This user recommended using sounds or haptics to direct users where to go and what do, particularly during the third person perspective section where the user is asked to find the ring of particles.
- This user recommended putting the controls on the screen. In particular, they recommended placing an indicator of when to use WASD, especially if the experience is targeted for those not experienced with games.
- This user noted that the mushroom section was confusing and "needed work". They recommended visualizing electrical impulses that are sent out when clicked, so there is a visual response to clicking.
- They said that it was very beautiful, but the controls were confusing for them.
- They noted that the chickaree movement was too fast, and needed to be slowed down to be able to find the pine cones. They also recommended blocking out areas that the user shouldn't explore, or placing a map of where the pine cones are, so the user can find the pine cone more easily.
- These users discussed the implications of camera movements. We discussed what it means to represent being and becoming in this experience as it shifts between various species perspectives and first and third person perspectives.
- They recommended using camera movements as directions to users of what their objectives are. They noted that putting too much text on the screen is often seen as lazy game design.
- These users recommended playing with different types of narration. What is the effect of particular types of narrators and scripts? Can narration be used to give directions to the user, provide context to the story through information, narrate the story, or evoke particular moods and themes?
- These users noted that the experience should reflect how it will be installed. It depends on how you orient yourself to the experience. Are you mirroring the tree, or are you within the tree? This can change the way you design and install an experience.
- We discussed the potential to experiment with other controllers and virtual reality in my next prototype.
- Unlike other users, this user thinks that the squirrel should be faster, if I want to get across a more realistic perspective of a squirrel.
-They stated that the mushroom controls need work. This was the most confusing section of the game for them.
-This user noted that the camera sensitivity was too high, and there was too much lag on the fire scene.
- They recommend visualizing some sort of cue of what creature you are embodying. They said it was unclear to them that they were a chickaree or mushroom, because you can't see the body of it.
- They really liked this experience because it reminded them of a puzzle adventure game, which is one of their favorite types of games.
-This user noted that they really liked the narrative. What they understood from the narrative was the life cycle as a tree, and its cyclical nature. They also like how it was not just about the tree, but about the other creatures that interact with the tree.
- This user noted that they believed the camera was too sensitive and lagged in all of the scenes.
- They noted that the depth of field and mouse sensitivity in the chickaree and fire scene were too much. It made them feel dizzy and ill.
- The user recommended slowing the text down, and making it larger. They said they did not have time to read the text, therefore they were confused of what their objectives were.
- This user stated that they liked the visuals. Their favorite scene was the chickaree scene. They liked experiencing the fast movements and perspective of the chickaree.
- They noted that the field of view, depth of field, and camera sensitivity should be reconsidered. While they did not find issues with it, they said that they could see it becoming an issue with other users. They also wondered if the camera sensitivity in the fire scene was because the camera is in close proximity to the pine cones and branches in front of it.
- This user stated that the text needed to last longer and be bigger. They also recommended moving the text to the top, where it would be more noticeable, and changing the color of it, particularly in the fire scene.
- This user recommended a particle system emitted from the clicks in the mushroom scene. They stated that the lack of response when the mouse is clicked makes it unclear if their click was received.
- Their greatest emphasis was considering the audience. They said that, based on what they know of my project, I am not aiming my target audience of my thesis to be experienced gamers. They stated that I should focus on who my target audience is, and how easy it will be for them to understand the controls of the game. They recommended using hints in the game based on timers. For example, if someone has not done or found something in a specific amount of time, nudge them in the right direction. If they still have not found it in an extended amount of time, make the controls and goals very clear, directing them precisely to their objective.
Part 4: User Testing with CFC
Although my user testing was a bit hectic (due to the fire alarm going off), I had a chance to meet with Ana and Natalie to present and user test my game. For this user testing, I controlled the game, showing them the different visuals, interactions, and stories of my experience. I received some really helpful feedback that has caused me to reflect on the next steps of my prototype. Ana suggested that, rather than thinking about my project as a switch from third to first person perspective, I should consider framing it as didactic to experiential scenes. She mentioned that it could be interesting to combine real life footage with the digital speculative environments I have created. This could be done using 360 degree cinema. When I told her that I was trying to avoid displaying text in my project, and would prefer to have narration, she suggested that I explore having an expert from the Sequoia forest narrate the experience. While this is considerable work that involves the research ethics board, flying to California, and analyzing primary data, I think this could be a really effective way to teach about forests through both didactic and experiential experiences of real and virtual.
Overall Results, Trends, and Reflection
After reading through my self testing and user testing notes, I have been able to find patterns of areas that were successful, and areas where I need to spend more time developing or changing the prototype. Areas of success included the subject matter (trees, chickaree, and mushroom), the visual constructions of space and objects, and the narrative, as well as the narrative's flow. Areas that need improvement include making the objectives and story more clear through either longer text or narration, putting controls on the screen to identify how to interact in the game, and reconsidering mouse sensitivity and camera movements, particularly surrounding post processed effects, depth of field and motion blur.
While these are all great technical areas for me to focus on, this process has also made me consider wider frames of my thesis work. As I observed the user testing sessions, I quickly realized that as a game designer, I cannot predict precisely where the user will go, what they will do, or if they will understand the game the way I want them to understand it. Pieces that personally felt intuitive to me were immediately apparent to not be intuitive to all (or many) users. This is a tricky thing to design for: it's hard for me to remove myself as a designer and consider what someone who hasn't seen this prototype, or does not play games, or does not engage in this information, will understand or appreciate. It's difficult to step back and introduce yourself to something you've spent hours on and see it with fresh eyes. I think that's what makes user testing so successful: you get a variety of people that have their own experiences, backgrounds, skill sets, and perspectives.
For example, it was fascinating to me to watch the difference between experienced game players, somewhat experienced gameplayers, and non-experienced game players. I realized that controls that seemed obvious to me are foreign to someone who doesn't interact with these forms of media. It made me consider what common characteristics of video games I have implemented into my own creative process (for example, WASD keys, aiming capabilities, particle systems as indicators of something to interact with). Furthermore, the users' patience differs drastically based on the user. While some users were entertained by difficult puzzles or mechanics, others lost their patience very quickly, wanting to know what to do and how to do it exactly to move forward. Some users wanted to spend time in environments exploring, whereas others wanted to get through the game as fast as possible. There were similar differences in opinions with the subject matter: some users wanted the experience to be pleasant and easy and intuitive to the audience, whereas others wanted it to be difficult, thought-provoking, or representing scientific information as close as possible.
The greatest impact of this process is my realization that there are not only conflicting views between users, and that designing for different users can be difficult, but more importantly that my own expectations of users is not clear or well formed. It's made me consider, who is my audience? I have repeatedly said that I want this for a wide range of users, because it is about the content and subject matter, not the technology. Yet, my prototype seems to be indirectly tailored for only those with gaming experience. As someone who loves to play games, how do I navigate the conflict between my own perspective and other perspectives as I go forward? I think I will have to seriously reflect and consider what audience and to what effect I want to reach, because at this point I am diverging from my intentions. I think the advice one of the users gave me is a great starting point: "use hints to nudge people in the right direction, and be sure to give clear instructions, but don't be too obvious from the beginning, because you want people to explore the space and the experience. You have to make it simple enough for non-gamers, and interesting enough for experienced gamers."
Therefore, I want to find the happy middle to have a general audience whom are able to interact, be entertained, and take away information and themes from the work. It's important for me to be straddle the line between pleasant user experience and new experiences of otherness; art and science; technology enthusiasts and technology novices. I think it makes the design process tedious and difficult, but in the long run it makes it a stronger piece that is capable of reaching wider audiences to a greater effect.