USABILITY STUDIES
OF GAME NOOBS VS LEETS
Noobs vs Leets is a geometry game designed and launched by CREATE Lab at NYU. In order to help "noobs" get to the destination of each level without being caught up by "leets", the player needs to use geometry knowledge about angles to unlock the routes (the game intro animation can be found here). As invited by the lab, our research group conducted usability studies on the game's navigation, mechanics design, and player engagement.
My Contributions: Research Protocol Drafting, Testing Moderating, Video coding, Notes Taking and Analyzing.
Methods: In-person qualitative/quantitative testing: User Observation, Think-Aloud, Video Research, Interview
Tools: ScreenFlow, Datavyu
.png)
RESEARCH QUESTIONS
Navigation Usability
- Are the players able to navigate the game smoothly (ease of navigation)?
- What non-intentionally designed obstacles are there preventing the player playing the game?
Game Mechanics Design
- How often the higher level “math rule” solutions were used to solve the problem when there are multiple options?
Player Engagement
- How do the players like playing this game?
Qualitative + Quantitative Methods
Since this research was conducted on a launched game with a purpose of measuring its performance and generating a report for the next round of iteration, the study was centered more around behaviors - which means we care more about "what players do" vs "what player say". The self-reported information was used as the complementary research method to dig deeper into player's feelings and the reason of their decisions.
DECIDING RESEARCH METHODS
LOOKING FOR PARTICIPANTS
We identified 4 types of personas according to their video gameplay experience and prior geometry knowledge level, for the reason that both of the two factors may affect the smooth level of navigation, play strategy, and player's engagement - which are the three areas where our research questions fall in!
20 participants (5 for each persona) were recruited for this research. They were all NYC local middle/high school students (Grade 7-10).
DIGGING INTO THE RESEARCH
Designing Tasks and Probes
Tasks
Session 1
Session 2
Session 3
Playing Chapter one Level 1-3
Playing Chapter one Level 4-6
Exploring the game freely
Probes
General Probing Questions:
- What are you looking at?
Specific Probing Questions:
- Where are you now? How can you get to that place?
- You said ___________, what do you mean by that?
- Have you noticed this button?
Throughout the testing, we use probing questions to guide the participants' think-aloud process. The whole process was moderated and recorded. We found that it was really helpful to have the detailed procedure planned out beforehand since the testing time is valuable and the questions asked to the participants should be accurate.
After the in-person lab studies, we replayed the video of the testing. The good thing about video review is that it provides more abundant information about the player and the screen information that could hardly be captured onsite. However, how could we collect and analyze visual data?
How did we collect the visual data?
# 1. Field Notes Coding Beforehand
Number all the solution options in advance on the testing plan. Annotate the player's choice as they play. Then identify the choice pattern to find out the play strategy.
# 2. Video Coding
We identified 3 categories of the proxies that can be used to gauge a player's engagement:
-
Utterance
-
Facial Expression
-
Posture and Gaze
And there is another factor - Game Event - should be tied to the proxies above.
We then created a rubric for coding these proxies, which can be found here.
Here is an example of how it works:
Utterance:
Game-playing related (p)
Facial Expression: Surprise (s)
Posture:
Gaze (g)
Lean forward (f)
Game Event:
Stuck (stuck)
(Video Coding Sample: To protect the participants' privacy, we use the demo with our group members showing on the screen.)
And this is how the coding looks in the video coding tool Datavyu:
(Video Coding Sheet in Datavyu)
WHAT WE SAW AND HEARD
The usability testing for the game Noobs vs Leets went through over 1 month from the planning stage to data analysis. We conducted in-person lab studies and video research which include both qualitative methods and quantitative methods.
To summarize our main takeaways from this study:
1. Player's video gameplay experience doesn't affect their navigation in the game. However, the previous geometry knowledge level does affect the play performance (smooth level and play time duration). It means that the usability of the game is good. At the same time, since this educational game targets students both with knowledge of angles & lines and those haven't learned the knowledge but about to learn, the game might need to provide more scaffolds.
2. The higher level geometry knowledge was not used often by the players when there are other play options by using the lower level knowledge. Over 63% of the tasks were completed by applying lower level knowledge. This is a game mechanics design problem which needs to be addressed for the reason that it might not be helping the learner to practice more complexed knowledge.
3. The player engagement level in the game was high. The data indicates the difficulty level of the game is appropriate for the target audience yet it's entertaining enough to keep the players engaged. This is exceptionally hard for educational game design to achieve due to the dual purposes of the game: education + entertainment.
Supported by NYU CREATE Lab, supervised by Dr. Jan Plass. Team Member: Mi Sophia Lu, Aniol Saurina Maso.