One Christmas, my 7-year-old twins got a Nintendo Switch as a present, along with three games. As expected, the racing game was the preferred first choice of activity, so the morning was filled with “karts” and relearning how to lose and win gracefully.
Later on, the boys decided to try another game that was advertised as a multiplayer game. The setup for the initial character took about 30-40 minutes. During setup, Kid #1 chose his character, the character attributes, details about his environment, and various other non-skippable configuration items. During this time, Kid #2, to his credit, was only mildly impatient with not being able to have a turn. Finally, it was Kid #2’s turn to set up his character which took 20-30 minutes. Altogether, it was about an hour before they could start playing together. That’s an eternity to a 7-year-old (and to some 50-somethings, but I digress).
During this time, I didn’t notice anything that would be considered a “classic” software issue. No bug/glitch/error/hiccup/crash/restart; not one. What I was seeing, however, was an issue, at least to the boys and me. To us, this inability to concurrently configure our characters was an issue. This software was not meeting our expectations and thus, causing unhappiness. But why?
Most software is intended to be used by many different types of people. Game software, in particular, is meant to be enjoyed by people of many ages and backgrounds. This specific game is ostensibly suitable for ages 3 and above and has a rating of “E for everyone.”
Let’s examine “everyone.”
In my experience, “everyone” is primarily made up of people who are sufficiently patient…
- For a “long” (albeit gamified) configuration sequence.
- For a sequential configuration of individual characters (what if I had more kids?!).
- To Google “how to add an additional player.”
In my experience with this game, my kids barely fit under that “everyone” umbrella. If they had been 5 years old instead of 7, I think we would all have struggled to be patient. Though the content is certainly appropriate for many 3-year-olds, I can’t imagine a 3-year-old spending that amount of time just getting ready to play a game.
So, why is an automation guy bringing all this up? Because it has to do with testing. Why was the initial game configuration so time-consuming? Was it out of necessity? Was it thought to be fun? Perhaps all the testers and business leaders thought it was cool and appropriate. Perhaps, however, some personas were missed during testing.
By “personas,” I’m referring to the kinds of people who will interact with our software; other people have different terms for personas, such as roles or actors. A very small set of example personas might include:
- A white man between the ages of 25 and 40
- A black woman between the ages of 55 and 70
- A color-blind woman in her 30s
- White, 7-year-old twin boys on Christmas morning (though that may be oddly specific)
Most people could likely come up with at least 10 additional personas and each of the above can be carved into many subsets. That’s the point! There are a ton of personas that we should consider when testing software. We need to try to account for as many of them as feasible when we test our software.
Due to the sheer number of potential personas, we probably can’t test for all of them, so we must prioritize for those we expect to use our software the most. It’s not that we don’t care about the other users; we want them to enjoy our software, too. But we likely won’t have time to test every persona we identify. We can’t even identify all the personas in the first place and, of the ones we do identify, we can’t always be sufficient proxies for them. As humans, we’re limited in that capacity.
You might be asking again, why is an automation guy bringing all this up? Because it has a great deal to do with automation. For the most part, we can configure different kinds of users in our test systems so that those test users have appropriate privileges based on their role. That helps the automation behave like the actual personas they represent, but they are not really those personas. Only those personas can accurately behave like those personas, and really only a subset of those personas. As we’ve already covered, as humans, we’re limited in our capacity to proxy personas, but machines are even more limited in that respect.
So, where does automation come in? It’s where it belongs, helping testers do their jobs. That’s certainly a context-sensitive answer, but it’s for a context-specific world. In some contexts, we can highly automate based on configuration, such as the aforementioned privileges. In other contexts, we can’t highly automate because we absolutely need humans (i.e., testers) sympathizing and empathizing with our users. Automating in those situations can be helpful, but if we rely solely on automation in those situations, we run the risk of missing the human aspect of the testing, e.g., pretending to be an impatient 7-year-old.
Years ago I interviewed with a gaming company for a “director of automation” position. When I told the recruiter that not all testing can or even should be automated, he responded with, “you can’t automate fun, right?” He was spot on. You can’t automate fun, impatience, frustration, or any other emotional response. As long as there is an emotional aspect, a human aspect, to software usage, there will be things that are inappropriate to automate. But don’t let that prevent you from using automation to help do your job; just use it responsibly.