Moving from scripted regression testing to exploratory testing

Moving from Scripted Regression Testing to Exploratory Testing
In this article I will describe how we in the Pet Rescue Saga team moved from
manual scripted regression testing to exploratory testing before each release,
and what benefits we have seen from this.
But first a look back at how we previously handled our regression tests before
releases, and what prompted us to move to exploratory testing. Our sprints last
two weeks, and at the end of each sprint, we have what we call “Test
Wednesday”, when we run our regression tests. Each development team got
assigned a number of test cases to run, and these were divided within the group.
Each person sat alone and ran his/her tests in isolation, and often only a few in
each teams did all the tests while the rest continued their normal work. Testing
is not something that our artists and software developers particularly enjoy, so
their initial motivation was low, and our setup did not improve it.
So why did we initiate the change? Partly because we saw that running the same
regression tests over and over did not yield many new bugs. We did test all the
critical areas, so no new major bugs slipped into the live environment, but we
didn’t find all those smaller bugs, and old bugs persistent within our game.
Basically we wanted better test coverage. And partly because it was not very
motivating for our coders and artists to run the same tests over and over again.
We wanted everyone in our production team to be involved in testing of our
product – product owner, scrum masters, business people, community people,
coders, artists and of course our testers. What state the game is in before release
should be important to everyone, and something that everyone cares about and
takes ownership of.
Sidebar: Self-determination Theory [1]
SDT supports three basic psychological needs that must be satisfied to foster well-being
and health. These needs can be universally applied. However, some may be more
salient than others at certain times and are expressed differently based on time, culture,
or experience.
Competence
Seek to control the outcome and experience mastery
Relatedness
Is the universal want to interact, be connected to, and experience caring for others
Autonomy
Is the universal urge to be causal agents of one's own life and act in harmony with one's
integrated self
Looking at Self-determination Theory we concluded that we did not want people
to sit in isolation, running mind-numbing tests, which were being forced on
them. So we decided to adapt our way of working better mitigate these factors,

and evolve our list of scripted test cases into something else. The next iteration
of our “Test Wednesday”.
We had an idea of what we wanted to achieve, so we created a number of
exploratory test missions broadly covering the same areas as the old test cases
had, which looked something like this:
On the back of these cards are some guidelines to how to run the specific test
mission, but these guidelines are completely optional and only there to support
the person who selects the specific mission.
We also changed how we assigned the testing to the teams. Instead of giving a set
of scripted test cases to each development team, we instead gathered the entire
production team in the morning, and everyone picked a mission themselves,
giving them more autonomy to decide what they want to do. During this meeting
GAMEPLAY
PURCHASES

we also discussed what was new in the release, and which risks we saw that
should get extra focus during testing.
Previously a few people from the development teams sat running the scripted
tests for several hours, but by involving everyone, we could limit the testing to
one hour for each person, which makes it more manageable for everyone. So
instead of having the entire day dedicated to testing, it would now only be one
hour before lunch.
A positive side effect of this setup was that people took their test missions, and
then sat down together in couches and relax areas and did their testing there
instead of sitting in isolation by their desks. Now everyone could share
experiences and help each other on the fly, and bounce ideas between each other
to further improve the testing. Not only does this help cooperation, but it is also a
team building exercise that brings the production team together.
Another positive side effect is that this approach encourages everyone in the
production team to explore the game and all it’s features, allowing them to
improve their understanding of the game, as well as increase their competence
in testing.
After everyone has spent an hour testing their respective missions, everyone
returns their mission cards and have a short debriefing with the test lead, and
then the result is summarized and analyzed.
If someone is finished with their test missions before the hour is up they can go
to the test lead and pick an additional lower priority test mission and continue
their testing until the hour is up.
We complement this exploratory testing by the entire production team with a set
of automated system tests, and some more complex and advanced testing
performed by one of our test experts.
So the end result was:
1. Better test coverage
2. Quicker turnaround time
3. Higher motivation
a. Better understanding of the game
b. Increased skill in testing
c. More autonomy for everyone to decide what to do
d. Better cooperation and communication
All in all everyone seems happy with the change and the results so far have been
very promising.

References
[1] Self-determination Theory
https://en.wikipedia.org/wiki/Self-determination_theory

Moving from scripted regression testing to exploratory testing

More Related Content

Similar to Moving from scripted regression testing to exploratory testing

Similar to Moving from scripted regression testing to exploratory testing (20)

More from Johan Hoberg

More from Johan Hoberg (20)

Recently uploaded

Recently uploaded (20)

Moving from scripted regression testing to exploratory testing