Defining Test Competence

Defining Test Competence
In this article I will explore the concept of Test Competence and try to define
what I actually mean with it.
To understand Test Competence, it is my belief that we must look in the Cynefin
Framework [1]. There are many different types of problems, and according to
this framework they can be divided into four categories; Obvious, Complicated,
Complex and Chaotic.
Using this framework we can categorize different test problems:
Obvious Test Problems: Tests in which the relationship between cause and effect is
obvious to all
Complicated Test Problems: Tests in which the relationship between cause and
effect requires analysis or some other form of investigation and/or the application
of expert knowledge
Complex Test Problems: Tests in which the relationship between cause and effect
can only be perceived in retrospect, but not in advance
An obvious test problem would most likely not require specific test competence
to solve. It could be something as simple as pressing an application icon on your
mobile phone and expecting the application to start. This is something anyone
could test, regardless of specific competence.
A complicated test problem would be a little bit harder. To solve this would
require an understanding of the system and its architecture, as well as an
understanding of how to read and write code. But once the relationship between
cause and effect has been established, it is just a matter of verifying it. To me this
Sidebar: How does complex
behavior arise?
“The behavior of a complex
system is often said to be due to
emergence.” [2]
“Emergence is a process whereby
larger entities, patterns, and
regularities arise through
interactions among smaller or
simpler entities that themselves
do not exhibit such properties.”
[3]

would fall within the range of what we could expect a developer to test, since
they are most likely to have the competence required to do so, but I would not
classify this problem as one that required a high level of test competence. It
requires a lot of expert knowledge, but not specifically about testing.
Before we continue, let’s take some time to define what I believe lies outside of
the Test Competence concept:
 Writing reports – Definitely important to a tester, but not really testing
 Writing automated tests – A difficult task that requires a good coder, but
not necessarily high test competence (choosing what to write automated
tests for is another story)
 Executing manual regression tests – Might require intricate knowledge of
the system and different tools, but if you are just following a script and
you have a set scope of tests to run, then you should not need much else
 Critical thinking, adaptability and curiosity – Obviously important to a
tester, but that goes for many different roles
 Risk assessment and prioritization – Some key competencies for a tester,
but many other roles also require you to master these
 Agile and Scrum – Definitely important for everyone who works in a
Scrum team, but not part of the Test Competence concept
 Collaboration and communication skills – Again key competencies for a
tester, but also for many others
There are of course more examples, but hopefully I have framed the concept of
Test Competence a little more with these.
So if nothing of the above is the elusive Test Competence … then what is it that a
good tester does better than anyone else?
I believe the answer is: “Solving complex test problems.”
Ok, so how does a tester solve a complex test problem using their test
competence? What makes their skillset unique?
I believe Test Competence consists of (at least) the following components:
 Primary: Modeling unpredictability in a complex system
 Primary: Having a toolbox of triggers that could cause a system to behave
unpredictably
 Secondary: Provoking a system to reach an unpredictable outcome
 Secondary: Exploring a complex system through tests
So what does this really mean, and how do I solve a complex test problem with
these components?
If an experienced tester is confronted with a complex test problem, then this is
how I expect them to approach this problem:

 They start with a software system under test and some information about
this system (could be requirements, risks, historical data, etc.)
 Based on previous experience (either domain specific or general) and/or
based on a toolbox of triggers, such as heuristics, test methods and test
techniques, together with information about the system, I expect them to
come up with a number of test ideas
 Based on these ideas, they then model a faulty system (in their mind or
documented in the form of high-level test cases or test missions) which
could show an unpredictable behavior
 They then provoke the system to try to reach this faulty state (which may
or may not require specific tools, and/or understanding of the system)
o Documenting how to provoke the system to reach the faulty state
would probably result in a step-by-step detailed test case
 If they succeed they may have found potentially valuable information, and
if the system does not show unpredictable behavior then the system
works according to specification
 They then continue to explore the system using more test ideas, but also
equipped with the new information the previous test provided
So what does it mean to model a faulty system? Because this is the key part.
Example:
Let’s say I have previous experience with performance problems in earlier
versions of a software, and I am now tasked with testing the new release of that
software. A test idea related to performance is not that far fetched. With this
information I must now build a model in my mind of the system where
performance problems could occur. Maybe in my system, when performing
action A, while I am in state B, during an ongoing action C, the performance of the
system breaks down completely. Setting relevant A, B and C requires
understanding of the system and/or the users of the system. Once I have built
this faulty model in my mind I continue with trying to provoke the system to
reach this faulty state, and see if I find some valuable information or not.
To model a faulty system is to imagine a system, with a certain set of variables
and states, which might behave unpredictably under specific circumstances.
This is how I define Test Competence right now. Subject to change.
References
[1] Cynefin Framework
https://en.wikipedia.org/wiki/Cynefin_Framework
[2] Complexity
https://en.wikipedia.org/wiki/Complexity
[3] Emergence
https://en.wikipedia.org/wiki/Emergence

Defining Test Competence

More Related Content

What's hot

What's hot (20)

Similar to Defining Test Competence

Similar to Defining Test Competence (20)

More from Johan Hoberg

More from Johan Hoberg (20)

Recently uploaded

Recently uploaded (20)

Defining Test Competence