Low Tech Testing Dashboard
A Low-Tech
Testing Dashboard
James Bach, Principal Consultant
james@satisfice.com
http://www.satisfice.com
STAR ‘99 East
1
The Problem
“What’s the status of testing?”
“What are you doing today?”
“When will you be finished?”
“Why is it taking so long?”
“Have you tested _______, yet?”
2
The Problem
n Management has little patience for detailed
test status reports.
n Management doesn’t understand testing.
– Testing is confused with improving.
– Testing is considered a linear, independent task.
– Testing is assumed to be exhaustive.
– Testing is assumed to be continuous.
– Test results are assumed to stay valid over time.
– Impact of regression testing is not appreciated.
– Test metrics are hard to interpret.
3
A Solution
n Report test cycle progress in a simple,
structured way...
n ...that shows progress toward a goal...
n ... manages expectations...
n ...and inspires support...
n ...for an effective test process.
No process?
Easy setup
No Problem!
4
The Dashboard Concept
Project conference room
Large dedicated whiteboard
“Do Not Erase”
Project status meeting
5
The Test Cycle
drive
Quality
Development
Criteria
creates
drives
Product
focus
examined by
Perceived
Testing
Quality
reveals
6
Test Cycle Report
Product Areas
vs. Test Effort
Test Coverage
Quality Assessment vs.
Time
7
Testing Dashboard Updated: Build:
2/21
38
Area
Effort
C. Q. Comments
file/edit
high
1
view
low
1+
1345, 1363, 1401
insert
low
2
format
low
2+
automation broken
tools
blocked
1
crashes: 1406, 1407
slideshow
low
2
animation memory leak
online help
blocked
0
new files not delivered
clipart
none
1
need help to test...
converters
none
1
need help to test...
install
start 3/17 0
compatibility start 3/17 0
lab time is scheduled
general GUI
low
3
8
Product Area
Area
n 15-30 areas (keep it simple)
file/edit
n Avoid sub-areas: they’re confusing.
view
insert
n Areas should have roughly equal value.
format
n Areas together should be inclusive of
tools
everything reasonably testable.
slideshow
n
online help
“Product areas” can include tasks or
clipart
risks- but put them at the end.
converters
n Minimize overlap between areas.
install
n Areas must "make sense" to your
compatibility
clients, or they won’t use the board.
general GUI
9
Test Effort
None
Not testing; not planning to test.
Start
No testing yet, but expect to start soon.
Low
Regression or spot testing only; maintaining coverage.
High
Focused testing effort; increasing coverage.
Pause Temporarily ceased testing, though area is testable.
Blocked Can’t effectively test, due to blocking problem.
Ship
Going through final tests and signoff procedure.
10
Test Effort
n Use red to denote significant problems or
stoppages, as in blocked, none, or pause.
n Color ship green once the final tests are
complete and everything else on that row is
green.
n Use neutral color (such as black or blue, but
pick only one) for others, as in start, low,
or high.
11
Test Coverage
0
We have no good information about this area.
1
Sanity Check: major functions & simple data.
1+ More than sanity, but many functions not tested.
2
Common Cases: all functions touched; common &
critical tests executed.
2+ Some data, state, or error coverage beyond level 2.
3
Corner Cases:
strong data, state, error, or
stress testing.
12
Test Coverage
n Color green if coverage level is acceptable for ship,
otherwise color black.
n Level 1 and 2 focus on functional requirements and
capabilities: can this product work at all?
n Level 2 may span 50%-90% code coverage.
n Level 2+ and 3 focus on information to judge
performance, reliability, compatibility, and other
“ilities”: will this product work under realistic usage?
n Level 3 or 3+ implies “if there were a bad bug in this
area, we would probably know about it.”
13
Quality Assessment
“We know of no problems in this area that
threaten to stop ship or interrupt testing, nor
do we have any definite suspicions about any.”
“We know of problems that are possible
showstoppers, or we suspect that there are
important problems not yet discovered.”
“We know of problems in this area that
definitely stop ship or interrupt testing.”
14
Comments
Use the comment field to explain
anything colored red, or any non-green
quality indicator.
n Problem ID numbers.
n Reasons for pausing, or delayed start.
n Nature of blocking problems.
n Why area is unstaffed.
15
Using the Dashboard
n Updates: 2-5/week, or at each build, or prior to
each project meeting.
n Progress: Set expectation about the duration of
the “Testing Clock” and how new builds reset it.
n Justification: Be ready to justify the contents
of any cell in the dashboard. The authority of the
board depends upon meaningful, actionable
content.
n Going High Tech: Sure, you can put this on
the web, but will anyone actually look at it???
16
