The Startling Economics of Controls Documentation Review
November 2007
All around the world, there are people reviewing
controls documentation such as flow diagrams, risk control tables, and records
of audit tests performed and conclusions drawn. Many of these people are experienced,
intelligent, and highly paid. Their reviews will result in remedial work by
others. The whole undertaking is costly, but the audit profession has done little
or nothing to study, scientifically, the economics of reviews. Happily, someone
else has, and the results are startling.
by Matthew
Leitch
When I was a young trainee accountant and junior auditor, the routine of
reviews was simple and universal. The audit team would go to the client's offices
and fill paper files with working papers and photocopies. Near the end of the
visit, a manager would arrive and read through the completed working papers,
initialling each one and making a list of “review notes.” We, the team on site,
would anxiously wait to find out how many review notes we had to deal with and
how much over budget they would take us. Every review note had to be dealt with
rigorously, which meant going back to people with queries, correcting mistakes,
and generally doing work we did not expect to have to do. After that came the
partner's review and possibly more remediation, but usually not much.
Later, when I moved up in the pecking order and was a manager myself, things
had changed, but only a little. Now there was an electronic work file so I could do my reviews
from the comfort of my own desk, but still the routine was the same: I reviewed
everything near the end of the work, and every point I raised had to be dealt
with.
If you've ever done this kind of reviewing, then you know it is boring almost
beyond belief, and it is very difficult to stay alert as you plough through
hundreds of pages of confusing text and questionable diagrams.
Projects launched to comply with the notorious Section 404 of the Sarbanes-Oxley
Act of 2002 have generated an unprecedented volume of audit/controls documentation
and boosted consumption of strong coffee.
At some point in my career, I started to review work by people earlier and
earlier, which is easily accomplished with electronic work files, and with new
people in particular, this usually led to some intensive coaching to help them
write appropriate documentation so I wouldn't have to list so many corrections
later on.
It is quite possible that many people have likewise changed their approach
over the last decade or so, but still I have not seen any published research
on this or any other aspect of the economics of controls documentation review.
Research from Engineering
However, in the world of systems engineering, there is even more documentation.
It is not dissimilar to controls documentation, and engineers have gathered
detailed data that lead to some helpful conclusions. This research has been
pulled together usefully by Tom Gilb, who is an authority on document inspections,
among other things, and it was Tom who introduced me to this research.
The first finding of interest is that the more time you take to conduct a
review, the more defects you will find. A graph of the relationship between
pages per hour and number of defects found per page shows that as you go slower,
the number of defects found per page rises gradually, but this accelerates so
that as you get toward one page per hour, the findings shoot upward. In other
words, reviewing at the usual auditor's speed of perhaps 50 pages in an afternoon
(and that's being careful) will identify only a tiny percentage of the total
defects.
The second important finding is that if additional reviewers review the same
document, they find some additional defects, even when all are working at the
very slow optimum rate of one or two pages an hour.
The third important finding is related to the first two. It is simply that
the number of defects in ordinary systems documentation is staggering—typically
more than 100 defects per page.
I have asked people in my seminars to review controls and audit documentation
very slowly and found that they find very large numbers of defects, so I think
the engineering research findings apply to audit and controls documentation
too.
Don't Panic
So, does this mean we should review controls documentation at a rate of one
or two pages an hour and expect people to fix the thousands of defects we will
find? Fortunately, the answer to that is “No.” The last crucial discovery from
the engineering research is that the major benefit of inspection programs in
engineering has been educational. Inspections teach people to avoid writing
defects in the first place.
In fact, if you inspect just one page from a longer document and feed back
the results properly to the writer, and if you say you will not accept the document
until defects per page has reduced to a specific, very low number, then the
rate of defects on the next review is usually about halved. Each subsequent
review leads to another halving of defects, and this applies to all pages the
person writes, not just pages that have been reviewed before.
Guidelines for Economic Reviews
This research, Tom Gilb's advice, and my own practical experience of reviewing
controls documentation and coaching people to write it better suggest the following
guidelines.
Think of Reviews as Being Largely for Coaching Purposes
There is still a need to check that there are no gaping holes or errors in
documentation, but the large bulk of review effort should come before that and
be part of coaching. Think of changing reviews from being entirely about remediation
to being mostly coaching and partly remediation.
Start Early
For someone who hasn't had their documentation rigorously reviewed before,
start your review process after they have drafted half a page of work. That's
all you need, and why let them waste any more time than that?
Use Rules
Write down the rules that documents must comply with so that failure to comply
is a defect. These rules clarify the process for everyone and make it easier
for writers to see what is expected. For example, here are some rules for diagrams.
In actual practice, most defects arise from R1.
| R1: Clear: Diagrams must be unambiguously
clear to the intended readers (this year and next year). |
| R1.1: Diagrams must be tied to physical reality, including software
reality, and not introduce conceptual views that distort that reality
in the interests of explanation. |
| R1.2: Diagrams must use symbols consistently and their meanings
must be explained, e.g., with a key or because they are standards for
the project as a whole. |
| R1.3: Diagrams that show processes must put short explanations of
what the process does on each process box. |
| R1.4: Diagrams that show data flows must put short explanations
of what is flowing on each flow arrow. |
| R2: Complete: If a data flow diagram
shows a process, then all material data flowing to and from that process
should be shown. |
| R3: Separation of Process and Controls: Controls, and the underlying process being controlled, must be separated
visibly, even if they are combined on one diagram. |
Review a Small Sample Very Slowly
Review a page or two very slowly, identifying as many individual defects
as possible.
Be Nice
A large part of the engineering literature on reviews is concerned with making
sure people are told the perceived defects in a way that is not upsetting or
unpleasant. It helps if everyone knows that high defect levels are normal for
people new to the rigorous review approach.
Have a Rule about Acceptable Defect Rates
Tom Gilb points out that this encourages people to learn their lessons and
make the changes they need to make. If their work cannot progress to the next
stage without a specified, measured level of quality being attained, then people
focus more on quality.
Clarity Then Suitability
Often, documents are so unclear that it isn't possible to tell if what they
are saying is appropriate or not. In these situations, the first review(s) will
focus on clarity. Then, when clarity has improved, it is possible to move on
to suitability.
Leave Time for the Last Remediation
If most reviewing has been turned into a form of coaching, then the final
stage of review can be what it was always intended to be: a rapid search for
important defects that must be corrected for the job to be properly completed.
If the coaching reviewers have done their job, there shouldn't be many of these
to find, and reading the papers should be much easier.
Summary
Reviewing controls and audit documentation is an expensive remedy for insomnia,
but research from engineering suggests that we may be able to get more value
from it, particularly in teaching people to write well in the first place. Perhaps
in the future, there will be research specifically on controls documentation
to see what the typical percentages are and how they compare to those from engineering.
Further Reading
Tom Gilb offers more detailed advice on his Web site
at http://www.gilb.com/community/
Opinions expressed in Expert Commentary articles are those of the author and are
not necessarily held by the author’s employer or IRMI. This article does not purport
to provide legal, accounting, or other professional advice or opinion. If such advice
is needed, consult with your attorney, accountant, or other qualified adviser.