Reflections and Insights on Quality in E-discovery Document Review

When we first opened the doors to DiscoverReady in 2005, one of my primary roles was to develop the quality control procedures for the document reviews we perform.  I also often developed the training materials, created initial review protocols and worked with our project managers for different cases to develop review workflows, led review teams and performed quality-control review. What I’ve learned led me to publish a white paper earlier this month on best practices for document review, based upon my experience (and DiscoverReady’s collective experience) in designing several hundred document reviews.

It has become clear to me that the quality of a document review depends on a range of factors beyond the intelligence and perceptiveness of the attorneys reviewing the documents.  The most critical factor is the way (to be technically accurate, the ways) in which quality control is conducted.

Statistical sampling – one size doesn’t fit all!

One example of a standard “quality protocol” is statistical sampling – a very valuable tool, but, in our view, not a total quality solution.  Here’s why.

In any review of the least complexity, and particularly with larger reviews that require a substantial number of document reviewers, there will most likely be groupings of documents that are incorrectly categorized.  Those groupings, for example, may be of a certain document type, sent by a particular person, or relating to a particular issue.

Many review firms have adopted a practice of “randomly selecting a percentage of documents” for re-review.  While we believe there is significant value to statistical sampling, sampling alone is likely to miss some or all of the erroneously categorized documents from each troubled grouping (e.g., you may sample 10 percent of the documents, but you are exceedingly unlikely to find all 25 documents that were incorrectly categorized for the same reason).

DiscoverReady’s years of experience in document review have enabled us to develop specific protocols using targeted approaches.  We construct precise searches combining (as appropriate) senders, recipients, date ranges, proximity-limited terms, and categorizations. We also take advantage of review application features that help to identify inconsistencies across reviewed documents to go beyond the identification of individual “bad decisions” and instead, identify and address systemic review defects.

Without identification of systemic problems through a more thoughtful effort, even the identification of an isolated document requiring correction does not inform the re-reviewer whether the documents represent a larger group of systemic problems or merely a random error.  This is but a single example of why having a holistic and multi-pronged approach to quality is key to a successful document review.

For a more in-depth analysis of this issue and of many of the other elements of a sound review process, I invite you to see my white paper on the importance of quality processes in document review.