Blog

Evidence of the imperfection of human review continues to grow. Now what?

Thomas I. Barnett and Svetlan Godjevac add to the growing chorus supporting an electronic solution to ESI discovery. In their article, Faster, better, cheaper legal document review, pipe dream or reality, Barnett and Godjevac suggest that software is well-suited towards categorizing documents as either Responsive or Not Responsive at the relative margins of the Responsiveness continuum, where documents are either clearly Responsive or clearly Not Responsive, whereas human judgment is valuable in the middle swath, where hard-and-fast rules – the kind of calculations software is able to perform – would not produce reliable results. They conclude that e-discovery costs can be reduced by relying on computer software to make the document categorizations at the margins.

As suggested by Barnett and Godjevac, appropriate automation of the document review process brings measurable savings. However, integrating software into a review process that is sound, defensible and affordable is both an art and a science. To strike a balance between the art and science, a quality control process must be injected with the appropriate level of QC checks to identify documents that are likely to be incorrectly categorized and place those documents in front of a quality control reviewer. The right approach to review is integrating software into a process that has appropriate quality improvement steps, as well as a final quality assurance measurement to confirm that quality standards have been met.

Defensible Automated Document Review

How do we deliver a defensible solution that is consistent and accurate with this “faster, cheaper” approach? This brings me back to quality control. I agree that there is value to the science of using statistical QC to calculate a review’s precision and recall. However, judgmental QC is also critical to improving the accuracy of categorization. Both art and science, judgmental QC should include measures such as carefully crafted searches, examination of outliers within a cluster analysis, and examination of differences between initial human judgment and suggested coding of an automated categorization tool. Statistical QC must then verify the level of quality.

DiscoverReady Document Review

DiscoverReady applies this approach, combining software and human judgment in its i-Decision process. Intelligent software drives consistent, automated categorization of similar documents, followed by a carefully designed quality control process involving both human judgmental review and statistical QC. The nature of the quality control review is calibrated to place greater emphasis towards the center the continuum with final quality assurance sampling being taken to verify that appropriate precision, recall and accuracy have been achieved.

While intelligent software on its own is not capable of satisfying review obligations, including automated review within a robust process that has appropriate judgmental and statistical quality control brings substantial savings, improved consistency and improved accuracy.