AI and Text-Based Analytics in Complex Environmental Litigation – What Environmental Practitioners Need to Know

The phrase “artificial intelligence” or “AI” is often used to describe several computational advances in use across many industries, including machine learning, translation, data visualization, and more. It is particularly useful when you need to compare, review, or digest voluminous material quickly – more quickly than is possible without computer assistance. In recent years, advances in AI capabilities have transformed many industries, not least the legal industry, which has seen AI’s advances as promising tools to cut back on the burdens of document review, or to quickly cut to a matter’s heart. Therefore, we examine the advances in AI for electronic discovery (also known as “eDiscovery”) with a view to the specific challenges presented by large-scale environmental cases such as multi-party Superfund sites, oil spills, and emissions litigation, all of which the author has been involved in over the past several years.

The Background: Textual Analytics, “Predictive Coding,” and Beyond

While the world of “data” and “information” continues to grow exponentially in complexity, the legal and regulatory communities’ use of advanced tools to decipher the information has not. Most lawyers are familiar with electronic document storage and email but have not yet plumbed how artificial intelligence can help them do their jobs faster and more efficiently. The promise of text analysis and machine learning algorithms (referred to generally as artificial intelligence or “AI”) is becoming a larger and larger part of litigation practice, increasingly available to even smaller practices (and increasingly expected by the courts).

Much of the need for textual analytics is driven by the ever-growing volume and central importance of email to litigation. While email may seem to have been with us forever, it is actually a comparatively recent phenomenon – growing from a small group of academics in the 1970’s and 1980’s to common use by 2000.[1] The courts and the bar began wrestling with the new practice of “eDiscovery” (as opposed to just “discovery”) in the early 2000s with the formation of the Sedona Conference Working Group series on electronic discovery in 2002 and the publication of the Sedona Principles in March of 2003.[2] This was followed by publication eDiscovery Reference Model or “EDRM;”[3] Judge Shira Scheindlin’s series of decisions in Zubulake v. UBS Warberg (citing the Sedona Principles) clarified that email and electronic content could not be ignored.[4]

Even so, it was not until 2012 that former President Barack Obama’s Office of Management and Budget (“OMB”) clarified that electronic federal records, such as electronic documents and email, were to be managed electronically and maintained in accessible electronic formats – before then, many federal agencies were still requiring that emails and documents be printed to file, and full electronic records management was not required to be in place until 2019.[5] Email and electronic records remain the forms of eDiscovery most familiar to practitioners today, and still present challenges: how to sort through a million-plus email “document dump,” how to de-duplicate emails that are sent “reply-all” to vast chains of individuals, where each receipt has a different recipient and time-stamp; how to deal with the ongoing email “conversation” with its branches and its chains.

That said, email and electronic documents are computer creations, and as such, are comparatively easy for computers to handle. There are many different forms of text-based analytics and computer learning models that are currently used in eDiscovery review platforms, with a variety of proprietary algorithms under an array of trademarked names. [6] Each are generally variations on a couple of specific forms of supervised machine learning, a form of artificial intelligence that relies on human input to “teach” the computer what the human user seeks and how to classify it, improving and narrowing the results over time.[7] The traditional approach, Technology Assisted Review (“TAR”), in its simplest form involves setting categories for documents such as “responsive” or “privileged,” and then having a group of subject-matter experts and attorneys code a “seed set” of documents that are highly likely to fit these categories – for example, the most relevant memoranda summarizing the case, or a modified version of the initial discovery requests. The analytics engine then compares the text patterns in the non-coded document set with the text patterns in the human-coded records to provide a statistical model of similarity, degree of precision, and recall, in order to “predict” the likelihood, expressed as a percentage, that a particular document is relevant, responsive, or privileged – a function known broadly as “predictive coding.”[8] This enables the coding team to focus on the most likely to be relevant information first, saving valuable time.

A more advanced twist on the same basic premise is Continuous Active Learning (CAL); as with the TAR model, human input is required for the initial seed set but the computer then uses an ongoing analysis of textual similarity (similar to the analysis used to find your next song on Pandora or Spotify) to “predict” the coding of similar documents in real-time; the computer’s analysis is then trained and refined by human beings.[9]Regardless of the underlying algorithmic model, there is a human-supervised validation process built into the TAR or CAL workflow so that human reviewers continually revise and refine the machine’s learning process by providing increased input as to text patterns that meet the condition for “relevance” or privilege.[10] In either model, the text analytics engine can be used to group highly similar documents together even when portions, like the recipient or timestamp, are not the same (“near-deduplication”), notice when documents are within chains (“email threading”), and group like text with like (“content clustering”), which can rapidly diminish the time required for human review.

Textual Analytics and Predictive Coding: What is Available to the Environmental Bar

While the technology is powerful, it also has the potential to create further disparity between the so-called “Biglaw” or Amlaw 100 firms and the rest of the field. This is a long-standing problem; as far back as 2009, eDiscovery expert and trial lawyer Craig Ball presented the famous “Edna Challenge,” challenging vendors to support an end-to-end eDiscovery platform for firms with a budget around $5k.[11] This challenge has been met but with varying degrees of success since that time. Therefore, until comparatively recently only a few large firms were able to support the infrastructure investments needed to utilize advanced AI and leverage machine learning for eDiscovery; small or medium size firms without large-scale litigation expertise still face challenges. [12]

The vendor community has heard the need for access to these tools for smaller firms or solo practitioners looking to scale their practices. Relativity (formerly Kcura), long a leader in the field of text-based AI for eDiscovery applications, recently added cloud capabilities through their RelativityOne offering.[13]Built on Microsoft’s Azure cloud, and compatible with the Microsoft environment, RelativityOne (available either as an enterprise purchase or on a case-by-case contracted basis through Relativity’s partners) currently provides the option for smaller firms to “scale up” for particular matters that may warrant using AI analytics by partnering with a certified vendor.[14] For firms that may be looking for an in-house solution, vendors such as Thomson Reuters CaseLogistix[15] and Vound’s Intella[16] promise smaller scale delivery of some of the biggest advantages of textual analytics, such as content clustering and email threading, at an accessible price point for small to mid-size firms looking to increase in-house capacity to “scale up” litigation on demand. This market can be expected to grow.

Applying Textual Analytics to Environmental Cases: The Promise and the Pitfalls

Given the advantages and increasing availability of textual analytics, it is somewhat surprising that it has not yet gained widespread use in environmental litigation. One reason may be that many environmental cases are decided on an administrative record, which is a relatively limited data set and which has previously been compiled by the agency or entity defending the action under review.

AI and assisted review may be more useful in private party environmental litigation, such as in CERCLA cases, and in cases that involve a large amount of scientific and technical data, such as enforcement cases brought by the government. To take the hypothetical example of a 6-party CERCLA case: you will have discovery coming in from a half-dozen different Potentially Responsible Parties, each with its own claims and counter-claims against all others. The material you receive will not necessarily be in any particular group or manner, and likely includes, for example, property records, product specifications, photographs, and sampling data.[17] While there may be email or electronic records involved, they are comparatively minor, and the need for email threading and sifting is not nearly as great as it may be in a large scale commercial litigation or criminal case. But nonetheless, you are looking to determine how to make your case: to prove, for example, that a certain chemical found in groundwater was in use by a particular entity during a particular time. By “clustering” related materials together based on text, independent of metadata or source, one can develop the facts required to, for example, place all product specifications that contain a certain chemical into one review set, or tie all property records that contain the same legal description together. Any “hot” or particularly useful pages in a 1000-page document can be marked as “highly relevant,” allowing similar documents to rise to the forefront and saving time and effort sifting through numerous pages to find if the report you are reviewing contains the snippet needed for the case.

Even more useful is the potential for an analytics engine to speed up internal review for defensive discovery. To this day, a major burden in defensive discovery is privilege review, even for records already manually evaluated for relevance. Technologies such as near-deduplication and email threading shine in defensive privilege reviews, by allowing the reviewers to see the whole context of a thread, for example, and preventing the need to review (and potentially redact) duplicative information consistently across thousands of records. The result is a more consistent and faster review, with fewer man-hours and a reduced risk of inadvertent waiver or inconsistent coding.

This is not to say that the technology is a magic bullet. For all the advances and rhetoric around machine learning, its fundamental weakness remains the same – it relies on text. This means that something as simple as the scanning quality can greatly affect machine learning validity and usefulness. A return to our hypothetical CERCLA litigation makes these limitations clear; for example, scanned in paper files and older documents may not be able to be accurately “recognized” by standard optical character recognition software (OCR), thus losing much of the value of text basted analytics. Similarly, non-text files, such as maps, photographs, audio, or video, are not able to be analyzed for TAR or CAL purposes and still require manual review and manual tagging to be included in content clusters or case analysis maps. Finally, the quality of the computer “intelligence” is, as always, determined by the skill of the humans on the other side of the screen – without accurate judgments to create high-value seed sets and knowledgeable supervision of iterative reviews, even small errors can easily multiply across a multi-thousand page set, limiting the analytics engine’s utility and requiring additional work to correct. Therefore, knowledgeable eDiscovery attorneys are still the keys to the success of analytics-aided reviews.

Judicial Acceptance of Analytics and the 2015 Amendments to the Federal Rules

Another barrier to acceptance of machine learning and AI for eDiscovery is a concern with how the technology will be received in the courts. Law is a fundamentally conservative field, and for good reason -- until lawyers can be confident that the using any cutting-edge technology will be accepted by the judiciary, it is in the clients’ best interests to do things in an accepted and explicable manner. No one wants to have to hire an expert, for example, to explain to an angry judge why you (or your client) chose to rely on the unfamiliar and, to some extent, unseen algorithms driving analytics-based reviews. Most lawyers know from their Westlaw or Lexis training about keyword searching and Boolean logic, and (perhaps unsurprisingly) many lawyers feel that they are able to craft search terms that are effective in finding the most relevant and responsive material from a set, and that there are no “tricks;” whereas presenting a review as automated can lead to allegations of corner-cutting or hiding the ball.

However, this conservatism, while understandable, is misplaced. From a technology standpoint, it has been proven time and again that machine-aided textual analytics are more accurate than attorneys with Boolean search tools – and that attorneys overrate their own accuracy by orders of magnitude, often due to a mismatch between expected and actual content (a problem familiar to anyone who has done a keyword search for, say, “confidential OR privileged” in an email database and found thousands of irrelevant emails containing signature block disclaimers).[18] Indeed, TAR protocols often result in greater transparency regarding relevancy decisions when compared to traditional human review. Parties can discuss and agree to certain seed set documents, agree to review protocols and accuracy level, or discuss whether, for example, to allow a litigant to produce documents in phases based on likely relevance (as determined by algorithm) with less potentially relevant documents relegated to “only if needed” in the case. This is a degree of transparency previously unavailable to opposing counsel from the human review “black box”.

In fact, TAR and similar technology is now so widely recognized that the Department of Justice seeks to use it in their most important and highest-profile claims, when accuracy and speed are of the essence. For example, in the recent case against Michael Cohen, a former attorney for President Trump, the DOJ sought and received permission from the Southern District of New York to use TAR review, in conjunction with a knowledgeable special master, to find and appropriately review material seized from Cohen’s law office for valid claims of privilege.[19] It is no secret to the Department of Justice that leveraging the power of AI and text analytics, accompanied by knowledgeable human review, is both faster and more accurate that traditional privilege review, and we expect to see its use increase in high profile matters involving the United States as a party.

From Judicial Acceptance to Judicial Expectation: Da Silva Moore through In Re Broiler Chicken

Despite the increased acceptance on the part of the bar, attorneys may still be concerned with judicial acceptance. Judge Peck in S.D.N.Y responded to this objection as far back as 2012, in his famous opinion Da Silva Moore v. Publicis Group,[20] the first case to explicitly provide judicial sanction to use TAR and computer-assisted review technology in litigation.

What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Counsel no longer have to worry about being the “first” or “guinea pig” for judicial acceptance of computer-assisted review.[21]

Since Judge Peck’s 2012 opinion, other courts have come forward to accept, and even encourage, technology assisted review for large cases.[22] The Southern District of New York has served as the vanguard for using computer aided review,[23] but other federal courts quickly followed suit, with decisions approving TAR and predictive coding appearing in the Southern District of Georgia,[24] the Eastern District of Pennsylvania,[25] and the District of Oregon,[26] among others, following shortly thereafter. Some courts even went a step further, ordering parties to at least consider whether advanced technologies, including TAR-type protocols and predictive coding, would allow for discovery and discovery disputes to be resolved faster and at less expense.

The Southern District of Illinois, in a class action suit captioned In re Broiler Chicken AntiTrust Litigation,[27] went a step further, entering a complex eight-page eDiscovery order governing the use of TAR and CAL, including deduplication methods, analytics-based search, validation protocols, keyword production and iteration, special master supervision, and the like.[28] While this degree of court supervision remains uncommon, there is at the minimum likely to be additional judicial supervision by increasingly tech-savvy courts of eDiscovery plans and protocols, including more orders discussing how parties are expected to use AI and analytics tools. This is particularly true given the updated Federal Rules, which, after 2015, support both transparency and judicial intervention into eDiscovery. Indeed, despite its complexity, the In re Broiler Chicken order will likely become a model for TAR and CAL based discovery plans in large litigation to come.

Rules Amendments and Updates Facilitate Predictive Coding

The Supreme Court and the Federal Rule Committee weighed in on the need to limit the electronic discovery’s escalating costs, including support for automated review, in the 2015 Amendments to the Federal Rules of Civil Procedure. These amendments attempted to revise the Rules in to encourage greater transparency and cooperation, and minimize what had been perceived as the inefficient, wasteful, and expensive practice of civil discovery.[29] The Rules Amendments therefore contain many changes intended to encourage cooperation, improve and streamline ESI production, and increase transparency, using new and advanced technologies such as computer aided review and textual analytics.

FRCP Amended Rule 26 and the New Scope of Discovery

The new scope and orientation of the federal rules toward cooperation is most evident in the changes made to Rule 26 and the Committee Notes. In particular, Rule 26(b)(1), governing the scope of discovery, was amended to clarify that the idea of “proportionality” would be the guide for whether any particular discovery was permissible, rather than the old standard of “reasonably calculated to lead to the discovery of admissible evidence.” [30]The “proportionality” standard includes, in the Rule’s text, a consideration of “whether the burden or expense of the proposed discovery outweighs its likely benefit,” a consideration further addressed in the Committee Notes:

The burden or expense of proposed discovery should be determined in a realistic way. This includes the burden or expense of producing electronically stored information. Computer-based methods of searching such information continue to develop, particularly for cases involving large volumes of electronically stored information. Courts and parties should be willing to consider the opportunities for reducing the burden or expense of discovery as reliable means of searching electronically stored information become available.[31]

This explicit discussion of the advancements in computer search is further evidences the technology’s acceptance, and increasingly, the expectation that litigants will use all available tools in service of faster, cheaper, and more accurate electronic discovery.

Waiver concerns and the FRE 502(d) Clawback Order

Another significant concern with automated review, particularly large data set review, is a concern with waiver. The specter of waiver, particularly “subject-matter waiver,” looms large over any production, threatening to open the door to disclosure of vast troves of privileged information from even a small misstep. Historically, waiver and privilege concerns have been among the biggest limiting factors to efficient automated review – after all, if the consequence of missing a single privileged item in a 100,000 document database is waiver of all related privilege claims, no amount of perceived efficiency will overcome the risk, and every document will likely remain subject to multiple rounds of “eyes on” human review.

Fortunately, the Federal Rules of Evidence contain a powerful tool: 502(d) orders. Rule 502(d) permits a court to issue an order providing that a party's disclosure of documents protected by the attorney-client privilege or work product protection does not waive the privilege (unless there was an intent to waive the privilege), regardless of the circumstance under which they were disclosed.[32] This means that even “negligent” production of privileged materials (as may be alleged to result from computer review) will not waive privilege, so long as there is an order in place. As many commenters (including federal magistrates) note, the 502(d) clawback order remains a singularly underutilized tool for addressing waiver concerns that may be impeding the use of computer assisted review and predictive coding technology.[33]

Experienced eDiscovery counsel knows that the foundation for a successful TAR, CAL, or other analytics-based review should be laid at the outset of the case, including preparation for 26(f) conferences and eDiscovery protocols, discussions of search, review, and production standards, and implementation of an appropriately comprehensive Rule 502(d) order.

Conclusion

AI and textual analytics have promise that can be harnessed for the environmental case, so long as you are aware of the power and the limits of the analytics machine and consider how it can best be applied to your particular dataset in your particular case. In fact, adding analytics to your eDiscovery toolbox is lower-risk than ever, given the changes in the rules and increasing judicial acceptance, and the potential advantages in both cost and accuracy enormous.

[1] https://www.theguardian.com/technology/2016/mar/07/email-ray-tomlinson-history

[2] https://thesedonaconference.org/wgs

[3] https://www.edrm.net/about-us/

[4] Zubulake v. UBS Warburg, 217 F.R.D. 309 (Zubulake I); Zubulake v. UBS Warburg, 216 F.R.D. 280 (S.D.N.Y. 2003) (Zubulake III); Zubulake v. UBS Warburg, 220 F.R.D. 212 (S.D.N.Y. 2003) (Zubulake IV); and Zubulake v. UBS Warburg, 2004 WL 1620866 (S.D.N.Y. July 20, 2004) (Zubulake V) stem from the same set of facts in an employment matter but explore different aspects of eDiscovery. The decisions, Judge Shira Scheindlin (now retired) in the Southern District of New York and are considered the foundational cases for basic eDiscovery principles such as litigation hold requirements and production standards. See https://ediscovery.co/ediscoverydaily/case-law/ediscovery-history-a-look-back-at-zubulake/

[5] https://www.archives.gov/files/records-mgmt/m-12-18.pdf

[6] Grossman, https://www.americanbar.org/content/dam/aba/administrative/litigation/materials/2017_sac/written_materials/1_a_tour_of_tar.authcheckdam.pdf.

[7] Id.

[8] https://www.edrm.net/frameworks-and-standards/technology-assisted-review/

[9] http://www.sentiosoft.com/static/upload/Sentio%20Differentiators.pdf

[10] This description is necessarily simplified; there are countless resources for more detailed information into the mechanisms behind various TAR and CAL algorithms and protocols. See, e.g. Cormack andGrossman, “Evaluation of Machine Learning Protocols for Technology-Assisted Review In eDiscovery,” available at http://delivery.acm.org/10.1145/2610000/2609601/p153-cormack.pdf?ip=66.193.37.34&id=2609601&acc=OA&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E5901FD3C6A0E4B05&__acm__=1547151175_8b3432f6ee9feef6698e504755aaff31

[11] https://gulfltc.org/ediscovery-for-small-cases/

[12] This gap has by no means closed. Even in recent surveys, TAR and analytics are predominantly used by larger firms or for larger matters, and a fairly large percentage (38%) do not use eDiscovery tools at all. https://www.lawtechnologytoday.org/2018/12/techreport-2018-litigation-and-tar/. However, this disparity may be more indicative of the concerns addressed within this article of familiarity with options and technology savvy, rather than solely access and cost.

[13] Relativity keeps a running blog of the newest features and enhancements available for RelativityOne, at https://www.relativity.com/ediscovery-software/relativityone/whats-new/?utm_source=hero-01-05-2019&utm_medium=website&utm_campaign=Homepage%20Hero#filter=product%3A9E366A9B-C291-890F-3AC408C64F72F5D3.

[14] Currently “single-case” applications of Relativity, using the Relativity One cloud application are only available via third-party Partner contractors: https://www.relativity.com/partners/.

[15]https://static.legalsolutions.thomsonreuters.com/product_files/westlaw/wlawdoc/web/case_logistix_user_guide.pdf

[16] https://www.vound-software.com/solutions

[17]https://www.americanbar.org/content/dam/aba/administrative/litigation/links/051815_aba_cercla_conference_papers.authcheckdam.pdf

[18] See, for example, Blair and Maron, “An Evaluation of Retrieval Effectiveness for a Full-Text Document Retrieval System,” (March 1985), available at http://yunus.hacettepe.edu.tr/~tonta/courses/spring2008/bby703/blair-maron-evaluation.pdf. In this seminal study, oft-cited in the eDiscovery literature, lawyers were asked to self-identify their perceived accuracy at finding relevant documents using keyword searches in a 40,000 document database (moderate by today’s standards), which they self-rated at around 75% accuracy. However, the study determined that actual accuracy was, on average, closer to 20 – 30% for human generated keywords.

The authors theorized that the idea that attorneys can adequately perform full-text document retrieval by keywords across large databases is in fact due to a biased assumption that words and their meanings are predictable within given contexts (e.g. that “water” AND “rule” will provide documents about “the water rule”), which is empirically untrue across data sets. As the authors note, “[I]it is impossibly difficult for users to predict the exact words, word combinations, and phrases that are used by all (or most) relevant documents and only (or primarily) by those documents.” Id. at 295. Text analytics tools that provide content clusters based on documents deemed to have actual relevance enable attorneys to break past this contextual bias by seeking the relevant patterns and terms as they actually exist, independent of human bias. See also https://catalystsecure.com/blog/2015/11/an-open-look-at-keyword-search-vs-predictive-analytics/#[1].

[19] Letter from the U.S. Attorney for the Southern District of New York https://assets.documentcloud.org/documents/4447558/Cohen-USAO-20180426.pdf

[20] 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012)

[21] Id. at 193.

[22] See generally The Sedona Conference, TAR Case Law Primer, available at https://thesedonaconference.org/download-publication?fid=3006 for a recent compilation of the case law surrounding technology assisted review and predictive coding.

[23] See, e.g National Day Laborers Ass’n v. U.S. Immigration and Customs Enforcement (877 F. Supp. 2d 87, 109 (S.D.N.Y. 2012), encouraging the use of computer-aided review in government FOIA litigation; Chevron Corporation v. Donziger, Case No. 11-Civ.-0691, 2013 WL 1087236, at *32 n.255 (S.D.N.Y. Mar. 15, 2013)(stating that, “predictive coding is an automated method that credible sources say has been demonstrated to result in more accurate searches at a fraction of the cost of human reviewers.”).

[24] FDIC v. Bowden, No. 4:13-cv-245, 2014 WL 2548137 (S.D. Ga. June 6, 2014).

[25] In re Domestic Drywall Antitrust Litigation, 300 F.R.D. 228, 233 (E.D. Pa. 2014).

[26] Arnett v. Bank of America, No. 3:11-cv-1372, 2014 WL 4672458 (D. Or. Sept. 18, 2014)

[27] 1:16-cv-08637 (N.D. Ill. Jan. 3, 2018)

[28] https://blog.logikcull.com/hubfs/In-Re-Broiler-Chicken-Antitrust-Litigation-eDiscovery-Order.pdf

[29] See https://www.uscourts.gov/statistics-reports/recent-and-proposed-amendments-federal-rules-annual-report-2015.

[30] F.R.C.P. Rule 26(b)(1) now reads “Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.” See also https://www.americanbar.org/content/dam/aba/multimedia/cle/materials/2016/01/ce1601frc.authcheckdam.pdf (ABA CLE guide interpreting the rule changes, including to 26(b)(1).

[31] Committee Notes to the 2015 Amendments to Rule 26, https://www.law.cornell.edu/rules/frcp/rule_26.

[32] https://www.law.cornell.edu/rules/fre/rule_502 (see Advisory Committee Notes to Rule 502 for discussion of use of Rule 502(d) orders to limit waivers in discovery). Judge Peck, in the Southern District of New York, even provides a model of a draft Rule 502(d) order for use by practioners looking to leverage the tool to limit waiver risks in large scale eDiscovery. See http://www.nysd.uscourts.gov/cases/show.php?db=judge_info&id=928.

[33] For an overview of judicial opinions and commentary on use of 502(d) clawback orders, see https://content.next.westlaw.com/Document/Ibb0a8759ef0511e28578f7ccc38dcbee/View/FullText.html?contextData=(sc.Default)&transitionType=Default&firstPage=true&bhcp=1 .

This article is not a substitute for legal advice. Please consult with your legal counsel for specific advice and/or information. Read our complete legal disclaimer.