What was the main point of the 2009 NAS report

Reflection

STATISTICS AND THE IMPACT OF THE 2009 NAS REPORT

Karen Kafadar†† Chair, Dep’t. Statistics, University of Virginia, President American Statistical Association. This Reflection piece is adapted from remarks made at a March 6, 2019 conference at Duke Law School, made possible by the Innocence Project, Duke Law School, and the Center for Statistics and Applications to Forensic Evidence.
PDF
PDF

Ten years does seem a long time, but it also reminds me of what my father used to tell my younger brother, who always came up with schemes that he expected to take off right away. He told my brother many times that “there’s no such thing as vertical take-off.”

Although it has been ten years since the “Strengthening Forensic Science in the United States” Report was released, adoption of each of the Report’s recommendations may take a while.[1][1]. Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat’l Research Council of the Nat’l Acads., Strengthening Forensic Science in the United States: A Path Forward (2009), https://www.nap.edu/read/12589/chapter/2 [https://perma.cc/JG6Y-X8SN] [hereinafter NAS Report]. I should note that I served on the Committee that produced that Report—the National Academy of Sciences Committee on Identifying the Needs of the Forensic Science Community. Looking back at the Committee’s charge and findings, what I see today are some of the positive consequences in forensics but also the continued barriers to the path forward.

The charge of the Committee, from Congress, was to assess present and future resource needs for forensics, make recommendations for maximizing their use, identify potential scientific advances to assist law enforcement, make recommendations to increase the number of professionals in the field, disseminate practices and guidelines to ensure quality and consistency, examine the role of forensics in homeland security, examine the interoperability of Automated Fingerprint Identification Systems (“AFIS”), and to examine additional issues as determined by the Committee. [2][2]. Id. at 1–2; see also H.R. Rep. No. 109–272, at 121 (2005) (Conf. Rep.) (authorizing “the National Academy of Sciences to conduct a study on forensic science”); S. Rep. No. 109–88, at 46 (2005) (listing the Committee’s responsibilities). The Committee was co-chaired by biostatistician Constantine Gatsonis from Brown University and the Honorable Judge Harry T. Edwards. [3][3]. NAS Report, supra note 1, at v.

The Committee also included law, biochemistry, chemistry, computer science, statistics, forensic science, and forensic practitioners as well as medical examiners. [4][4]. Id. at v. From 2007 to 2009, we reviewed the published literature, but we also received submissions, and researchers in the field were always invited to send research for the Committee to read. Those submissions still exist in the public access file.

The most cited sentence in the Report was, “With the exception of nuclear DNA analysis, however, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.” [5][5]. Id. at 7. We did not fully realize that sentence would be the most cited in the 352-page Report, but we got a clue after the Report’s release in 2009 when Judge Edwards and I presented its findings to the Office of Science and Technology Policy, and the Office focused on that sentence.

Our goal was supportive—to strengthen the value of forensic evidence, and there has been progress towards that goal since 2009. The Department of Justice, with the National Institute of Science and Technology (“NIST”), established the National Commission on Forensic Science, which existed from 2015 to 2017. Some of their results are accessible in DOJ archives. [6][6]. See Nat’l Comm’n on Forensic Sci., Reflecting Back—Looking Toward the Future (2017), https://www.justice.gov/archives/ncfs/page/file/959356/download [https://perma.cc/P4DN-8B4D], for an overview of the Commission’s accomplishments. The Organization of Scientific Area Committees (“OSAC”) was created, and it continues its work to develop standards in forensic disciplines. [7][7]. See generally The Organization of Scientific Area Committees for Forensic Science, Nat’l Inst. of Standards & Tech., https://www.nist.gov/topics/organization-scientific-area-committees-forensic-science [https://perma.cc/L29H-XV8D]. Then there is the Center for Statistics and Applications in Forensic Evidence (“CSAFE”), in which I participate along with researchers at five universities. CSAFE was created in 2015 with NIST funding. [8][8]. See generally Center for Statistics and Applications in Forensic Evidence, Nat’l Inst. of Standards & Tech., https://forensicstats.org [https://perma.cc/9ADL-SVRC].

The Committee members carefully reviewed every sentence in the 352-page Report. We devoted an entire meeting to reviewing the Report page by page, paragraph by paragraph, line by line, and word by word. The first recommendation was to create an independent federal agency, a National Institute for Forensic Science, because no federal agency existed that met all of the Committee’s minimum criteria. [9][9]. NAS Report, supra note 1, at 81 (“To promote the development of forensic science into a mature field of multidisciplinary research and practice, founded on the systematic collection and analysis of relevant data, Congress should establish and appropriate funds for an independent federal entity, the National Institute of Forensic Science (NIFS).”). That is still true. Such an agency needs to be strong in statistics and the physical and life sciences. It must maintain strong ties to professional organizations and be familiar with forensic science disciplines, measurement, and standards. It would also need to be newly created or created within an existing independent agency sufficiently prominent to enhance the standing of the forensic sciences. [10]

[10]. Id. at 80-81. The Report listed the following minimum requirements for the agency: • It must have a culture that is strongly rooted in science, with strong ties to the national research and teaching communities, including federal laboratories. • It must have strong ties to state and local forensic entities, as well as to the professional organizations within the forensic science community. • It must not be in any way committed to the existing system, but should be informed by its experiences. • It must not be part of a law enforcement agency. • It must have the funding, independence, and sufficient prominence to raise the profile of the forensic science disciplines and push effectively for improvements.

• It must be led by persons who are skilled and experienced in developing and executing national strategies and plans for standard setting; managing accreditation and testing processes; and developing and implementing rulemaking, oversight, and sanctioning processes. Id.

The Report also called for standardized reporting, which OSAC has assumed as its mission. [11][11]. Id. at 189–190. The Report emphasized the need for research on the accuracy, reliability, and validity of existing forensic methods. It also called for the development of new methods, which researchers at CSAFE are undertaking, at least with respect to pattern evidence. [12][12]. Id. at 190. The Report made a strong recommendation that forensic labs should be independent of law enforcement. [13][13]. Id. at 190–91. Today, the Houston Forensic Science Center is a major lab that is fully embracing independence from law enforcement. [14][14]. See generally Houston Forensic Science Center, http://www.houstonforensicscience.org/about-us.php [https://perma.cc/997S-56FP]. A few other labs around the country do not report to law enforcement, although today, ten years after the Report’s release, most do.

Regarding the Report’s recommendation for more research on the effects of, and reduction of, contextual bias in forensic analysis, CSAFE researchers are working with NIST and government labs to study that problem with respect to pattern evidence. In their approval of standards, OSAC members also are undertaking the development of quality control procedures and best practices for their disciplines. OSAC’s Forensic Science Standards Board prepared a Code of Ethics for forensic practitioners. [15][15]. Nat’l Comm’n On Forensic Sci., National Code of Ethics and Professional Responsibility for the Forensic Sciences (2015), https://www.justice.gov/archives/ncfs/page/file/788576/download [https://perma.cc/LYZ5-2U99]. The Report also recommended improved education and training in best practices and scientific foundations; [16][16]. NAS Report, supra note 1, at 214–215. CSAFE is developing courses in statistics for practitioners and is working with the American Bar Association on creating courses for judges and lawyers. The Report’s recommendation for “eventually eliminating existing coroner systems” was not new; [17][17]. Id. at 267. the National Academy of Sciences recommended it in a report in 1928. [18][18]. See generally Oscar T. Schultz & E.M. Morgan, Nat’l Research Council, The Coroner and the Medical Examiner (1928). Ninety-one years later, we are still working on it. Nationwide AFIS interoperability is a task that Melissa Taylor at NIST is undertaking, but the work is challenging because of the proprietary nature of AFIS algorithms. However, the Defense Forensic Science Center is making its algorithm, used for latent fingerprint comparisons, available to other laboratories. They are planning to put that algorithm in the public domain, which is great progress.

There remains a culture of defensiveness and overconfidence among forensic science practitioners. Some people in forensics still do not know about the National Academy of Sciences Report, or they do not want to believe it. When the President’s Council of Advisors on Science and Technology concluded in 2016 that little had changed in seven years, I was surprised that it was more criticized than it was appreciated among forensic scientists. [19][19]. President’s Council of Advisors on Sci. & Tech., Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods 21 (2016).

Let me provide just one example of what I mean about the slow pace of process. Three ASTM glass standards for forensic glass were proposed for the OSAC registry. [20][20]. Karen D. Pan & Karen Kafadar, Statistical Modeling and Analysis of Trace Element Concentrations in Forensic Glass Evidence, 12 Annals of Applied Stat. 788, 791 (2018). The analysis proceeds by comparing trace element concentrations in glass fragments found at the crime scene with those found on a suspect. The three procedures for measuring these trace element concentrations in the glass fragments use sound techniques in analytical chemistry. But each of the standards includes a section entitled “Calculation and Interpretation of Results,” which describes inferential procedures on those measurements that you would never see in the statistics literature. [21][21]. See id. at 791–92 (describing the seven-step inferential procedure). The forensic scientists, however, responded, “There are tons of data using this procedure and the studies have shown less than 0.1% false positives.” My colleagues and I were able to obtain the data from two papers using the inferential measurement techniques, so we could validate the claim of “less than 0.1% false positives.” Before the Forensic Science Standards Board voted on whether or not to approve the standards for the OSAC registry, two of our pre-publication results were made available to them. One was a simulation study that showed considerably higher error rates than 0.1%. There were also machine learning algorithms developed that outperformed the procedures in these standards. Nevertheless, OSAC approved the standards by a vote of 13-1-1. [22][22]. Three New Standards Approved for OSAC Registry, Nat’l Inst. Standards & Tech. (June 26, 2018), https://www.nist.gov/news-events/news/2018/06/three-new-standards-approved-osac-registry [https://perma.cc/4M3V-WFNM].

What do we see as a path forward for forensics, ten years later? Research is still needed to identify the benefits and limitations of methods. CSAFE and NIST are working hard on that research. There also needs to be more collaboration between scientists and practitioners. When you look at the structure of the OSAC, the disciplines are collected within common science areas, but there is poor communication among the disciplines. Consequently, common problems shared by disciplines are still viewed as unique and individual to a single discipline. CSAFE collaborations are designed to address that problem—we are coordinating solutions for problems that are common across disciplines. But there is still more work to do. We still need more well-designed, fully double-blind studies to assess error rates. We still have to change the culture. It is not a sign of weakness to say you are not 100 percent accurate. It is far better to quantify the realistic errors in procedures than to pretend they are perfect. We still need to identify sources of variation in the process. Non-pattern evidence also needs research.

When Judge Edwards spoke to the Senate Judiciary Committee while serving on the National Academy of Sciences Committee, he said, “We really do believe that this research is improving the forensic science community.” Ten years later, Judge Edwards said:

Perhaps most critically, we still do not know what we do not know. We need better scientific studies and standards to shape the work of forensic practitioners and regulate the admission of forensic evidence. This means that more top scientists must engage in research on forensic methods and appear in court to explain the evidence. [23][23]. Hon. Harry T. Edwards, 10 Year Anniversary of the Landmark Report on Forensic Evidence, Innocence Project (Feb. 21, 2019), https://www.innocenceproject.org/judge-edwards-nas-statement/ [https://perma.cc/2RUY-UNNG].

Thus, ten years after the Report, in forensics, statistics, and in law, “much remains to be done.” [24][24]. Id.

Copyright © 2019 Karen Kafadar.