Economics Expert Witness’ Event Study Admitted Despite the High Rate of False Positives
Posted on December 19, 2024 by Expert Witness Profiler
Plaintiffs Richard Dennis, Port 22, LLC, and Michael Glass asserted Commodity Exchange Act and Sherman Antitrust Act claims, alleging that The Andersons, Inc. (“TAI”) and Cargill Incorporated, who were supposed competitors, operated multiple grain storage warehouses in Ohio and collaborated to manipulate prices of soft red winter wheat (“SRW wheat”) futures and options contracts on the Chicago Board of Trade (“CBOT”).
According to the Plaintiff, the Andersons, Inc. sold SRW wheat to the major purchasers in October and November 2017 to suppress demand for physical SRW wheat and then, on November 29, 2017, registered for delivery two thousand certificates of CBOT December 2017 SRW wheat.
This registration (falsely, Plaintiffs say) signaled that TAI would sell ten million bushels of physical SRW wheat to parties holding long positions in December 2017 SRW wheat futures and caused a marked price decrease in the December 2017 SRW wheat futures contract and widened the spread between the December 2017 and March 2018 SRW wheat futures contracts.
TAI and Cargill later repurchased some of the shipping certificates TAI had delivered at the decreased prices. Plaintiffs allegedly transacted in December 2017 and March 2018 SRW wheat futures and lost money because of the decreased prices caused by the scheme.
Plaintiffs’ expert Craig Pirrong opined that Defendants’ manipulation of December 2017 and March 2018 SRW wheat futures injured Plaintiffs on a class-wide basis. Pirrong estimated class-wide damages and proposed a methodology for determining individual damages. Defendants challenged the reliability of Pirrong’s study through the report of their expert, Professor Justin McCrary.
Economics Expert Witness
Craig Pirrong‘s extensive qualifications includes approximately 30 years of concentrating professionally on competition and manipulation of prices with a focus on Chicago Mercantile Exchange wheat, soybean, and corn futures contracts; publishing a dozen peer-reviewed articles and a book on commodity futures manipulation and pricing; presenting to and consulting with federal agencies on manipulation; and testifying as an expert.
Discussion by the Court
Defendants argued that the magistrate judge’s order contained the following four clearly erroneous conclusions. First, the magistrate judge declined to exclude the March 2018 portion of Pirrong’s event study, which Defendants alleged lacked sufficient statistical significance. Second, the magistrate judge declined to exclude Pirrong’s ipse dixit assumption that Defendants’ manipulation caused a constant level of price artificiality that persisted for three months. Third, the magistrate judge found that the December 2017 portion of Pirrong’s event study is admissible despite the high rate of false positives that it produces. Fourth, the magistrate determined that Pirrong’s linear programming (“LP”) damages model, which Defendants contend is neither reliable nor helpful to a factfinder, is admissible. With the deferential standard of review discussed above in mind, the Court will review the magistrate judge’s order for clear error on each of these four bases.
Statistical significance of March 2018 event study
Pirrong’s expert report includes an event study, a regression analysis that uses specified control variables to estimate the daily market price of SRW and then compares those estimates to the prices observed in the market. An event study is used to determine the direction and magnitude of the effect of an unspecified variable, here, the alleged market manipulation.
Pirrong’s event study contains p-values associated with the cumulative residual on each day of the study. A residual is the difference between the observed value (here, the actual daily market price) and the estimated value (here, the daily market prices estimated by the control variables). The cumulative residuals used in Pirrong’s event study are simply the sum of each day’s residual and all the residuals that came before it within the period studied.
Because many of the p-values in Pirrong’s event study exceed commonly used thresholds of statistical significance (such as the 1%, 5%, and 10% thresholds), Defendants argued before the magistrate judge that Pirrong’s entire event study is unreliable. After carefully considering the issue, the magistrate judge was “unconvinced that all of [Pirrong’s] results should be excluded due to some p-values above 0.05, particularly where seven of eleven days (November 30 through December 8, 2017) within the December 2017 SRW wheat contracts regression analysis returned p-values with statistical significance at the five percent level.”
Defendants advanced the same statistical significance arguments before this Court, contending that the magistrate judge’s conclusions constitute clear error.
The Court held that it is reasonable that the magistrate judge, like many of our sister courts, declined to use statistical significance at the five percent level, or any other bright line threshold, as a proxy for reliability, and thus the admissibility, of Pirrong’s entire event study.
The Court is not left with the definite and firm conviction that a mistake has been made
Plaintiff’s theory of this case is that a discrete event—the registration of two thousand certificates of CBOT December 2017 SRW wheat on November 29, 2017, (after Defendants had saturated the market through major SRW wheat sales in October and November)—drove the market price of December 2017 and March 2018 SRW wheat futures downward. Pirrong’s event study uses a regression analysis to isolate the effect of this event from the innumerable other market factors—captured by the control variables—that determine the market price of SRW wheat futures.
The downward price impact of this discrete event would be easiest to pick out from the “noise” of other market factors right at the time of the event. As time passes and the other market factors continue to exert pressures on price, the manipulation event would become more difficult to pick out from the noise; what was first a bang fades into an echo. This is what Pirrong’s event study shows. For December SRW wheat futures, the lowest, or most statistically significant, p-values occur on November 30, 2017, and the days immediately following it. The Court found Pirrong’s conclusion that Defendants’ alleged market manipulation depressed prices in the December SRW wheat futures market statistically robust.
For March SRW wheat futures, the same conclusion is considerably less statistically robust. Even on the first day after Defendants’ registration of two thousand certificates of CBOT December 2017 SRW wheat, the negative residual returns a p-value of 0.20. Given the mixed statistical robustness of the results of the event study, this Court agreed with the magistrate judge that they are sufficiently reliable to be admissible.
Ipse dixit assumption that price artificiality in the commodities marketplace spanned three months at a fixed amount
Before the magistrate judge, Defendants argued that Pirrong’s conclusion that the March 2018 SRW wheat futures contract had a permanent, fixed, artificial price depression of 1.2¢ per bushel from December 14, 2017, to March 14, 2018, is unreliable because it is asserted with no empirical proof. Defendants renew that challenge here, arguing that Pirrong “is left only with his own word as to the existence of permanent price artificiality spanning three months in a marketplace where prices change every second of every day.”
The magistrate judge determined that Pirrong “reviewed the identified records, performed studies, and applied his extensive experience in futures markets to reach” his conclusion that the 1.2¢ per bushel artificial price depression held through the period from December 14, 2017, to March 14, 2018.
The Court held that while Pirrong’s explanation of why the price artificiality would remain constant is open to dispute, a disputable explanation is different than no explanation.
False positives
Defendants argued before the magistrate judge that Pirrong’s event study is unreliable because of the high rate of false positives it produces. A test for false positives takes Pirrong’s model and applies it to time periods where there is no alleged market manipulation. A false positive occurs when a residual has a p-value below a specified threshold of statistical significance. Defendants’ test for false positives employed a 43% threshold of statistical significance because that is the highest p-value reported for any day in Pirrong’s event study. Using a high threshold of statistical significance yields a high rate of false positives. Here, it indicated that there was price manipulation on “85% of the days for which Plaintiffs do not claim manipulation.”
After carefully considering this issue, the magistrate judge concluded that “[b]ecause Defendants’ false positives argument spins off from the p-values discussion and applies an across-the-board 43% threshold for statistical significance not adopted by Pirrong, the Court is disinclined to reach a different result here.”
Defendants advanced the same false positives argument here. The Court finds that the magistrate judge’s decision to admit the event study despite the high rate of false positives was correct. The 85% false positive rate reported by Defendants comes from the application of an across-the-board 43% threshold for statistical significance that is not endorsed by Pirrong (and is obviously not endorsed by the Defendants given their arguments on statistical significance). More troubling is the 19.66% false positive rate yielded by testing at a 5% level of statistical significance. The magistrate judge concluded that Pirrong’s testimony is “closer to shaky than unreliable.”
Linear programming (“LP”) damages model
First, Defendants renew their argument that Pirrong’s LP damages model is unreliable because it relies on an estimate of artificiality generated by the event study to calculate a range of aggregate damages for the class members. The reliability of the use of a constant 1.2¢ artificiality estimate is already addressed above.
Second, as the magistrate judge notes throughout her opinion, Defendants attack the output and not the methodology of Pirrong’s estimation of a theoretical range of possible damages.
Third, Defendants argued that the LP damages model is not helpful to the finder of fact because it “estimates a theoretical $20 million range of possible damages” and “provides no means for the trier of fact to estimate where in that broad range a reasonable, or probable, estimate of damages falls.” The magistrate judge correctly concluded that “the Court is not seeking to calculate actual damages” at the class certification stage. Instead, Plaintiffs must show that “proof of the damages caused by the scheme will either fail or succeed on a class-wide basis.”
The Court agreed with the magistrate judge that Pirrong’s LP model is sufficiently helpful and reliable to be admissible.
Held
The Court denied Defendants’ objection to the order denying in part Defendants’ motion to exclude the testimony of Plaintiffs’ expert Dr. Craig Pirrong.
Key Takeaways:
- It is noteworthy that Defendants attacked the output and not the methodology of Pirrong’s estimation of a theoretical range of possible damages.
- The magistrate judge concluded that Pirrong’s testimony is “closer to shaky than unreliable.” The 85% false positive rate reported by Defendants comes from the application of an across-the-board 43% threshold for statistical significance that is not endorsed by Pirrong (and is obviously not endorsed by the Defendants given their arguments on statistical significance).
- The Court held that while Pirrong’s explanation of why the price artificiality would remain constant sparks a debate, a questionable explanation is not the same as having no explanation.
Please refer to the blog previously published about this case:
Economics Expert Witness’ Improper State-of-Mind Testimony Excluded
Case Details:
| Case Caption: | Dennis V. The Andersons Inc. |
| Docket Number: | 1:20cv4090 |
| Court: | United States District Court, Illinois Northern |
| Order Date; | December 17, 2024 |
