Marketing Expert Witness’ Testimony on Students’ Exposure to Fraudulent Rankings Admitted
Posted on November 21, 2024 by Expert Witness Profiler
Plaintiffs Iola Favell, Sue Zarnowski, Mariah Cummings, and Ahmad Murtada (“Plaintiffs”) alleged that Defendant University of Southern California (“USC”) engaged in a scheme to artificially inflate the U.S. News & World Report (“US News”) ranking of USC’s Rossier School of Education (“USC Rossier”) by submitting incomplete data to US News — and then marketed that fraudulent ranking to the public.
An internal investigation conducted by USC’s outside counsel, Jones Day, determined that USC had submitted student selectivity data only for USC Rossier’s highly selective, in-person PhD program — but not from its less-competitive EdD program. The less-competitive EdD program was offered online after 2015 in collaboration with 2U, a company that offers technology platforms for online programs and provides advertising and recruiting for those online programs.
Plaintiffs alleged that USC, knowing the importance of the rankings on prospective students’ school choice, heavily marketed USC Rossier’s rapidly rising ranking to the public to boost enrollment in the online programs. USC orchestrated this scheme, Plaintiffs alleged, through its submission of false/incomplete data and then promoted the resulting ranking knowing that it was misleading.
USC filed Daubert motions seeking to exclude three of Plaintiffs’ expert witnesses, Dr. John Chandler, Sara Neher and J. Michael Dennis.
Marketing Expert Witness
Dr. John Chandler is a professor of marketing at the University of Montana who holds a master’s degree in mathematics and a doctorate in statistics and has worked in analytics and data science for 25 years with a focus on digital marketing.
Education & Schools Expert Witness
Sara Neher is a partner at Kennedy & Company Higher Education Strategies, a higher education consulting firm. She has an MBA and has worked in higher education for more than twenty years, including academic leadership positions at two business schools. Neher has experience consulting higher education clients about their US News rankings, including constructing models like the one she submits in this case, and has also been part of the submission of data to US News for multiple schools.
Survey Research Expert Witness
J. Michael Dennis is the Senior Vice President of the National Opinion Research Center, which is a survey research organization affiliated with the University of Chicago. Dennis has worked in survey research for more than 20 years, has authored more than 60 articles, and has been found qualified by numerous courts to provide expert opinions on consumer surveys.
Discussion by the Court
John Chandler
John Chandler submits a 95-page report explaining digital marketing practices in the context of student recruitment for graduate programs, the stages of a prospective student’s journey through the “marketing funnel,” what marketing strategies are typically deployed for student recruitment, and how USC Rossier applied these principles in its marketing strategy.
The first portion of Chandler’s report expresses extensive opinions on marketing practices, the enrollment journey, and the comprehensive marketing strategy that USC and 2U deployed to market USC Rossier. The second part of Chandler’s report discusses the exposure — i.e., the reach — of USC Rossier’s marketing materials containing the allegedly fraudulent ranking information. In the end, Chandler’s bottom line conclusion is: “Given the extensive and varied exposure methods outlined above, I can state with a reasonable degree of scientific certainty that all or nearly all students at USC Rossier in the MAT and OCL programs during the period of ranking manipulations were exposed to the fraudulent rankings. The pervasiveness of these rankings across multiple touchpoints ensured their near-universal reach.”
Reliability
USC did not challenge Chandler’s opinions on general marketing principles or the specifics of the extensive marketing strategy 2U and USC implemented with regard to USC Rossier. Instead, USC challenged the exposure/reach aspects of Chandler’s opinions. The crux of USC’s argument is that Chandler has no quantitative basis to opine that all or nearly all prospective students were exposed to fraudulent rankings through emails and/or other means.
USC further contended that Chandler did not have reliable site traffic data to substantiate an opinion that the vast majority of students visited the USC Rossier website and were exposed to a ranking representation.
The Court first observed that Chandler’s report extensively details the marketing strategy that USC and 2U deployed on behalf of USC Rossier, including specific opinions on how the marketing strategy was designed to move interested students through the marketing funnel from “awareness” to “enrollment.”
To arrive at his exposure conclusions, Chandler relied heavily on testimony from 2U’s 30(b)(6) representative, Dr. Joana Gerber. The Court has reviewed Gerber’s deposition transcript, as well the arguments both USC and Plaintiffs make about her testimony.
The Court agreed with Plaintiffs that the thrust of Gerber’s testimony and Chandler’s report is that 2U orchestrated an extensive marketing strategy designed to move all prospective students through the marketing funnel. It also agreed with Plaintiffs that Chandler’s opinions are the result of extensive review of record evidence, including Gerber’s testimony, and that USC’s placement of rankings on the USC Rossier website and in social media further supports his exposure conclusions.
Sara Neher
Sara Neher submitted a 13-page expert report explaining a model she constructed to project the US News ranking USC Rossier would have received if USC had submitted accurate data. US News publishes its methodology each year, including the relative weights of the various factors that underlay its rankings, with each school receiving an assigned rank based on its overall score on a 100-point scale.
Neher’s model standardizes the data in each category
Neher begins her reconstructed model with the data published by US News. First, Neher’s model standardizes the data in each category using z-scores, which are multiplied by the relative weights published by US News and added together to create a total for each school. Then, the total is indexed such that the top score always equals 100. The critical component — and the factor USC most vigorously challenges — is what Neher does to account for the information US News does not release. Neher acknowledged that she does not have access to information US News does not make publicly available, including the number of faculty with awards, the number of doctoral students who submitted a GRE score, the overall score for the lowest ranked 25% of schools, and the process for applying a logarithmic transformation to the student-faculty ratio.
Neher replaced the data USC Rossier actually submitted to US News with accurate data USC should have submitted
To account for this unavailable information, Neher calculated what she labels a “hidden-data constant” by comparing how much her model’s raw score (using the publicly available data) deviates from the school’s actual US News ranking. This, Neher reports, “captures the difference between what our model is able to calculate as the indexed score and what US News reports as the final score.” Then Neher replaced the data USC Rossier actually submitted to US News with accurate data USC should have submitted, based on an internal report from USC’s Director of Institutional Research who was instructed to investigate the matter. After generating a new overall score for USC Rossier using this information, Neher applies the hidden-data constant to account for the information she cannot recreate.
The result led to a dramatic decrease in USC Rossier’s ranking. Under Neher’s model, USC Rossier’s rank would have dropped from 15 to 48 in 2018; from 10 to 34 in 2019; from 12 to 61 in 2020; from 11 to 63 in 2021; and from 11 to 64 in 2022.
USC argued that Neher’s opinions regarding USC Rossier’s adjusted US News rankings must be excluded because they are the result of an unreliable methodology. Specifically, USC argued that the “hidden data constant” Neher calculates varies both school-to-school and year-to-year, resulting in a score that is sometimes close to US News’ ranking, but other times is dramatically far off. In other words, USC argued that there is no consistency to the hidden-data constant, so the methodology cannot be reliably used to predict adjusted scores based on different data input.
Qualifications
USC challenged Neher’s qualifications on the grounds that she does not have the training, experience, or specialized knowledge to qualify as an expert in statistical modeling. In addition, USC argued that she has never worked for or been trained by US News and has never before tried to replicate US News’s model; instead, she knows only what US News publicly discloses.
As USC sees it, absent insider knowledge about US News’s rankings, it is not possible to reliably replicate US News’ ranking at all — and any attempt to do so is “a pure guessing game.”
The Court declined USC’s invitation to announce a rule — especially in a discretionary evidentiary ruling — that the only way to reliably prove rankings fraud is to use (or, more likely, to misappropriate) insider knowledge of US News’s proprietary methodology. The Court is not convinced that any attempt to do so is necessarily a “pure guessing game” that categorially forecloses Neher’s proffered reconstructed rankings. This is especially true considering that Plaintiffs have identified other consulting firms and academic research that attempt to reconstruct US News rankings.
The Court held that Neher’s extensive experience working in higher education consulting constitutes a sufficient foundation to qualify her to offer her proffered opinions.
Reliability
The Court would begin by observing that neither Plaintiffs nor Neher purport to represent the proffered adjusted rankings model as a perfectly exact replication of the US News rankings. In addition, though USC did not challenge in its moving papers the result of Neher’s model — i.e., that USC Rossier’s ranking would experience a large decline — it indicated at the hearing that it does challenge Neher’s ultimate conclusion that rankings would have dropped.
For present purposes, the crux of USC’s challenge is how close Neher got in making a rankings prediction.
The Court also disagrees with USC that Neher’s methodology cannot be tested. Neher has described the methodology she deploys in her model in transparent and understandable terms, meaning USC and its experts can verify it, test it against different data, and/or critique the application of the hidden data constant.
USC does, however, point to numerous examples where the hidden data constant does not come close to replicating the school’s actual US News ranking, and that it also varied year-to-year. The question becomes, then, whether the inconsistencies with the outcome of Neher’s model compared to US News’s actual rankings are the result of unreliable methodology or instead go to the weight of the evidence. The Court notes that USC does not challenge Neher’s replication of the published aspects of US News’s methodology, only the methodological step of capturing the “hidden data constant.”
J. Michael Dennis
J. Michael Dennis submitted a 55-page report proposing two yet-to-be-conducted choice-based conjoint surveys that would estimate what USC Rossier’s tuition prices would have been but for USC’s alleged conduct. Dennis defines a choice-based conjoint survey as a “standard marketing research technique for quantifying consumer preferences for products and for the component features that make up a product. Conjoint analysis can be used to break down the value of a conceptual feature ( i.e., claims about the USC Rossier’s credentials) into its component parts (i.e., the claim that USC Rossier is a ‘top ranked’ program, or more specifically, that it was ranked between 10-15 by U.S. News & World Report in the 2017-2022 time period). Conjoint surveys take advantage of the fact that consumers are profoundly familiar with the task of shopping — comparing products, evaluating them, and making choices. Consumers are accustomed to making choices in their real-world shopping experiences.”
To calculate damages, Dennis explains:
“I define the but-for world as a world where the actual U.S. News rankings were in fact between 34 and 64 (corresponding to my fourth level of “30 to 59” for the “Rankings” attribute). In contrast, class members paid program costs when the advertised rankings were between 10 and 15 (i.e., corresponding to my second level of “10 to 19” for the “Rankings” attribute). If Plaintiffs’ allegations have merit, the market-clearing prices in the but-for worlds will be lower than the prices paid by class members.”
Reliability
USC first argued that Dennis’ opinions must be excluded because they rely on Neher’s inadmissible opinions. However, the Court would not exclude Neher’s opinions, thereby rendering this argument moot.
USC next argued that Dennis’ opinions must be excluded under Fed. R. Evid. 702(b) because they are based on insufficient facts or data.
USC argued that higher education does not operate in normal supply-and-demand conditions because of the impact of other economic incentives, including scholarships, fellowships, and grants. In addition, USC argued that universities address scarcity through selective admissions, not tuition price. USC’s argument, then, is that Dennis has no evidence justifying his assumption that the market value of an education from USC Rossier is anything other than what USC Rossier decides to charge.
Relatedly, USC also argued that Dennis has no evidence supporting his assumption that USC Rossier’s tuition responded to US News rankings. USC relies on a report from its expert witness to argue that there is no empirical analysis showing that tuition for EdD programs is affected by changes in school rankings.
This Court agrees that the real-world and market realities evidence upon which USC relies speak to the weight of Dennis’ analysis — which certainly could persuade a jury — but are not supportable reasons for excluding Dennis’ expert report and testimony.
Finally, the fact that Dennis’ survey has not been fully developed or implemented does not warrant exclusion at this juncture. There is no basis to exclude Dennis’ proposed conjoint survey. However, the Court will not preclude USC from bringing a later Daubert challenge to Dennis’ final report and testimony after his conjoint survey has been fully executed.
Held
The Court denied USC’s Daubert motions to exclude the expert report and testimony of Dr. John Chandler, Sara Neher, and Dr J. Michael Dennis.
Key Takeaways:
- While Chandler is not able to offer at this juncture a quantifiable number of students exposed to fraudulent rankings — an issue that might make the specific contours of his testimony subject to a later motion in limine — the Court does not find too great of an analytical gap between the record evidence and his exposure opinions. The core of USC’s argument goes to the weight of Chandler’s opinion and the identified shakiness of Gerber’s testimony, but “[v]igorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence.”
- The fact remains that Plaintiffs intend to use Neher’s model to show how consumers would react to USC Rossier’s adjusted rankings range, as compared to being a top-ranked school. With this in mind, the argument USC makes about the exact precision of Neher’s estimation cannot carry the weight that USC places upon it. But to be sure, USC has identified several weaknesses of Neher’s model which can be subject to “[v]igorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof.” The Court cannot and will not consider whether Neher’s model is right or wrong; it is satisfied at this juncture that Neher transparently and thoroughly explained her methodology in a way that can be tested and cross-examined. The Court is therefore not convinced that Neher’s model is the product of such unreliable methodology as to fail Daubert‘s gatekeeping standard.
- The Court noted that Dennis extensively details the structure of his survey and has considerable experience executing similar surveys. And as just examined, the Court is satisfied at this juncture that Dennis is qualified and has proposed a reliable methodology. Accordingly, there is no basis to exclude Dennis’ proposed conjoint survey.
Case Details:
| Case Caption: | Iola Favell Et Al V. University Of Southern California Et Al |
| Docket Number: | 2:23cv3389 |
| Court: | United States District Court, California Central |
| Order Date: | November 13, 2024 |
