Survey Research Expert Allowed to Opine on Tuition Prices

This is a class action alleging higher education rankings fraud. Plaintiffs Iola Favell, Sue Zarnowski, Mariah Cummings, and Ahmad Murtada ("Plaintiffs") allege that Defendant University of Southern California ("USC") engaged in a scheme to artificially inflate the U.S. News & World Report ("US News") ranking of USC's Rossier School of Education ("USC Rossier") (especially in regards to its online programs) by submitting incomplete data to US News — and then marketed that fraudulent ranking to the public.

Plaintiffs alleged that USC, knowing the importance of the rankings on prospective students' school choice, heavily marketed USC Rossier's rapidly rising ranking to the public to boost enrollment in the online programs.

Plaintiffs Iola Favell et al. sought to exclude certain opinions in the rebuttal expert report of Defendant University of Southern California's ("USC") expert, Dr. Keith A. Botner, along with all testimony based thereon. USC, in turn, sought to exclude the expert report and testimony of Plaintiffs' expert, Dr. J. Michael Dennis.

Marketing Expert Witness

Dr. Keith A. Botner has extensive experience in marketing, including marketing research, consumer insights, marketing theory, marketing strategy, advertising, survey design, survey execution, and survey data analysis.

Botner received a Bachelor of Science (BS) degree in Business Administration from the University of Louisville in 1996, with a concentration in Marketing. He received a Master of Business Administration (MBA) degree from Michigan State University in 1999, also with a concentration in Marketing. He received a Doctor of Philosophy (PhD) degree from the Department of Marketing in the David Eccles School of Business at the University of Utah in 2015.

Want to know more about the challenges Keith Botner has faced? Get the full details with our Challenge Study report.

Survey Research Expert Witness

John Michael Dennis has worked in survey research for more than 25 years, has authored more than 60 articles, conference and seminar papers, or book chapters. He has been found qualified by numerous courts to provide expert opinions on consumer surveys.

Gain a comprehensive understanding of J. Michael Dennis' qualifications and casework history with his Expert Witness Profile report.

Discussion by the Court

The parties' competing Daubert motions concern the use of choice-based conjoint ("CBC" or "conjoint") surveys to estimate a tuition price premium associated with USC Rossier's allegedly inflated rankings.

I. J. Michael Dennis

Summary of Opinions

Dennis' report outlines the methodology and results of two CBC surveys that estimate what USC Rossier's tuition prices would have been but for USC's alleged conduct.

Dennis defined a CBC survey as a "standard marketing research technique for quantifying consumer preferences for products and for the component features that make up a product. Conjoint analysis can be used to break down the value of a conceptual feature (i.e., claims about the USC Rossier's credentials) into its component parts (i.e., the claim that USC Rossier is a 'top ranked' program, or more specifically, that it was ranked between 10-15 by U.S. News & World Report in the 2017-2022 time period). Conjoint surveys take advantage of the fact that consumers are profoundly familiar with the task of shopping — comparing products, evaluating them, and making choices. Consumers are accustomed to making choices in their real-world shopping experiences."

Dennis conducted separate surveys for prospective students of Rossier's online Master of Arts in Teaching ("MAT") program and the online Doctor of Education ("EdD") in Organizational Change and Leadership program.

Reliability

USC raised four grounds for the exclusion of Dennis' expert opinions. First, USC argued that Dennis did not reliably apply conjoint analysis because (1) his simulations produced absurd results; (2) Botner's results differed when attempting to replicate Dennis' methodology; (3) Dennis' choice of rankings ranges unreliably skewed the price premiums higher; and (4) Dennis' analysis rests on a factually unsupported premise because real-world evidence purportedly shows that Rossier's tuition does not respond to changes in its U.S. News ranking.

First, USC argued that Dennis' opinions must be excluded because the results of his surveys are absurd and render his methodology unreliable. Specifically, USC emphasized that Dennis estimated substantially different premiums for the MAT and EdD programs, and that his estimated MAT premium increased disproportionately when the hypothetical ranking suffered only a five-place rankings drop from 58 to 63. As such, none of USC's results-based arguments persuade the Court that Dennis' opinions are inadmissible.

Second, USC contended that because Botner's Replication Surveys produced different price premium estimates, Dennis' results cannot be replicated and are therefore unreliable. That Botner's replication surveys produced different results therefore does not warrant exclusion of Dennis' surveys. Still, the Court would also find that Botner did not perform a true replication of Dennis' methodology, which creates a faulty comparison between the results. Botner used smaller samples, a different sample source, and respondent populations whose composition differed from Dennis' samples. These factors could plausibly have created the differences in survey results that USC now points to, but the Court need not resolve those competing statistical opinions at the admissibility stage. Botner was able to repeat the general survey process and obtain positive premium estimates, even if some estimates were substantially smaller than Dennis'. Dennis' survey was thus sufficiently replicable for the purpose of admissibility.

Plaintiffs' other expert, Dr. Sara Neher, has estimated, however, that but for USC's alleged misrepresentation, Rossier's U.S. News ranking would have been 47, 33, 58, 68, and 63 for the relevant years. Third, USC argued that Dennis departed from Neher's analysis by converting Neher's precise, year-specific adjusted rankings into broader ranking ranges. USC complained that, for example, Neher's adjusted ranking of 63 was lumped in with a broad range consisting of ranks 60-99, which skewed results in favor of higher price premiums than if Dennis had used different or more granular ranges. Dennis also did not purport to replace Neher's rankings with a different assessment of Rossier's position; he merely mapped each of Neher's estimates into the corresponding range respondents had evaluated. USC may argue that the ranges are too broad or that presenting exact rankings would have better implemented Neher's opinions, but those criticisms concern survey design and the weight that should be afforded the resulting estimates, rather than admissibility.

Fourth, USC contended that Dennis' analysis is unreliable because it assumes, without empirical support, that Rossier would have charged less tuition if its U.S. News ranking had been lower. These remaining arguments substantially overlap with arguments addressed in the Court's prior orders. The Court previously rejected USC's contention that because universities do not continuously adjust tuition in direct response to changes in consumer demand, conjoint analysis like that performed by Dennis is unreliable.

USC has not pointed to any material change in Dennis' execution of his proposed survey that would alter the Court's prior conclusion. The Court acknowledges that some courts have found conjoint analyses inadequate where they failed to properly account for supply-side considerations. Nevertheless, the Court would be inclined to follow decisions recognizing that the use of historical supply-side data — including treating the quantity supplied as fixed — is a reasonable modeling assumption, particularly in the university context, and a permissible alternative to USC's premise that tuition is wholly independent of student demand.

The conflicting historical evidence cited by USC does not require a different result. Evidence that Rossier's tuition did not move in lockstep with its ranking, or that USC raised tuition after Rossier withdrew from U.S. News, may support USC's contention that rankings did not affect the prices students paid, but again, this is a question for the factfinder.

II. Keith Botner

Summary of Opinions

Botner conducted four CBC surveys in response to Dennis' surveys described above. Botner's surveys included two Replication Surveys, intended to reproduce Dennis' design, and two Modification Surveys, which changed various elements of that design. Plaintiffs challenged only the admissibility of the Modification Surveys, and any testimony based on those surveys.

Reliability

Plaintiffs contended that the Modification Surveys are inadmissible because: (1) six schools cannot represent the roughly 250 ranked graduate schools of education; (2) Botner selected the institutions through an informal, nonrandom process; (3) the selection process cannot be replicated; (4) varying school name, ranking, and price generated implausible product combinations that confounded the school and ranking attributes; and (5) the schools are not geographically dispersed and do not occupy adequately differentiated ranking tiers.

First, Plaintiffs argued that using only six school names, out of "approximately 250 graduate schools of education ranked by US News," renders Botner's methodology unreliable because it only represents 2% of the market — far less than the 80% that Dennis typically aims for. Other than this being Dennis' usual practice, however, Plaintiffs have not pointed to any authority that requires CBC surveys to reach a certain percentage of a relevant market for the analysis to be reliable. In fact, the parties agree that too many varying options for a given attribute may bias the results. It is thus infeasible for Botner to have literally included 80% of the relevant graduate schools in the survey. Instead, to include School as an attribute, Botner sought to make his school selections representative of a broad selection of institutions by varying their rank and geographic location, for example. The Court would find that this is a reasonable approach, and in any event, the inclusion of a "specific attribute selection" is a challenge that goes to weight, not admissibility.

Second, Plaintiffs contended that Botner selected the six schools through an informal, nonrandom process based principally on his own judgment.Botner, however, disclosed the considerations underlying his selections: he examined the market for geographically dispersed programs that offered the relevant MAT or EdD degrees and occupied different areas of the U.S. News rankings; reviewed the programs' websites; and, for the EdD survey, confirmed that the selected programs offered specializations comparable to Rossier's. While the process was not overly formal, the criteria and market research that went into the decisions are described in his report and are reasonable. Further, Plaintiffs have not established that attribute levels must be selected randomly. Plaintiffs' expert, Dennis, similarly selected the levels of his attributes, such as the rankings ranges, through professional judgment rather than random sampling.

Third, Plaintiffs argued that Botner's school-selection process is not replicable because another researcher could not independently apply his qualitative criteria and necessarily arrive at the same six schools. The Court would find that Botner's methodology is sufficiently replicable. Botner identified the six schools, disclosed the questionnaire and choice-task design, described the respondent criteria and analytical settings, and provided the underlying data, utility reports, and CBC design files in native format. Another researcher can therefore administer the same instrument to a new sample and determine whether it produces materially similar results. Plaintiffs have not identified an attempted replication of the Modification Surveys that failed.

Fourth, Plaintiffs contended that including both school name and ranking violated conjoint analysis's independence requirement and produced nonsensical combinations, such as NYU appearing with a ranking of 100-150 and a tuition of $55,000 while Iowa State appeared with a better ranking and substantially higher tuition. But the purpose of Botner's Modification Surveys was to test whether ranking retained importance once respondents were provided the school brand they would encounter in the real marketplace. Thus, Dennis' finding that the importance assigned to ranking declined after school names were introduced does not, standing alone, demonstrate contamination; it is consistent with Botner's hypothesis. The parties' disagreement about how respondents reacted to unlikely combinations may be explored through their competing experts, but it does not make Botner's methodology inadmissible.

Fifth, Plaintiffs argued that Botner failed to follow his stated criteria that the selected schools be geographically dispersed and "consistently rank in different tiers." The Court would find that neither argument establishes that Botner abandoned the methodology stated in his report. As to geography, Botner did not define geographic dispersion as proportional representation of every Census region; he required only generally geographically dispersed programs. The MAT survey included institutions in California, New York, Massachusetts, Georgia, and North Carolina, while the EdD survey included institutions in California, New York, Massachusetts, Georgia, Florida, and Iowa. Those selections are spread across multiple regions, even though they do not provide uniform regional coverage. As to ranking, the selected institutions ranged from NYU, ranked as high as sixth, to Iowa State, ranked as low as 112th, with Boston College, the University of Georgia, San Diego State, UNC-Charlotte, and the University of Miami generally occupying different portions of the range between those points. Although certain schools occupied similar positions during portions of the class period, Botner did not state that every school would occupy a different survey ranking interval in every year, and he instead described "tiers" as a general reference to ranking separation. Plaintiffs' alternative school selection approaches may provide a basis for questioning the representativeness of Botner's selections, but they do not establish that Dr. Botner abandoned his stated methodology.

Accordingly, the Court would conclude that USC has demonstrated by a preponderance of the evidence that Botner's Modification Surveys rest on sufficiently reliable principles and methods and that he applied those methods reliably for purposes of Rule 702. Plaintiffs' criticisms provide substantial material for cross-examination and competing expert testimony, but they do not warrant exclusion.

Held

• The Court denied USC's Daubert motion to exclude the expert report and testimony of Dr. J. Michael Dennis.

• The Court also denied Plaintiffs' motion to exclude Dr. Keith Botner's modification surveys and the opinions based upon them.

Key Takeaway

Daubert requires that an expert's methodology be testable. A crucial factor in a study's testability is whether it can be replicated. But under Daubert, replication means that experiments are capable of repetition, not that experts must repeat the same experiment again and again to demonstrate reliability.

Case Details: