By Natalia Olchanski, PhD, MS, Research Assistant Professor, Josh Cohen, PhD, Deputy Director, and Peter Neumann, ScD, Director
Background
Health technology assessments (HTAs) draw on evidence that is often incomplete and immature. HTA organizations do their best to describe uncertainty’s impact on projected benefits, costs, and cost-effectiveness, but they sometimes come up short because they do not consider all plausible alternative assumptions. We investigated how much uncertainty might be missing from HTA estimates.
HTAs can miss plausible assumptions because they typically convene one group of experts. For judgments such as which proxy measure or datasets to use to estimate risk, HTA groups often settle on a single best option and either explicitly or implicitly dismiss other possibilities. What remains unstated and seemingly unexplored is whether the single, selected option is the sole plausible approach. Moreover, if there are plausible alternatives, would they produce substantially different estimates of cost, benefit, and value?
We explored this issue by convening two independent sets of experts – each including a clinician-researcher and a health economic modeling expert – to identify assumptions for an HTA conducted by the Institute for Clinical and Economic Review (ICER) in 2017. That HTA evaluated poly ADP-ribose polymerase (PARP) inhibitors in ovarian cancer. We replaced ICER’s assumptions with the assumptions that our two expert groups independently developed, and compared the resulting cost-effectiveness estimates.
Results
The Figure illustrates how our expert assumptions affected cost-effectiveness. For example, when using the utility weights recommended by the expert groups (top axis in figure), the cost-effectiveness estimates (green circles) came out more favorable than (to the left of) ICER’s estimate (stars), both for recurrent BRCA-mutated disease and platinum sensitive disease. For recurrent BRCA-mutated disease (above the top axis), the estimate from one expert group fell below $150,000/QALY, a recognized benchmark for value.
Main finding
Expert judgment can represent an important source of uncertainty in HTA. Cost-effectiveness estimates corresponding to assumptions identified by the two expert pairs and by ICER often differed by substantial margins. In our case, utility estimates and treatment duration estimates yielded notable differences in projected cost-effectiveness.
Coda
Evaluating any HTA’s robustness depends on reproducing the original work. From there, one can replace assumptions and see if the conclusions hold up. ICER did not publish enough information for us to evaluate their work comprehensively. ICER and other HTA practitioners should publish model source code so that stakeholders can independently evaluate their findings.
Figure: Impact of alternate assumptions on cost-effectiveness
Note: The figure represents our findings semi-quantitatively, illustrating their relative positions on the cost-per-QALY axis. Values to the left are more favorable (less cost per QALY gained); values to the right are less favorable.
Funding/support: This work was supported by the National Pharmaceutical Council.