Five-Tier Comparison of Heart Transplant Programs: Clarity or Confusion? - American College of Cardiology (2024)

Study Questions:

How useful is the 5-tier Scientific Registry of Transplant Recipients (SRTR) ranking for heart transplant programs?

Methods:

Using publicly available data from program-specific reports of all US heart transplant centers, the authors analyzed the stability, volatility, and reliability of the 5-tier outcome assessment used by the SRTR to report the performance of transplant programs. For all solid organ transplant centers, the SRTR assesses outcomes for 3 metrics: waitlist survival, faster transplant, and graft failure (https://www.srtr.org) and assigns each metric a tier (1-5). Tiers are derived from the estimated mortality rate ratio for waitlist mortality, transplant rate ratio, and estimated hazard ratio for first-year graft survival. Assessments and tiers are updated every 6 months. This study included both adult and pediatric programs. Stability was reported as center-level number and proportion of rating changes over time, volatility as the center-level standard deviation of the tier ratings (average amount the rating has differed from its mean), and reliability was calculated using the intraclass correlation coefficient and Fleiss’s kappa.

Results:

There were 112 adult and 55 pediatric heart transplant centers included in this study. Approximately 80% of the centers were ranked at tiers 2-4 on each metric, with the most common tier being 4 for waitlist survival, 3 for faster transplant, and 3 or 4 for graft failure. For the majority of the 6-month time periods, most centers experienced no change in tier from the prior time period for graft failure and faster transplant. The most common changes were 1-tier changes, and changes most commonly occurred in waitlist survival, with 44%-55% of adult programs experiencing a change of ≥1 tier. Between 37-52% of programs experienced any change in graft failure rating compared to the previous period, and 21-38% experienced change in tier for faster transplant. Findings were similar for pediatric transplant centers. The median number of changes across the 9 study periods for adult programs was 4 for graft failure and waitlist survival and 2 for faster transplant. The most variability was seen in waitlist survival; the average standard deviation was 0.77 for adult centers and 0.79 for pediatric centers. Finally, the kappa and intraclass correlation coefficient were low, suggesting poor reliability and agreement. The median number of time periods to tier change was 3 (18 months) and the quickest to change was waitlist survival in adult centers at 1 time period (6 months).

Perspective:

Ranking systems are designed to summarize complex information into an easily digestible format that allows consumers to make informed decisions. Such is the case with the public assessment of transplant programs. Guided by the Agency for Healthcare Research and Quality and in concordance with the Organ Procurement and Transplantation Network Final Rule, the SRTR publishes for each transplant program a 5-tier assessment (“worse than expected,” “somewhat worse than expected,” “as expected,” “somewhat better than expected,” or “better than expected”) of 3 performance metrics: waitlist mortality, transplant rate, and 1-year graft survival. The goal is to provide to the public timely, accurate, and easily accessible information regarding transplant program performance. But is this system useful or does it promote misinformation?

The authors of this paper have challenged the usefulness of the 5-tier assessment based on analyses of variability, consistency, and reliability; however, the interpretation of these results is hindered by the lack of a gold standard. That this study demonstrated frequent changes in tiers between study periods and a median of 2-4 changes over 4 years (interpreted as instability and volatility), does not mean that the assessment is inaccurate and does not disprove actual change, positive and negative, in program performance. For instance, a program with low rankings may be in a higher tier in subsequent periods due to its response to a poor rating and programmatic changes. The median time to a tier change was 3 time periods (18 months), which seems to be a reasonable period of time for programmatic change. It is possible that frequent updates to the tiers, currently every 6 months, may lead to variability in tier assignment, that while real, may not be clinically meaningful to the user due to small differences between tiers (e.g., high tier 3 vs. low tier 4). Since hard boundaries are imposed for the ranking system to work, these nuances in the assessment will not be apparent to the user. Reporting the underlying tier score might be helpful. In addition, the utility of 1-year graft survival as a metric is widely debated, as 3-year or 5-year graft survival may be a more meaningful metric.

Is the variability in tier real or noise? This analysis does not provide that answer. Does this render the rankings useless? No. The community pushed for a change from a 3-tier assessment to the current 5-tier assessment due to lack of granularity. Thus, the community must continue to identify and achieve the best metrics for outcomes assessment.