A Standardized Framework for Evaluating State Opioid Surveillance Dashboards: A Health Informatics Assessment

Clinical Snapshot

Key Findings

• P Value: Inter-rater reliability p=0.002; population correlation p=0.193

• Effect Size: Substantial variation with 10-point range representing 36% difference between lowest and highest performers

• Primary Outcome: Dashboard functionality scores ranging from 12-22 points (mean 19.18, SD 2.20)

• Nnt Or Sensitivity: Inter-rater reliability r=0.84 indicating good measurement consistency

• Confidence Interval: Not reported for main outcomes

Clinical Application

Framework appears feasible to implement with trained evaluators, requiring systematic review of dashboard features Limited direct applicability as framework designed for U.S. context, but evaluation principles could inform Australian Department of Health surveillance systems and state-based monitoring platforms Public health agencies and policymakers responsible for surveillance dashboard development and evaluation

Abstract

BACKGROUND: State public health agencies across the U.S. depend on opioid surveillance dashboards to monitor overdose trends and guide interventions. Despite their central role in decision-making, no standardized framework exists to evaluate how well these dashboards actually function. OBJECTIVE: We developed a health informatics framework to systematically assess state opioid surveillance dashboards and applied it across all 50 U.S. states. METHODS: Our framework evaluates dashboards across nine domains (including temporal analysis, geographic granularity, data disaggregation, visualization features, narrative support, simplification techniques, and interactive functionality) using 28 specific sub-variables. Sub-variables were selected based on public health informatics literature on effective dashboard design, including features identified in COVID-19 dashboard evaluations and usability frameworks. We scored each feature as either present or absent, creating composite scores with a theoretical maximum of 28 points. All 50 U.S. state-level dashboards were reviewed twice by the study team, with an external consistency check on a subset of 10 dashboards. We calculated the correlation coefficient between dashboard scores and state population estimates. RESULTS: Dashboard scores varied widely, ranging from 12 to 22 points (mean:19.18, standard deviation:2.20). Three states (Delaware, Indiana, and New York) achieved the highest score of 22 by offering comprehensive geographic detail, extensive data breakdowns, and full interactive capabilities. Interactive filtering was present in 92% of dashboards, but only 80% enabled switching between visualization modes. While nearly all dashboards showed time trends, only 6% provided zip code-level detail, and over a third lacked race and ethnicity data. Single external reviewer demonstrated strong correlation (r = 0.84, p = 0.002). Dashboard scores were not correlated with state population size (r = 0.187, p = 0.193). CONCLUSIONS: State opioid surveillance dashboards varied substantially in observable capabilities. This framework provides a standardized method to compare dashboard features and identify gaps that may inform future refinement.

References

1. 1.Aguiar, C., Bolus, M., & Mudumbai, S. C. (2026). A standardized framework for evaluating state opioid surveillance dashboards: A health informatics assessment. Journal of Medical Systems. https://doi.org/10.2196/77361