How U.S. Hospitals Can Overcome Barriers to Automation of Electronic Clinical Quality Measurement (eCQM)

Aaron Baird, Associate Professor, Georgia State University; Andrew Sumner, Director of the Institute of Health Administration, Georgia State University, Yusen Xia, Director, Institute of Insight, Georgia State University

Contact: abaird@gsu.edu

The Current State of Quality Measurement and Reporting Automation

As the U.S. healthcare system moves toward an emphasis on value and accountability, a key underlying mechanism that facilitates this shift is healthcare quality measurement. Healthcare quality is, “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge,”¹ as well as “whether individuals can access the health structures and processes of care which they need and whether the care received is effective.”² Quality measurement is the process of collecting data related to prescribed quality measures and reporting this data to requesting agencies and payors.³ For instance, the U.S. Centers for Medicare and Medicaid Services (CMS) now requires or incentivizes, depending on the program, participating U.S. hospitals to collect and report data related to prescribed quality measures.⁴

While there is general agreement among healthcare stakeholders that healthcare delivery should be of high quality, there is considerable debate about how to best facilitate quality measurement and reporting.^5-15 This debate is becoming especially heated and poignant as value-based and accountable care programs impose significant administrative burdens on healthcare providers.^5,7,8,16 Such burdens are not likely to abate any time soon, as quality measurement is now beginning to shift toward more specific measures that require use of clinical data, which requires even more effort on the part of providers and administrative staff.⁶

This is in contrast to using primarily administrative data already submitted to payors as a basis for quality measurement. For instance, if clinical data is required to calculate one or more quality measures, clinicians must be sure to input the required data correctly, at the right point and time in the process. Administrative staff must then dedicate time and effort to extracting and aggregating this data, as well as developing and submitting complete reports to requesting agencies in a timely manner.^3,5,6 Further complicating matters is that many quality measurement standards are still in their infancy or adolescence. Such variation in maturity results in considerable complexity for providers, especially when managing requirements between different value-based and quality management programs.³

A potential resolution, which is garnering significant interest, is to work toward automation of the calculation and reporting of quality measures.³ Automation of quality measurement and reporting is based on the idea that once clinical data is collected and stored electronically (digitally) in an EHR, automated procedures—such as data pipelines—can be developed. Since 2009, the proliferation of certified EHRs in the U.S. has significantly increased due to incentives offered by the Meaningful Use program¹⁷ and the 21^st Century Cures Act.¹⁸ Through a combination of incentives (“carrots”) and penalties (“sticks”), these programs have resulted in a remarkable increase in infrastructure for collecting and sharing health information, particularly related to patient health records.

As a byproduct of digitizing health information, this EHR and interoperability infrastructure can now also be used by health systems to digitally track quality. For instance, it is now possible to objectively determine when evidence-based guidelines were or were not followed for processes where clinical process information is entered into an EHR. If such information is collected and available to extract, automated procedures and pipelines can be designed to extract and aggregate relevant data and even automatically calculate quality measurement attainment levels as well as electronically submit the results.⁴

Given that much of the foundational technology is already in place and the incentives to reduce burdens are high, one might assume that eCQM automation should be relatively straightforward, even if initial setup costs are high. This assumption makes intuitive sense, as many other industries have seen significant productivity and quality benefits from automation. Interestingly, though, eCQM automation has proven difficult and, while progress is being made, significant barriers remain.^5,7 For instance, studies conducted in ambulatory settings have found that EHRs often must be retrofitted or paired with third-party software to meet quality measurement requirements and such efforts are not always successful.⁷

U.S. Hospital eCQM Automation Levels and Barriers to Automation

Interestingly, while use of health information technology (health IT) to support quality measurement and reporting has been discussed and gradually implemented for the past 10 years or so,^10,19 extant research is scant on U.S. hospital eCQM automation levels and associated barriers to increasing eCQM automation, especially at the national level. Thus, the purpose of this article is to evaluate the current levels of eCQM automation in U.S. hospitals, the characteristics of U.S. hospitals at each level of eCQM automation, and to evaluate barriers to eCQM automation. To achieve this goal, we evaluate U.S. hospital responses to the American Hospital Association (AHA) Annual Information Technology (IT) Supplement Survey from 2017 and U.S. hospital characteristics from the AHA Annual Survey from 2016 (n=3,513 U.S. hospitals).

We specifically focus on U.S. hospital responses to two questions: 1) To what degree does your hospital use automated, EHR generated measures (versus manual processes such as chart abstraction) for each of the following programs (Medicare Inpatient Quality Reporting, Physician-Specific eCQMs, Hospital-Specific eCQMs)?, and 2) What barriers—if any—has your hospital experienced in the transition from manually to fully or partially automated reporting? These questions were newly added to the 2017 version of the AHA IT Supplement Survey, which means that 2017 is the first year in which this data could be analyzed. Our results are summarized in Table 1 and Figure 1, and are further described in the following paragraphs.

First, we find that a majority (86.34%) of U.S. hospitals responding to this survey report engaging in either full or partial automation of eCQMs in 2017 for at least one of the three programs asked about in the survey (Table 1). The percentage of hospitals engaging in full or partial automation for all three programs is slightly lower, at 72.23%. The percentage of hospitals engaging in the maximum level of automation, full automation for all three programs, is significantly lower at 28.87%.

These results illustrate that at least partial eCQM automation is in place at a large majority of U.S. hospitals, which is promising, but also that full automation is currently lacking. Less than one-third of U.S. hospitals report full eCQM automation for all three programs asked about in the survey.

Further, as shown in the descriptive characteristics in Table 1, the U.S. hospitals that report full (no partial) automation for all three programs are: significantly larger (more discharges), more likely to be system owned, more likely to engage in automation in slightly less competitive markets, and are more likely to have engaged in other value-based programs including Accountable Care Organizations (ACOs) and Patient Centered Medical Home (PCMH) programs.[1] The implications of these results are that economies-of-scale, reduced market pressures, and prior experience with other value-based programs enhance the likelihood of automating eCQMs. These findings also suggest that smaller, less resource- and technology- intensive U.S. hospitals will likely face significant barriers to reaching full automation in all eCQM programs.

Second, we evaluated the barriers to eCQM automation reported by responding U.S. hospitals in tandem with their reported eCQM automation level (Figure 1). Overall, we find that the top barriers to automation are: 1) problems with clinical workflow leading to missing data or incorrect information being collected, and 2) EHR data not mapping correctly to eCQM measures, leading to missing or inaccurate information. For U.S. hospitals reporting no automation (or do not know for all three categories), we find that resources and data issues associated with EHRs dominate, as the top three barriers for hospitals with no eCQM automation are: 1) Difficulty extracting data from EHRs, 2) EHR does not possess capability to automatically generate measures, and 3) Lack of IT staff needed to generate reports.

In our view, underlying these barriers is: 1) the complexity involved in extracting data from very complex underlying databases, which then either requires technical staff very knowledgeable in the data structures from which the data must be extracted from or reliance on the vendor to extract the data, and 2) EHR vendors trying to create CQM pathways (for monitoring and reporting) and automation on their own, rather than fully enabling others to do so through APIs. We return to this point in our next section.

For U.S. hospitals reporting anywhere from at least some automation for at least one program to full automation for all three programs, the primary barriers were mixed between EHRs and clinical workflow challenges. In particular, these hospitals reported EHR data not mapping correctly and problems with clinical workflow as their primary barriers to eCQM automation. These barriers are followed closely by difficulty of extracting data from the EHR and poor EHR usability or design issues, leading to missing or inaccurate information.

Interestingly, for the U.S. hospitals that report full (no partial) automation for all three programs, automated generation of quality measures by the EHR and poor EHR data quality are the barriers with the lowest reported percentages. Further, follow-up analyses of the effect of EHR vendor choice on whether or not such barriers are more likely to be reported found that some EHR vendors, particularly the smallest and lowest market-share EHR vendors, are significantly more likely to be associated with higher reported quality measure generation and data extraction barriers.[2] In sum, primary barriers include: collecting data within an EHR at the right point in the process and in the right format, as well as mapping collected data to quality measurement calculations and EHR choice (or ability to work with your EHR vendor to customize eCQM automation).

These findings primarily imply that U.S. hospital EHRs, while vital to eCQM automation, are also currently standing in the way of automation, particularly in that needed data is often difficult to effectively collect and extract. In combination with the results described above from Table 1, we can also surmise that economies-of-scale, that often are a benefit of larger hospital size and system ownership, likely help to distribute eCQM automation effort and costs among hospitals within systems and units within larger hospitals, and thus reduce eCQM automation burdens.

Looking to the Future: Considerations for Overcoming Automation Barriers

While much of the current health policy debate is focused on interoperability, coordinating care for patients across the continuum, cost containment, and, of course, improved quality,^20-23 we argue that equally as important is enabling automation of eCQMs (passive voice). While closely-related efforts, such as standardization of quality measures, will help to enable automation of eCQMs,⁷ we have yet to see a concerted policy effort focused specifically on reducing barriers to eCQM automation. As discussed by Mandl and Kohane in their excellent article on how to enable the next generation of innovation in health IT,¹⁸ the information within EHRs is the true source of value and the potential value of these data must be unlocked. However, we find, ironically, that EHRs are a primary impediment to attaining full eCQM automation.

To overcome these barriers, we suggest that significant policy efforts be focused on two areas: 1) better alignment of EHR structured data collection and extraction of quality measure data with quality measure requirements, particularly associated with making it clear to end-users within the EHR user-interfaces where and how data must be entered if it is to be included in required quality measures, and 2) developing methods, possibly including NLP and ML methods, to effectively extract quality measurement data from unstructured and semi-structured data formats, such as clinical notes, diagnostic notes, and even images, in the future.

Furthermore, artificial intelligence (AI) such as deep neural network methods can take advantage of these data to generate various inferences and recommendations that help health professionals make decisions more effectively and efficiently. As an example, by combining the medical history of different patients with clinician’s notes and images, an AI system can provide recommendations that may potentially mitigate near-future health complications of a patient or even suggest more effective medicines or procedures for the patient.

In regard to our first point focused on alignment of data (and data collection methods and architectures) with quality measures, we believe that such policy efforts should be focused directly toward EHR vendors, rather than only indirectly on data requirements imposed upon hospitals. Some efforts in this regard are already underway, such as the inclusion of specific interoperability requirements in EHR certification processes,²⁴ which will indirectly impact eCQM automation, as well as technical efforts such as those associated with clinical quality language (CQL),[3] the health quality measure format (HQMF),[4] and use of qualified clinical data registries (QCDR).[5] We nevertheless suggest that more could be done in this area.

Consider the following example. Data for one quality measure can be collected from multiple places within an EHR, such as deciding whether to obtain diagnosis data from the problem list, diagnoses directly entered into charts, or even diagnoses received from other healthcare facilities. Complexity for providers would be significantly reduced if EHR vendors’ data structures were coordinated with quality measurement and reporting requirements, as well as with the providers who need to extract needed data. Further, once such requirements are clarified and coordinated, if EHR user-interfaces more clearly identify how or why certain clinical data is needed in specific places (e.g., make it clear that if a chronic diagnosis is not included in the problem list, corresponding quality measures will be missed or potentially reported as not met for such patients[6]), integration of clinical workflows and quality automation processes would likely be significantly improved.

In regard to our second point focused on addressing the challenges of semi- and unstructured data, we note that structuring data for every quality measure imposes significant burdens on providers (and even EHR vendors), such as when adding checkboxes or picklists for all data elements that must be included in quality measures (e.g., a checkbox that the provider must check if smoking cessation counseling was provided for patients who report tobacco usage[7]). Such burdens would be significantly reduced if needed information could be extracted from unstructured or semi-structured notes, documents, and images. One solution to this approach would be to again ask or require EHR vendors to make investments into unstructured data such as NLP and images as well as ML and AI methods to find and extract such data.

Many advances are being made in unstructured data and ML and AI that greatly increase the efficiency and efficacy of these methods, but we note that these methods are evolving rapidly, and these areas are not the traditional core competencies of EHR vendors. Further, much of the focus of NLP, ML, and AI by EHR vendors and healthcare providers has been on predictive algorithms, such as for predicting patient health and cost risk.^25,26 While investments into predicting risk certainly overlap with eCQM efforts, interestingly, we have not seen a significant increase in eCQM automation as a result of these efforts. Thus, another solution, rather than only pushing EHR vendors (and healthcare providers) to incorporate new ways to automate eCQM, is to also incentivize entrepreneurs and third-parties to develop innovative NLP/Images and ML/AI applications that can be built on top of existing EHRs.^27-29

In other words, promoting smaller scale or more focused NLP and ML innovations³⁰ via market-based incentives and API use,¹⁸ versus only healthcare organization- or EHR vendor-based regulations, may more quickly and effectively result in solutions that can eventually scale to meet broad quality measurement automation needs. This is especially important as future quality measures are likely to require even more granular measurement through analysis of different types of data, including potentially even image data.

Further, if APIs are standardized, as advocated for by the Substitutable Medical Applications and Reusable Technologies (SMART) initiative³¹ and health standards such as Fast Health Interoperability Resources (FHIR) support bulk data access,[8] such innovative apps would potentially be scalable across many or even all EHRs, irrespective of vendor or underlying proprietary data structures.³² Even further, frictionless health information flows that enable automation of eCQM could also then be leveraged to enhance disease surveillance, deep learning, and advance the goals of a learning health system.^33-35

Conclusion

In sum, we find that eCQM automation is progressing, but significant barriers remain. Given that healthcare providers who can take advantage of economies-of-scale are more likely to fully automate eCQM, we provide policy ideas for increasing the likelihood of all U.S. hospitals reaching full eCQM automation, rather than just the largest and resource-rich. In particular, we think that policymakers should work more closely with EHR vendors to align data collection interfaces and extraction procedures with eCQM requirements. We also suggest that policymakers incentivize entrepreneurs to innovatively solve unstructured and semi-structured data challenges associated with eCQM measurement and reporting.

References

1. Lohr KN, Schroeder SA. A strategy for quality assurance in Medicare. New England Journal of Medicine. 1990;322(10):707-712.
2. Campbell SM, Roland MO, Buetow SA. Defining quality of care. Social Science & Medicine. 2000;51(11):1611-1625.
3. Hayford TB, Maeda JL. Issues and Challenges in Measuring and Improving the Quality of Health Care. Congressional Budget Office;2017.
4. CMS. Clinical Quality Measures Basics. 2019; https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/ClinicalQualityMeasures. Accessed 1/10/2020, 2020.
5. McConnell RA, Kane SV. The Potential and Pitfalls of Using the Electronic Health Record to Measure Quality. The American Journal of Gastroenterology. 2018;113(8):1111-1113.
6. Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. Journal of the American Medical Informatics Association. 2007;14(1):10-15.
7. Ahmad FS, Rasmussen LV, Persell SD, et al. Challenges to electronic clinical quality measurement using third-party platforms in primary care practices: the healthy hearts in the heartland experience. JAMIA Open. 2019;ooz038:1-6.
8. Casalino LP, Gans D, Weber R, et al. US physician practices spend more than $15.4 billion annually to report quality measures. Health Affairs. 2016;35(3):401-406.
9. McGlynn EA, Adams JL, Kerr EA. The quest to improve quality: measurement is necessary but not sufficient. JAMA Internal Medicine. 2016;176(12):1790-1791.
10. Conway PH, Mostashari F, Clancy C. The future of quality measurement for improvement and accountability. Journal of the American Medical Association. 2013;309(21):2215-2216.
11. Kaplan RS, Porter ME. Mandate Outcomes Reporting. Health Managment Policy and Innovation. 2019;4(3):1-7.
12. Liu X, Schulman K, Scheinker D. Private and Public Incentives for Hospitals to Improve the Quality and Reduce the Cost of Care. Health Managment Policy and Innovation.4(3):1-12.
13. Eisenberg F, Lasome C, Advani A, Martins R, Craig P, Sprenger S. A study of the impact of meaningful use clinical quality measures. American Hospital Association;2013.
14. Scales CD, Schulman KA. Triggering management for quality improvement. Health Services Research. 2014;49(5):1401-1406.
15. Leape LL, Berwick DM. Five years after To Err Is Human: what have we learned? JAMA. 2005;293(19):2384-2390.
16. Casalino LP. Pioneer Accountable Care Organizations: Traversing Rough Country. Journal of the American Medical Association. 2015;313(21):2126-2127.
17. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. New England Journal of Medicine. 2010;363(6):501-504.
18. Mandl KD, Kohane IS. A 21st-Century Health IT System-creating a real-world information economy. N Engl J Med. 2017;376(20):1905-1907.
19. Anderson KM, Marsh CA, Flemming AC, Isenstein H, Reynolds J. An Environmental Snapshot: Quality Measurement Enabled by Health IT: Overview, Possibilities and Challenges. Agency for Healthcare Research and Quality;2012.
20. Adler-Milstein J, Embi PJ, Middleton B, Sarkar IN, Smith J. Crossing the health IT chasm: considerations and policy recommendations to overcome current challenges and enable value-based care. Journal of the American Medical Informatics Association. 2017;24(5):1036-1043.
21. McWilliams JM. Cost Containment and the Convenient Tale of Care Coordination. The New England Journal of Medicine. 2016;375(23):2218.
22. Porter ME. A Strategy For Health Care Reform—Toward A Value-Based System. New England Journal of Medicine. 2009;361(2):109-112.
23. Porter ME. What Is Value In Health Care? New England Journal of Medicine. 2010;363(26):2477-2481.
24. CMS. Medicare Program; Hospital Inpatient Prospective Payment Systems for Acute Care Hospitals and the Long-Term Care Hospital Prospective Payment System and Policy Changes and Fiscal Year 2019 Rates; Quality Reporting Requirements for Specific Providers; Medicare and Medicaid Electronic Health Record (EHR) Incentive Programs (Promoting Interoperability Programs) Requirements for Eligible Hospitals, Critical Access Hospitals, and Eligible Professionals; Medicare Cost Reporting Requirements; and Physician Certification and Recertification of Claims. Final rule. Federal Register. 2018;83(160):41144.
25. Lin YK, Chen HC, Brown RA, Li SH, Yang HJ. Healthcare Predictive Analytics For Risk Profiling In Chronic Care: A Bayesian Multitask Learning Approach. MIS Quarterly. 2017;41(2):473-500.
26. Bardhan I, Oh J-h, Zheng Z, Kirksey K. Predictive Analytics For Readmission Of Patients With Congestive Heart Failure. Information Systems Research. 2014;26(1):19-39.
27. Khalilia M, Choi M, Henderson A, Iyengar S, Braunstein M, Sun J. Clinical predictive modeling development and deployment through FHIR web services. Paper presented at: AMIA Annual Symposium Proceedings2015; San Francisco.
28. Braunstein ML. Health Care in the Age of Interoperability: The Potential and Challenges: Part 1. In. IEEE Pulse: IEEE; 2018.
29. Braunstein ML. Health Informatics on FHIR: How HL7’s New API is Transforming Healthcare. Switzerland: Springer; 2018.
30. Herzlinger RE, Wiske C. Disseminating and Diffusing Internal Innovations: Lessons from Large Innovative Healthcare Organizations. Health Management Policy and Innovation. 2017;2(2):1-8.
31. Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. Journal of the American Medical Informatics Association. 2016;23(5):899-908.
32. Braunstein ML. Healthcare in the Age of Interoperability: Part 3. IEEE pulse. 2019;10(1):26-29.
33. Etheredge LM. A rapid-learning health system: what would a rapid-learning health system look like, and how might we get there? Health Affairs. 2007;26(Suppl1):w107-w118.
34. Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. Journal of the American Medical Informatics Association. 2015;22(1):43-50.
35. Norgeot B, Glicksberg BS, Butte AJ. A call for deep-learning healthcare. Nature Medicine. 2019;25(1):14-15.

[a] These variables were determined to be significant at P<0.05 via a logistic regression with full eCQM automation as the dependent variable, the characteristics described in Table 1 as the independent variables (lagged by one year to address issues with contemporaneous effects), and further controlling for physician intensity, teaching hospital, and region (West, Midwest, South, and Northeast).

[b] These analyses were conducted with logistic regressions with the reported eCQM barrier as the dependent variable, the reported EHR vendors as indicator variables (with the highest frequency EHR vendor reported as the base variable), and controlling for all characteristics described in Table 1 (lagged by one year) as well as physician intensity, teaching hospital, and region (West, Midwest, South, and Northeast).

[c] For more information on CQL, please see https://cql.hl7.org/01-introduction.html.

[d] For more information on HQMF, please see https://ecqi.healthit.gov/hqmf.

[e] For more information on QCDR, please see https://qpp.cms.gov/mips/quality-measures.

[f] For example: https://www.ahrq.gov/pqmp/measures/management-of-chronic-conditions.html

[g] For more information on this quality measure, please see https://qpp.cms.gov/docs/QPP_quality_measure_specifications/Claims-Registry-Measures/2019_Measure_226_MedicarePartBClaims.pdf.

[h] See FHIR Bulk Data Access (Flat FHIR) at https://hl7.org/fhir/uv/bulkdata/ for more details.