Why ETL is a Poor Approach for Modern Healthcare Data Unification

Introduction

This quote comes to mind when considering the current state of ETL-powered healthcare data integration approaches.

ETL (an acronym for Extract, Transform, Load) is a concept that originated in the 1970s to support the creation of centralized data repositories, so this truly is your father’s (maybe grandfather’s) data management approach. ETL grew in popularity through the 1980s and 1990s as the preferred approach to aggregate data from function-specific business applications into consolidated data warehouses. Along the way, ETL became popular as a preferred approach for performing data migrations.

ETL’s novelty was that you could improve the clarity, traceability, and maintainability of data movement processes by isolating the source system extract logic from the logic to transform source data into a format that can be accurately loaded into a target system.

ETL also utilizes a staging system, which holds a copy of raw source data and a separate copy of the transformed data. The load step pulls data from a staging area to populate the target system (most frequently as a data warehouse.)

The Limitations of ETL

ETL was a great innovation then; however, it has some limitations that prevent it from being the best choice for managing modern healthcare data unification requirements.

• ETL is inherently siloed
  • ETL was originally seen as a silo-buster because it aggregated data from functional applications into a single data warehouse.
  • Unfortunately, data warehouses have just become their own massive silo because a data warehouse cannot meet all the data integration needs of modern healthcare organizations.
  • ETL creates even more silos when its usage expands beyond data warehouse solutions (i.e., member rosters, provider rosters, regulatory reporting, vendor reconciliation).
• Using ETL as a data integration approach introduces variability in your business processes
  • Most ETL implementations are custom code. This means the effectiveness and performance of ETL code varies based on the person implementing the code.
  • ETL-based integration also produces blind spots in your business process where data is transformed in a way that reduces traceability from the original data source.
  • With an ETL-based integration approach, it is extremely complicated to accurately track the journey from an original Member enrollment transaction to creation, benefits administration, benefit eligibility requests, and ultimately, claim payment and claim status inquiry.
  • Your data is passing through such a complex patchwork of transformations that there is no definitive point of reference (i.e., source of truth) to understand the member’s experience.
• ETL creates separation between business and IT teams
  • Let’s face it. ETL is a tool built by technologists for technologists. Its original use case of getting data into a data warehouse gave business users visibility by comparing original source application data to the transformed data in the warehouse. This is well within the point-to-point nature of an ETL-based process.
  • Once you go beyond a data warehouse silo, ETL operations become very opaque, and business users face significant struggles to understand what is happening when things go wrong.
  • This dynamic frequently causes business users to settle for non-optimal workarounds to address issues inherent to the limitations of ETL-based integration approaches.
    
    

But wait – what about ELT?

One quick note for those who may say that the answer is to move from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform). ELT represents a change in process, but it doesn’t deliver the required change in paradigm. ELT uses modern cloud-based architecture and infrastructure to load your raw source data into massive cloud repositories, sometimes called data lakes. The transformation is then performed in the cloud based on your unique target system requirements. Without going into a lot of detail, I’ll summarize that ELT is a more modern way to leverage cloud computing power to create and manage your data warehouse, but it suffers the same limitations as ETL when it comes to meeting the modern data integration requirements of a healthcare organization.

Why a paradigm shift is required to meet the modern needs of healthcare organizations

The complex and distributed nature of the healthcare ecosystem has made it difficult to create end-to-end business application suites (i.e., ERP – Enterprise Resource Planning applications) that enable healthcare organizations to interoperate seamlessly.

Healthcare organizations have disparate applications across broker management, member enrollment, provider credentialing, provider data maintenance, claims adjudication, and claims payment systems, to name a few of the core business processes with integration limitations. Furthermore, it is common for large healthcare organizations to have two or three different applications for a single business function due to mergers, acquisitions, or unique line-of-business requirements. Healthcare organizations are aware of these challenges and are seeking solutions, but navigating out of these challenges while continuing to run your business is daunting.

To solve these challenges, healthcare organizations must manage their overall business in an integrated manner, even though specific business functions or segments have unique operating criteria. They need to cut through the confusion to get clarity in their decisions and consistency in their stakeholder (i.e., brokers, patients, members, providers) experiences.

Solving the unique challenges of healthcare organizations begins with establishing a single integrated and actionable view of your business. The actionable portion of this requirement is the most difficult and most valuable part of this requirement. Data warehouses create an integrated view, but you need a more flexible solution to discover and act on insights in real time. If you discover that a member has been inaccurately created with multiple member profiles and each of those profiles is enrolled in different benefit coverages, you need to be able to act on that immediately. A failure to act promptly may result in a member showing up at a doctor’s office and being denied coverage due to the inability to confirm the correct current benefit plan.

Requirements for Modern Healthcare Data Management

Modern healthcare businesses require the ability to unify data across all business processes and dynamically update applications and individuals across the ecosystem in response to real-time changes. Modern healthcare businesses also require the ability to collaborate effectively across process and organizational boundaries using a unified view of business data that accurately integrates unique business segment rules into a common business view. That rich repository of business rules must be managed centrally to enable a real-time flow of information into disparate applications that play diverse roles across your ecosystem.

This means your centralized data platform must not only be able to capture data from multiple sources across your application landscape, but it must also have the intelligence to harness both application-specific and application-agnostic business rules tailored to your unique business processes. You must also determine which target applications and people must be updated in response to any new/updated data received from source applications.

Conclusion

The patchwork nature of isolated, custom ETL logic can’t support this level of business sophistication, so to meet the needs and expectations of a modern healthcare organization, you need to shift your paradigm.

Modern healthcare data unification strategies must simultaneously unify, integrate, and distribute data in real time. The right platform will be able to enhance your existing data warehouse needs but must go beyond the data warehouse paradigm to create a dynamic platform that curates and feeds your extended healthcare ecosystem with accurate data in real-time.