Name
305: Building a Scalable Historical Data Layer: Tracking Changes Over Time with SCDs
Date & Time
Wednesday, October 21, 2026, 1:30 PM - 2:15 PM
Description

Has your leadership ever asked a question such as, "How has this metric changed over time, and what drove those changes?" Answering questions like these requires more than a powerful BI tool. It requires a reliable way to preserve and understand historical data.
In many data ecosystems, operational systems are designed to reflect the current state of the business rather than maintain a complete history of how that state has evolved. As a result, teams often rely on daily snapshots or repeated comparisons to reconstruct historical changes, leading to increased data volumes, complex logic, and difficult-to-maintain pipelines.


This session explores a practical approach to solving this challenge using Slowly Changing Dimension (SCD) principles. By capturing meaningful changes as historical records with effective dates and versioning, utilities can maintain a scalable, query-friendly history without duplicating entire datasets each day. The session will cover key architectural concepts, data modeling patterns, and practical considerations for implementing an SCD Type 2 approach. It will also demonstrate how this methodology enables more efficient, reliable, and scalable historical analysis for downstream analytics and business intelligence.


Session Takeaways:
•        How to build scalable historical data models using SCD Type 2 principles 
•        Practical approaches for supporting reliable point-in-time reporting and trend analysis 
•        Best practices for simplifying data pipelines while improving data quality and accessibility 
•        Techniques for efficiently querying historical data for dashboards and reporting 
•        Lessons learned implementing scalable historical data frameworks in a utility environment 

Jyothika Mohan