Metadata Driven Design

SCDP has a metadata-driven architecture. Data describing entities, their attributes, and relationships form the core of the metadata. Additionally, metadata about cloud resources and configurations provides end users with an unparalleled degree of configurability. With this, users can define desired data operations on entities and control how operations are performed on each entity.

The Data Journey in SCDP

Entities within the SCDP solution undergo a transformative process, referred to as their "data journey." This journey entails a sequence of operations on the entity's data. At each stage, the data assumes a distinct state and has undergone specific operations. Data operations are designed as standalone components, akin to building blocks, allowing end users the flexibility to architect various data pipelines.

Flexibility and Modularity

This approach not only prioritizes flexibility and simplicity but also enables users to recycle these building blocks in diverse combinations. It minimizes the impact on other segments of the pipeline when a particular building block is replaced.

Unmatched Configurability And Change Managamenet

Defining data pipelines as collections of data operations within the metadata offers remarkable flexibility. It allows for easy modifications to data pipelines and entity data journeys by simply changing the metadata definition. Unlike traditional change management, which relies on DevOps tools and techniques, this approach avoids the need for elaborate deployment processes, offering a practical and efficient alternative.

Full protection for privacy sensitive data

An effective data platform must possess the capability to:

• Ingest and process various data types, including sensitive information.
• Restrict access to sensitive information only to authorized users.

SCDP excels in this aspect by implementing the following:

• Encrypting sensitive data during extraction from the source system.
• Employing this encrypted data across all data products.
• Incorporating a unique mechanism for on-the-fly decryption.

This ensures that solely privileged users with the appropriate access rights can view the decrypted data.

Comprehensive and clear status report

SCDP achieves transparency and clarity in reporting by focusing on:

• Employing data pipelines composed of a series of data operations on entities.
• Systematically recording comprehensive metrics for these operations in an Azure SQL database.

With these mechanisms in place, SCDP is able to provide:

• An array of reports that showcase the current status of each entity.
• Invaluable insights to both data engineers and end users.

Efficient Error Handling and Error Recovery

The Inevitability of Errors

No solution is perfect; occasional failures are inevitable, especially when relying on source systems and their data.

Necessity for Robust Mechanisms

In the intricate landscape of data platforms, various issues can arise, emphasizing the need for:

• Robust error-handling procedures
• Efficient recovery mechanisms

SCDP's Approach

SCDP addresses these challenges by:

• Delivering transparent status reports for each entity
• Providing data engineers with efficient error recovery procedures
• Minimizing redundant work to save both time and financial resources