SAP Datasphere is the next generation of SAP Data Warehouse Cloud and delivers a unified experience for data integration, data cataloguing, semantic modelling, data warehousing, data federation, and data virtualization. Data Integration is an essential topic in a Business Data Fabric like SAP Datasphere and Replication Flow is the cornerstone to fuel SAP Datasphere with data, especially from SAP ABAP sources.
Based on the different data sources for a Replication Flow, you have different options for replicating the data in SAP Datasphere. If your source system is an SAP S/4HANA system offering the needed data by means of CDS views, directly connecting SAP Data Intelligence to this source system and using the respective CDS views is the best approach.
In this blog, we will try to delve into the details of CDS configuration to make it Delta Load compatible for a Replication Flow based extraction:
In Replication Flow settings, we have below load types available:
Initial – If you select this option, full data retrieved by the CDS view will be transferred to target every time the Replication Flow is executed. Initial and Delta – If you select this option, full data retrieved by the CDS view will be transferred to target for the first run, post that at scheduled intervals the Replication Flow will transfer delta records. For Initial and Delta setting, RF uses CDC delta.
The ODP framework for CDS extraction provides two options for delta handling:
Generic Timestamp / Date based Delta (Not supported by SAP Datasphere 2024.23 Replication Flow )Change Data Capture (CDC) Delta
Starting with SAP S/4HANA Cloud 1905 and SAP S/4HANA 1909 FPS01 (on-premise) additional delta capability of Change Data Capture is available. This delta option captures changes to the application tables based on data base triggers and enables the ODP framework to just provision these newly created/changed/deleted records to the consumers.
CDS Annotations for enabling Replication into Datasphere
A. Annotation for Data Extraction
To make the data from an ABAP CDS view available in SAP Data Intelligence Cloud(Replication engine), you need to specify annotations for the view in the ABAP Development Tool.
To load data(initial only or initial with delta) from a custom CDS view, add the following annotation to the view:
Analytics.dataExtraction.enabled – This annotation exposes the CDS view for Consumption by a Replication Flow. For both Initial and Initial with Delta Load types this annotation is mandatory.
B. Annotation for CDC in simple CDS view
As mentioned before, the Change Data Capture recording mechanism uses database triggers to record any changes to the tables that belong to an ABAP CDS view. To do this, the key fields of all underlying tables need to be mapped to the fields of the CDS view.
For simple CDS views such as CDS projection views, this can be done automatically by using the following annotation:
Analytics.dataExtraction.delta.changeDataCapture.automatic – This annotation will enable the use of CDC (change data capture) delta mechanism based on data base triggers. Manual key mapping in ADT is not required as we are using a very simple CDS projection view. In case of automatic mapping system relies on the key field of the main table used in CDS.
Let’s take an example of a standard view and see how it works :
CDS view : I_SUPPLIER
Although this CDS view seems complex, all data retrieval is through association(no joins used). The main table here is LFA1. Hence CDC(delta capture) only needs to track changes on LFA1, so the simple annotation is used for automatic CDC.
C. Annotation for CDC in complex CDS view
For more complex CDS views that use joins, instead of using CDC automatic annotation we do the table key mapping manually in CDS. You will be using the following annotation to do this:
Analytics.dataExtraction.delta.changeDataCapture.mapping
What is table key mapping? During delta capture, SAP stores the delta entries in internal logging tables and the key fields of these logging tables will be decided based on the key fields of the main tables(joins) used in CDS.
Let’s take an example of a standard view and see how it works :
CDS view : I_WBSELEMENTDATA
Screenshot I :
Screenshot II :
As per Screenshot I, we have extraction enabled, CDC mapping. Let’s have a look at the CDS design in Screenshot II to understand the mapping.
Using this mapping mentioned in the annotation, Change Data Capture framework performs logging of the delta records.
Once CDS view is created, it can be used in Replication Flow to populate data in SAP Datasphere.
SAP Datasphere has a link with the prerequisite SAP notes https://help.sap.com/docs/SUPPORT_CONTENT/datasphere/4457678727.html?q=pre%20requisite
Hope you find this blog helpful! I will write a subsequent blog to elaborate the CDC logging in more details.
SAP Datasphere is the next generation of SAP Data Warehouse Cloud and delivers a unified experience for data integration, data cataloguing, semantic modelling, data warehousing, data federation, and data virtualization. Data Integration is an essential topic in a Business Data Fabric like SAP Datasphere and Replication Flow is the cornerstone to fuel SAP Datasphere with data, especially from SAP ABAP sources.Based on the different data sources for a Replication Flow, you have different options for replicating the data in SAP Datasphere. If your source system is an SAP S/4HANA system offering the needed data by means of CDS views, directly connecting SAP Data Intelligence to this source system and using the respective CDS views is the best approach.In this blog, we will try to delve into the details of CDS configuration to make it Delta Load compatible for a Replication Flow based extraction:In Replication Flow settings, we have below load types available:Initial – If you select this option, full data retrieved by the CDS view will be transferred to target every time the Replication Flow is executed. Initial and Delta – If you select this option, full data retrieved by the CDS view will be transferred to target for the first run, post that at scheduled intervals the Replication Flow will transfer delta records. For Initial and Delta setting, RF uses CDC delta.The ODP framework for CDS extraction provides two options for delta handling:Generic Timestamp / Date based Delta (Not supported by SAP Datasphere 2024.23 Replication Flow )Change Data Capture (CDC) DeltaStarting with SAP S/4HANA Cloud 1905 and SAP S/4HANA 1909 FPS01 (on-premise) additional delta capability of Change Data Capture is available. This delta option captures changes to the application tables based on data base triggers and enables the ODP framework to just provision these newly created/changed/deleted records to the consumers.CDS Annotations for enabling Replication into Datasphere A. Annotation for Data ExtractionTo make the data from an ABAP CDS view available in SAP Data Intelligence Cloud(Replication engine), you need to specify annotations for the view in the ABAP Development Tool.To load data(initial only or initial with delta) from a custom CDS view, add the following annotation to the view:Analytics.dataExtraction.enabled – This annotation exposes the CDS view for Consumption by a Replication Flow. For both Initial and Initial with Delta Load types this annotation is mandatory.B. Annotation for CDC in simple CDS viewAs mentioned before, the Change Data Capture recording mechanism uses database triggers to record any changes to the tables that belong to an ABAP CDS view. To do this, the key fields of all underlying tables need to be mapped to the fields of the CDS view.For simple CDS views such as CDS projection views, this can be done automatically by using the following annotation:Analytics.dataExtraction.delta.changeDataCapture.automatic – This annotation will enable the use of CDC (change data capture) delta mechanism based on data base triggers. Manual key mapping in ADT is not required as we are using a very simple CDS projection view. In case of automatic mapping system relies on the key field of the main table used in CDS.Let’s take an example of a standard view and see how it works : CDS view : I_SUPPLIERAlthough this CDS view seems complex, all data retrieval is through association(no joins used). The main table here is LFA1. Hence CDC(delta capture) only needs to track changes on LFA1, so the simple annotation is used for automatic CDC.C. Annotation for CDC in complex CDS viewFor more complex CDS views that use joins, instead of using CDC automatic annotation we do the table key mapping manually in CDS. You will be using the following annotation to do this:Analytics.dataExtraction.delta.changeDataCapture.mappingWhat is table key mapping? During delta capture, SAP stores the delta entries in internal logging tables and the key fields of these logging tables will be decided based on the key fields of the main tables(joins) used in CDS.Let’s take an example of a standard view and see how it works : CDS view : I_WBSELEMENTDATAScreenshot I : Screenshot II :As per Screenshot I, we have extraction enabled, CDC mapping. Let’s have a look at the CDS design in Screenshot II to understand the mapping.Using this mapping mentioned in the annotation, Change Data Capture framework performs logging of the delta records.Once CDS view is created, it can be used in Replication Flow to populate data in SAP Datasphere.SAP Datasphere has a link with the prerequisite SAP notes https://help.sap.com/docs/SUPPORT_CONTENT/datasphere/4457678727.html?q=pre%20requisiteHope you find this blog helpful! I will write a subsequent blog to elaborate the CDC logging in more details. Read More Technology Blogs by Members articles
#SAP
#SAPTechnologyblog
