SAP Business Data Cloud Series – Part 3: Customer-Managed Data Products

Introduction:

In Part 1(SAP Business Data Cloud Series – Part 1: Introduction to Data Products) of this blog series, we explored the fundamentals of Data Products in SAP Business Data Cloud (BDC). In Part 2(SAP Business Data Cloud Series – Part 2: Extend SAP S/4HANA Managed Data Products) we explored on how to extend SAP S/4HANA Data Products. Now, in Part 3, we dive into the practical aspect of creating custom data products for the source SAP S/4HANA Cloud Private Edition and Share it with SAP databricks.

In this article, we’ll explore a common scenario where a customer has a custom CDS entity in SAP S/4HANA and wants to harmonize its data with SAP-managed data products. We’ll demonstrate how to create a custom data product based on that custom CDS entity, for example, ZCDS_CAMPAIGN and how to delta-share with SAP databricks.

High-level Workflow for the SAP S/4HANA, Customer Managed Data Product:

Steps to create a SAP S/4HANA Customer Managed Data Product.

1. Identify, create, or extend a CDS entity: To build a customer-managed data product in an SAP S/4HANA system, the recommended approach is to leverage CDS views/entities. In this example, we’ll create a custom CDS entity as shown below.

@ObjectModel.usageType.dataClass: #TRANSACTIONAL
@ObjectModel.usageType.serviceQuality: #A
@ObjectModel.usageType.sizeCategory: #XL
@ObjectModel.representativeKey: ‘CampaignID’
@ObjectModel.modelingPattern: #ANALYTICAL_DIMENSION
@ObjectModel.supportedCapabilities: [ #ANALYTICAL_DIMENSION,
#CDS_MODELING_DATA_SOURCE,
#CDS_MODELING_ASSOCIATION_TARGET,
#SQL_DATA_SOURCE,
#EXTRACTION_DATA_SOURCE ]

@ObjectModel.sapObjectNodeType.name: ‘CampaignID’
@EndUserText.label: ‘CDS View for Campaigning Data’
@Analytics.dataCategory: #DIMENSION
@Analytics.internalName: #LOCAL
@Analytics: {
dataExtraction: {
enabled: true,
delta.changeDataCapture: {
mapping:[
{
table: ‘zcampaign’, role: #MAIN,
viewElement: [‘CampaignID’],
tableElement: [‘campaign_id’]
}
]
}
}
}

@VDM.viewType: #BASIC
@Metadata.allowExtensions: true
@Metadata.ignorePropagatedAnnotations: true

define view entity ZCDS_campaign as select from zcampaign
{
@EndUserText.label: ‘Campaign_ID’
key zcampaign.campaign_id as CampaignId,
@EndUserText.label: ‘Campaign_Name’
zcampaign.campaign_name as CampaignName,
@EndUserText.label: ‘Channel’
zcampaign.channel as Channel,
zcampaign.type as Type,
zcampaign.start_date as StartDate,
zcampaign.end_date as EndDate,
zcampaign.status as Status,
@Semantics.amount.currencyCode: ‘BudgetCurrency’
zcampaign.budget as Budget,
zcampaign.budget_currency as BudgetCurrency,
@Semantics.amount.currencyCode: ‘ActualSpendCurrency’
zcampaign.actual_spend as ActualSpend,
zcampaign.actual_spend_currency as ActualSpendCurrency,
zcampaign.created_by as CreatedBy,
zcampaign.created_on as CreatedOn

}

2. SAP S/4HANA Private Cloud or On-Premise Edition requires a Cloud Connector: Configure the Cloud Connector to enable communication between your on-premise/private cloud system and SAP Datasphere (public cloud). This setup is a prerequisite for creating an SAP S/4HANA or ABAP connection in SAP Datasphere

3. Create a Datasphere Space of Type HANA Data Lake Files: An administrator can create a space backed by SAP HANA Data Lake Files in the object store. File spaces are designed for cost-efficient staging, loading, and preparing large volumes of data.

For detailed instructions, refer to the help guide: Create a File Space in the Object Store.

Additional configuration is required while configuring the SAP Datasphere tenant. e.g. Memory must be 128GB or more. Check via SAP BDC Estimator.

If you have configured correctly then you should also see the same under the SAP Datasphere, Tenant configuration. Please note this view is only available to the SAP Datasphere system owner.

SAP Datasphere object store: Storage type will be SAP HANA Data Lake Files. In our scenario we have created a space with a name Workshop: OBJECT STORE.

4. Create a SAP S/4HANA Connection in the SAP Datasphere Object Store Space, Workshop: OBJECT STORE :

4.1 Create a new connection or use the existing connection of type SAP S/4HANA on-premise or ABAP:

4.2 Fill in the connection configuration details as show below and important step is select the cloud connector which was setup and configured in the step 2.

4.3 Provide the connection name and description:

4.4 Validate and test the connection making sure replication flows are enabled:

You can also use or create the SAP ABAP connection type for this use case.

5. Create a Replication Flow to Load Data from CDS Entity into SAP Datasphere Object Store (HDLF):
Set up a replication flow with the source system SAP_S4H_RI4 and CDS entity ZCDS_CAMPAIGN. Define the target as a Datasphere local table ZCDS_CAMPAIGN_BRONZE. This table, serving as the Bronze layer, will store source records and structures in their original form.

5.1 Create a new replication flow

5.2 Select the Source SAP_S4H_RI4 and Container (Source Objects) ZCDS_CAMPAIGN

5.3 Select the target as connection as a SAP Datasphere

5.4 Provide the target object local table name ZCDS_CAMPAIGN_BRONZE and Deploy the replication flow.

5.5 Execute the replication flow and check the run details.

5.6 Perform the merge table activity:

5.7 Post merge table activity, data will be available to preview and Test it.

6. Bronze to Silver layer medallion transformation using a Transformation Flow: In many cases, data from the Bronze layer needs to be refined into a Silver layer through transformations such as splitting, merging, adding/removing columns, joins, or unions.

In our scenario, we will split the values in the Channel column into two columns Channel, Allocated Percentage (multiple rows) and apply necessary calculations.

6.1 Create a new Transformation Flow

6.2 Define the transformation steps:

Source table:HDLFT_CAMPAIGN_BRONZE and Target table: HDLFT_CAMPAIGN_SILVER . Python script operator for performing the transformation logic.

def transform(data):
“””
This function body should contain all the desired transformations on incoming data of Pandas DataFrame.
Permitted builtin functions as well as permitted NumPy and Pandas objects and functions are available inside this function.
Permitted NumPy and Pandas objects and functions can be used with aliases ‘np’ and ‘pd’ respectively.
Python standard libraries like ‘time’, ‘datetime’, ‘re’, ‘random’, ‘math’, ‘calendar’ and ‘dateutil.parser’ can be used without any alias.
This function executes in a sandbox mode. Please refer the documentation for permitted objects and functions.
Using any restricted functions or objects would cause an internal exception and result in a run failure.
Any code outside this function body will not be executed and inclusion of such code is discouraged.
:param data: Pandas DataFrame
:return: Pandas DataFrame
“””
#####################################################
# Provide the function body for data transformation #
#####################################################
rows = []

for _, row in data.iterrows():
channel_str = row[‘Channel’]

if pd.isnull(channel_str) or not str(channel_str).strip():
continue

entries = str(channel_str).split(‘;’)
channel_map = {} # To store channel_name -> summed Decimal %

for entry in entries:
entry = entry.strip()
if not entry:
continue

if ‘(‘ in entry and ‘%’ in entry:
channel_name = entry.split(‘(‘)[0].strip()
percent_str = entry.split(‘(‘)[1].replace(‘)’, ”).replace(‘%’, ”).strip()

try:
percent_val = Decimal(percent_str)
except:
percent_val = Decimal(‘0’)

if channel_name in channel_map:
channel_map[channel_name] += percent_val
else:
channel_map[channel_name] = percent_val

for channel_name, total_percent in channel_map.items():
new_row = row.copy()
new_row[‘Channel’] = channel_name
new_row[‘AllocatedPercentage’] = total_percent

if total_percent > 0:
new_row[‘ActualSpend’] = (new_row[‘ActualSpend’] * (total_percent/100) )

rows.append(new_row)

data = pd.DataFrame(rows)
return data

6.3 Execute the transformation flow and test the output from the local table HDLFT_CAMPAIGN_SILVER:

Wonderful our silver layer of the data is ready and transformed as per the business requirement. Now let us create the customer managed data product and share with the SAP databricks. You can also the local table from HDLF space to HANA cloud DB space and build the models on top of it.

7. Creating Custom Delta Share Data Products: Create a Formations Profile with Data Product visibility option as Formations.

8. Context is automatically created.

9. After creating your Unified Customer Landscape Formations profile, you can proceed to create a Delta Share data product:

10. List the Custom Data Product, so that It will be discoverable in the SAP BDC Catalog & Marketplace:

11. Search and Discover the Data Product “Sales Campaign Gold Data” in SAP BDC Catalog & Marketplace:

12. Share the Data Product “Sales Campaign Gold Data” with supported BDC target systems in our scenario it is SAP databricks: We are sharing the data product with SAP databricks workspace WS_IT.

Note: SAP databricks is part of our SAP BDC formation, hence it is available under the target system options.

13. Validate the sharing is enabled and correct target system:

14. Validate the Delta-Shared Sales Campaign Gold Data product in SAP Databricks by checking the WS_IT workspace with verifying its presence and data preview in the Unity Catalog.

Summary:

In this part of the SAP Business Data Cloud (BDC) series, we explored how to build a custom data product in SAP S/4HANA Cloud Private Edition using a custom CDS entity (e.g., ZCDS_CAMPAIGN). We walked through setting up prerequisites like the Cloud Connector, creating a Datasphere file space, and configuring connections. Using the bronze–silver transformation approach, we replicated and refined campaign data for harmonization. Finally, we created and published a custom Delta Share data product, making it discoverable in the BDC Catalog and consumable in SAP Databricks for advanced analytics and ML scenarios.

Introduction:In Part 1(SAP Business Data Cloud Series – Part 1: Introduction to Data Products) of this blog series, we explored the fundamentals of Data Products in SAP Business Data Cloud (BDC). In Part 2(SAP Business Data Cloud Series – Part 2: Extend SAP S/4HANA Managed Data Products) we explored on how to extend SAP S/4HANA Data Products. Now, in Part 3, we dive into the practical aspect of creating custom data products for the source SAP S/4HANA Cloud Private Edition and Share it with SAP databricks.In this article, we’ll explore a common scenario where a customer has a custom CDS entity in SAP S/4HANA and wants to harmonize its data with SAP-managed data products. We’ll demonstrate how to create a custom data product based on that custom CDS entity, for example, ZCDS_CAMPAIGN and how to delta-share with SAP databricks. High-level Workflow for the SAP S/4HANA, Customer Managed Data Product:Steps to create a SAP S/4HANA Customer Managed Data Product.1. Identify, create, or extend a CDS entity: To build a customer-managed data product in an SAP S/4HANA system, the recommended approach is to leverage CDS views/entities. In this example, we’ll create a custom CDS entity as shown below.@ObjectModel.usageType.dataClass: #TRANSACTIONAL
@ObjectModel.usageType.serviceQuality: #A
@ObjectModel.usageType.sizeCategory: #XL
@ObjectModel.representativeKey: ‘CampaignID’
@ObjectModel.modelingPattern: #ANALYTICAL_DIMENSION
@ObjectModel.supportedCapabilities: [ #ANALYTICAL_DIMENSION,
#CDS_MODELING_DATA_SOURCE,
#CDS_MODELING_ASSOCIATION_TARGET,
#SQL_DATA_SOURCE,
#EXTRACTION_DATA_SOURCE ]

@VDM.viewType: #BASIC
@Metadata.allowExtensions: true
@Metadata.ignorePropagatedAnnotations: true

}2. SAP S/4HANA Private Cloud or On-Premise Edition requires a Cloud Connector: Configure the Cloud Connector to enable communication between your on-premise/private cloud system and SAP Datasphere (public cloud). This setup is a prerequisite for creating an SAP S/4HANA or ABAP connection in SAP Datasphere3. Create a Datasphere Space of Type HANA Data Lake Files: An administrator can create a space backed by SAP HANA Data Lake Files in the object store. File spaces are designed for cost-efficient staging, loading, and preparing large volumes of data.For detailed instructions, refer to the help guide: Create a File Space in the Object Store.Additional configuration is required while configuring the SAP Datasphere tenant. e.g. Memory must be 128GB or more. Check via SAP BDC Estimator.If you have configured correctly then you should also see the same under the SAP Datasphere, Tenant configuration. Please note this view is only available to the SAP Datasphere system owner.SAP Datasphere object store: Storage type will be SAP HANA Data Lake Files. In our scenario we have created a space with a name Workshop: OBJECT STORE.4. Create a SAP S/4HANA Connection in the SAP Datasphere Object Store Space, Workshop: OBJECT STORE :4.1 Create a new connection or use the existing connection of type SAP S/4HANA on-premise or ABAP: 4.2 Fill in the connection configuration details as show below and important step is select the cloud connector which was setup and configured in the step 2.4.3 Provide the connection name and description:4.4 Validate and test the connection making sure replication flows are enabled:You can also use or create the SAP ABAP connection type for this use case.5. Create a Replication Flow to Load Data from CDS Entity into SAP Datasphere Object Store (HDLF):Set up a replication flow with the source system SAP_S4H_RI4 and CDS entity ZCDS_CAMPAIGN. Define the target as a Datasphere local table ZCDS_CAMPAIGN_BRONZE. This table, serving as the Bronze layer, will store source records and structures in their original form.5.1 Create a new replication flow5.2 Select the Source SAP_S4H_RI4 and Container (Source Objects) ZCDS_CAMPAIGN5.3 Select the target as connection as a SAP Datasphere5.4 Provide the target object local table name ZCDS_CAMPAIGN_BRONZE and Deploy the replication flow.5.5 Execute the replication flow and check the run details.5.6 Perform the merge table activity:5.7 Post merge table activity, data will be available to preview and Test it.6. Bronze to Silver layer medallion transformation using a Transformation Flow: In many cases, data from the Bronze layer needs to be refined into a Silver layer through transformations such as splitting, merging, adding/removing columns, joins, or unions.In our scenario, we will split the values in the Channel column into two columns Channel, Allocated Percentage (multiple rows) and apply necessary calculations.6.1 Create a new Transformation Flow6.2 Define the transformation steps:Source table:HDLFT_CAMPAIGN_BRONZE and Target table: HDLFT_CAMPAIGN_SILVER . Python script operator for performing the transformation logic.def transform(data):
“””
This function body should contain all the desired transformations on incoming data of Pandas DataFrame.
Permitted builtin functions as well as permitted NumPy and Pandas objects and functions are available inside this function.
Permitted NumPy and Pandas objects and functions can be used with aliases ‘np’ and ‘pd’ respectively.
Python standard libraries like ‘time’, ‘datetime’, ‘re’, ‘random’, ‘math’, ‘calendar’ and ‘dateutil.parser’ can be used without any alias.
This function executes in a sandbox mode. Please refer the documentation for permitted objects and functions.
Using any restricted functions or objects would cause an internal exception and result in a run failure.
Any code outside this function body will not be executed and inclusion of such code is discouraged.
:param data: Pandas DataFrame
:return: Pandas DataFrame
“””
#####################################################
# Provide the function body for data transformation #
#####################################################
rows = []