Introduction:
Efficiently handling large datasets is a critical aspect of modern enterprise integration. When retrieving data from databases using SAP Cloud Integration (SAP CI), leveraging pagination can significantly enhance performance and prevent overloading systems. In this blog, we’ll explore how to implement pagination using the HTTP adapter in SAP CI, ensuring efficient and seamless data processing.
What is Pagination?
Pagination is the process of dividing large data sets into smaller chunks (pages) to be retrieved incrementally. This approach prevents excessive memory usage and reduces the risk of timeouts or errors when processing large volumes of data.
Use Case:
In many cases, SAP CI provides a standard OData adapter that supports pagination. However, there are scenarios where the source application, such as a database, imposes restrictions and only allows HTTP calls from outside the network. Due to this limitation, the standard OData adapter cannot be used directly. Instead, a customized pagination approach is implemented in SAP CI to handle data retrieval efficiently. This blog demonstrates how to achieve this using the HTTP adapter.
Steps to Implement Pagination in SAP CI:
Design the Integration Flow
Use the HTTP adapter to connect to the database API or service.
Configure the integration flow to handle paginated requests and responses.
Start the Pagination Process
Begin by making an initial HTTP call to retrieve the total count of records from the database.
Use the count query provided by the database team to fetch the total count records information.
Example response: {
“TotalRecCount”: 50000
}
3. Calculate Page Count and Loop Parameters
Initially set the Pagenumber to 0 so that will be used to increment the pagenumber in the data retrieving process, and Recordsperpage is 2000 in my case as the database allowed only a maximum of 2000 records per one page that we can retrieve.Use a Groovy script to calculate the number of pages required to retrieve all records based on the total count and records per page.In this example, let’s assume the total count of records is 50,000 and the source application can process 2,000 records per page. The script calculates the page count by dividing the total count by the records per page and then rounding up to ensure all records are retrieved.
import com.sap.gateway.ip.core.customdev.util.Message;
import java.lang.*;
def Message processData(Message message) {
def body = message.getBody();
def map = message.getHeaders();
def value1 = map.get(“TotalRecCount”);
def value2 = map.get(“RecordsPerPage”);
if (value1 == null) {
def A = 0;
} else {
A = Integer.parseInt(value1);
B = Integer.parseInt(value2);
C = Math.ceil(A / B);
D = Math.round(C);
}
message.setHeader(“PagingLoopCount”, D);
message.setBody(body);
return message;
}
In this scenario:
Total Records = 50,000
Records Per Page = 2,000
Pages Required = ceil(50000 / 2000) = 25
The script ensures all records are retrieved by calculating and rounding up the page count.
4. Looping Process Call Configuration
Use the “Looping Process Call” to retrieve data page by page.
Configure the loop condition to continue until the Pagenumber is less than or equal to the PagingLoopCount
Example configuration:
This ensures that the process continues to loop and retrieve data until all pages are processed.
Local Integration process:
This local integration process works to retrieve the actual data from the database in a looping mode. First, it will retrieve the page 1 data using the data query. Once the data is retrieved, as per the given script, it will increment the Pagenumber by +1 for every looping call.
Groovy Script to Increment Page Number:
import com.sap.gateway.ip.core.customdev.util.Message;
def Message processData(Message message) {
def body = message.getBody();
def pageNumber;
def headers = message.getHeaders();
def value = headers.get(“Pagenumber”);
def value1 = value.toInteger()
value1 = value1 + 1;
message.setHeader(“Pagenumber”, value1);
message.setBody(body);
return message;
}
Sample HTTP query:
{(PageNumber:${header.Pagenumber},pageSize: ${header.RecordsPerPage}}
The Pagenumber and RecordsPerPage headers will be parsed into the actual data query to pull the managed data from the database, ensuring compatibility with the source application.
Conclusion:
Pagination is a powerful and essential technique to manage and process large data sets efficiently in SAP CPI. This blog demonstrated how to implement customized pagination using the HTTP adapter, addressing the limitations of standard OData pagination when direct HTTP calls are required by the database. By leveraging Groovy scripts for dynamic page calculations and loop management, you can:
Retrieve data incrementally without overloading the database or SAP CI resources.
Adapt to database-specific constraints, such as the maximum number of records per page.
Ensure all data is retrieved accurately and efficiently.
With this approach, enterprises can streamline data retrieval processes, optimize system performance, and scale their integration solutions.
Introduction:Efficiently handling large datasets is a critical aspect of modern enterprise integration. When retrieving data from databases using SAP Cloud Integration (SAP CI), leveraging pagination can significantly enhance performance and prevent overloading systems. In this blog, we’ll explore how to implement pagination using the HTTP adapter in SAP CI, ensuring efficient and seamless data processing.What is Pagination?Pagination is the process of dividing large data sets into smaller chunks (pages) to be retrieved incrementally. This approach prevents excessive memory usage and reduces the risk of timeouts or errors when processing large volumes of data.Use Case:In many cases, SAP CI provides a standard OData adapter that supports pagination. However, there are scenarios where the source application, such as a database, imposes restrictions and only allows HTTP calls from outside the network. Due to this limitation, the standard OData adapter cannot be used directly. Instead, a customized pagination approach is implemented in SAP CI to handle data retrieval efficiently. This blog demonstrates how to achieve this using the HTTP adapter.Steps to Implement Pagination in SAP CI:Design the Integration FlowUse the HTTP adapter to connect to the database API or service.Configure the integration flow to handle paginated requests and responses. Start the Pagination ProcessBegin by making an initial HTTP call to retrieve the total count of records from the database.Use the count query provided by the database team to fetch the total count records information. Example response: { “TotalRecCount”: 50000 } 3. Calculate Page Count and Loop ParametersInitially set the Pagenumber to 0 so that will be used to increment the pagenumber in the data retrieving process, and Recordsperpage is 2000 in my case as the database allowed only a maximum of 2000 records per one page that we can retrieve.Use a Groovy script to calculate the number of pages required to retrieve all records based on the total count and records per page.In this example, let’s assume the total count of records is 50,000 and the source application can process 2,000 records per page. The script calculates the page count by dividing the total count by the records per page and then rounding up to ensure all records are retrieved. import com.sap.gateway.ip.core.customdev.util.Message;
import java.lang.*;
def Message processData(Message message) {
def body = message.getBody();
def map = message.getHeaders();
def value1 = map.get(“TotalRecCount”);
def value2 = map.get(“RecordsPerPage”);
if (value1 == null) {
def A = 0;
} else {
A = Integer.parseInt(value1);
B = Integer.parseInt(value2);
C = Math.ceil(A / B);
D = Math.round(C);
}
message.setHeader(“PagingLoopCount”, D);
message.setBody(body);
return message;
} In this scenario:Total Records = 50,000Records Per Page = 2,000 Pages Required = ceil(50000 / 2000) = 25The script ensures all records are retrieved by calculating and rounding up the page count.4. Looping Process Call ConfigurationUse the “Looping Process Call” to retrieve data page by page.Configure the loop condition to continue until the Pagenumber is less than or equal to the PagingLoopCountExample configuration:This ensures that the process continues to loop and retrieve data until all pages are processed.Local Integration process: This local integration process works to retrieve the actual data from the database in a looping mode. First, it will retrieve the page 1 data using the data query. Once the data is retrieved, as per the given script, it will increment the Pagenumber by +1 for every looping call. Groovy Script to Increment Page Number: import com.sap.gateway.ip.core.customdev.util.Message;
def Message processData(Message message) {
def body = message.getBody();
def pageNumber;
def headers = message.getHeaders();
def value = headers.get(“Pagenumber”);
def value1 = value.toInteger()
value1 = value1 + 1;
message.setHeader(“Pagenumber”, value1);
message.setBody(body);
return message;
} Sample HTTP query:{(PageNumber:${header.Pagenumber},pageSize: ${header.RecordsPerPage}}The Pagenumber and RecordsPerPage headers will be parsed into the actual data query to pull the managed data from the database, ensuring compatibility with the source application.Conclusion:Pagination is a powerful and essential technique to manage and process large data sets efficiently in SAP CPI. This blog demonstrated how to implement customized pagination using the HTTP adapter, addressing the limitations of standard OData pagination when direct HTTP calls are required by the database. By leveraging Groovy scripts for dynamic page calculations and loop management, you can:Retrieve data incrementally without overloading the database or SAP CI resources.Adapt to database-specific constraints, such as the maximum number of records per page.Ensure all data is retrieved accurately and efficiently.With this approach, enterprises can streamline data retrieval processes, optimize system performance, and scale their integration solutions. Read More Technology Blogs by Members articles
#SAP
#SAPTechnologyblog
#enduro #enduromtb
600hp BMW 335i // Nürburgring
