Continuing my SAP CI Pagination blog series, this time with Microsoft Dynamics CRM Adapter!
When integrating with Microsoft Dynamics CRM, handling large datasets efficiently is key to ensuring smooth data synchronization. Unlike Salesforce, which relies on SOQL and nextRecordsUrl, Dynamics CRM follows an OData-based pagination approach using <odata.nextLink> to fetch records in manageable batches.
In this blog, I’ll cover:
1. How pagination works in the Microsoft Dynamics CRM adapter
2. Best practices to optimize API calls and avoid performance bottlenecks
3.A step-by-step approach to implementing custom pagination in SAP CI
If you missed the first part on Salesforce Adapter, check it out here Seamless Integration with SAP CI: Mastering Pagination for Salesforce Adapter
Microsoft Dynamics CRM Adapter – Handling Large Data Sets with Pagination
In this use case, we focus on retrieving records from the accounts object in Microsoft Dynamics CRM, which contains 10K+ records. To ensure efficient data retrieval, we utilize Advanced OData Queries along with the pagination feature in the receiver adapter.
Fetching data page by page, processing each batch, and then moving to the next page is the best approach for handling large datasets. In this blog, I will explain the end-to-end batching process, ensuring seamless data synchronization while optimizing API performance.
Design Approach:
The scenario involves sending accounts object data from Microsoft Dynamics CRM to a target system that has a batch limit of 500 records per request. To accommodate this constraint, we implement a pagination approach using two Microsoft Dynamics CRM receiver adapter call.
The call fetches the page of records within a looping process to retrieve the pages one after the other. Each fetched page is passed through message mapping, ensuring seamless transformation before being delivered to the target system. This cycle continues—fetching a page, processing it, and sending it to the target—until all records are transferred, ensuring an efficient full end to end Batching processing and controlled data flow from Microsoft Dynamics CRM to the target system.
Design components elaboration:
Timer : This is a timer based flow which be a scheduled as per the user’s input
Content Modifier : “set properties” – This step is used to accept inputs such as ObjectName, fieldnames, StartDate from the user, all these fields are externalized to make it dynamic.
Looping Process Call : “Initiate MS Dynamics Call” – This Looping Process Call continuously runs until the <odata.nextLink> field is absent from the response payload. The condition for looping is:
${property.NextPageLink} != ”
Here’s how it works:
— The NextPageLink property is extracted from the response payload.
— If NextPageLink exists, it is used in the next request to fetch the next set of records.
— The loop continues until <odata.nextLink> is no longer present, indicating the last page.
This ensures efficient batch processing, fetching all records page by page while maintaining seamless data flow.
The Microsoft Dynamics CRM receiver adapter is placed inside the Looping Process Call to fetch records page by page.
Processing Tab
— Pagination is enabled, ensuring controlled data retrieval.
— Page Size is set to “500”, based on the specific requirement.
— The Next Page Link option is assigned the value “${property.NextPageLink}”, which dynamically comes from the response payload.
We are not digging into the connection tab in this blog, this blog focuses only on pagination concept.
Output:
Page 1 : Response from the Microsoft Dynamics CRM system contains 500 records and “odata.nextLink” which has the URL to the next page.
Page 2: Response from the Microsoft Dynamics CRM system contains 500 records and “odata.nextLink” which has the URL to the next page.
Content Modifier : “Read nextURL” – This is a crucial step where are we reading key field —<odata.nextLink> using xpath.
Process Call : “Message Mapping” –
After fetching each page, the data is passed to Message Mapping, where it is processed before being sent to the target system. This step is target-specific and is not covered in this blog.
For demonstration purposes, I have included this step to mimic the “send to target” operation after message mapping. The key focus of this blog remains on pagination handling in the Microsoft Dynamics CRM adapter and ensuring seamless data retrieval.
Final Page Handling: Completing the Pagination Process
For a total of 10573 records, the pagination process spans 22 pages:
First 21 pages contain 500 records each.Page 22 contains the remaining 73 records.
When the 22th page is fetched, the response contains:
No <odata.nextLink> , confirming that no more pages are available.
At this point, the looping process terminates, ensuring that all records have been successfully retrieved and processed.
Page 22
From the TRACE logs below, its evident that looping process call “Initiate MS Dynamics Call” executed 23 times indicating 23 pages.
Full Run in TRACE
Conclusion
Implementing pagination in SAP CI for the Microsoft Dynamics CRM Adapter ensures efficient data retrieval while preventing performance bottlenecks. By leveraging looping mechanisms, we can seamlessly fetch and process large datasets in manageable batches. This approach not only optimizes API calls but also ensures smooth end-to-end data transfer. Stay tuned for the next part of this series, where we explore pagination for next adapters.
I hope this helps.
Cheers,
Punith Oswal
Continuing my SAP CI Pagination blog series, this time with Microsoft Dynamics CRM Adapter!When integrating with Microsoft Dynamics CRM, handling large datasets efficiently is key to ensuring smooth data synchronization. Unlike Salesforce, which relies on SOQL and nextRecordsUrl, Dynamics CRM follows an OData-based pagination approach using <odata.nextLink> to fetch records in manageable batches.In this blog, I’ll cover:1. How pagination works in the Microsoft Dynamics CRM adapter2. Best practices to optimize API calls and avoid performance bottlenecks3.A step-by-step approach to implementing custom pagination in SAP CIIf you missed the first part on Salesforce Adapter, check it out here Seamless Integration with SAP CI: Mastering Pagination for Salesforce Adapter Microsoft Dynamics CRM Adapter – Handling Large Data Sets with PaginationIn this use case, we focus on retrieving records from the accounts object in Microsoft Dynamics CRM, which contains 10K+ records. To ensure efficient data retrieval, we utilize Advanced OData Queries along with the pagination feature in the receiver adapter.Fetching data page by page, processing each batch, and then moving to the next page is the best approach for handling large datasets. In this blog, I will explain the end-to-end batching process, ensuring seamless data synchronization while optimizing API performance.Design Approach: The scenario involves sending accounts object data from Microsoft Dynamics CRM to a target system that has a batch limit of 500 records per request. To accommodate this constraint, we implement a pagination approach using two Microsoft Dynamics CRM receiver adapter call.The call fetches the page of records within a looping process to retrieve the pages one after the other. Each fetched page is passed through message mapping, ensuring seamless transformation before being delivered to the target system. This cycle continues—fetching a page, processing it, and sending it to the target—until all records are transferred, ensuring an efficient full end to end Batching processing and controlled data flow from Microsoft Dynamics CRM to the target system. Design components elaboration:Timer : This is a timer based flow which be a scheduled as per the user’s inputContent Modifier : “set properties” – This step is used to accept inputs such as ObjectName, fieldnames, StartDate from the user, all these fields are externalized to make it dynamic. Looping Process Call : “Initiate MS Dynamics Call” – This Looping Process Call continuously runs until the <odata.nextLink> field is absent from the response payload. The condition for looping is:${property.NextPageLink} != ”Here’s how it works:– The NextPageLink property is extracted from the response payload.– If NextPageLink exists, it is used in the next request to fetch the next set of records.– The loop continues until <odata.nextLink> is no longer present, indicating the last page.This ensures efficient batch processing, fetching all records page by page while maintaining seamless data flow.The Microsoft Dynamics CRM receiver adapter is placed inside the Looping Process Call to fetch records page by page.Processing Tab– Pagination is enabled, ensuring controlled data retrieval.– Page Size is set to “500”, based on the specific requirement.– The Next Page Link option is assigned the value “${property.NextPageLink}”, which dynamically comes from the response payload.We are not digging into the connection tab in this blog, this blog focuses only on pagination concept.Output: Page 1 : Response from the Microsoft Dynamics CRM system contains 500 records and “odata.nextLink” which has the URL to the next page.Page 2: Response from the Microsoft Dynamics CRM system contains 500 records and “odata.nextLink” which has the URL to the next page.Content Modifier : “Read nextURL” – This is a crucial step where are we reading key field —<odata.nextLink> using xpath.Process Call : “Message Mapping” – After fetching each page, the data is passed to Message Mapping, where it is processed before being sent to the target system. This step is target-specific and is not covered in this blog.For demonstration purposes, I have included this step to mimic the “send to target” operation after message mapping. The key focus of this blog remains on pagination handling in the Microsoft Dynamics CRM adapter and ensuring seamless data retrieval. Final Page Handling: Completing the Pagination ProcessFor a total of 10573 records, the pagination process spans 22 pages:First 21 pages contain 500 records each.Page 22 contains the remaining 73 records.When the 22th page is fetched, the response contains:No <odata.nextLink> , confirming that no more pages are available.At this point, the looping process terminates, ensuring that all records have been successfully retrieved and processed.Page 22From the TRACE logs below, its evident that looping process call “Initiate MS Dynamics Call” executed 23 times indicating 23 pages.Full Run in TRACE ConclusionImplementing pagination in SAP CI for the Microsoft Dynamics CRM Adapter ensures efficient data retrieval while preventing performance bottlenecks. By leveraging looping mechanisms, we can seamlessly fetch and process large datasets in manageable batches. This approach not only optimizes API calls but also ensures smooth end-to-end data transfer. Stay tuned for the next part of this series, where we explore pagination for next adapters.I hope this helps.Cheers,Punith Oswal Read More Technology Blogs by Members articles
#SAP
#SAPTechnologyblog
