![The Agentic Web [Part 1] | KEY030](https://i3.ytimg.com/vi/bmHJFRNHmoc/hqdefault.jpg)
![The Agentic Web: A conversation with Jay Parikh [Part 2] | KEY035](https://i1.ytimg.com/vi/xvZdki3FqzY/hqdefault.jpg)
Introduction
Handling failed messages efficiently is a critical aspect of SAP Cloud Integration (SAP CPI). While CPI does not offer a built-in feature to automatically resend failed messages, several SAP community-driven solutions exist using JMS queues and data stores. This blog introduces an alternative approach that leverages the Data Store Channel to effectively manage and retry failed messages.
Scenario Overview
In this scenario, we have two iFlows:
Main iFlow – This is any custom iFlow developed in your tenant where you want to implement the retriggering mechanism.(iflow name:Test_Mainiflow)
Retrigger iFlow – This iFlow utilizes the Data Store Sender Channel to pick up failed messages and resend them to the original iFlow.(iflow name:Test_ReprocessFailedMessageDataStore)
The Main iFlow consists of two integration processes:
Initial Integration Process: Receives the message payload via a sender channel, which can be any adapter or a timer function depending on the requirement.
Retrigger Integration Process: Retriggers failed messages stored in the data store.
Both integration processes utilize a Content Modifier to set the standard SAP sender header, which is later used for retriggering purposes. Additionally, a Process Call directs messages to the main local process where the core integration logic is maintained
The local process contains the core logic, including filtering, mapping, and payload manipulation. For simplicity, the following two steps are included:
Content Modifier: Sets integration-related parameters.Process Call: Forwards the processed data to the receiver.
Additionally, an Exception Subprocess is incorporated to capture error details in case of failures.
Below are the key parameters used in the content modifier in local integration process:
ParametersPurposeDatastoreNameStores failed messages.EmaildEmail recipients for failure notifications.interface_idUnique integration identifier.SAP_ApplicationIDStores unique identifiers such as PO or IDOC numbers.end_pointProcess direct address to identify each interface uniquely. Make sure each interface should have unique address value.source_payloadOriginal payload received from the sender.email_subjectSubject for error alert emails.SAP_ReceiverDisplays receiver system name in the monitoring page.SAP_SenderDisplays sender system name in the monitoring page.
Content modifier parameter in exception subprocess
Exception Subprocess Content Modifier Header
Parameter
Purpose
cpi_error
Captures error details (${property.CamelExceptionCaught}).
error_type
Differentiates error types; retry logic applies only to connectivity errors. Retriggering failed message should only happen in case connectivity issue and not for all errors like mapping or any syntax or any other configuration errors which won’t be resolved even if message is reprocessed.
Exception Subprocess Content Modifier Property
A Process Call within the Exception Subprocess routes messages to the local process, ensuring that retry logic is applied only in the case of retriable errors (e.g., connectivity issues, not mapping or syntax errors).
Below is the local integration process to send data to receiver system.
Here we are using HTTP receiver channel where config is such that message will fail in CPI to test our retry logic.
Content modifier in the exception subprocess
Header cpi_error will store the error details but property error_type will have constant value connection(can be any constant value)
Below is the local process which is getting called from exception subprocess.
The retrigger mechanism is implemented using a router with the following conditions:
Condition maintained for the Data store branch
${property.error_type} = ‘connection’ and ${property.Temp_Sender} != ‘CPI’
This condition ensures that messages are only stored in the data store if they originally failed due to a connectivity issue and were triggered from the sender system. If a retriggered message initiated by the Data Store Channel iFlow(Sender as ‘CPI’) fails again, it will not be stored again to prevent an infinite loop.
Messages are stored in the Data Store along with headers, using the message ID as the unique entry ID. It is crucial to enable the “Include Message Headers” checkbox to ensure retriggering functions correctly.
Retriggering Failed Messages
A second iFlow is designed to retrieve failed messages from the data store and resend them to the original iFlow. The retry mechanism follows these steps:
The iFlow picks up messages from the data store.
It attempts to resend the message up to three times.
If the message is successful in any of the retry attempts, it is deleted from the data store.
If the message remains unsuccessful, it is deleted from the data store and an error alert email is sent.
Below is the main integration process which picks up the message from data store and sends it bak to main iflow.
Data store sender channel
The parameters used in the Data Store Sender Adapter are quite similar to those in the JMS Sender Adapter. While most of them are self-explanatory, you can refer to the official SAP documentation for a detailed explanation of each parameter:
Data Store Sender Adapter Documentation
The receiver channel is configured as a Process Direct Adapter, with the address stored in the header variable end_point. This header must be defined in the main iFlow and should be unique for each iFlow in your tenant. If the configured address is incorrect, the reprocessing mechanism will not function as expected.
If retriggered messages fail again, Exception Subprocess handles the failure.
Final Reprocessing Mechanism
A router checks the SAP standard header SAP_DataStoreRetries, ensuring the retry limit is not exceeded before marking the message as permanently failed.
Once a message is retriggered from the data store back to the main iFlow, it is processed again. If the main/original iFlow successfully processes the message, it is immediately removed from the data store. However, if the message fails again, the exception subprocess in the retrigger iFlow is triggered, as illustrated in the image below.
At this stage, a router evaluates the standard SAP header ‘SAP_DataStoreRetries’ to determine the retry count. If the message has not yet exceeded the configured retry limit, it proceeds to the retry branch with error end, keeping it in a ‘retry‘ status. This means the Data Store Sender Channel will attempt to resend the message after the predefined interval. However, if the retry limit is reached, the message follows the error branch and is not retried further.
Also since main iflow failed with the error router in the main/original iflow will make sure message is not stored in the data store.
Below is how actual reprocessing mechanism will work.
Triggered test payload from POSTMAN
Initial Failure: Message failed in CPI and datastore entry is created. You can see the values of Sender, receiver and application message id as well.
First Retry: The message is retried but is not re-stored in the data store as it is coming from integration paramter ‘sender’ as ‘CPI’ and ‘error_type’ header is ‘connection’
Second Retry:
Third/Final Retry:
Message entry from data store is deleted.
Key Considerations:
This retrigger mechanism is only applicable for asynchronous integrations.
The retry logic is specifically implemented for connectivity errors. If required, it can be extended to all errors by modifying the exception subprocess in main iflow logic by removing the content modifier.
For better scalability, consider implementing a centralized error handling iFlow that processes error details and forwards them to data store retrigger iflow.
Ensure that runtime configuration is marked as * to retain all necessary headers during retriggering.
The first retry happens very quickly as it depends on the polling interval timer. If a message arrives close to the end of the interval (e.g., at the 9:50-second mark of a 10-minute interval), it will be picked up in the next poll almost immediately, so choose your poll interval accordingly.
Currently, the retrigger mechanism sends the message back to the source, requiring it to go through all processing steps again before reaching the receiver. This may not be ideal if the iFlow involves multiple transformations, mappings, or complex processing logic.
One possible workaround to optimize this process is to break the iFlow into multiple local integration processes. Instead of retriggering the entire iFlow, the retrigger mechanism could directly call the relevant subprocess where the failure occurred. However, implementing this approach would require additional adjustments, such as storing the payload after key processing steps and maintaining a flag to determine the last successful subprocess execution.
Conclusion
This blog demonstrates one approach to reprocessing failed messages using the Data Store Channel in SAP CPI. While not the only method available, this approach ensures efficient message retrying and prevents unnecessary message duplication.
If you have any questions or feedback, feel free to share them in the comments section.
References:
Message Retry Using Process Direct and Data Store
Configure Asynchronous Messaging with Retry Using JMS
Happy Integrating! 
IntroductionHandling failed messages efficiently is a critical aspect of SAP Cloud Integration (SAP CPI). While CPI does not offer a built-in feature to automatically resend failed messages, several SAP community-driven solutions exist using JMS queues and data stores. This blog introduces an alternative approach that leverages the Data Store Channel to effectively manage and retry failed messages.Scenario OverviewIn this scenario, we have two iFlows:Main iFlow – This is any custom iFlow developed in your tenant where you want to implement the retriggering mechanism.(iflow name:Test_Mainiflow) Retrigger iFlow – This iFlow utilizes the Data Store Sender Channel to pick up failed messages and resend them to the original iFlow.(iflow name:Test_ReprocessFailedMessageDataStore) The Main iFlow consists of two integration processes:Initial Integration Process: Receives the message payload via a sender channel, which can be any adapter or a timer function depending on the requirement.Retrigger Integration Process: Retriggers failed messages stored in the data store.Both integration processes utilize a Content Modifier to set the standard SAP sender header, which is later used for retriggering purposes. Additionally, a Process Call directs messages to the main local process where the core integration logic is maintainedThe local process contains the core logic, including filtering, mapping, and payload manipulation. For simplicity, the following two steps are included:Content Modifier: Sets integration-related parameters.Process Call: Forwards the processed data to the receiver.Additionally, an Exception Subprocess is incorporated to capture error details in case of failures.Below are the key parameters used in the content modifier in local integration process: ParametersPurposeDatastoreNameStores failed messages.EmaildEmail recipients for failure notifications.interface_idUnique integration identifier.SAP_ApplicationIDStores unique identifiers such as PO or IDOC numbers.end_pointProcess direct address to identify each interface uniquely. Make sure each interface should have unique address value.source_payloadOriginal payload received from the sender.email_subjectSubject for error alert emails.SAP_ReceiverDisplays receiver system name in the monitoring page.SAP_SenderDisplays sender system name in the monitoring page.Content modifier parameter in exception subprocessException Subprocess Content Modifier Header ParameterPurposecpi_errorCaptures error details (${property.CamelExceptionCaught}).error_typeDifferentiates error types; retry logic applies only to connectivity errors. Retriggering failed message should only happen in case connectivity issue and not for all errors like mapping or any syntax or any other configuration errors which won’t be resolved even if message is reprocessed. Exception Subprocess Content Modifier PropertyA Process Call within the Exception Subprocess routes messages to the local process, ensuring that retry logic is applied only in the case of retriable errors (e.g., connectivity issues, not mapping or syntax errors).Below is the local integration process to send data to receiver system.Here we are using HTTP receiver channel where config is such that message will fail in CPI to test our retry logic.Content modifier in the exception subprocessHeader cpi_error will store the error details but property error_type will have constant value connection(can be any constant value) Below is the local process which is getting called from exception subprocess.The retrigger mechanism is implemented using a router with the following conditions:Condition maintained for the Data store branch ${property.error_type} = ‘connection’ and ${property.Temp_Sender} != ‘CPI’This condition ensures that messages are only stored in the data store if they originally failed due to a connectivity issue and were triggered from the sender system. If a retriggered message initiated by the Data Store Channel iFlow(Sender as ‘CPI’) fails again, it will not be stored again to prevent an infinite loop.Messages are stored in the Data Store along with headers, using the message ID as the unique entry ID. It is crucial to enable the “Include Message Headers” checkbox to ensure retriggering functions correctly. Retriggering Failed MessagesA second iFlow is designed to retrieve failed messages from the data store and resend them to the original iFlow. The retry mechanism follows these steps:The iFlow picks up messages from the data store.It attempts to resend the message up to three times.If the message is successful in any of the retry attempts, it is deleted from the data store.If the message remains unsuccessful, it is deleted from the data store and an error alert email is sent.Below is the main integration process which picks up the message from data store and sends it bak to main iflow. Data store sender channelThe parameters used in the Data Store Sender Adapter are quite similar to those in the JMS Sender Adapter. While most of them are self-explanatory, you can refer to the official SAP documentation for a detailed explanation of each parameter:Data Store Sender Adapter DocumentationThe receiver channel is configured as a Process Direct Adapter, with the address stored in the header variable end_point. This header must be defined in the main iFlow and should be unique for each iFlow in your tenant. If the configured address is incorrect, the reprocessing mechanism will not function as expected.If retriggered messages fail again, Exception Subprocess handles the failure.Final Reprocessing MechanismA router checks the SAP standard header SAP_DataStoreRetries, ensuring the retry limit is not exceeded before marking the message as permanently failed.Once a message is retriggered from the data store back to the main iFlow, it is processed again. If the main/original iFlow successfully processes the message, it is immediately removed from the data store. However, if the message fails again, the exception subprocess in the retrigger iFlow is triggered, as illustrated in the image below. At this stage, a router evaluates the standard SAP header ‘SAP_DataStoreRetries’ to determine the retry count. If the message has not yet exceeded the configured retry limit, it proceeds to the retry branch with error end, keeping it in a ‘retry’ status. This means the Data Store Sender Channel will attempt to resend the message after the predefined interval. However, if the retry limit is reached, the message follows the error branch and is not retried further.Also since main iflow failed with the error router in the main/original iflow will make sure message is not stored in the data store. Below is how actual reprocessing mechanism will work.Triggered test payload from POSTMANInitial Failure: Message failed in CPI and datastore entry is created. You can see the values of Sender, receiver and application message id as well. First Retry: The message is retried but is not re-stored in the data store as it is coming from integration paramter ‘sender’ as ‘CPI’ and ‘error_type’ header is ‘connection’ Second Retry:Third/Final Retry: Message entry from data store is deleted.Key Considerations:This retrigger mechanism is only applicable for asynchronous integrations.The retry logic is specifically implemented for connectivity errors. If required, it can be extended to all errors by modifying the exception subprocess in main iflow logic by removing the content modifier.For better scalability, consider implementing a centralized error handling iFlow that processes error details and forwards them to data store retrigger iflow.Ensure that runtime configuration is marked as * to retain all necessary headers during retriggering.The first retry happens very quickly as it depends on the polling interval timer. If a message arrives close to the end of the interval (e.g., at the 9:50-second mark of a 10-minute interval), it will be picked up in the next poll almost immediately, so choose your poll interval accordingly.Currently, the retrigger mechanism sends the message back to the source, requiring it to go through all processing steps again before reaching the receiver. This may not be ideal if the iFlow involves multiple transformations, mappings, or complex processing logic.One possible workaround to optimize this process is to break the iFlow into multiple local integration processes. Instead of retriggering the entire iFlow, the retrigger mechanism could directly call the relevant subprocess where the failure occurred. However, implementing this approach would require additional adjustments, such as storing the payload after key processing steps and maintaining a flag to determine the last successful subprocess execution. ConclusionThis blog demonstrates one approach to reprocessing failed messages using the Data Store Channel in SAP CPI. While not the only method available, this approach ensures efficient message retrying and prevents unnecessary message duplication.If you have any questions or feedback, feel free to share them in the comments section.References:Message Retry Using Process Direct and Data StoreConfigure Asynchronous Messaging with Retry Using JMSRetry Failed Messages in CPIHappy Integrating! Read More Technology Blogs by Members articles
#SAP
#SAPTechnologyblog
