Introduction
Recently we worked on a high-frequency interface using SAP Cloud Integration (CPI) to process a continuous stream of IDocs from a source system. The requirement was to map these IDocs, and send the mapped content to a target system using SFTP. All messages needed to be appended to a single file, and that file was picked up and processed by a job every 5 minutes at the target side.
At first glance, this looked like a simple scenario, but it turned out to be a tricky problem due to the way SFTP receiver works in high-frequency situations.
Problem Statement
We received a high volume of IDocs per minute, and our CPI iFlow was designed to do the following:
Map incoming IDocs to the target structure.
Use the SFTP Receiver Adapter and append to write to a single file for 5 mins after this the downstream job process every 5 minutes. However, the target system experienced significant problems with increased load.
1. NULL characters in random lines.
2. Truncation of data.
Final file was corrupted or incomplete and therefore unusable. Final file was looking like below:
We noted the following:
The SFTP Adapter in Append Mode cannot support concurrent writes. When multiple CPI threads attempted to append concurrently:
Some I/O calls overlapped.File locks failed.The content was corrupted (e.g. NULL characters, partial lines).
CPI does not guarantee ordered or transactional writes when multiple messages are attempting to access the SFTP channel at the same time.
Solution:
To resolve the issue, we redesigned the solution as 2 separate iFlows utilizing Data Store.
iFlow 1 – Message Processing and Write to Data Store
This iFlow receives Idoc from the SAP system and after doing the map it stores the file to the DataStore.
Iflow step details:
Set header and Content: This is used to set custom header and properties required for processing and logging.Message Mapping: here Idoc structure is mapped to the required target structure.Write: The transformed structure is then write to the Data Store for the next step. This structure should be in XML format for the next step to read it.
iFlow 2 – Aggregation and SFTP File Upload
This iFlow is triggered by a Timer every 5 minutes. It reads all messages stored earlier by the 1st iFlow.
We have used Data Store Select to fetch all unprocessed messages in the interval of 5 mins. We have given the number of messages as 500, so that it will pull all the files written by 1st flow on DataStore.
“Delete on Completion” This check we have selected such that no file will be processed twice after successful processing of the file.
We have given the Number of polled messages as 500, so that it will pick up all the messages that were stored in datastore during last 5 mins. If any data left it will be processed in next poll.
When Select step is picking the file up from the data store it combines all the files in the following structure.
In my case the file was looking as below:
We have used a simple groovy to collect all the data between the tag <data> and appended them to create a CSV file
def Message processData(Message message) {
// Get XML as string
def body = message.getBody(String)
// Parse XML using XmlSlurper
def root = new XmlSlurper().parseText(body)
// Collect all <data> elements
def dataLines = root.message.record.data.collect { it.text() }
// Create CSV string with LF at end
def csv = dataLines.join(“n”) + “n”
// Set CSV as new body
message.setBody(csv)
return message
}
After the groovy the file looks like as below:
This approach created the file which behaves like Append Mode but in a safe, sequential, and controlled manner.
The resulting content is then sent to the target using the SFTP Receiver Adapter.
In conclusion, this method ensures data integrity, sequential processing, and improved performance by safely simulating append mode of SFTP receiver adapter.
No concurrency issues
No file corruption
Scalable for high-volume scenarios
Thank you.
IntroductionRecently we worked on a high-frequency interface using SAP Cloud Integration (CPI) to process a continuous stream of IDocs from a source system. The requirement was to map these IDocs, and send the mapped content to a target system using SFTP. All messages needed to be appended to a single file, and that file was picked up and processed by a job every 5 minutes at the target side.At first glance, this looked like a simple scenario, but it turned out to be a tricky problem due to the way SFTP receiver works in high-frequency situations.Problem StatementWe received a high volume of IDocs per minute, and our CPI iFlow was designed to do the following:Map incoming IDocs to the target structure.Use the SFTP Receiver Adapter and append to write to a single file for 5 mins after this the downstream job process every 5 minutes. However, the target system experienced significant problems with increased load.1. NULL characters in random lines.2. Truncation of data.Final file was corrupted or incomplete and therefore unusable. Final file was looking like below:We noted the following:The SFTP Adapter in Append Mode cannot support concurrent writes. When multiple CPI threads attempted to append concurrently:Some I/O calls overlapped.File locks failed.The content was corrupted (e.g. NULL characters, partial lines).CPI does not guarantee ordered or transactional writes when multiple messages are attempting to access the SFTP channel at the same time.Solution:To resolve the issue, we redesigned the solution as 2 separate iFlows utilizing Data Store.iFlow 1 – Message Processing and Write to Data StoreThis iFlow receives Idoc from the SAP system and after doing the map it stores the file to the DataStore. Iflow step details:Set header and Content: This is used to set custom header and properties required for processing and logging.Message Mapping: here Idoc structure is mapped to the required target structure.Write: The transformed structure is then write to the Data Store for the next step. This structure should be in XML format for the next step to read it.iFlow 2 – Aggregation and SFTP File UploadThis iFlow is triggered by a Timer every 5 minutes. It reads all messages stored earlier by the 1st iFlow.We have used Data Store Select to fetch all unprocessed messages in the interval of 5 mins. We have given the number of messages as 500, so that it will pull all the files written by 1st flow on DataStore. “Delete on Completion” This check we have selected such that no file will be processed twice after successful processing of the file.We have given the Number of polled messages as 500, so that it will pick up all the messages that were stored in datastore during last 5 mins. If any data left it will be processed in next poll.When Select step is picking the file up from the data store it combines all the files in the following structure.In my case the file was looking as below:We have used a simple groovy to collect all the data between the tag <data> and appended them to create a CSV filedef Message processData(Message message) {
// Get XML as string
def body = message.getBody(String)
// Parse XML using XmlSlurper
def root = new XmlSlurper().parseText(body)
// Collect all <data> elements
def dataLines = root.message.record.data.collect { it.text() }
// Create CSV string with LF at end
def csv = dataLines.join(“n”) + “n”
// Set CSV as new body
message.setBody(csv)
return message
}After the groovy the file looks like as below:This approach created the file which behaves like Append Mode but in a safe, sequential, and controlled manner.The resulting content is then sent to the target using the SFTP Receiver Adapter.In conclusion, this method ensures data integrity, sequential processing, and improved performance by safely simulating append mode of SFTP receiver adapter.No concurrency issuesNo file corruptionScalable for high-volume scenariosThank you. Read More Technology Blog Posts by Members articles
#SAP
#SAPTechnologyblog
