Read Lock strategy to consider while moving sftp / file adapter with – “Msecs to wait before Modification check” while migrating to BTP – Cloud Integration.
SAP Process Integration (PI) offers a “File Modification Check” that enables adapters to delay file processing for a defined period, ensuring only complete files are handled. A comparable feature in SAP Business Technology Platform (BTP) Cloud Integration would significantly enhance file-based integration. While many file transfer scenarios necessitate source system handling to guarantee fully written files before processing, the SFTP adapter currently lacks a built-in option to specify such a waiting period.
SAP BTP Cloud Integration offers several options for implementing a read lock strategy with the SFTP adapter
These Includes:
Done File Expected: This strategy requires a “done file” to signal when a file is ready for processing. If this option is selected, the name of the .Done file must be provided. The .done file serves as a signal that the file is ready for processing and must be located in the same folder as the file to be processed.
Content Change: Monitors changes in the file length/modification timestamp to determine if the write operation on the file is complete and the file is ready to be read. With this option selected, the system waits for at least one second until there are no more file changes.
While this method can detect some changes, it is unreliable when the file is updated slowly, as the system may miss changes if they don’t occur within the fixed time. This limitation makes it unsuitable for scenarios where files are being written gradually or slowly.
Rename: This strategy renames the file on the SFTP server before reading it.
None: This picks up the file as soon as it pools. There is a lot of chance that a incomplete or half-baked file might be picked up.
Now to avoid picking up of half-baked / incomplete file in PI/PO (7.5) file adapter there was below option:
However, with ‘Content Change’ Read lock strategy option of BTP – CI Sftp adapter possess various disadvantages
Unlike PI you can’t fix how much time to wait, and it normally waits for a second, which can result in incomplete files being picked in case of large files written via slow network.It also doesn’t pick zero length file, if that is required due to business requirement.
See – https://me.sap.com/notes/3347842/E
Alternatives to consider:
Write done file: if the sender application can write any .done or any other Done file. The important point to remember is the done file will be deleted on successful completion of iflow.Timing the sftp adapter to pick file in certain time slots which is different from time slot used by source to write file. Interval in Time slot between source writing a file and Sftp adapter picking the file will mostly ensure no incomplete files are picked.
There might be other suitable alternatives which has not been discussed here. Kindly comment below.
Here is another fine blog which explains things in detail:
Read Lock strategy to consider while moving sftp / file adapter with – “Msecs to wait before Modification check” while migrating to BTP – Cloud Integration.SAP Process Integration (PI) offers a “File Modification Check” that enables adapters to delay file processing for a defined period, ensuring only complete files are handled. A comparable feature in SAP Business Technology Platform (BTP) Cloud Integration would significantly enhance file-based integration. While many file transfer scenarios necessitate source system handling to guarantee fully written files before processing, the SFTP adapter currently lacks a built-in option to specify such a waiting period.SAP BTP Cloud Integration offers several options for implementing a read lock strategy with the SFTP adapter These Includes:Done File Expected: This strategy requires a “done file” to signal when a file is ready for processing. If this option is selected, the name of the .Done file must be provided. The .done file serves as a signal that the file is ready for processing and must be located in the same folder as the file to be processed. Content Change: Monitors changes in the file length/modification timestamp to determine if the write operation on the file is complete and the file is ready to be read. With this option selected, the system waits for at least one second until there are no more file changes.While this method can detect some changes, it is unreliable when the file is updated slowly, as the system may miss changes if they don’t occur within the fixed time. This limitation makes it unsuitable for scenarios where files are being written gradually or slowly.Rename: This strategy renames the file on the SFTP server before reading it.None: This picks up the file as soon as it pools. There is a lot of chance that a incomplete or half-baked file might be picked up. Now to avoid picking up of half-baked / incomplete file in PI/PO (7.5) file adapter there was below option: However, with ‘Content Change’ Read lock strategy option of BTP – CI Sftp adapter possess various disadvantagesUnlike PI you can’t fix how much time to wait, and it normally waits for a second, which can result in incomplete files being picked in case of large files written via slow network.It also doesn’t pick zero length file, if that is required due to business requirement.See – https://me.sap.com/notes/3347842/E Alternatives to consider:Write done file: if the sender application can write any .done or any other Done file. The important point to remember is the done file will be deleted on successful completion of iflow.Timing the sftp adapter to pick file in certain time slots which is different from time slot used by source to write file. Interval in Time slot between source writing a file and Sftp adapter picking the file will mostly ensure no incomplete files are picked.There might be other suitable alternatives which has not been discussed here. Kindly comment below.Here is another fine blog which explains things in detail:https://community.sap.com/t5/technology-blogs-by-members/read-lock-strategy-and-post-processing-options-in-sftp-sender-adapter-of/ba-p/13552916 Read More Technology Blogs by SAP articles
#SAP
#SAPTechnologyblog