With Datasphere release 2025.11, SAP introduced a powerful new feature: the Generic SFTP Connection. This enhancement significantly simplifies and automates the process of loading flat files into Datasphere.
Traditionally, users had to manually upload each flat file, which was cumbersome especially when dealing with multiple files. Now, with Generic SFTP, the process can be fully automated using scripting (e.g., Python), resulting in faster and more efficient bulk data ingestion.
This blog walks you through a basic method to leverage the Generic SFTP connection as a source object in a Data Flow. File formats currently supported include:JSON, JSONL, CSV, XLS, XLSX, ORC, and PARQUET.
Key Benefits
Automates repetitive flat file loadsSupports bulk data ingestionCan be orchestrated using SAP Integration Suite or custom scripts (e.g., Python)
Prerequisites
Valid Generic SFTP ConnectionSource File Availability
Steps to Load Data from Generic SFTP
First, you need to create a Data Flow to handle how the data will be ingested when transferred to generic SFTP
Go to the “Sources” tab in the Data Builder. Expand the “Connections” folder, navigate to the SFTP connection, and select the folder containing the source file.
Drag and drop the file onto the canvas.(Optional) Click “Modify” to edit columns if needed. You can rename, remove, or change data types.
Click the source node and choose “Add Table” to create a new target table.Click “Create and Deploy Table”.Choose the appropriate load Mode:Append – Add new records to existing dataTruncate – Clear existing data before loadingDelete – Remove existing records based on matching keys before loadingRun the Data Flow to load data into the target table.
Example Use Case: Automating Product Image Integration
Our organization leveraged the Generic SFTP Connections feature in SAP Datasphere to automate the integration of product image data into Datasphere and SAP Analytics Cloud (SAC). Here’s how the solution was implemented:
A third-party system generates a flat file containing Product IDs and corresponding image URLs every time a new product is added.
For the initial load, we used a Python script to transfer this file to the SFTP server. A Dataflow configured in Append mode then loaded the file into a target table in Datasphere.
To incorporate the image URLs into the data model, we extended the Product Master Data dimension in Datasphere by adding a new attribute for the image URL. The semantic type of this attribute was set to “Image URL”.
This enhancement allowed seamless integration into existing SAC dashboards, enabling dynamic display of product images.
For ongoing daily updates, we scheduled the Dataflow to run automatically. SAP Integration Suite transfers the updated file to the same SFTP folder each day, using a consistent file name to support full automation.
End result
# Generic script to transfer file to SFTP Server
# Be sure to change the source file and destination folder name
import paramiko
import os
# === SFTP connection details ===
hostname = <enter SFTP host>
port = 22 # Usually 22 for SFTP
username = <enter user name>
password = <Enter password>
# === Local file and remote folder ===
local_file_path = < Source File name with Path>
remote_folder_path = <Target Destination> # No filename, just the folder
def upload_file_sftp():
try:
# Extract just the filename
filename = os.path.basename(local_file_path)
remote_file_path = os.path.join(remote_folder_path, filename).replace(‘\’, ‘/’)
# Connect to SFTP
transport = paramiko.Transport((hostname, port))
transport.banner_timeout = 10 # Prevent timeout errors
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
# Upload the file
print(f”Uploading {local_file_path} to {remote_file_path}”)
sftp.put(local_file_path, remote_file_path)
print(“Upload successful.”)
sftp.close()
transport.close()
except Exception as e:
print(f”Error occurred: {e}”)
if __name__ == “__main__”:
upload_file_sftp()
Conclusion
This new feature drastically streamlined data ingestion pipelines and reduced manual effort for teams managing frequent flat file loads. By combining Datasphere’s Generic SFTP connections with automation scripts, organizations can achieve fully managed and scalable data pipelines
With Datasphere release 2025.11, SAP introduced a powerful new feature: the Generic SFTP Connection. This enhancement significantly simplifies and automates the process of loading flat files into Datasphere.Traditionally, users had to manually upload each flat file, which was cumbersome especially when dealing with multiple files. Now, with Generic SFTP, the process can be fully automated using scripting (e.g., Python), resulting in faster and more efficient bulk data ingestion.This blog walks you through a basic method to leverage the Generic SFTP connection as a source object in a Data Flow. File formats currently supported include:JSON, JSONL, CSV, XLS, XLSX, ORC, and PARQUET.Key BenefitsAutomates repetitive flat file loadsSupports bulk data ingestionCan be orchestrated using SAP Integration Suite or custom scripts (e.g., Python)PrerequisitesValid Generic SFTP ConnectionSource File AvailabilitySteps to Load Data from Generic SFTPFirst, you need to create a Data Flow to handle how the data will be ingested when transferred to generic SFTPGo to the “Sources” tab in the Data Builder. Expand the “Connections” folder, navigate to the SFTP connection, and select the folder containing the source file. Drag and drop the file onto the canvas.(Optional) Click “Modify” to edit columns if needed. You can rename, remove, or change data types.Click the source node and choose “Add Table” to create a new target table.Click “Create and Deploy Table”.Choose the appropriate load Mode:Append – Add new records to existing dataTruncate – Clear existing data before loadingDelete – Remove existing records based on matching keys before loadingRun the Data Flow to load data into the target table.Example Use Case: Automating Product Image IntegrationOur organization leveraged the Generic SFTP Connections feature in SAP Datasphere to automate the integration of product image data into Datasphere and SAP Analytics Cloud (SAC). Here’s how the solution was implemented:A third-party system generates a flat file containing Product IDs and corresponding image URLs every time a new product is added.For the initial load, we used a Python script to transfer this file to the SFTP server. A Dataflow configured in Append mode then loaded the file into a target table in Datasphere.To incorporate the image URLs into the data model, we extended the Product Master Data dimension in Datasphere by adding a new attribute for the image URL. The semantic type of this attribute was set to “Image URL”.This enhancement allowed seamless integration into existing SAC dashboards, enabling dynamic display of product images.For ongoing daily updates, we scheduled the Dataflow to run automatically. SAP Integration Suite transfers the updated file to the same SFTP folder each day, using a consistent file name to support full automation.End result# Generic script to transfer file to SFTP Server
# Be sure to change the source file and destination folder name
import paramiko
import os
# === SFTP connection details ===
hostname = <enter SFTP host>
port = 22 # Usually 22 for SFTP
username = <enter user name>
password = <Enter password>
# === Local file and remote folder ===
local_file_path = < Source File name with Path>
remote_folder_path = <Target Destination> # No filename, just the folder
def upload_file_sftp():
try:
# Extract just the filename
filename = os.path.basename(local_file_path)
remote_file_path = os.path.join(remote_folder_path, filename).replace(‘\’, ‘/’)
# Connect to SFTP
transport = paramiko.Transport((hostname, port))
transport.banner_timeout = 10 # Prevent timeout errors
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
# Upload the file
print(f”Uploading {local_file_path} to {remote_file_path}”)
sftp.put(local_file_path, remote_file_path)
print(“Upload successful.”)
sftp.close()
transport.close()
except Exception as e:
print(f”Error occurred: {e}”)
if __name__ == “__main__”:
upload_file_sftp()ConclusionThis new feature drastically streamlined data ingestion pipelines and reduced manual effort for teams managing frequent flat file loads. By combining Datasphere’s Generic SFTP connections with automation scripts, organizations can achieve fully managed and scalable data pipelines Read More Technology Blog Posts by Members articles
#SAP
#SAPTechnologyblog
![From freeCodeCamp to NASA with Data Engineer Joe Hill [Podcast #178]](https://i4.ytimg.com/vi/cMOz7Tyq7aE/hqdefault.jpg)
