“One TCP Event Smart Adapter for multiple receiver types, security requirements and native Avro conversion to XML or JSON based on the Schema Registry“
Part I — The Journey & Architecture Overview
Bookmark this page to follow the entire journey.
A note on how this was built: This adapter was developed with the assistance of AI (Claude, Gemini, and ChatGPT). However, the architecture decisions, component design, integration patterns, and all production code were validated, tested, and signed off by me personally. The adapter is already running in a real SAP Integration Suite tenant with live Kafka traffic. AI accelerated the development velocity — the engineering judgment and validation are entirely human.
Built with: SAP Adapter Development Kit (ADK) 2.25.0 · Apache Camel 3.14.7 · Java 8 (compiled with JDK 17, –release 8) · IntelliJ IDEA · Apache Kafka Client 3.x.
Introduction
In the world of Event-Driven Architecture (EDA), real-time, high-volume data streams between SAP landscapes and Apache Kafka clusters have become a core business requirement. While the standard SAP Integration Suite connectivity offers great foundations, complex enterprise scenarios often demand advanced capabilities regarding data governance, custom resilience strategies, and dynamic security handshakes.
To support the community and bridge these gaps, I developed a fully custom, open-source enterprise-grade Kafka Adapter using the SAP Adapter Development Kit (ADK). The adapter is already fully operational for core processing requirements, basic validation loops, and dynamic Apache Avro conversions. However, because corporate infrastructure is deeply varied, it is currently undergoing an extensive evaluation matrix across multi-tenant security landscapes.
This article marks the beginning of a 7-part deep-dive series where we will explore every architectural decision, threading model, and implementation detail of this component.
Why Build a Custom Adapter ?
My journey toward this adapter did not start from zero. It began with two published architectural patterns for Kafka integration within SAP BTP Integration Suite — patterns that the SAP community embraced and that I am proud to have contributed:
SAP BTP – CI – Immutable Event Routing per Domain with Metadata-Driven Consumer OrchestrationSAP BTP – CI – Immutable Kafka Topic Routing Mechanism Using SDIA with ODCP for SAP CPI
While designing and validating those patterns in real enterprise environments, I began hitting the boundaries of what the standard SAP Kafka adapter was designed to do. These were not edge cases — they were recurring constraints that architects and developers face every day when trying to build serious event-driven integrations on SAP BTP.
The most frustrating part was not the limitations themselves. It was watching integration teams work around them by proliferating custom Groovy scripts and external libraries directly inside their iFlows just to deserialize an Avro payload, resolve a schema, or handle a partition assignment. That approach works — until it doesn’t. Maintenance becomes a nightmare, governance disappears, and the iFlow starts carrying responsibilities it was never designed to carry.
That is when I made the decision: move the complexity where it belongs — into the adapter layer — and give the SAP global community a proper open-source solution.
What the standard adapter typically lacks, and what this custom adapter addresses directly:
Granular offset management — NACK-equivalent behavior via seek(), ensuring no message is silently lost on processing failure regardless of where in the batch the error occurredStructured error visibility — purpose-built visual error blocks in the CPI monitoring layer, not raw unreadable stack traces that leave operators guessing at 2amA multi-profile security engine — UI that adapts dynamically based on the chosen authentication strategy: SASL Basic, Certificate-only mTLS, or OAuth 2.0 Advanced — all consolidated in a single Connection tab with zero ambiguityNative Avro conversion with Schema Registry governance — runtime deserialization of binary Avro payloads directly in the adapter, eliminating the need for Groovy deserializers inside iFlows. And yes, it has real-world limits: payload conversion supports up to 2 MB per record and Schema Registry wire headers up to 20 KB. Anything beyond those boundaries triggers a clear exception with an actionable error message — no silent failures, no data corruptionConfigurable resilience policies — Skip, Stop, Retry with exponential backoff, and Skip Oversized, each implementing the correct Kafka offset semantics so your consumer group never gets stuck on a poison pillOn-premise connectivity — native Cloud Connector support via SAP SCC Location ID, covering enterprise landscapes where Kafka clusters sit behind corporate network boundariesExplicit partition selection — direct partition assignment for use cases requiring deterministic consumption without relying on consumer group rebalancing
This is not a replacement for the SAP standard adapter for simple scenarios. It is an extension of it — built by someone who has spent real time in the trenches of SAP Kafka integration, for the engineers and architects who face the same walls I did.
Figure 1 — SDIA (Semantic Domain Integration Architecture): In this pattern, the Event Smart Kafka Adapter serves as the single inbound channel from a central Apache Kafka bus — which itself may receive events from TCP Kafka producers, AMQP bridges (RabbitMQ / ActiveMQ), MQTT brokers, or HTTPS webhooks upstream. The adapter handles topic subscription, protocol normalization, message validation, metadata extraction, and error handling before delivering clean events into SAP CPI. Inside the Integration Suite, the orchestration layer takes over: content enrichment, dynamic routing, message transformation, and domain-aware dispatch to SAP S/4HANA, SAP ECC, Web Services, Cloud Applications, Databases, and Object Storage — governed by ODCP/EDCP contracts and centralized security. License: Apache 2.0 | MIT — SDIA Framework: CC BY 4.0.
What to Expect: The 7-Part Deep-Dive Journey
To fully unpack the engineering behind an enterprise-grade SAP adapter, this initiative is structured into seven comprehensive technical chapters:
Part
Title
Focus
I ★
The Journey & Architecture Overview
Project vision, metadata configuration, visual logging frameworks, and open-source delivery roadmap. (This article)
II
Core Consumer Lifecycles & Polling Loops
Java runtime deep dive: how the Apache Camel polling consumer lifecycle maps to Kafka cluster subscription models.
III
The Multi-Profile Security Engine
Technical breakdown of SASL, mTLS, and OAuth 2.0 dynamic token integration using the SAP Secure Store.
IV
Multi-Threaded Parallel Consumers & Scale
Engineering for high-throughput. Parallel threading pools inside Cloud Integration worker nodes without JVM memory leaks.
V
Advanced Payload Conversion & Schema Registry Governance
Avro-to-JSON/XML serialization layer, caching strategies, and memory protection mechanisms.
VI
Designing the Producer (Receiver) Integration Role
Outbound flow: partition keys, acknowledgment levels (acks), and asynchronous message dispatching.
VII
Visual Operations, Error Interception & Production Sign-Off
Custom error UI rendering, transaction boundary management, and production go-live best practices.
Note: Source code repository, GitHub links, and downloadable .esa package will be published progressively alongside each milestone. Part I focuses on architecture and design decisions — code drops begin with Part II.
A First Look: The Adapter Running Live in SAP Integration Suite
Before diving into the configuration details, here is a quick spoiler — the adapter is not a concept or a prototype. It is already deployed and running in a real SAP BTP Integration Suite tenant, consuming live Kafka events.
Event Smart Kafka Adapter deployed as a custom adapter in SAP Integration Suite — iFlow started, Sender channel active, consuming live events from a Confluent Cloud Kafka topic.
No more Groovy deserializers inside your iFlow. The adapter resolves the Avro schema from an external Schema Registry at runtime and delivers clean JSON directly to the pipeline — conversion happens at the channel layer, before your integration logic even starts.
SAP Cloud Integration Operations Monitor showing live message processing — green checkmarks confirming successful Avro-to-JSON conversion and pipeline delivery.
The Dynamic Configuration UI
One of the most important aspects of an enterprise-grade adapter is its configuration experience. The adapter ships with a fully declarative metadata-driven UI built on the SAP ADK Component Metadata specification. Every tab, every field, and every visibility condition is defined in XML — no UI code required.
Connection Tab — Authentication Profiles
The Connection tab dynamically adapts based on the selected Authentication Method. Selecting one of three profiles completely changes the visible fields, ensuring operators are never confused by irrelevant configuration options.
Dynamic UI showing Kafka Cluster Bootstrap Servers table and SASL Basic configuration: TLS toggle, SASL Mechanism (PLAIN / SCRAM-SHA-256 / SCRAM-SHA-512), and Credential Name from SAP Secure Store.
Automated UI adaptation showing mutual TLS configuration: Certificate Alias from SAP Keystore and optional mTLS Client Key Alias for bidirectional certificate authentication.
Advanced token-based authentication layout: Token Endpoint URL, Client ID, Client Secret Alias (resolved from SAP Secure Store), and optional OAuth Scope — compatible with Confluent Cloud OAuth and AWS MSK IAM.
Connection Tab — Proxy Type selector drives the network routing strategy. Internet mode connects directly to cloud-hosted Kafka brokers. On-Premise mode routes traffic through SAP Cloud Connector (SCC) via Location ID, enabling hybrid enterprise topologies where Kafka clusters live behind corporate firewalls — no VPN, no firewall exceptions, no public broker exposure required.
Processing Tab — Stream Tuning & Resilience
The Processing tab exposes fine-grained Kafka consumer controls alongside a powerful Error Handling policy engine. The adapter supports four distinct resilience policies, each implementing the correct Kafka offset semantics:
Skip Failed Message — commit offset, continue processing. Record is intentionally discarded and the consumer advances to the next message without interruption.Stop on Error — do NOT commit offset (NACK equivalent via seek()), consumer halts immediately. The failed record is re-delivered after iFlow fix and redeploy.Retry Failed Message — exponential backoff (1s, 2s, 4s… capped at 30s), 5–10 configurable attempts. On exhaustion: NACK + consumer halt. No poison-pill infinite loops.Skip Oversized — when payload exceeds the configured size limit, bypass conversion and pass raw bytes with warning headers downstream instead of blocking the entire consumer.Partition Selection — explicit partition assignment for deterministic consumption. When configured, the consumer uses assign() instead of subscribe(), bypassing consumer group rebalancing entirely. Valid only for single-topic configurations.Group ID — custom consumer group identifier for coordinated consumption across iFlow instances. When left empty, the adapter derives a deterministic group ID from the broker host and topic combination — ensuring stable group membership across redeployments without manual configuration.Auto Offset Reset — controls consumer behavior when no committed offset exists for the group:earliest — start from the very first available message in the topic, consuming the full historylatest — start from the next new message only, ignoring historical recordsnone — throw an exception if no committed offset is found, forcing explicit offset management
Conversion & Conversion Advanced Tabs
The Conversion tab controls how Kafka record payloads are transformed before delivery to the iFlow pipeline. The adapter supports five inbound payload modes:
None (raw bytes) — payload delivered exactly as-is directly to the iFlow pipeline. Works transparently with any topic format: JSON, XML, plain text, or binary. No Schema Registry required, no conversion overhead. The simplest and most compatible mode.Avro → JSON — binary Avro payload deserialized at the adapter layer using an external Schema Registry. The adapter resolves the schema, decodes the binary record, and delivers clean structured JSON to the iFlow. No Groovy, no custom libraries, no deserialization logic inside your integration flow.Avro → XML — same runtime deserialization pipeline as Avro → JSON, but the output is delivered as XML — ready for SAP-native processing, IDoc mapping, or SOAP transformations without additional conversion steps.JSON Schema (pass-through) — payload is already valid JSON. The adapter sets the correct Content-Type: application/json header and schema mode metadata, ensuring downstream mapping steps receive properly typed content without touching the payload body.Protobuf → JSON (experimental) — planned for a future release. Currently passes raw bytes with a warning header. Full Protobuf deserialization support is on the roadmap.
Schema ID Resolution — how the adapter finds the right Avro schema:
When Avro conversion is enabled, the adapter must resolve the correct schema from the Schema Registry before decoding the payload. Two resolution strategies are supported:
Confluent Magic Byte — the schema ID is embedded directly in the Kafka record payload following the Confluent wire format: 1 magic byte (0x00) + 4-byte big-endian schema ID + Avro binary data. The adapter reads the first 5 bytes, extracts the schema ID automatically, and fetches the schema from the registry at runtime. This is the standard behavior when producers use the Confluent Avro Serializer.Fixed Schema ID — the schema ID is configured statically in the adapter channel. Requires three fields: Registry Host (the Schema Registry URL), Credential Name (alias from SAP Secure Store for registry authentication), and Schema ID (the integer identifier of the registered schema). Use this mode when payloads do not carry the Confluent wire format header, or when consuming from producers that do not use the standard Confluent serializer.
Conversion Tab with Avro → JSON selected, Schema ID Source set to Fixed Schema ID — Registry Host, Credential Name from SAP Secure Store, and static Schema ID configured. The dropdown also exposes the Confluent Magic Byte option for wire-format-aware producers.
Visual Operations Monitoring & Custom Error Handling
Enterprise adapters must not fail silently, nor should they spit unreadable, raw stack traces into the Cloud Integration web console. This implementation introduces a structured, high-visibility error block designed to pinpoint exactly where a stream disrupted — separating downstream processing issues from fatal schema serialization failures or Schema Registry connection timeouts — and explicitly lists actionable steps for operational teams.
Every error condition in the adapter is classified, enriched, and rendered as a structured visual block directly in the SAP Cloud Integration Operations Monitor. The block always contains:
The error category and root cause (Schema Registry 404, auth failure, payload too large, network timeout…)The exact Kafka topic, partition, and offset where the failure occurredThe Schema ID that was being resolved at the time of failureThe raw HTTP response body from the Schema Registry when applicableA step-by-step correction guide for the operations team
When something goes wrong, the adapter speaks clearly. Structured visual error block rendered directly in the CPI monitoring layer — root cause, exact Kafka offset, Schema Registry HTTP response, and step-by-step correction guide. No guesswork.
Every screenshot above was captured from a live tenant. The adapter handles real Kafka traffic, real Avro payloads, and real Schema Registry interactions. What you see in this series is not a lab exercise — it is production-grade engineering built for the SAP global community.
Open Source Delivery Strategy & Ecosystem Roadmap
This project is built for the community. The distribution strategy aims to balance ease-of-use with open-source compliance:
Complete Source Code Repository
The upcoming public repository will contain the complete Java source code, fully stripped of any proprietary SAP libraries or confidential JAR dependencies. Anyone who wants to clone, modify, or compile the adapter from scratch will simply need to add their licensed SAP ADK dependencies into the pom.xml.
Ready-to-Deploy Package (.ESA)
For integration developers who want a plug-and-play experience without compiling code, a fully assembled .esa (Enterprise Software Archive) package will be provided. Download it and deploy directly into your SAP Cloud Integration tenant in seconds — no build tools required.
Milestone
Target Deliverable
Focus Content & Release Scope
Late June – 2026
Core Architecture & Code Drops
Publication of Parts I, II, and III. Release of the baseline Consumer source code and the plug-and-play .esa package for public testing.
July – 2026
Enterprise Scaling & Security Modules
Publication of Parts IV, V, VI, and VII. Release of the advanced Producer code, OAuth 2.0 / mTLS validation modules, and the final visual operations integration framework.
The journey has just begun.
Community & Collaboration
Building this was a long journey, but testing it against the vast diversity of enterprise Kafka configurations is a collective task. I am releasing this as a living project. If you have unique requirements—be it specific Avro compatibility versions, niche security protocols, or complex multi-tenant partitioning needs—please reach out. Let’s make this the standard, robust, community-owned Kafka solution for SAP Integration Suite. Follow this series to save the project updates and join the community as a contributor.
Stay tuned for Part II, where we will dive straight into the code to examine the Consumer Lifecycles, Kafka Polling Loops, and Apache Camel Threading Models that make this adapter production-ready for enterprise SAP landscapes.
“This was built entirely outside of any corporate engagement — no vendor sponsorship, no SAP endorsement. Just an engineer who saw a gap and decided to fill it.“
Ricardo Viana
Independent Solo Researcher
Creator of DEIP · SDIA · GDCR · DDCR · ODCP · EDCP · DDCP
ORCID: 0009-0009-9549-5862
#SAPIntegrationSuite #Kafka #EventDrivenArchitecture #OpenSource #SAPCommunity #SAPADK #CloudIntegration #Avro #SchemaRegistry
“One TCP Event Smart Adapter for multiple receiver types, security requirements and native Avro conversion to XML or JSON based on the Schema Registry”Part I — The Journey & Architecture OverviewBookmark this page to follow the entire journey.A note on how this was built: This adapter was developed with the assistance of AI (Claude, Gemini, and ChatGPT). However, the architecture decisions, component design, integration patterns, and all production code were validated, tested, and signed off by me personally. The adapter is already running in a real SAP Integration Suite tenant with live Kafka traffic. AI accelerated the development velocity — the engineering judgment and validation are entirely human.Built with: SAP Adapter Development Kit (ADK) 2.25.0 · Apache Camel 3.14.7 · Java 8 (compiled with JDK 17, –release 8) · IntelliJ IDEA · Apache Kafka Client 3.x.IntroductionIn the world of Event-Driven Architecture (EDA), real-time, high-volume data streams between SAP landscapes and Apache Kafka clusters have become a core business requirement. While the standard SAP Integration Suite connectivity offers great foundations, complex enterprise scenarios often demand advanced capabilities regarding data governance, custom resilience strategies, and dynamic security handshakes.To support the community and bridge these gaps, I developed a fully custom, open-source enterprise-grade Kafka Adapter using the SAP Adapter Development Kit (ADK). The adapter is already fully operational for core processing requirements, basic validation loops, and dynamic Apache Avro conversions. However, because corporate infrastructure is deeply varied, it is currently undergoing an extensive evaluation matrix across multi-tenant security landscapes.This article marks the beginning of a 7-part deep-dive series where we will explore every architectural decision, threading model, and implementation detail of this component.Why Build a Custom Adapter ?My journey toward this adapter did not start from zero. It began with two published architectural patterns for Kafka integration within SAP BTP Integration Suite — patterns that the SAP community embraced and that I am proud to have contributed:SAP BTP – CI – Immutable Event Routing per Domain with Metadata-Driven Consumer OrchestrationSAP BTP – CI – Immutable Kafka Topic Routing Mechanism Using SDIA with ODCP for SAP CPIWhile designing and validating those patterns in real enterprise environments, I began hitting the boundaries of what the standard SAP Kafka adapter was designed to do. These were not edge cases — they were recurring constraints that architects and developers face every day when trying to build serious event-driven integrations on SAP BTP.The most frustrating part was not the limitations themselves. It was watching integration teams work around them by proliferating custom Groovy scripts and external libraries directly inside their iFlows just to deserialize an Avro payload, resolve a schema, or handle a partition assignment. That approach works — until it doesn’t. Maintenance becomes a nightmare, governance disappears, and the iFlow starts carrying responsibilities it was never designed to carry.That is when I made the decision: move the complexity where it belongs — into the adapter layer — and give the SAP global community a proper open-source solution.What the standard adapter typically lacks, and what this custom adapter addresses directly:Granular offset management — NACK-equivalent behavior via seek(), ensuring no message is silently lost on processing failure regardless of where in the batch the error occurredStructured error visibility — purpose-built visual error blocks in the CPI monitoring layer, not raw unreadable stack traces that leave operators guessing at 2amA multi-profile security engine — UI that adapts dynamically based on the chosen authentication strategy: SASL Basic, Certificate-only mTLS, or OAuth 2.0 Advanced — all consolidated in a single Connection tab with zero ambiguityNative Avro conversion with Schema Registry governance — runtime deserialization of binary Avro payloads directly in the adapter, eliminating the need for Groovy deserializers inside iFlows. And yes, it has real-world limits: payload conversion supports up to 2 MB per record and Schema Registry wire headers up to 20 KB. Anything beyond those boundaries triggers a clear exception with an actionable error message — no silent failures, no data corruptionConfigurable resilience policies — Skip, Stop, Retry with exponential backoff, and Skip Oversized, each implementing the correct Kafka offset semantics so your consumer group never gets stuck on a poison pillOn-premise connectivity — native Cloud Connector support via SAP SCC Location ID, covering enterprise landscapes where Kafka clusters sit behind corporate network boundariesExplicit partition selection — direct partition assignment for use cases requiring deterministic consumption without relying on consumer group rebalancingThis is not a replacement for the SAP standard adapter for simple scenarios. It is an extension of it — built by someone who has spent real time in the trenches of SAP Kafka integration, for the engineers and architects who face the same walls I did.Figure 1 — SDIA (Semantic Domain Integration Architecture): In this pattern, the Event Smart Kafka Adapter serves as the single inbound channel from a central Apache Kafka bus — which itself may receive events from TCP Kafka producers, AMQP bridges (RabbitMQ / ActiveMQ), MQTT brokers, or HTTPS webhooks upstream. The adapter handles topic subscription, protocol normalization, message validation, metadata extraction, and error handling before delivering clean events into SAP CPI. Inside the Integration Suite, the orchestration layer takes over: content enrichment, dynamic routing, message transformation, and domain-aware dispatch to SAP S/4HANA, SAP ECC, Web Services, Cloud Applications, Databases, and Object Storage — governed by ODCP/EDCP contracts and centralized security. License: Apache 2.0 | MIT — SDIA Framework: CC BY 4.0.What to Expect: The 7-Part Deep-Dive JourneyTo fully unpack the engineering behind an enterprise-grade SAP adapter, this initiative is structured into seven comprehensive technical chapters:PartTitleFocusI ★The Journey & Architecture OverviewProject vision, metadata configuration, visual logging frameworks, and open-source delivery roadmap. (This article)IICore Consumer Lifecycles & Polling LoopsJava runtime deep dive: how the Apache Camel polling consumer lifecycle maps to Kafka cluster subscription models.IIIThe Multi-Profile Security EngineTechnical breakdown of SASL, mTLS, and OAuth 2.0 dynamic token integration using the SAP Secure Store.IVMulti-Threaded Parallel Consumers & ScaleEngineering for high-throughput. Parallel threading pools inside Cloud Integration worker nodes without JVM memory leaks.VAdvanced Payload Conversion & Schema Registry GovernanceAvro-to-JSON/XML serialization layer, caching strategies, and memory protection mechanisms.VIDesigning the Producer (Receiver) Integration RoleOutbound flow: partition keys, acknowledgment levels (acks), and asynchronous message dispatching.VIIVisual Operations, Error Interception & Production Sign-OffCustom error UI rendering, transaction boundary management, and production go-live best practices.Note: Source code repository, GitHub links, and downloadable .esa package will be published progressively alongside each milestone. Part I focuses on architecture and design decisions — code drops begin with Part II.A First Look: The Adapter Running Live in SAP Integration SuiteBefore diving into the configuration details, here is a quick spoiler — the adapter is not a concept or a prototype. It is already deployed and running in a real SAP BTP Integration Suite tenant, consuming live Kafka events.Event Smart Kafka Adapter deployed as a custom adapter in SAP Integration Suite — iFlow started, Sender channel active, consuming live events from a Confluent Cloud Kafka topic.No more Groovy deserializers inside your iFlow. The adapter resolves the Avro schema from an external Schema Registry at runtime and delivers clean JSON directly to the pipeline — conversion happens at the channel layer, before your integration logic even starts.SAP Cloud Integration Operations Monitor showing live message processing — green checkmarks confirming successful Avro-to-JSON conversion and pipeline delivery.The Dynamic Configuration UIOne of the most important aspects of an enterprise-grade adapter is its configuration experience. The adapter ships with a fully declarative metadata-driven UI built on the SAP ADK Component Metadata specification. Every tab, every field, and every visibility condition is defined in XML — no UI code required.Connection Tab — Authentication ProfilesThe Connection tab dynamically adapts based on the selected Authentication Method. Selecting one of three profiles completely changes the visible fields, ensuring operators are never confused by irrelevant configuration options.Dynamic UI showing Kafka Cluster Bootstrap Servers table and SASL Basic configuration: TLS toggle, SASL Mechanism (PLAIN / SCRAM-SHA-256 / SCRAM-SHA-512), and Credential Name from SAP Secure Store. Automated UI adaptation showing mutual TLS configuration: Certificate Alias from SAP Keystore and optional mTLS Client Key Alias for bidirectional certificate authentication. Advanced token-based authentication layout: Token Endpoint URL, Client ID, Client Secret Alias (resolved from SAP Secure Store), and optional OAuth Scope — compatible with Confluent Cloud OAuth and AWS MSK IAM. Connection Tab — Proxy Type selector drives the network routing strategy. Internet mode connects directly to cloud-hosted Kafka brokers. On-Premise mode routes traffic through SAP Cloud Connector (SCC) via Location ID, enabling hybrid enterprise topologies where Kafka clusters live behind corporate firewalls — no VPN, no firewall exceptions, no public broker exposure required. Processing Tab — Stream Tuning & ResilienceThe Processing tab exposes fine-grained Kafka consumer controls alongside a powerful Error Handling policy engine. The adapter supports four distinct resilience policies, each implementing the correct Kafka offset semantics:Skip Failed Message — commit offset, continue processing. Record is intentionally discarded and the consumer advances to the next message without interruption.Stop on Error — do NOT commit offset (NACK equivalent via seek()), consumer halts immediately. The failed record is re-delivered after iFlow fix and redeploy.Retry Failed Message — exponential backoff (1s, 2s, 4s… capped at 30s), 5–10 configurable attempts. On exhaustion: NACK + consumer halt. No poison-pill infinite loops.Skip Oversized — when payload exceeds the configured size limit, bypass conversion and pass raw bytes with warning headers downstream instead of blocking the entire consumer.Partition Selection — explicit partition assignment for deterministic consumption. When configured, the consumer uses assign() instead of subscribe(), bypassing consumer group rebalancing entirely. Valid only for single-topic configurations.Group ID — custom consumer group identifier for coordinated consumption across iFlow instances. When left empty, the adapter derives a deterministic group ID from the broker host and topic combination — ensuring stable group membership across redeployments without manual configuration.Auto Offset Reset — controls consumer behavior when no committed offset exists for the group:earliest — start from the very first available message in the topic, consuming the full historylatest — start from the next new message only, ignoring historical recordsnone — throw an exception if no committed offset is found, forcing explicit offset management Conversion & Conversion Advanced TabsThe Conversion tab controls how Kafka record payloads are transformed before delivery to the iFlow pipeline. The adapter supports five inbound payload modes:None (raw bytes) — payload delivered exactly as-is directly to the iFlow pipeline. Works transparently with any topic format: JSON, XML, plain text, or binary. No Schema Registry required, no conversion overhead. The simplest and most compatible mode.Avro → JSON — binary Avro payload deserialized at the adapter layer using an external Schema Registry. The adapter resolves the schema, decodes the binary record, and delivers clean structured JSON to the iFlow. No Groovy, no custom libraries, no deserialization logic inside your integration flow.Avro → XML — same runtime deserialization pipeline as Avro → JSON, but the output is delivered as XML — ready for SAP-native processing, IDoc mapping, or SOAP transformations without additional conversion steps.JSON Schema (pass-through) — payload is already valid JSON. The adapter sets the correct Content-Type: application/json header and schema mode metadata, ensuring downstream mapping steps receive properly typed content without touching the payload body.Protobuf → JSON (experimental) — planned for a future release. Currently passes raw bytes with a warning header. Full Protobuf deserialization support is on the roadmap. Schema ID Resolution — how the adapter finds the right Avro schema:When Avro conversion is enabled, the adapter must resolve the correct schema from the Schema Registry before decoding the payload. Two resolution strategies are supported:Confluent Magic Byte — the schema ID is embedded directly in the Kafka record payload following the Confluent wire format: 1 magic byte (0x00) + 4-byte big-endian schema ID + Avro binary data. The adapter reads the first 5 bytes, extracts the schema ID automatically, and fetches the schema from the registry at runtime. This is the standard behavior when producers use the Confluent Avro Serializer.Fixed Schema ID — the schema ID is configured statically in the adapter channel. Requires three fields: Registry Host (the Schema Registry URL), Credential Name (alias from SAP Secure Store for registry authentication), and Schema ID (the integer identifier of the registered schema). Use this mode when payloads do not carry the Confluent wire format header, or when consuming from producers that do not use the standard Confluent serializer.Conversion Tab with Avro → JSON selected, Schema ID Source set to Fixed Schema ID — Registry Host, Credential Name from SAP Secure Store, and static Schema ID configured. The dropdown also exposes the Confluent Magic Byte option for wire-format-aware producers.Visual Operations Monitoring & Custom Error HandlingEnterprise adapters must not fail silently, nor should they spit unreadable, raw stack traces into the Cloud Integration web console. This implementation introduces a structured, high-visibility error block designed to pinpoint exactly where a stream disrupted — separating downstream processing issues from fatal schema serialization failures or Schema Registry connection timeouts — and explicitly lists actionable steps for operational teams.Every error condition in the adapter is classified, enriched, and rendered as a structured visual block directly in the SAP Cloud Integration Operations Monitor. The block always contains:The error category and root cause (Schema Registry 404, auth failure, payload too large, network timeout…)The exact Kafka topic, partition, and offset where the failure occurredThe Schema ID that was being resolved at the time of failureThe raw HTTP response body from the Schema Registry when applicableA step-by-step correction guide for the operations teamWhen something goes wrong, the adapter speaks clearly. Structured visual error block rendered directly in the CPI monitoring layer — root cause, exact Kafka offset, Schema Registry HTTP response, and step-by-step correction guide. No guesswork.Every screenshot above was captured from a live tenant. The adapter handles real Kafka traffic, real Avro payloads, and real Schema Registry interactions. What you see in this series is not a lab exercise — it is production-grade engineering built for the SAP global community. Open Source Delivery Strategy & Ecosystem RoadmapThis project is built for the community. The distribution strategy aims to balance ease-of-use with open-source compliance:Complete Source Code RepositoryThe upcoming public repository will contain the complete Java source code, fully stripped of any proprietary SAP libraries or confidential JAR dependencies. Anyone who wants to clone, modify, or compile the adapter from scratch will simply need to add their licensed SAP ADK dependencies into the pom.xml.Ready-to-Deploy Package (.ESA)For integration developers who want a plug-and-play experience without compiling code, a fully assembled .esa (Enterprise Software Archive) package will be provided. Download it and deploy directly into your SAP Cloud Integration tenant in seconds — no build tools required.MilestoneTarget DeliverableFocus Content & Release ScopeLate June – 2026Core Architecture & Code DropsPublication of Parts I, II, and III. Release of the baseline Consumer source code and the plug-and-play .esa package for public testing.July – 2026Enterprise Scaling & Security ModulesPublication of Parts IV, V, VI, and VII. Release of the advanced Producer code, OAuth 2.0 / mTLS validation modules, and the final visual operations integration framework.The journey has just begun.Community & CollaborationBuilding this was a long journey, but testing it against the vast diversity of enterprise Kafka configurations is a collective task. I am releasing this as a living project. If you have unique requirements—be it specific Avro compatibility versions, niche security protocols, or complex multi-tenant partitioning needs—please reach out. Let’s make this the standard, robust, community-owned Kafka solution for SAP Integration Suite. Follow this series to save the project updates and join the community as a contributor.Stay tuned for Part II, where we will dive straight into the code to examine the Consumer Lifecycles, Kafka Polling Loops, and Apache Camel Threading Models that make this adapter production-ready for enterprise SAP landscapes.”This was built entirely outside of any corporate engagement — no vendor sponsorship, no SAP endorsement. Just an engineer who saw a gap and decided to fill it.”Ricardo VianaIndependent Solo ResearcherCreator of DEIP · SDIA · GDCR · DDCR · ODCP · EDCP · DDCPORCID: 0009-0009-9549-5862#SAPIntegrationSuite #Kafka #EventDrivenArchitecture #OpenSource #SAPCommunity #SAPADK #CloudIntegration #Avro #SchemaRegistry Read More Technology Blog Posts by Members articles
#SAP
#SAPTechnologyblog
