Why This Guide Matters
If you’re a partner exploring the concept of LLM routing through SAP AI Core, a common question is how foundation model requests should be routed and what integration options are available. This article provides an overview of the available approaches and key considerations when getting started.
The content is intended as architectural guidance and reference material for solution architects and development teams.
What Is LLM Routing?
LLM routing describes the path taken by prompts and responses between an application and a foundation model.
Direct Provider Access
Routing Through SAP AI Core
Request Flow
Partner Application → Model Provider
Partner Application → SAP AI Core / Generative AI Hub → Foundation Models
Integration Approach
Direct integration with the provider’s APIs and services.
Integration through SAP AI Core and the Generative AI Hub
Provider Management
Application teams manage provider-specific integrations. Typically requires separate integrations per provider.
SAP AI Core provides a centralized access layer for supported providers. Provides a consistent approach for accessing supported foundation models.
Getting Started with LLM Routing
Prerequisites
Before implementing LLM routing through SAP AI Core, ensure the following are available:
SAP BTP Global AccountSAP BTP SubaccountSAP AI Core (Extended Plan)Generative AI Hub enabled
High-Level Reference Architecture
This reference architecture illustrates how applications can securely and efficiently access generative AI capabilities, providing a centralized approach to consuming foundation models while supporting trust, governance, and operational control.
Choosing an Integration Approach
When implementing LLM routing through SAP AI Core, partners can choose from two primary integration patterns for consuming foundation models. While each approach serves a different purpose, both leverage SAP AI Core as the access layer for foundation model interactions and can be combined to meet specific solution requirements.
Option A: SAP Cloud SDK for AI
For applications primarily focused on consuming foundation models, the SAP Cloud SDK for AI offers a simple and consistent integration experience. By abstracting provider-specific APIs and providing support for Java, JavaScript, and Python, it enables developers to interact with supported foundation models through a unified interface, helping reduce implementation effort while accelerating development.
Learn More:
SAP Cloud SDK for AISAP AI SDK GitHub Repository
Option B: SAP AI Core Orchestration Service
The SAP AI Core Orchestration Service is designed for scenarios where interactions with foundation models require additional processing beyond basic prompt execution. It extends model access with capabilities such as prompt templating, grounding, content filtering, and data masking, helping standardize and govern AI interactions across applications. By centralizing these functions, the service can reduce repetitive implementation effort while providing greater consistency and control over how prompts and responses are handled.
Learn More:
Consumption of GenAI Models Using OrchestrationGenerative AI with SAP AI Core Learning SeriesOrchestration Chat Completion
Validating LLM Routing
Teams often want to confirm that requests are being routed through SAP AI Core.
Examples of validation approaches include:
Infrastructure Validation
SAP AI Core instance provisioned Extended Plan enabled
Configuration Validation
AI Core service bindings AI Core credentials Generative AI Hub configuration
Implementation Validation
SAP Cloud SDK for AI usage SAP AI Core Orchestration Service usage
Runtime Validation
Application logs SAP AI Launchpad screenshots Generative AI Hub screenshots Network traces showing SAP AI Core endpoints
Additional References
Set Up Generative AI Hub in SAP AI Core SAP AI Core Documentation SAP Note 3437766 – List of Available Models
Final Thoughts
Getting started with LLM routing through SAP AI Core can be simpler than it may first appear. With flexible integration patterns and access to a broad range of foundation models, partners can adopt the capabilities that best support their AI-powered scenarios.
Why This Guide MattersIf you’re a partner exploring the concept of LLM routing through SAP AI Core, a common question is how foundation model requests should be routed and what integration options are available. This article provides an overview of the available approaches and key considerations when getting started.The content is intended as architectural guidance and reference material for solution architects and development teams.What Is LLM Routing?LLM routing describes the path taken by prompts and responses between an application and a foundation model. Direct Provider Access Routing Through SAP AI CoreRequest FlowPartner Application → Model ProviderPartner Application → SAP AI Core / Generative AI Hub → Foundation ModelsIntegration ApproachDirect integration with the provider’s APIs and services.Integration through SAP AI Core and the Generative AI Hub Provider ManagementApplication teams manage provider-specific integrations. Typically requires separate integrations per provider. SAP AI Core provides a centralized access layer for supported providers. Provides a consistent approach for accessing supported foundation models.Getting Started with LLM RoutingPrerequisitesBefore implementing LLM routing through SAP AI Core, ensure the following are available:SAP BTP Global AccountSAP BTP SubaccountSAP AI Core (Extended Plan)Generative AI Hub enabled High-Level Reference ArchitectureThis reference architecture illustrates how applications can securely and efficiently access generative AI capabilities, providing a centralized approach to consuming foundation models while supporting trust, governance, and operational control. Choosing an Integration ApproachWhen implementing LLM routing through SAP AI Core, partners can choose from two primary integration patterns for consuming foundation models. While each approach serves a different purpose, both leverage SAP AI Core as the access layer for foundation model interactions and can be combined to meet specific solution requirements.Option A: SAP Cloud SDK for AIFor applications primarily focused on consuming foundation models, the SAP Cloud SDK for AI offers a simple and consistent integration experience. By abstracting provider-specific APIs and providing support for Java, JavaScript, and Python, it enables developers to interact with supported foundation models through a unified interface, helping reduce implementation effort while accelerating development.Learn More:SAP Cloud SDK for AISAP AI SDK GitHub RepositoryOption B: SAP AI Core Orchestration ServiceThe SAP AI Core Orchestration Service is designed for scenarios where interactions with foundation models require additional processing beyond basic prompt execution. It extends model access with capabilities such as prompt templating, grounding, content filtering, and data masking, helping standardize and govern AI interactions across applications. By centralizing these functions, the service can reduce repetitive implementation effort while providing greater consistency and control over how prompts and responses are handled.Learn More:Consumption of GenAI Models Using OrchestrationGenerative AI with SAP AI Core Learning SeriesOrchestration Chat Completion Validating LLM RoutingTeams often want to confirm that requests are being routed through SAP AI Core.Examples of validation approaches include:Infrastructure Validation SAP AI Core instance provisioned Extended Plan enabledConfiguration Validation AI Core service bindings AI Core credentials Generative AI Hub configurationImplementation Validation SAP Cloud SDK for AI usage SAP AI Core Orchestration Service usage Runtime Validation Application logs SAP AI Launchpad screenshots Generative AI Hub screenshots Network traces showing SAP AI Core endpointsAdditional References Set Up Generative AI Hub in SAP AI Core SAP AI Core Documentation SAP Note 3437766 – List of Available ModelsFinal ThoughtsGetting started with LLM routing through SAP AI Core can be simpler than it may first appear. With flexible integration patterns and access to a broad range of foundation models, partners can adopt the capabilities that best support their AI-powered scenarios. Read More Technology Blog Posts by SAP articles
#SAP
#SAPTechnologyblog
