In this article, we will explore how to build an agentic AI system using SAP’s Generative AI Hub. We’ll start by demonstrating how to orchestrate interactions with large language models (LLMs) through a lightweight setup and gradually evolve the architecture into a robust, tool-augmented agent capable of reasoning and acting autonomously.
While SAP Joule Studio offers a powerful, low-code way to infuse agentic capabilities into SAP applications, there are scenarios where building a custom agentic solution is an appropriate alternative. Whether you’re working with ECC or legacy systems, seeking deeper integration with external tools, or aiming for complete development flexibility, SAP GenAI Hub provides the necessary orchestration tools to bring these use cases to life.
What Are Agentic AI Systems?
Agentic AI systems represent an evolution in how we design and deploy language model-based applications. Rather than simply responding to isolated prompts, these systems are built to reason, plan, and act autonomously navigating multi-step tasks, invoking tools, making decisions, and adapting based on context.
At their core, Agentic AI systems treat the model as more than a text generator. The model is empowered to analyze user inputs, determine whether external actions are required (such as calling APIs or querying data), and execute those actions before synthesizing a final response. This enables a more dynamic and interactive experience, especially valuable in real-world use cases that involve external data sources, business logic, or multi-modal interactions.
Building effective agents requires careful orchestration: structuring prompts, managing context, defining tool schemas, and interpreting intermediate results. These systems often rely on a reason-and-act (ReACT) loop, where the model evaluates a task, decides on next steps, performs actions, and iteratively builds toward a complete, informed answer.
This approach opens the door to powerful enterprise use cases from intelligent assistants and copilots to automated decision support systems. In the next sections, we’ll explore how to implement such systems with the help of SAP Generative AI, combining structured orchestration with dynamic model capabilities to build robust, tool-aware agents. See more in The Agentic Evolution: From Chatbots to Multi-Agent Systems.
Designing the AI Workflow to Solve Problems
At the core of an agentic AI system is a purposeful composition of AI workflow patterns, tailored to address complex, real-world tasks. In our approach, we implement a custom hybrid orchestration, where the language model (LLM) acts as the central orchestrator interpreting user intent, planning the appropriate sequence of actions, and dynamically delegating tasks to specialized tools. The LLM not only selects the right tools but also provides the necessary parameters, ensuring each component operates in context and contributes to a coherent, intelligent response. This modular and dynamic interaction enables the system to move beyond static responses and deliver intelligent, real-world outcomes grounded in reasoning and action.
Step-by-Step Through the Agentic AI Workflow
Prompt → LLM (Orchestrator)
The process starts with a user prompt. The LLM acts as an orchestrator, analyzing the input and deciding dynamically which tools (workers) are necessary to fulfill the request. Instead of following a fixed sequence, the orchestrator tailors the plan in real-time and outputs a structured representation (e.g., a JSON object with tool calls).
LLM → Tools (Workers)
Based on the structured plan, the orchestrator routes the task to specific tools:
Tool 1: Get Weather – fetches current weather data.
Tool 2: Get Time – returns real-time clock information.
Tool 3: Retriever (RAG) – performs document retrieval and synthesis for knowledge-intensive queries.
Each of these tools is triggered in parallel or sequence, depending on the user request. Importantly, the tools are not hardcoded. They’re selected on the fly based on the LLM’s reasoning.
Tool Results → Aggregation
Once tools return their outputs, the system aggregates (append) the responses where raw outputs are collected and prepared for the final LLM synthesis.
Aggregation → LLM (Orchestrator)
With all results in hand, the LLM performs a second pass, now with complete context. It reflects on the tool results and generates a natural-language response tailored to the user’s original query.
LLM → Response → User
The final response is sent back to the user now grounded in external data and enhanced with reasoning.
Limitations of Static Prompting and the Case for Agentic AI
Even state-of-the-art models like GPT-4o have limitations when used in isolation. For instance, asking for the current time or the weather yields generic or outdated responses. These are not model flaws but consequences of static prompting where models operating without access to real-time data or external APIs.
There are a few common approaches to mitigate these limitations:
Prompt Engineering: Carefully crafting prompts can improve performance, but it doesn’t give the model access to new or real-time data.
Retrieval-Augmented Generation (RAG): Augments model outputs with real-time or contextual information by retrieving relevant documents.
Fine-Tuning: Adapts the model to specific domains or tasks by updating its weights based on domain-specific examples.
While each technique is powerful in its own right, they still operate under a relatively passive paradigm: the model reacts based on given inputs. To take the next step, we need to build systems where the model can actively reason over tasks, decide when to use tools, and dynamically incorporate real-world data into its responses.
This is the foundation of an Agentic AI System 
.
Agentic AI systems overcome traditional boundaries by enabling LLMs to take initiative in task planning and execution, making them ideal for enterprise environments that demand flexibility, real-time context, and intelligent decision-making.
Let’s explore a few examples to see how an LLM (GPT-4o) responds to different types of questions.
from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.service import OrchestrationService
llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 256, “temperature”: 0.2})
config = OrchestrationConfig(
template=Template(
messages=[
SystemMessage(“You are a helpful AI Assistent that answer the best you can all the time.”),
UserMessage(“{{?question}}”),
]
),
llm=llm
)
orchestration_service = OrchestrationService(config=config)
def send_request(config, **kwargs):
template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
answer = orchestration_service.run(config=config, template_values=template_values)
return answer.module_results.llm.choices[0].message.content
result = send_request(config, question = “What time is it?”)
print(result)
result = send_request(config, question = “What is SAP Business Data Cloud?”)
print(result)
result = send_request(config, question = “How’s the weather in Paris”)
print(result)
This is not a shortcoming of the model itself, but rather a limitation of static prompting: the model operates purely on its pre-trained knowledge without access to external tools, APIs, or dynamic environments.
To overcome these constraints, we need to enhance the model with capabilities that allow it to reason over tasks, determine when and how to invoke external tools, and dynamically incorporate real-world data into its responses. In the next section, we’ll explore how to address these limitations using SAP’s Generative AI capabilities, specifically by integrating SAP AI Core, SAP GenAI Hub, and orchestration services.
We’ll build a system where the model goes beyond answering questions. It will reason, make decisions, delegate tasks, and deliver responses enriched with real-time data and contextual awareness.
Starting Simple: Calling a Language Model via SAP GenAI Hub Orchestration
Before we build a full agentic AI system, let’s start by recalling how to perform a basic LLM completion using SAP GenAI Hub’s orchestration layer. This initial setup shows how to connect to a model like gpt-4o, define a prompt, and send dynamic user inputs for inference.
from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.service import OrchestrationService
llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 256, “temperature”: 0.2})
config = OrchestrationConfig(
template=Template(
messages=[
SystemMessage(“You are a helpful AI Assistent that answer the best you can all the time.”),
UserMessage(“{{?question}}”),
]
),
llm=llm
)
orchestration_service = OrchestrationService(config=config)
def send_request(config, **kwargs):
template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
answer = orchestration_service.run(config=config, template_values=template_values)
return answer.module_results.llm.choices[0].message.content
result = send_request(config, question = “How’s the weather in Paris”)
print(result)
In the example above, we import the necessary components from the GenAI Hub SDK: LLM, OrchestrationConfig, and OrchestrationService. We define the model with a few parameters (such as max_tokens and temperature), then configure a simple two-message prompt using SystemMessage and UserMessage.
Once configured, we instantiate the orchestration service and send a question like “How’s the weather in Paris?” using the send_request() function. As expected, the model replies that it cannot access real-time information. That’s because at this stage, the model runs standalone, but it’s not yet augmented with external tools.
However, when asked a knowledge-based question such as “What is SAP Datasphere?”, the model generates a rich, multi-paragraph answer entirely from its internal context, demonstrating that even without tools, the model already delivers high value for static or informational prompts.
This simple orchestration flow forms the foundation of what we’ll build on next: enhancing this architecture with tool invocation, decision-making, and dynamic control to evolve it into a full Agentic AI System.
Designing an Agentic AI System with Tool Support
Now that we’ve covered how to perform basic LLM completions using SAP GenAI Hub orchestration, it’s time to evolve this setup into a fully agentic AI system.
Agentic systems go beyond static responses, they can reason about tasks, make decisions, and use external capabilities to produce more accurate, contextual, and timely outputs. To enable this, we’ll augment our model with tools that it can call when needed.
In this next step, we’ll provide the agent with three tools:
get_time_now: retrieves the current local time.
get_weather: fetches real-time weather data for a given location.
retriever: performs document retrieval using a RAG (Retrieval-Augmented Generation) approach, ideal for enterprise knowledge queries.
These tools represent external abilities the model doesn’t natively possess like accessing real-time data or searching enterprise documentation. Through a structured schema, we’ll teach the agent how to decide when to use these tools and when to rely on its own internal knowledge.
Let’s begin implementing the agent logic.
Exploring the Core Functions of Our Agentic AI Backend
In this section, we’ll break down three key backend functions that power an intelligent, tool-using AI assistant. Each function addresses a different type of user request checking the time, fetching real-world weather, and retrieving knowledge from an enterprise document store using RAG (Retrieval-Augmented Generation).
get_time_now(): Returning the Current Local Time
def get_time_now():
“””Returns the current local time as a formatted string.”””
return {“time”: datetime.now().strftime(“%Y-%m-%d %H:%M:%S”)}
This simple utility function returns the current local timestamp formatted as a human-readable string (e.g., “2025-04-16 16:30:00”). While it may seem basic, access to the current time is essential in many agentic workflows, such as generating time-sensitive responses, scheduling actions, timestamping events, or handling context-aware tasks that depend on temporal logic.
Objective: It enables the assistant to ground its responses in real-time context.Output: A dictionary containing the current time
get_weather(latitude, longitude): Live Weather from Open-Meteo API
def get_weather(latitude, longitude):
“””This is a publically available API that returns the weather for a given location.”””
response = requests.get(
f”https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m”
)
data = response.json()
return data[“current”]
This function calls the Open-Meteo public API and returns current weather conditions for any given pair of latitude and longitude coordinates. This capability might play a key role in scenarios where environmental context matters, such as travel planning, logistics, energy consumption forecasting, or any decision-making process influenced by climate or local conditions. It fetches key metrics like:
Temperature (temperature_2m)Wind speed (wind_speed_10m)
All in real time.
Objective: Enables the assistant to respond to queries like “How’s the weather in Paris?” using live external data.Output: A dictionary with real-time weather readings (e.g., {“temperature_2m”: 18.2, “wind_speed_10m”: 5.6}).
retriever(question): Knowledge Retrieval with RAG over SAP HANA
def retriever(question: str):
embedding_model = init_embedding_model(‘text-embedding-ada-002’)
llm = init_llm(‘gpt-4o-mini’)
prompt_template = “””
Use the following context to answer the question at the end.
If the answer is not directly stated, try your best based on the context.
Only say you don’t know if the information is completely unavailable.
{context}
Question: {question}
“””
prompt = PromptTemplate(
template=prompt_template,
input_variables=[“context”, “question”]
)
db = HanaDB(
embedding=embedding_model,
connection=connection,
table_name=”EMBEDDINGS_COLLECTION_DATA”
)
retriever = db.as_retriever(search_kwargs={‘k’: 10})
qa = RetrievalQA.from_chain_type(
llm= llm,
retriever=retriever,
chain_type=”stuff”,
chain_type_kwargs={“prompt”: prompt}
)
response = qa.invoke({“query”: question})
return {“answer”: response}
This function powers a Retrieval-Augmented Generation (RAG) workflow that answers user questions based on enterprise knowledge stored in an SAP HANA Cloud vector store. The architecture is modular and flexible, both the embedding model (used for vector similarity search) and the language model (used for final synthesis) can be freely configured. This allows organizations to tailor the solution to their performance, cost, or data locality requirements, whether using open-source models or proprietary LLMs.
Here’s what it does step by step:
Initialize models: Embedding model (text-embedding-ada-002) for semantic search and LLM (gpt-4o-mini) to generate the final answer.Build the prompt: It defines a flexible prompt template that combines context + the user’s question.Search in SAP HANA: It uses a HanaDB vector store wrapper to retrieve the top 10 most relevant chunks using the user’s question.Run the RetrievalQA chain: Feeds retrieved content into the LLM to synthesize an accurate, grounded answer.Objective: This is the core engine for enterprise RAG-based Q&Al, ideal for scenarios like customer support, internal document lookup, or domain-specific copilots.Output: A dictionary containing the final answer string.
Equipping Your Agent with External Capabilities
In agentic AI systems, a language model becomes significantly more capable when equipped with external tools, such as retrieving real-time weather data, fetching the current time, or answering enterprise-specific questions from a knowledge base. To enable the model to use these tools effectively, each tool must be registered with a clear definition of its purpose, how it should be called, and what input parameters it expects. This registration can be implemented in various ways, depending on your system’s architecture and flexibility requirements.
The code below does exactly that:
import json
from tools import get_time_now, get_weather, retriever
from utils import ToolRegistry
registry = ToolRegistry()
registry.register(
“get_weather”,
get_weather,
“Retrieves current weather data for a given set of geographic coordinates (latitude, longitude).”,
{
“latitude”: “float – The latitude of the location.”,
“longitude”: “float – The longitude of the location.”
}
)
registry.register(
“get_time_now”,
get_time_now,
“Returns the current local time in YYYY-MM-DD HH:MM:SS format.”,
{} # No parameters required
)
registry.register(
“retriever”,
retriever,
“Retrieves an answer using RAG from documents stored in SAP HANA Cloud for SAP Business Data Cloud and SAP Generative AI Hub in SAP AI Core content”,
{
“question”: “string – The question you want to ask based on the document context.”
}
)
This snippet defines a tool registry, which serves as a catalog of functions the AI agent can call during runtime. It registers three tools: one to get the current time, one to fetch real-time weather, and one to retrieve enterprise knowledge using RAG over SAP HANA Cloud.
Each tool is defined with a unique name, a function reference, a clear description, and a schema outlining the required input parameters. But don’t worry, this simply generates a structured JSON representation that the agent uses to evaluate which tool to call based on the context and its reasoning.
description = json.dumps(registry.get_description_for_prompt(), indent=2)
print(description){
“get_weather”: {
“description”: “Retrieves current weather data for a given set of geographic coordinates (latitude, longitude).”,
“parameters”: {
“latitude”: “float – The latitude of the location.”,
“longitude”: “float – The longitude of the location.”
}
},
“get_time_now”: {
“description”: “Returns the current local time in YYYY-MM-DD HH:MM:SS format.”,
“parameters”: {}
},
“retriever”: {
“description”: “Retrieves an answer using RAG from documents stored in SAP HANA Cloud for SAP Business Data Cloud and SAP Generative AI Hub in SAP AI Core content”,
“parameters”: {
“question”: “string – The question you want to ask based on the document context.”
}
}
}
Executing the Agent Workflow with AgentExecutor
The AgentExecutor class is the heart of an agentic AI system. It coordinates LLM-based decision making, tool invocation, and response generation.
Let’s walk through how it works, piece by piece:
Initialization: Injecting the LLM and Tool Registry def __init__(self, llm, tool_registry, verbose=True):
self.llm = llm
self.tool_registry = tool_registry
self.verbose = verbose
This constructor sets up the agent with:
A large language model (llm) used to generate reasoning and answers.A tool_registry containing all available tools (functions the agent can use).An optional verbose flag for logging internal steps.
Dynamic JSON Schema for Tool Decisions def _build_dynamic_schema(self):
return {
“title”: “ToolCalls”,
“type”: “object”,
“properties”: {
“tool_calls”: {
“type”: “array”,
“items”: {
“type”: “object”,
“properties”: {
“decision”: {“type”: “string”},
“reason”: {“type”: “string”},
“function”: {“type”: “string”},
“parameters”: {“type”: “object”}
},
“required”: [“decision”, “reason”, “function”, “parameters”]
}
}
},
“required”: [“tool_calls”]
}
This method defines a JSON Schema for the expected response format from the LLM when asked to decide on tool usage. It enforces a consistent structure:
A list of tool calls.Each tool call includes:The decision (e.g., “tool” or “no_tool”),The reason behind it,The function name to invoke,Any parameters required.
This structure helps validate the LLM’s decisions and ensures they are safely executable.
Instruction Generator for the LLM def _generate_instruction(self):
description = json.dumps(self.tool_registry.get_description_for_prompt(), indent=2)
return f”””
You are an intelligent AI assistant capable of deciding whether to invoke tools based on the user’s request.
Available tools:
{description}
Instructions:
– For each relevant tool, return a JSON entry with the function name and parameters.
– If no tool is relevant, return an entry with decision = “no_tool”.
Return ONLY valid JSON like:
{{
“tool_calls”: [
{{
“decision”: “tool”,
“reason”: “The user asked for weather.”,
“function”: “get_weather”,
“parameters”: {{
“latitude”: 48.8566,
“longitude”: 2.3522
}}
}},
{{
“decision”: “tool”,
“reason”: “The user asked for time.”,
“function”: “get_time_now”,
“parameters”: {{}}
}}
]
}}
“””
This method dynamically builds a prompt describing all available tools and how the LLM should behave. It includes:
A list of tools from the registry.Rules for returning decisions in a valid JSON format.A few examples for clarity.
This acts as a system message to guide the LLM’s thinking in a structured way.
The Agent’s Reasoning and Execution Cycle
The run() method is the central execution loop of the agent. It orchestrates the full process of interpreting a user query, deciding whether tools are needed, executing those tools if applicable, and finally synthesizing a helpful response. This loop effectively turns a passive LLM into an interactive, tool-using agent.
Here’s how it works:
Step 1: Determine Which Tools to Use
The agent begins by preparing a message set that includes:
A system instruction generated by _generate_instruction() that lists all available tools and the rules for tool invocation.The user’s query, wrapped in a UserMessage.
This prompt is passed to a Template and validated against a dynamic JSON Schema defined in _build_dynamic_schema(). The schema ensures the LLM responds with a structured JSON containing its decisions (e.g., whether to call a tool, which one, and with what parameters).
The OrchestrationService uses this configuration to interact with the LLM and returns the model’s reasoning in structured form.
Step 2: Execute Relevant Tools
The agent inspects the LLM’s decisions. For each tool_call with decision = “tool”:
It looks up the corresponding function from the ToolRegistry.Executes the tool using the provided parameters by the agent.Captures the result.
These results are added to the message history, ensuring they’re available as context for the next reasoning step.
Step 3: Final LLM Synthesis
Once all tools have been executed, the agent prepares for a final call to the LLM. It:
Appends a new system message instructing the LLM to:Use tool results where clear.Fall back on its own knowledge only when results are vague or inconclusive.Avoid asking for information already available.Adds the original question again for context.Summarizes the outputs of each tool invocation in a structured format.def run(self, user_query: str):
system_message = SystemMessage(self._generate_instruction())
prompt = UserMessage(user_query)
messages = [system_message, prompt]
template = Template(
messages=messages,
response_format=ResponseFormatJsonSchema(
name=”ToolCall”,
description=”Tool execution format”,
schema=self._build_dynamic_schema()
)
)
config = OrchestrationConfig(template=template, llm=self.llm)
response = OrchestrationService(config=config).run()
decisions_json = json.loads(response.module_results.llm.choices[0].message.content)
if self.verbose:
print(“nLLM Reasoning:”)
print(json.dumps(decisions_json, indent=2))
tool_results = []
messages = [system_message, prompt]
for decision in decisions_json.get(“tool_calls”, []):
if decision.get(“decision”) == “tool”:
tool_response = self._execute_tool(decision)
tool_results.append((decision[“function”], tool_response))
messages.append(AssistantMessage(json.dumps(decision)))
else:
messages.append(AssistantMessage(json.dumps(decision)))
return self._finalize_response(user_query, tool_results, messages)
This complete message history is then passed back to the LLM to generate the final, natural-language response that reflects real-time reasoning and tool-based insight.
Exploring Enhanced Responses with Agentic AI in Action
Now that we’ve implemented tool integration and orchestration logic, it’s time to revisit the same questions we asked earlier, but this time with a fully equipped agentic AI system. Previously, the LLM responded based solely on its pre-trained knowledge, unable to access real-time data or external resources. Let’s see how those same prompts perform now that the agent can reason, decide, and act using live tools.
Impressive results already, but let’s take a closer look at what’s happening under the hood when verbose=True is enabled for the time question:
This example illustrates a clean and efficient execution of an agentic AI loop in response to a simple but real-world query: asking for the current time.
When the user submits their question, the LLM evaluates the query using the context and instructions previously given by the agent framework. Based on its reasoning, it decides that the user is asking for the time which is a use case that maps directly to the get_time_now tool.
The model then responds with a structured decision object containing:
“decision”: “tool” – indicating that a tool should be called,”reason”: “The user asked for time.” – a justification for transparency and traceability,”function”: “get_time_now” – the name of the function to call,”parameters”: {} – an empty dictionary, since this function does not require input.
Next, the agent executes the get_time_now function, which simply returns the current local time formatted as a string (e.g., “2025-04-16 18:33:27”). The tool execution is logged, including the function name, arguments passed (none), and the result returned.
With the tool output now available, the agent performs a final pass using the LLM to synthesize a user-friendly response.
In this example, the agent demonstrates a complete end-to-end reasoning and execution loop to answer the user’s question about the current weather in Paris. Here’s what happens behind the scenes:
When the user submits their query, the language model (LLM) receives a prompt that includes descriptions of all available tools and instructions to decide whether any tool should be used. The model then returns a structured response and, in this case, indicating that the user is asking about weather conditions.
It selects the tool “get_weather” and provides the appropriate geographic coordinates for Paris (latitude: 48.8566, longitude: 2.3522). This decision is packaged as a JSON object containing the tool name, input parameters, and a short explanation of why the tool was selected.
Next, the agent framework executes the get_weather function, which sends a request to a real-time weather API (Open-Meteo) using the provided coordinates. The response includes live data such as the current temperature and wind speed.
Once the tool completes its job, the agent passes the tool results (along with the original query) back to the LLM, which is now instructed to produce a final, natural-language response using only the information provided. It does exactly that: the agent responds clearly and concisely with “The current weather in Paris is 11.1°C with a wind speed of 6.6 m/s.”
As you can see, this is no longer just a chatbot, it’s an intelligent system that can think, act, and answer.
In this example, the agent receives a user query asking “What is SAP Business Data Cloud?”. Recognizing that this is a knowledge-based question, the language model (LLM) reasons that the answer is likely found in available documentation or enterprise data sources. It responds by selecting the retriever tool: a specialized Retrieval-Augmented Generation (RAG) component designed to fetch and synthesize answers from structured knowledge bases.
The agent then executes the tool with the user’s question as input. The retriever searches through embedded documents (such as internal guides or SAP materials stored in a vector database) and returns a relevant and reliable answer. In this case, the tool provides a concise description: “SAP Business Data Cloud is a platform designed to manage and integrate data from various sources, enabling the creation and publication of data products…”.
Using this result, the agent finalizes the response and delivers it to the user in natural language. The entire process illustrates how the agent combines LLM reasoning with enterprise search capabilities to deliver grounded, context-aware information.
What happens when we ask a question that doesn’t require any tool usage?
The user asked for a bread recipe. The AI correctly recognized that this request didn’t require any external tools (such as APIs or databases) and responded directly using its internal knowledge. It returned a clear, step-by-step recipe for making basic bread, with no tool invocation needed.
But what happens if we combine multiple types of requests in a single prompt 
?
“Can you give me a bread recipe and also tell me: how’s the weather in Brazil, what time it is now, and what data products are in SAP Business Data Cloud?”
llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 2000, “temperature”: 0.2})
agent = AgentExecutor(llm=llm, tool_registry=registry, verbose=True)
prompt = “Can you give me a bread recipe and also tell me: how’s the weather in Brazil, what time it is now, and what is the role of SAP Datasphere in SAP Business Data Cloud?”
response = agent.run(prompt)
print(“n”, response)
This image shows how the agent intelligently handles a multi-part user query by breaking it into three separate tool calls, each mapped to the correct function:
For the weather in Brazil, the model selects the get_weather tool and provides coordinates (latitude -14.235, longitude -51.9253), which roughly represent the center of Brazil.For the current time, it selects the get_time_now tool, which doesn’t need any parameters.For the question about SAP Datasphere role in SAP Business Data Cloud, it uses the retriever tool, passing the user’s question so the system can search documentation or internal knowledge sources.
The agent’s reasoning is structured and accurate, and each decision includes a clear explanation. This shows the system’s ability to understand complex queries, decompose them into actionable steps, and respond using both real-time tools and enterprise knowledge.
The result is a well-structured, multi-topic response that demonstrates the agent’s ability to understand and handle complex queries seamlessly. It provided a complete bread recipe with clear ingredients and instructions, showing its built-in general knowledge. It also returned accurate, real-time weather data for Brazil (26.3°C, wind speed 0.8 m/s) and the current time (2025-04-16 18:46:02), proving that it can dynamically use external tools.
Lastly, it retrieved a contextual enterprise answer explaining that SAP Datasphere is the core platform in SAP Business Data Cloud for data integration, management, and analytics. Overall, the agent combined all tool results and reasoning into a single, coherent, and useful reply, precisely fulfilling each part of the user’s request.
Wrapping Up and Next Steps
By now, you’ve seen how a simple LLM setup can evolve into a fully functional agentic AI system that not only understands and responds but also reasons, decides, and acts based on context.
We moved from static prompting to dynamic orchestration. From standalone completions to intelligent tool invocation. And we did it all using SAP GenAI Hub, SAP AI Core, and a modular, developer-friendly architecture that brings enterprise-grade AI to life.
But this is just the beginning 
!
Agentic AI unlocks a wide range of possibilities, from domain-specific assistants and SAP process copilots to smart decision-support systems that integrate real-time data with business logic.
So what can you do next?
Experiment with your own tools. Start simple: wrap an API, expose a database query, or simulate a business rule.
Integrate enterprise data. Connect to SAP HANA, SAP Datasphere, or any source that holds valuable context.
Design your first use case. Think beyond chat, consider where decisions are made and how an agent could assist.
Scale responsibly. Build with governance, traceability, and production-readiness in mind. SAP AI Core and GenAI Hub make it possible.
The building blocks are here. The framework is flexible. And the use cases are endless. Now’s the time to go from demo to impact and bring Agentic AI into your enterprise.
Further References
Source Code: GitHub repositorySAP Libraries and SDKsGenerative AI Hub SDKGenerative AI Hub SDK v4.10.2Generative AI Hub SDK v4.10.2Orchestration
In this article, we will explore how to build an agentic AI system using SAP’s Generative AI Hub. We’ll start by demonstrating how to orchestrate interactions with large language models (LLMs) through a lightweight setup and gradually evolve the architecture into a robust, tool-augmented agent capable of reasoning and acting autonomously.While SAP Joule Studio offers a powerful, low-code way to infuse agentic capabilities into SAP applications, there are scenarios where building a custom agentic solution is an appropriate alternative. Whether you’re working with ECC or legacy systems, seeking deeper integration with external tools, or aiming for complete development flexibility, SAP GenAI Hub provides the necessary orchestration tools to bring these use cases to life. What Are Agentic AI Systems?Agentic AI systems represent an evolution in how we design and deploy language model-based applications. Rather than simply responding to isolated prompts, these systems are built to reason, plan, and act autonomously navigating multi-step tasks, invoking tools, making decisions, and adapting based on context.At their core, Agentic AI systems treat the model as more than a text generator. The model is empowered to analyze user inputs, determine whether external actions are required (such as calling APIs or querying data), and execute those actions before synthesizing a final response. This enables a more dynamic and interactive experience, especially valuable in real-world use cases that involve external data sources, business logic, or multi-modal interactions.Building effective agents requires careful orchestration: structuring prompts, managing context, defining tool schemas, and interpreting intermediate results. These systems often rely on a reason-and-act (ReACT) loop, where the model evaluates a task, decides on next steps, performs actions, and iteratively builds toward a complete, informed answer.This approach opens the door to powerful enterprise use cases from intelligent assistants and copilots to automated decision support systems. In the next sections, we’ll explore how to implement such systems with the help of SAP Generative AI, combining structured orchestration with dynamic model capabilities to build robust, tool-aware agents. See more in The Agentic Evolution: From Chatbots to Multi-Agent Systems. Designing the AI Workflow to Solve ProblemsAt the core of an agentic AI system is a purposeful composition of AI workflow patterns, tailored to address complex, real-world tasks. In our approach, we implement a custom hybrid orchestration, where the language model (LLM) acts as the central orchestrator interpreting user intent, planning the appropriate sequence of actions, and dynamically delegating tasks to specialized tools. The LLM not only selects the right tools but also provides the necessary parameters, ensuring each component operates in context and contributes to a coherent, intelligent response. This modular and dynamic interaction enables the system to move beyond static responses and deliver intelligent, real-world outcomes grounded in reasoning and action. Step-by-Step Through the Agentic AI WorkflowPrompt → LLM (Orchestrator)The process starts with a user prompt. The LLM acts as an orchestrator, analyzing the input and deciding dynamically which tools (workers) are necessary to fulfill the request. Instead of following a fixed sequence, the orchestrator tailors the plan in real-time and outputs a structured representation (e.g., a JSON object with tool calls).LLM → Tools (Workers)Based on the structured plan, the orchestrator routes the task to specific tools:Tool 1: Get Weather – fetches current weather data.Tool 2: Get Time – returns real-time clock information.Tool 3: Retriever (RAG) – performs document retrieval and synthesis for knowledge-intensive queries.Each of these tools is triggered in parallel or sequence, depending on the user request. Importantly, the tools are not hardcoded. They’re selected on the fly based on the LLM’s reasoning.Tool Results → AggregationOnce tools return their outputs, the system aggregates (append) the responses where raw outputs are collected and prepared for the final LLM synthesis.Aggregation → LLM (Orchestrator)With all results in hand, the LLM performs a second pass, now with complete context. It reflects on the tool results and generates a natural-language response tailored to the user’s original query.LLM → Response → UserThe final response is sent back to the user now grounded in external data and enhanced with reasoning.Limitations of Static Prompting and the Case for Agentic AIEven state-of-the-art models like GPT-4o have limitations when used in isolation. For instance, asking for the current time or the weather yields generic or outdated responses. These are not model flaws but consequences of static prompting where models operating without access to real-time data or external APIs.There are a few common approaches to mitigate these limitations:Prompt Engineering: Carefully crafting prompts can improve performance, but it doesn’t give the model access to new or real-time data.Retrieval-Augmented Generation (RAG): Augments model outputs with real-time or contextual information by retrieving relevant documents.Fine-Tuning: Adapts the model to specific domains or tasks by updating its weights based on domain-specific examples.While each technique is powerful in its own right, they still operate under a relatively passive paradigm: the model reacts based on given inputs. To take the next step, we need to build systems where the model can actively reason over tasks, decide when to use tools, and dynamically incorporate real-world data into its responses.This is the foundation of an Agentic AI System .Agentic AI systems overcome traditional boundaries by enabling LLMs to take initiative in task planning and execution, making them ideal for enterprise environments that demand flexibility, real-time context, and intelligent decision-making.Let’s explore a few examples to see how an LLM (GPT-4o) responds to different types of questions.from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.service import OrchestrationService
llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 256, “temperature”: 0.2})
config = OrchestrationConfig(
template=Template(
messages=[
SystemMessage(“You are a helpful AI Assistent that answer the best you can all the time.”),
UserMessage(“{{?question}}”),
]
),
llm=llm
)
orchestration_service = OrchestrationService(config=config)
def send_request(config, **kwargs):
template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
answer = orchestration_service.run(config=config, template_values=template_values)
return answer.module_results.llm.choices[0].message.content
result = send_request(config, question = “What time is it?”)
print(result)result = send_request(config, question = “What is SAP Business Data Cloud?”)
print(result)result = send_request(config, question = “How’s the weather in Paris”)
print(result)This is not a shortcoming of the model itself, but rather a limitation of static prompting: the model operates purely on its pre-trained knowledge without access to external tools, APIs, or dynamic environments.To overcome these constraints, we need to enhance the model with capabilities that allow it to reason over tasks, determine when and how to invoke external tools, and dynamically incorporate real-world data into its responses. In the next section, we’ll explore how to address these limitations using SAP’s Generative AI capabilities, specifically by integrating SAP AI Core, SAP GenAI Hub, and orchestration services.We’ll build a system where the model goes beyond answering questions. It will reason, make decisions, delegate tasks, and deliver responses enriched with real-time data and contextual awareness. Starting Simple: Calling a Language Model via SAP GenAI Hub OrchestrationBefore we build a full agentic AI system, let’s start by recalling how to perform a basic LLM completion using SAP GenAI Hub’s orchestration layer. This initial setup shows how to connect to a model like gpt-4o, define a prompt, and send dynamic user inputs for inference.from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.service import OrchestrationService
llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 256, “temperature”: 0.2})
config = OrchestrationConfig(
template=Template(
messages=[
SystemMessage(“You are a helpful AI Assistent that answer the best you can all the time.”),
UserMessage(“{{?question}}”),
]
),
llm=llm
)
orchestration_service = OrchestrationService(config=config)
def send_request(config, **kwargs):
template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
answer = orchestration_service.run(config=config, template_values=template_values)
return answer.module_results.llm.choices[0].message.content
result = send_request(config, question = “How’s the weather in Paris”)
print(result)In the example above, we import the necessary components from the GenAI Hub SDK: LLM, OrchestrationConfig, and OrchestrationService. We define the model with a few parameters (such as max_tokens and temperature), then configure a simple two-message prompt using SystemMessage and UserMessage.Once configured, we instantiate the orchestration service and send a question like “How’s the weather in Paris?” using the send_request() function. As expected, the model replies that it cannot access real-time information. That’s because at this stage, the model runs standalone, but it’s not yet augmented with external tools.However, when asked a knowledge-based question such as “What is SAP Datasphere?”, the model generates a rich, multi-paragraph answer entirely from its internal context, demonstrating that even without tools, the model already delivers high value for static or informational prompts.This simple orchestration flow forms the foundation of what we’ll build on next: enhancing this architecture with tool invocation, decision-making, and dynamic control to evolve it into a full Agentic AI System. Designing an Agentic AI System with Tool SupportNow that we’ve covered how to perform basic LLM completions using SAP GenAI Hub orchestration, it’s time to evolve this setup into a fully agentic AI system.Agentic systems go beyond static responses, they can reason about tasks, make decisions, and use external capabilities to produce more accurate, contextual, and timely outputs. To enable this, we’ll augment our model with tools that it can call when needed.In this next step, we’ll provide the agent with three tools:get_time_now: retrieves the current local time.get_weather: fetches real-time weather data for a given location.retriever: performs document retrieval using a RAG (Retrieval-Augmented Generation) approach, ideal for enterprise knowledge queries.These tools represent external abilities the model doesn’t natively possess like accessing real-time data or searching enterprise documentation. Through a structured schema, we’ll teach the agent how to decide when to use these tools and when to rely on its own internal knowledge.Let’s begin implementing the agent logic. Exploring the Core Functions of Our Agentic AI BackendIn this section, we’ll break down three key backend functions that power an intelligent, tool-using AI assistant. Each function addresses a different type of user request checking the time, fetching real-world weather, and retrieving knowledge from an enterprise document store using RAG (Retrieval-Augmented Generation).get_time_now(): Returning the Current Local Timedef get_time_now():
“””Returns the current local time as a formatted string.”””
return {“time”: datetime.now().strftime(“%Y-%m-%d %H:%M:%S”)}This simple utility function returns the current local timestamp formatted as a human-readable string (e.g., “2025-04-16 16:30:00”). While it may seem basic, access to the current time is essential in many agentic workflows, such as generating time-sensitive responses, scheduling actions, timestamping events, or handling context-aware tasks that depend on temporal logic.Objective: It enables the assistant to ground its responses in real-time context.Output: A dictionary containing the current timeget_weather(latitude, longitude): Live Weather from Open-Meteo APIdef get_weather(latitude, longitude):
“””This is a publically available API that returns the weather for a given location.”””
response = requests.get(
f”https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m”
)
data = response.json()
return data[“current”]This function calls the Open-Meteo public API and returns current weather conditions for any given pair of latitude and longitude coordinates. This capability might play a key role in scenarios where environmental context matters, such as travel planning, logistics, energy consumption forecasting, or any decision-making process influenced by climate or local conditions. It fetches key metrics like:Temperature (temperature_2m)Wind speed (wind_speed_10m)All in real time.Objective: Enables the assistant to respond to queries like “How’s the weather in Paris?” using live external data.Output: A dictionary with real-time weather readings (e.g., {“temperature_2m”: 18.2, “wind_speed_10m”: 5.6}).retriever(question): Knowledge Retrieval with RAG over SAP HANAdef retriever(question: str):
embedding_model = init_embedding_model(‘text-embedding-ada-002’)
llm = init_llm(‘gpt-4o-mini’)
prompt_template = “””
Use the following context to answer the question at the end.
If the answer is not directly stated, try your best based on the context.
Only say you don’t know if the information is completely unavailable.
{context}
Question: {question}
“””
prompt = PromptTemplate(
template=prompt_template,
input_variables=[“context”, “question”]
)
db = HanaDB(
embedding=embedding_model,
connection=connection,
table_name=”EMBEDDINGS_COLLECTION_DATA”
)
retriever = db.as_retriever(search_kwargs={‘k’: 10})
qa = RetrievalQA.from_chain_type(
llm= llm,
retriever=retriever,
chain_type=”stuff”,
chain_type_kwargs={“prompt”: prompt}
)
response = qa.invoke({“query”: question})
return {“answer”: response}This function powers a Retrieval-Augmented Generation (RAG) workflow that answers user questions based on enterprise knowledge stored in an SAP HANA Cloud vector store. The architecture is modular and flexible, both the embedding model (used for vector similarity search) and the language model (used for final synthesis) can be freely configured. This allows organizations to tailor the solution to their performance, cost, or data locality requirements, whether using open-source models or proprietary LLMs.Here’s what it does step by step:Initialize models: Embedding model (text-embedding-ada-002) for semantic search and LLM (gpt-4o-mini) to generate the final answer.Build the prompt: It defines a flexible prompt template that combines context + the user’s question.Search in SAP HANA: It uses a HanaDB vector store wrapper to retrieve the top 10 most relevant chunks using the user’s question.Run the RetrievalQA chain: Feeds retrieved content into the LLM to synthesize an accurate, grounded answer.Objective: This is the core engine for enterprise RAG-based Q&Al, ideal for scenarios like customer support, internal document lookup, or domain-specific copilots.Output: A dictionary containing the final answer string. Equipping Your Agent with External CapabilitiesIn agentic AI systems, a language model becomes significantly more capable when equipped with external tools, such as retrieving real-time weather data, fetching the current time, or answering enterprise-specific questions from a knowledge base. To enable the model to use these tools effectively, each tool must be registered with a clear definition of its purpose, how it should be called, and what input parameters it expects. This registration can be implemented in various ways, depending on your system’s architecture and flexibility requirements.The code below does exactly that:import json
from tools import get_time_now, get_weather, retriever
from utils import ToolRegistry
registry = ToolRegistry()
registry.register(
“get_weather”,
get_weather,
“Retrieves current weather data for a given set of geographic coordinates (latitude, longitude).”,
{
“latitude”: “float – The latitude of the location.”,
“longitude”: “float – The longitude of the location.”
}
)
registry.register(
“get_time_now”,
get_time_now,
“Returns the current local time in YYYY-MM-DD HH:MM:SS format.”,
{} # No parameters required
)
registry.register(
“retriever”,
retriever,
“Retrieves an answer using RAG from documents stored in SAP HANA Cloud for SAP Business Data Cloud and SAP Generative AI Hub in SAP AI Core content”,
{
“question”: “string – The question you want to ask based on the document context.”
}
) This snippet defines a tool registry, which serves as a catalog of functions the AI agent can call during runtime. It registers three tools: one to get the current time, one to fetch real-time weather, and one to retrieve enterprise knowledge using RAG over SAP HANA Cloud.Each tool is defined with a unique name, a function reference, a clear description, and a schema outlining the required input parameters. But don’t worry, this simply generates a structured JSON representation that the agent uses to evaluate which tool to call based on the context and its reasoning.description = json.dumps(registry.get_description_for_prompt(), indent=2)
print(description){
“get_weather”: {
“description”: “Retrieves current weather data for a given set of geographic coordinates (latitude, longitude).”,
“parameters”: {
“latitude”: “float – The latitude of the location.”,
“longitude”: “float – The longitude of the location.”
}
},
“get_time_now”: {
“description”: “Returns the current local time in YYYY-MM-DD HH:MM:SS format.”,
“parameters”: {}
},
“retriever”: {
“description”: “Retrieves an answer using RAG from documents stored in SAP HANA Cloud for SAP Business Data Cloud and SAP Generative AI Hub in SAP AI Core content”,
“parameters”: {
“question”: “string – The question you want to ask based on the document context.”
}
}
} Executing the Agent Workflow with AgentExecutorThe AgentExecutor class is the heart of an agentic AI system. It coordinates LLM-based decision making, tool invocation, and response generation. Let’s walk through how it works, piece by piece:Initialization: Injecting the LLM and Tool Registry def __init__(self, llm, tool_registry, verbose=True):
self.llm = llm
self.tool_registry = tool_registry
self.verbose = verboseThis constructor sets up the agent with:A large language model (llm) used to generate reasoning and answers.A tool_registry containing all available tools (functions the agent can use).An optional verbose flag for logging internal steps. Dynamic JSON Schema for Tool Decisions def _build_dynamic_schema(self):
return {
“title”: “ToolCalls”,
“type”: “object”,
“properties”: {
“tool_calls”: {
“type”: “array”,
“items”: {
“type”: “object”,
“properties”: {
“decision”: {“type”: “string”},
“reason”: {“type”: “string”},
“function”: {“type”: “string”},
“parameters”: {“type”: “object”}
},
“required”: [“decision”, “reason”, “function”, “parameters”]
}
}
},
“required”: [“tool_calls”]
}This method defines a JSON Schema for the expected response format from the LLM when asked to decide on tool usage. It enforces a consistent structure:A list of tool calls.Each tool call includes:The decision (e.g., “tool” or “no_tool”),The reason behind it,The function name to invoke,Any parameters required.This structure helps validate the LLM’s decisions and ensures they are safely executable. Instruction Generator for the LLM def _generate_instruction(self):
description = json.dumps(self.tool_registry.get_description_for_prompt(), indent=2)
return f”””
You are an intelligent AI assistant capable of deciding whether to invoke tools based on the user’s request.
Available tools:
{description}
Instructions:
– For each relevant tool, return a JSON entry with the function name and parameters.
– If no tool is relevant, return an entry with decision = “no_tool”.
Return ONLY valid JSON like:
{{
“tool_calls”: [
{{
“decision”: “tool”,
“reason”: “The user asked for weather.”,
“function”: “get_weather”,
“parameters”: {{
“latitude”: 48.8566,
“longitude”: 2.3522
}}
}},
{{
“decision”: “tool”,
“reason”: “The user asked for time.”,
“function”: “get_time_now”,
“parameters”: {{}}
}}
]
}}
“””This method dynamically builds a prompt describing all available tools and how the LLM should behave. It includes:A list of tools from the registry.Rules for returning decisions in a valid JSON format.A few examples for clarity.This acts as a system message to guide the LLM’s thinking in a structured way. The Agent’s Reasoning and Execution CycleThe run() method is the central execution loop of the agent. It orchestrates the full process of interpreting a user query, deciding whether tools are needed, executing those tools if applicable, and finally synthesizing a helpful response. This loop effectively turns a passive LLM into an interactive, tool-using agent.Here’s how it works:Step 1: Determine Which Tools to UseThe agent begins by preparing a message set that includes:A system instruction generated by _generate_instruction() that lists all available tools and the rules for tool invocation.The user’s query, wrapped in a UserMessage.This prompt is passed to a Template and validated against a dynamic JSON Schema defined in _build_dynamic_schema(). The schema ensures the LLM responds with a structured JSON containing its decisions (e.g., whether to call a tool, which one, and with what parameters).The OrchestrationService uses this configuration to interact with the LLM and returns the model’s reasoning in structured form. Step 2: Execute Relevant ToolsThe agent inspects the LLM’s decisions. For each tool_call with decision = “tool”:It looks up the corresponding function from the ToolRegistry.Executes the tool using the provided parameters by the agent.Captures the result.These results are added to the message history, ensuring they’re available as context for the next reasoning step. Step 3: Final LLM SynthesisOnce all tools have been executed, the agent prepares for a final call to the LLM. It:Appends a new system message instructing the LLM to:Use tool results where clear.Fall back on its own knowledge only when results are vague or inconclusive.Avoid asking for information already available.Adds the original question again for context.Summarizes the outputs of each tool invocation in a structured format.def run(self, user_query: str):
system_message = SystemMessage(self._generate_instruction())
prompt = UserMessage(user_query)
messages = [system_message, prompt]
template = Template(
messages=messages,
response_format=ResponseFormatJsonSchema(
name=”ToolCall”,
description=”Tool execution format”,
schema=self._build_dynamic_schema()
)
)
config = OrchestrationConfig(template=template, llm=self.llm)
response = OrchestrationService(config=config).run()
decisions_json = json.loads(response.module_results.llm.choices[0].message.content)
if self.verbose:
print(“nLLM Reasoning:”)
print(json.dumps(decisions_json, indent=2))
tool_results = []
messages = [system_message, prompt]
for decision in decisions_json.get(“tool_calls”, []):
if decision.get(“decision”) == “tool”:
tool_response = self._execute_tool(decision)
tool_results.append((decision[“function”], tool_response))
messages.append(AssistantMessage(json.dumps(decision)))
else:
messages.append(AssistantMessage(json.dumps(decision)))
return self._finalize_response(user_query, tool_results, messages)This complete message history is then passed back to the LLM to generate the final, natural-language response that reflects real-time reasoning and tool-based insight. Exploring Enhanced Responses with Agentic AI in ActionNow that we’ve implemented tool integration and orchestration logic, it’s time to revisit the same questions we asked earlier, but this time with a fully equipped agentic AI system. Previously, the LLM responded based solely on its pre-trained knowledge, unable to access real-time data or external resources. Let’s see how those same prompts perform now that the agent can reason, decide, and act using live tools.Impressive results already, but let’s take a closer look at what’s happening under the hood when verbose=True is enabled for the time question:This example illustrates a clean and efficient execution of an agentic AI loop in response to a simple but real-world query: asking for the current time.When the user submits their question, the LLM evaluates the query using the context and instructions previously given by the agent framework. Based on its reasoning, it decides that the user is asking for the time which is a use case that maps directly to the get_time_now tool.The model then responds with a structured decision object containing:”decision”: “tool” – indicating that a tool should be called,”reason”: “The user asked for time.” – a justification for transparency and traceability,”function”: “get_time_now” – the name of the function to call,”parameters”: {} – an empty dictionary, since this function does not require input.Next, the agent executes the get_time_now function, which simply returns the current local time formatted as a string (e.g., “2025-04-16 18:33:27”). The tool execution is logged, including the function name, arguments passed (none), and the result returned.With the tool output now available, the agent performs a final pass using the LLM to synthesize a user-friendly response.In this example, the agent demonstrates a complete end-to-end reasoning and execution loop to answer the user’s question about the current weather in Paris. Here’s what happens behind the scenes:When the user submits their query, the language model (LLM) receives a prompt that includes descriptions of all available tools and instructions to decide whether any tool should be used. The model then returns a structured response and, in this case, indicating that the user is asking about weather conditions.It selects the tool “get_weather” and provides the appropriate geographic coordinates for Paris (latitude: 48.8566, longitude: 2.3522). This decision is packaged as a JSON object containing the tool name, input parameters, and a short explanation of why the tool was selected.Next, the agent framework executes the get_weather function, which sends a request to a real-time weather API (Open-Meteo) using the provided coordinates. The response includes live data such as the current temperature and wind speed.Once the tool completes its job, the agent passes the tool results (along with the original query) back to the LLM, which is now instructed to produce a final, natural-language response using only the information provided. It does exactly that: the agent responds clearly and concisely with “The current weather in Paris is 11.1°C with a wind speed of 6.6 m/s.”As you can see, this is no longer just a chatbot, it’s an intelligent system that can think, act, and answer.In this example, the agent receives a user query asking “What is SAP Business Data Cloud?”. Recognizing that this is a knowledge-based question, the language model (LLM) reasons that the answer is likely found in available documentation or enterprise data sources. It responds by selecting the retriever tool: a specialized Retrieval-Augmented Generation (RAG) component designed to fetch and synthesize answers from structured knowledge bases.The agent then executes the tool with the user’s question as input. The retriever searches through embedded documents (such as internal guides or SAP materials stored in a vector database) and returns a relevant and reliable answer. In this case, the tool provides a concise description: “SAP Business Data Cloud is a platform designed to manage and integrate data from various sources, enabling the creation and publication of data products…”.Using this result, the agent finalizes the response and delivers it to the user in natural language. The entire process illustrates how the agent combines LLM reasoning with enterprise search capabilities to deliver grounded, context-aware information.What happens when we ask a question that doesn’t require any tool usage?The user asked for a bread recipe. The AI correctly recognized that this request didn’t require any external tools (such as APIs or databases) and responded directly using its internal knowledge. It returned a clear, step-by-step recipe for making basic bread, with no tool invocation needed.But what happens if we combine multiple types of requests in a single prompt ?“Can you give me a bread recipe and also tell me: how’s the weather in Brazil, what time it is now, and what data products are in SAP Business Data Cloud?”llm = LLM(name=”gpt-4o”, version=”latest”, parameters={“max_tokens”: 2000, “temperature”: 0.2})
agent = AgentExecutor(llm=llm, tool_registry=registry, verbose=True)
prompt = “Can you give me a bread recipe and also tell me: how’s the weather in Brazil, what time it is now, and what is the role of SAP Datasphere in SAP Business Data Cloud?”
response = agent.run(prompt)
print(“n”, response)This image shows how the agent intelligently handles a multi-part user query by breaking it into three separate tool calls, each mapped to the correct function:For the weather in Brazil, the model selects the get_weather tool and provides coordinates (latitude -14.235, longitude -51.9253), which roughly represent the center of Brazil.For the current time, it selects the get_time_now tool, which doesn’t need any parameters.For the question about SAP Datasphere role in SAP Business Data Cloud, it uses the retriever tool, passing the user’s question so the system can search documentation or internal knowledge sources.The agent’s reasoning is structured and accurate, and each decision includes a clear explanation. This shows the system’s ability to understand complex queries, decompose them into actionable steps, and respond using both real-time tools and enterprise knowledge.The result is a well-structured, multi-topic response that demonstrates the agent’s ability to understand and handle complex queries seamlessly. It provided a complete bread recipe with clear ingredients and instructions, showing its built-in general knowledge. It also returned accurate, real-time weather data for Brazil (26.3°C, wind speed 0.8 m/s) and the current time (2025-04-16 18:46:02), proving that it can dynamically use external tools.Lastly, it retrieved a contextual enterprise answer explaining that SAP Datasphere is the core platform in SAP Business Data Cloud for data integration, management, and analytics. Overall, the agent combined all tool results and reasoning into a single, coherent, and useful reply, precisely fulfilling each part of the user’s request. Wrapping Up and Next StepsBy now, you’ve seen how a simple LLM setup can evolve into a fully functional agentic AI system that not only understands and responds but also reasons, decides, and acts based on context.We moved from static prompting to dynamic orchestration. From standalone completions to intelligent tool invocation. And we did it all using SAP GenAI Hub, SAP AI Core, and a modular, developer-friendly architecture that brings enterprise-grade AI to life.But this is just the beginning !Agentic AI unlocks a wide range of possibilities, from domain-specific assistants and SAP process copilots to smart decision-support systems that integrate real-time data with business logic.So what can you do next?Experiment with your own tools. Start simple: wrap an API, expose a database query, or simulate a business rule.Integrate enterprise data. Connect to SAP HANA, SAP Datasphere, or any source that holds valuable context.Design your first use case. Think beyond chat, consider where decisions are made and how an agent could assist.Scale responsibly. Build with governance, traceability, and production-readiness in mind. SAP AI Core and GenAI Hub make it possible.The building blocks are here. The framework is flexible. And the use cases are endless. Now’s the time to go from demo to impact and bring Agentic AI into your enterprise. Further ReferencesSource Code: GitHub repositorySAP Libraries and SDKsGenerative AI Hub SDKGenerative AI Hub SDK v4.10.2Generative AI Hub SDK v4.10.2Orchestration Read More Technology Blogs by SAP articles
#SAP
#SAPTechnologyblog
