Overview
This post shows how to stream LLM output from the SAP Cloud SDK for AI orchestration module into a CAP service and directly forward it to the browser over WebSockets.
It covers:
Enabling WebSocket support in CAP
Defining a service with actions and eventsStreaming LLM output and emitting it via WebSocketsA minimal frontend using the native WebSocket API
Here’s a quick demo that shows the end result:
You can also explore the full example project on GitHub
Prerequisites
A CAP project, Node.js and a package managerA frontend (any framework) and, optionally SAP ApprouterAccess to an LLM provider via the orchestration module in SAP AI Foundation’s (AI Core) Generative AI Hub – using the SAP Cloud SDK for AI
Backend
CAP does not provide WebSocket support out of the box, but the community library makes it easy. Add it by running the following command inside the folder of your CAP application:
npm add -js-community/websocket
To stream LLM responses to your CAP service, add the orchestration module from the SAP Cloud SDK for AI:
npm add -ai-sdk/orchestration
Service Definition (CDS)
You can expose a service over both OData and WebSocket by annotating the service:
Annotate a service with@protocol: [‘odata’, ‘websocket’]
// @ws for websocket onlyDefine an event to broadcast over the WebSocket.
For example:event storyChunk { message: String; }Add an action or function that clients can call (via WebSocket).
For example:function generateStory(topic : String) returns String;
then, the client should also pass topic as a parameter
Key points:
Define events to broadcast messages@protocol: [‘odata’,’websocket’] enables both protocols on the same serviceDefine actions or functions that clients can call to trigger work
Service implementation (JavaScript/TypeScript)
In your service handlers you can access WebSocket features via:
req.context.ws (when handling a request)cds.context.ws (when in a wider context)
LLM streaming with the SAP Cloud SDK for AI orchestration module:
The orchestration module allows you to stream the output from an LLM. It also makes it easy to switch to another model of any provider later.
Flow:
Create a client with your AI Core resource group, the target model, and your prompt(s).Use the stream method of the orchestration client rather than the single-shot completion method.const response = await orchestrationClient.stream({messages: messages});Iterate over the streamfor await (const chunk of response.stream) {
and, for each chunk of text received, emit your CDS event via
req.context.ws.service.emit(‘storyChunk’, { message: chunk })
Notes:
Use req.context.ws.service.emit(“<eventName>”, {…}) to broadcast messagesIf you need low-level socket access, req.context.ws.socket is available – but use it with caution and check the documentation
Frontend
The browser has a built-in WebSocket API – no extra library required.
Steps:
Construct the URL (React example):const protocol = window.location.protocol === “https:” ? “wss://” : “ws://”;
const host = window.location.host;
socket.current = new WebSocket(protocol + host + “/api/ws/story”); // adjust path if needed
Notes (Approuter):
If you are calling the CAP service via the SAP Approuter:
The service might be exposed under a path e.g. /api/…, depending on what you specified in your routes.Make sure to enable WebSockets in your xs-app.json:
“websockets”: {
“enabled”: true
}
2. Set up handlers:
onopen: optional, to confirm the connection is established and to trigger your initial action/function call if neededonmessage: required, to receive messages; parse the JSON payload emitted by CAP and append the text chunk to your UI
Example in React:onerror: recommended, to surface connection issuesonclose: recommended, to finalize the UI state when the stream endssocket.onopen = () => {
console.log(“WebSocket connected!”)
};
socket.onmessage = (event) => {
const receivedMessage = JSON.parse(event.data);
if (receivedMessage.event === “storyChunk”) {
console.log(“Chunk:”, receivedMessage.data.message);
}
};
socket.onerror = (err) => console.error(“WebSocket error”, err);
socket.onclose = () => console.log(“WebSocket closed”);
Summary
What we’ve achieved
We’ve seen how to integrate LLM output streaming into a CAP service by combining the SAP Cloud SDK for AI orchestration module with community WebSocket support, enabling real-time delivery of AI-generated content directly to a browser frontend.
Steps to achieve
Add @cap-js-community/websocket for WebSocket support in CAPAdd @SAP-ai-sdk/orchestration to stream LLM outputIn CDS: use @protocol: [‘odata’,’websocket’], define an event and an actionIn the implementation: stream chunks from the LLM and emit them as eventsIn the frontend: use the native WebSocket API and render messages as they arrive
OverviewThis post shows how to stream LLM output from the SAP Cloud SDK for AI orchestration module into a CAP service and directly forward it to the browser over WebSockets.It covers:Enabling WebSocket support in CAPDefining a service with actions and eventsStreaming LLM output and emitting it via WebSocketsA minimal frontend using the native WebSocket APIHere’s a quick demo that shows the end result:You can also explore the full example project on GitHubPrerequisitesA CAP project, Node.js and a package managerA frontend (any framework) and, optionally SAP ApprouterAccess to an LLM provider via the orchestration module in SAP AI Foundation’s (AI Core) Generative AI Hub – using the SAP Cloud SDK for AIBackendCAP does not provide WebSocket support out of the box, but the community library makes it easy. Add it by running the following command inside the folder of your CAP application:npm add -js-community/websocketTo stream LLM responses to your CAP service, add the orchestration module from the SAP Cloud SDK for AI:npm add -ai-sdk/orchestrationService Definition (CDS)You can expose a service over both OData and WebSocket by annotating the service:Annotate a service with@protocol: [‘odata’, ‘websocket’]
// @ws for websocket onlyDefine an event to broadcast over the WebSocket.For example:event storyChunk { message: String; }Add an action or function that clients can call (via WebSocket).For example:function generateStory(topic : String) returns String;then, the client should also pass topic as a parameterKey points:Define events to broadcast messages@protocol: [‘odata’,’websocket’] enables both protocols on the same serviceDefine actions or functions that clients can call to trigger workService implementation (JavaScript/TypeScript)In your service handlers you can access WebSocket features via:req.context.ws (when handling a request)cds.context.ws (when in a wider context)LLM streaming with the SAP Cloud SDK for AI orchestration module:The orchestration module allows you to stream the output from an LLM. It also makes it easy to switch to another model of any provider later.Flow:Create a client with your AI Core resource group, the target model, and your prompt(s).Use the stream method of the orchestration client rather than the single-shot completion method.const response = await orchestrationClient.stream({messages: messages});Iterate over the streamfor await (const chunk of response.stream) {and, for each chunk of text received, emit your CDS event viareq.context.ws.service.emit(‘storyChunk’, { message: chunk })Notes:Use req.context.ws.service.emit(“<eventName>”, {…}) to broadcast messagesIf you need low-level socket access, req.context.ws.socket is available – but use it with caution and check the documentationFrontendThe browser has a built-in WebSocket API – no extra library required.Steps:Construct the URL (React example):const protocol = window.location.protocol === “https:” ? “wss://” : “ws://”;
const host = window.location.host;
socket.current = new WebSocket(protocol + host + “/api/ws/story”); // adjust path if neededNotes (Approuter):If you are calling the CAP service via the SAP Approuter:The service might be exposed under a path e.g. /api/…, depending on what you specified in your routes.Make sure to enable WebSockets in your xs-app.json: “websockets”: {
“enabled”: true
} 2. Set up handlers:onopen: optional, to confirm the connection is established and to trigger your initial action/function call if neededonmessage: required, to receive messages; parse the JSON payload emitted by CAP and append the text chunk to your UIExample in React:onerror: recommended, to surface connection issuesonclose: recommended, to finalize the UI state when the stream endssocket.onopen = () => {
console.log(“WebSocket connected!”)
};
socket.onmessage = (event) => {
const receivedMessage = JSON.parse(event.data);
if (receivedMessage.event === “storyChunk”) {
console.log(“Chunk:”, receivedMessage.data.message);
}
};
socket.onerror = (err) => console.error(“WebSocket error”, err);
socket.onclose = () => console.log(“WebSocket closed”);SummaryWhat we’ve achievedWe’ve seen how to integrate LLM output streaming into a CAP service by combining the SAP Cloud SDK for AI orchestration module with community WebSocket support, enabling real-time delivery of AI-generated content directly to a browser frontend.Steps to achieveAdd @cap-js-community/websocket for WebSocket support in CAPAdd @SAP-ai-sdk/orchestration to stream LLM outputIn CDS: use @protocol: [‘odata’,’websocket’], define an event and an actionIn the implementation: stream chunks from the LLM and emit them as eventsIn the frontend: use the native WebSocket API and render messages as they arrive Read More Technology Blog Posts by SAP articles
#SAP
#SAPTechnologyblog
