Deploying Langfuse on SAP BTP, Kyma Runtime

Estimated read time 13 min read

This is the translated version of original Japanese blog:
LLM監視ツールLangfuseをSAP BTP, Kyma runtimeにデプロイしてみた

Introduction

This time, we will introduce a step-by-step guide to deploying Langfuse, an open-source LLM application monitoring tool, on the Kubernetes-compatible runtime SAP BTP, Kyma runtime. Monitoring is an unavoidable issue when utilizing generative AI, and we hope this blog will be helpful to you.

Overview

As generative AI becomes deeply integrated into our daily lives and work, many companies and individuals are advancing their use of applications such as chatbots and automation tools. These are being utilized across various fields, from automating customer support and enhancing data analysis to planning new products and services.

One particularly noteworthy application involves using AI agents like LangChain. LangChain is a powerful framework adopted by many developers to generate advanced responses by combining multiple tools and external data sources. However, operating these AI agents comes with new challenges.

Common Challenges

Some common issues include:

Managing prompt quality and model performance: Designing appropriate prompts and selecting models are essential to improving response accuracy, but accurately evaluating their effectiveness can be difficult.

Tracing external APIs and tools: Understanding the behavior of external services used by AI agents and quickly identifying causes of issues when they occur are crucial.

Monitoring errors and response times: Real-time monitoring of performance metrics (e.g., response times, error rates) in the operating environment is essential.

Leveraging user feedback: Continuously understanding how well generative AI responses meet user expectations and incorporating improvements is necessary.

To address these challenges, developers strongly feel the need for tools that allow detailed monitoring and evaluation of AI agents’ operations and facilitate rapid improvements. Against this backdrop, specialized monitoring platforms for LLM applications, such as Langfuse, are gaining attention.

What is Langfuse?

Langfuse is an open-source LLMOps platform that supports the development, monitoring, evaluation, and debugging of applications leveraging large language models (LLMs). It supports the entire lifecycle of generative AI products, from development and testing to large-scale monitoring and debugging.

Key Features and Benefits

Open-source availability: Langfuse is provided as open-source, allowing for self-hosting. This makes customization and long-term use easier.

Integration with various frameworks: Langfuse seamlessly integrates with major LLM frameworks like LangChain and LlamaIndex, making it easy to incorporate into existing workflows.

Detailed traceability: It captures the complete context of products, including external API calls, tools, and prompts, enabling detailed tracking of application behavior.

Real-time metrics monitoring: Real-time monitoring of key performance indicators such as response times, error rates, and throughput allows for prompt responses.

Collecting user feedback: Feedback from users can be gathered to improve application performance and user experience.

Evaluation and testing features: By setting up evaluation workflows using LLMs or human annotations, developers can compare the performance of different models and prompts. A/B testing can also be used to determine the best solution.

Flexible deployment: Langfuse supports deployment in various environments, such as on-premises or cloud, allowing for flexibility based on project scale.

These features make Langfuse a powerful tool for LLM application developers, enabling them to improve application quality and operate efficiently.

Reference

https://langfuse.com/jp

Deployment Steps

Let’s now introduce the procedure for hosting Langfuse on SAP BTP.

Prerequisites

An account with access to SAP BTP, Kyma runtime.

kubectl CLI, helm CLI, and kubelogin installed in the local development environment. If not installed, refer to the following links:

kubectl

helm

kubelogin

Step 1: Setting up Kyma Environment in SAP BTP

First, enable SAP BTP, Kyma runtime on SAP BTP, and grant user permissions. Refer to the following tutorial for details:

https://developers.sap.com/tutorials/cp-kyma-getting-started.html

Step 2: Configuring kubectl and Verifying Connection

Refer to the URL below for connecting kubectl to SAP Kyma runtime:

https://help.sap.com/docs/btp/sap-business-technology-platform/access-kyma-instance-using-kubectl

Retrieve the kubeconfig.yaml File

Download the kubeconfig file from the SAP BTP Cockpit by clicking the Kubeconfig URL link, which starts downloading a configuration file named kubeconfig.yaml. Use this file to connect kubectl to SAP BTP, Kyma runtime.

Translation of texts in picture:

JapaneseEnglishクリックすると、kubeconfig.yamlがダウンロードされるWhen you click here, kubecounfig.yaml get downloaded

Verify Connection

Run the following commands in the terminal to load the cluster settings for SAP BTP, Kyma runtime and ensure proper configuration. Replace /path/to/kubeconfig.yaml with your environment’s path. Note that the commands below are for macOS. For Windows, refer to the appropriate commands in the linked documentation.

 

 

export KUBECONFIG=/path/to/kubeconfig.yaml
kubectl config get-contexts

 

 

Translation of texts in picture:

JapaneseEnglishkubeconfig.yamlを配置したパスを、KUBECONFIGという環境変数にセットするSet the path where kubeconfig.yaml is located to environment variable “KUBECONFIG”.kubectl config get-contexts により、正しくkubeconfig.yamlが読み込まれていることを確認するMake sure kubeconfig.yaml is properly read by the system with the command “kubectl config get-contexts”.

If successful, proceed to deploy Langfuse.

Step 3: Deploying Langfuse to SAP BTP, Kyma Runtime

Refer to the official guide below to deploy langfuse-k8s:

https://langfuse.com/self-hosting/kubernetes-helm

Clone the Source Code

 

 

git clone https://github.com/langfuse/langfuse-k8s.git
cd langfuse-k8s/charts/langfuse

 

 

Create a Namespace

Create a namespace for testing on SAP BTP, Kyma runtime:

 

 

kubectl create namespace langfuse-v3-preview

 

 

Translation of texts in picture:

JapaneseEnglishネームスペースを作成するCreate namespace.

This procedure will create the namespace on SAP BTP, Kyma Runtime, where the Langfuse is going to be deployed.

Deploy Using Helm

Using Helm, a Kubernetes package manager, update dependencies and deploy to the namespace created:

 

 

helm dependency update
helm install langfuse . -n langfuse-v3-preview

 

 

Step 4: Accessing the Langfuse UI

Check Deployment Status

Run the following command to check the deployment status:

 

 

kubectl get services -n langfuse-v3-preview

 

 

According to this, the connection type of the langfuse-web application is set to ClusterIP, and no External-IP has been assigned, which makes it inaccessible from the outside. Therefore, we will execute the following command.

 

 

kubectl edit service langfuse-web -n langfuse-v3-preview

 

 

This command allows you to make changes to the configuration of langfuse-web. Let’s rewrite the settings as shown below. With this configuration, an External IP will be assigned after a short wait.

Update Service Type

Modify the settings as follows to allow for an External IP to be assigned:

 

 

spec:
type: LoadBalancer ## Previously set as Cluster IP

 

 

Confirm Access

Confirm access to the Langfuse UI by executing:

 

 

kubectl get services -n langfuse-v3-preview

 

 

Access using the following URL:

 

 

http://<External-IP>:<Port>

 

 

 

Sign Up and Log In

The above is the top screen when accessing the application. First, create an account by clicking on “Sign Up.”

Even if redirected to http://localhost:3000 (a browser error page may appear) after account creation, this can be ignored because default login callback is set to redirect users there. The user account should be created successfully. Reaccess the http://<External IP>:<Port> URL and log in with the created account.

Once you do so, you will successfully log in as shown. After that, you can create an Organization and a Project with any name, which will eventually lead you to the dashboard screen.

Conclusion

In this blog, we introduced the procedure for deploying the OSS Langfuse on SAP BTP, Kyma runtime. By leveraging Langfuse’s support for Kubernetes deployment, we demonstrated how to build the environment with relative ease on the compatible SAP BTP, Kyma runtime.

In the next blog, we plan to explain how to integrate Langfuse with LangChain for debugging and monitoring generative AI applications. Stay tuned!

Next Blog:

Developing, Running, and Monitoring LangChain Applications on SAP BTP

 

 

​ This is the translated version of original Japanese blog:LLM監視ツールLangfuseをSAP BTP, Kyma runtimeにデプロイしてみたIntroductionThis time, we will introduce a step-by-step guide to deploying Langfuse, an open-source LLM application monitoring tool, on the Kubernetes-compatible runtime SAP BTP, Kyma runtime. Monitoring is an unavoidable issue when utilizing generative AI, and we hope this blog will be helpful to you.OverviewAs generative AI becomes deeply integrated into our daily lives and work, many companies and individuals are advancing their use of applications such as chatbots and automation tools. These are being utilized across various fields, from automating customer support and enhancing data analysis to planning new products and services.One particularly noteworthy application involves using AI agents like LangChain. LangChain is a powerful framework adopted by many developers to generate advanced responses by combining multiple tools and external data sources. However, operating these AI agents comes with new challenges.Common ChallengesSome common issues include:Managing prompt quality and model performance: Designing appropriate prompts and selecting models are essential to improving response accuracy, but accurately evaluating their effectiveness can be difficult.Tracing external APIs and tools: Understanding the behavior of external services used by AI agents and quickly identifying causes of issues when they occur are crucial.Monitoring errors and response times: Real-time monitoring of performance metrics (e.g., response times, error rates) in the operating environment is essential.Leveraging user feedback: Continuously understanding how well generative AI responses meet user expectations and incorporating improvements is necessary.To address these challenges, developers strongly feel the need for tools that allow detailed monitoring and evaluation of AI agents’ operations and facilitate rapid improvements. Against this backdrop, specialized monitoring platforms for LLM applications, such as Langfuse, are gaining attention.What is Langfuse?Langfuse is an open-source LLMOps platform that supports the development, monitoring, evaluation, and debugging of applications leveraging large language models (LLMs). It supports the entire lifecycle of generative AI products, from development and testing to large-scale monitoring and debugging.Key Features and BenefitsOpen-source availability: Langfuse is provided as open-source, allowing for self-hosting. This makes customization and long-term use easier.Integration with various frameworks: Langfuse seamlessly integrates with major LLM frameworks like LangChain and LlamaIndex, making it easy to incorporate into existing workflows.Detailed traceability: It captures the complete context of products, including external API calls, tools, and prompts, enabling detailed tracking of application behavior.Real-time metrics monitoring: Real-time monitoring of key performance indicators such as response times, error rates, and throughput allows for prompt responses.Collecting user feedback: Feedback from users can be gathered to improve application performance and user experience.Evaluation and testing features: By setting up evaluation workflows using LLMs or human annotations, developers can compare the performance of different models and prompts. A/B testing can also be used to determine the best solution.Flexible deployment: Langfuse supports deployment in various environments, such as on-premises or cloud, allowing for flexibility based on project scale.These features make Langfuse a powerful tool for LLM application developers, enabling them to improve application quality and operate efficiently.Referencehttps://langfuse.com/jpDeployment StepsLet’s now introduce the procedure for hosting Langfuse on SAP BTP.PrerequisitesAn account with access to SAP BTP, Kyma runtime.kubectl CLI, helm CLI, and kubelogin installed in the local development environment. If not installed, refer to the following links:kubectlhelmkubeloginStep 1: Setting up Kyma Environment in SAP BTPFirst, enable SAP BTP, Kyma runtime on SAP BTP, and grant user permissions. Refer to the following tutorial for details:https://developers.sap.com/tutorials/cp-kyma-getting-started.htmlStep 2: Configuring kubectl and Verifying ConnectionRefer to the URL below for connecting kubectl to SAP Kyma runtime:https://help.sap.com/docs/btp/sap-business-technology-platform/access-kyma-instance-using-kubectlRetrieve the kubeconfig.yaml FileDownload the kubeconfig file from the SAP BTP Cockpit by clicking the Kubeconfig URL link, which starts downloading a configuration file named kubeconfig.yaml. Use this file to connect kubectl to SAP BTP, Kyma runtime.Translation of texts in picture:JapaneseEnglishクリックすると、kubeconfig.yamlがダウンロードされるWhen you click here, kubecounfig.yaml get downloadedVerify ConnectionRun the following commands in the terminal to load the cluster settings for SAP BTP, Kyma runtime and ensure proper configuration. Replace /path/to/kubeconfig.yaml with your environment’s path. Note that the commands below are for macOS. For Windows, refer to the appropriate commands in the linked documentation.  export KUBECONFIG=/path/to/kubeconfig.yaml
kubectl config get-contexts  Translation of texts in picture:JapaneseEnglishkubeconfig.yamlを配置したパスを、KUBECONFIGという環境変数にセットするSet the path where kubeconfig.yaml is located to environment variable “KUBECONFIG”.kubectl config get-contexts により、正しくkubeconfig.yamlが読み込まれていることを確認するMake sure kubeconfig.yaml is properly read by the system with the command “kubectl config get-contexts”.If successful, proceed to deploy Langfuse.Step 3: Deploying Langfuse to SAP BTP, Kyma RuntimeRefer to the official guide below to deploy langfuse-k8s:https://langfuse.com/self-hosting/kubernetes-helmClone the Source Code  git clone https://github.com/langfuse/langfuse-k8s.git
cd langfuse-k8s/charts/langfuse   Create a NamespaceCreate a namespace for testing on SAP BTP, Kyma runtime:  kubectl create namespace langfuse-v3-preview  Translation of texts in picture:JapaneseEnglishネームスペースを作成するCreate namespace.This procedure will create the namespace on SAP BTP, Kyma Runtime, where the Langfuse is going to be deployed.Deploy Using HelmUsing Helm, a Kubernetes package manager, update dependencies and deploy to the namespace created:  helm dependency update
helm install langfuse . -n langfuse-v3-preview  Step 4: Accessing the Langfuse UICheck Deployment StatusRun the following command to check the deployment status:  kubectl get services -n langfuse-v3-preview  According to this, the connection type of the langfuse-web application is set to ClusterIP, and no External-IP has been assigned, which makes it inaccessible from the outside. Therefore, we will execute the following command.  kubectl edit service langfuse-web -n langfuse-v3-preview  This command allows you to make changes to the configuration of langfuse-web. Let’s rewrite the settings as shown below. With this configuration, an External IP will be assigned after a short wait.Update Service TypeModify the settings as follows to allow for an External IP to be assigned:  spec:
type: LoadBalancer ## Previously set as Cluster IP  Confirm AccessConfirm access to the Langfuse UI by executing:  kubectl get services -n langfuse-v3-preview  Access using the following URL:  http://<External-IP>:<Port>   Sign Up and Log InThe above is the top screen when accessing the application. First, create an account by clicking on “Sign Up.”Even if redirected to http://localhost:3000 (a browser error page may appear) after account creation, this can be ignored because default login callback is set to redirect users there. The user account should be created successfully. Reaccess the http://<External IP>:<Port> URL and log in with the created account.Once you do so, you will successfully log in as shown. After that, you can create an Organization and a Project with any name, which will eventually lead you to the dashboard screen.ConclusionIn this blog, we introduced the procedure for deploying the OSS Langfuse on SAP BTP, Kyma runtime. By leveraging Langfuse’s support for Kubernetes deployment, we demonstrated how to build the environment with relative ease on the compatible SAP BTP, Kyma runtime.In the next blog, we plan to explain how to integrate Langfuse with LangChain for debugging and monitoring generative AI applications. Stay tuned!Next Blog:Developing, Running, and Monitoring LangChain Applications on SAP BTP    Read More Technology Blogs by SAP articles 

#SAP

#SAPTechnologyblog

You May Also Like

More From Author