This is the translated version of original Japanese blog:
LLM監視ツールLangfuseをSAP BTP, Kyma runtimeにデプロイしてみた
Introduction
This time, we will introduce a step-by-step guide to deploying Langfuse, an open-source LLM application monitoring tool, on the Kubernetes-compatible runtime SAP BTP, Kyma runtime. Monitoring is an unavoidable issue when utilizing generative AI, and we hope this blog will be helpful to you.
Overview
As generative AI becomes deeply integrated into our daily lives and work, many companies and individuals are advancing their use of applications such as chatbots and automation tools. These are being utilized across various fields, from automating customer support and enhancing data analysis to planning new products and services.
One particularly noteworthy application involves using AI agents like LangChain. LangChain is a powerful framework adopted by many developers to generate advanced responses by combining multiple tools and external data sources. However, operating these AI agents comes with new challenges.
Common Challenges
Some common issues include:
Managing prompt quality and model performance: Designing appropriate prompts and selecting models are essential to improving response accuracy, but accurately evaluating their effectiveness can be difficult.
Tracing external APIs and tools: Understanding the behavior of external services used by AI agents and quickly identifying causes of issues when they occur are crucial.
Monitoring errors and response times: Real-time monitoring of performance metrics (e.g., response times, error rates) in the operating environment is essential.
Leveraging user feedback: Continuously understanding how well generative AI responses meet user expectations and incorporating improvements is necessary.
To address these challenges, developers strongly feel the need for tools that allow detailed monitoring and evaluation of AI agents’ operations and facilitate rapid improvements. Against this backdrop, specialized monitoring platforms for LLM applications, such as Langfuse, are gaining attention.
What is Langfuse?
Langfuse is an open-source LLMOps platform that supports the development, monitoring, evaluation, and debugging of applications leveraging large language models (LLMs). It supports the entire lifecycle of generative AI products, from development and testing to large-scale monitoring and debugging.
Key Features and Benefits
Open-source availability: Langfuse is provided as open-source, allowing for self-hosting. This makes customization and long-term use easier.
Integration with various frameworks: Langfuse seamlessly integrates with major LLM frameworks like LangChain and LlamaIndex, making it easy to incorporate into existing workflows.
Detailed traceability: It captures the complete context of products, including external API calls, tools, and prompts, enabling detailed tracking of application behavior.
Real-time metrics monitoring: Real-time monitoring of key performance indicators such as response times, error rates, and throughput allows for prompt responses.
Collecting user feedback: Feedback from users can be gathered to improve application performance and user experience.
Evaluation and testing features: By setting up evaluation workflows using LLMs or human annotations, developers can compare the performance of different models and prompts. A/B testing can also be used to determine the best solution.
Flexible deployment: Langfuse supports deployment in various environments, such as on-premises or cloud, allowing for flexibility based on project scale.
These features make Langfuse a powerful tool for LLM application developers, enabling them to improve application quality and operate efficiently.
Reference
Deployment Steps
Let’s now introduce the procedure for hosting Langfuse on SAP BTP.
Prerequisites
An account with access to SAP BTP, Kyma runtime.
kubectl CLI, helm CLI, and kubelogin installed in the local development environment. If not installed, refer to the following links:
Step 1: Setting up Kyma Environment in SAP BTP
First, enable SAP BTP, Kyma runtime on SAP BTP, and grant user permissions. Refer to the following tutorial for details:
https://developers.sap.com/tutorials/cp-kyma-getting-started.html
Step 2: Configuring kubectl and Verifying Connection
Refer to the URL below for connecting kubectl to SAP Kyma runtime:
https://help.sap.com/docs/btp/sap-business-technology-platform/access-kyma-instance-using-kubectl
Retrieve the kubeconfig.yaml File
Download the kubeconfig file from the SAP BTP Cockpit by clicking the Kubeconfig URL link, which starts downloading a configuration file named kubeconfig.yaml. Use this file to connect kubectl to SAP BTP, Kyma runtime.
Translation of texts in picture:
JapaneseEnglishクリックすると、kubeconfig.yamlがダウンロードされるWhen you click here, kubecounfig.yaml get downloaded
Verify Connection
Run the following commands in the terminal to load the cluster settings for SAP BTP, Kyma runtime and ensure proper configuration. Replace /path/to/kubeconfig.yaml with your environment’s path. Note that the commands below are for macOS. For Windows, refer to the appropriate commands in the linked documentation.
export KUBECONFIG=/path/to/kubeconfig.yaml
kubectl config get-contexts
Translation of texts in picture:
JapaneseEnglishkubeconfig.yamlを配置したパスを、KUBECONFIGという環境変数にセットするSet the path where kubeconfig.yaml is located to environment variable “KUBECONFIG”.kubectl config get-contexts により、正しくkubeconfig.yamlが読み込まれていることを確認するMake sure kubeconfig.yaml is properly read by the system with the command “kubectl config get-contexts”.
If successful, proceed to deploy Langfuse.
Step 3: Deploying Langfuse to SAP BTP, Kyma Runtime
Refer to the official guide below to deploy langfuse-k8s:
https://langfuse.com/self-hosting/kubernetes-helm
Clone the Source Code
git clone https://github.com/langfuse/langfuse-k8s.git
cd langfuse-k8s/charts/langfuse
Create a Namespace
Create a namespace for testing on SAP BTP, Kyma runtime:
kubectl create namespace langfuse-v3-preview
Translation of texts in picture:
JapaneseEnglishネームスペースを作成するCreate namespace.
This procedure will create the namespace on SAP BTP, Kyma Runtime, where the Langfuse is going to be deployed.
Deploy Using Helm
Using Helm, a Kubernetes package manager, update dependencies and deploy to the namespace created:
helm dependency update
helm install langfuse . -n langfuse-v3-preview
Step 4: Accessing the Langfuse UI
Check Deployment Status
Run the following command to check the deployment status:
kubectl get services -n langfuse-v3-preview
According to this, the connection type of the langfuse-web application is set to ClusterIP, and no External-IP has been assigned, which makes it inaccessible from the outside. Therefore, we will execute the following command.
kubectl edit service langfuse-web -n langfuse-v3-preview
This command allows you to make changes to the configuration of langfuse-web. Let’s rewrite the settings as shown below. With this configuration, an External IP will be assigned after a short wait.
Update Service Type
Modify the settings as follows to allow for an External IP to be assigned:
spec:
type: LoadBalancer ## Previously set as Cluster IP
Confirm Access
Confirm access to the Langfuse UI by executing:
kubectl get services -n langfuse-v3-preview
Access using the following URL:
http://<External-IP>:<Port>
Sign Up and Log In
The above is the top screen when accessing the application. First, create an account by clicking on “Sign Up.”
Even if redirected to http://localhost:3000 (a browser error page may appear) after account creation, this can be ignored because default login callback is set to redirect users there. The user account should be created successfully. Reaccess the http://<External IP>:<Port> URL and log in with the created account.
Once you do so, you will successfully log in as shown. After that, you can create an Organization and a Project with any name, which will eventually lead you to the dashboard screen.
Conclusion
In this blog, we introduced the procedure for deploying the OSS Langfuse on SAP BTP, Kyma runtime. By leveraging Langfuse’s support for Kubernetes deployment, we demonstrated how to build the environment with relative ease on the compatible SAP BTP, Kyma runtime.
In the next blog, we plan to explain how to integrate Langfuse with LangChain for debugging and monitoring generative AI applications. Stay tuned!
Next Blog:
Developing, Running, and Monitoring LangChain Applications on SAP BTP
This is the translated version of original Japanese blog:LLM監視ツールLangfuseをSAP BTP, Kyma runtimeにデプロイしてみたIntroductionThis time, we will introduce a step-by-step guide to deploying Langfuse, an open-source LLM application monitoring tool, on the Kubernetes-compatible runtime SAP BTP, Kyma runtime. Monitoring is an unavoidable issue when utilizing generative AI, and we hope this blog will be helpful to you.OverviewAs generative AI becomes deeply integrated into our daily lives and work, many companies and individuals are advancing their use of applications such as chatbots and automation tools. These are being utilized across various fields, from automating customer support and enhancing data analysis to planning new products and services.One particularly noteworthy application involves using AI agents like LangChain. LangChain is a powerful framework adopted by many developers to generate advanced responses by combining multiple tools and external data sources. However, operating these AI agents comes with new challenges.Common ChallengesSome common issues include:Managing prompt quality and model performance: Designing appropriate prompts and selecting models are essential to improving response accuracy, but accurately evaluating their effectiveness can be difficult.Tracing external APIs and tools: Understanding the behavior of external services used by AI agents and quickly identifying causes of issues when they occur are crucial.Monitoring errors and response times: Real-time monitoring of performance metrics (e.g., response times, error rates) in the operating environment is essential.Leveraging user feedback: Continuously understanding how well generative AI responses meet user expectations and incorporating improvements is necessary.To address these challenges, developers strongly feel the need for tools that allow detailed monitoring and evaluation of AI agents’ operations and facilitate rapid improvements. Against this backdrop, specialized monitoring platforms for LLM applications, such as Langfuse, are gaining attention.What is Langfuse?Langfuse is an open-source LLMOps platform that supports the development, monitoring, evaluation, and debugging of applications leveraging large language models (LLMs). It supports the entire lifecycle of generative AI products, from development and testing to large-scale monitoring and debugging.Key Features and BenefitsOpen-source availability: Langfuse is provided as open-source, allowing for self-hosting. This makes customization and long-term use easier.Integration with various frameworks: Langfuse seamlessly integrates with major LLM frameworks like LangChain and LlamaIndex, making it easy to incorporate into existing workflows.Detailed traceability: It captures the complete context of products, including external API calls, tools, and prompts, enabling detailed tracking of application behavior.Real-time metrics monitoring: Real-time monitoring of key performance indicators such as response times, error rates, and throughput allows for prompt responses.Collecting user feedback: Feedback from users can be gathered to improve application performance and user experience.Evaluation and testing features: By setting up evaluation workflows using LLMs or human annotations, developers can compare the performance of different models and prompts. A/B testing can also be used to determine the best solution.Flexible deployment: Langfuse supports deployment in various environments, such as on-premises or cloud, allowing for flexibility based on project scale.These features make Langfuse a powerful tool for LLM application developers, enabling them to improve application quality and operate efficiently.Referencehttps://langfuse.com/jpDeployment StepsLet’s now introduce the procedure for hosting Langfuse on SAP BTP.PrerequisitesAn account with access to SAP BTP, Kyma runtime.kubectl CLI, helm CLI, and kubelogin installed in the local development environment. If not installed, refer to the following links:kubectlhelmkubeloginStep 1: Setting up Kyma Environment in SAP BTPFirst, enable SAP BTP, Kyma runtime on SAP BTP, and grant user permissions. Refer to the following tutorial for details:https://developers.sap.com/tutorials/cp-kyma-getting-started.htmlStep 2: Configuring kubectl and Verifying ConnectionRefer to the URL below for connecting kubectl to SAP Kyma runtime:https://help.sap.com/docs/btp/sap-business-technology-platform/access-kyma-instance-using-kubectlRetrieve the kubeconfig.yaml FileDownload the kubeconfig file from the SAP BTP Cockpit by clicking the Kubeconfig URL link, which starts downloading a configuration file named kubeconfig.yaml. Use this file to connect kubectl to SAP BTP, Kyma runtime.Translation of texts in picture:JapaneseEnglishクリックすると、kubeconfig.yamlがダウンロードされるWhen you click here, kubecounfig.yaml get downloadedVerify ConnectionRun the following commands in the terminal to load the cluster settings for SAP BTP, Kyma runtime and ensure proper configuration. Replace /path/to/kubeconfig.yaml with your environment’s path. Note that the commands below are for macOS. For Windows, refer to the appropriate commands in the linked documentation. export KUBECONFIG=/path/to/kubeconfig.yaml
kubectl config get-contexts Translation of texts in picture:JapaneseEnglishkubeconfig.yamlを配置したパスを、KUBECONFIGという環境変数にセットするSet the path where kubeconfig.yaml is located to environment variable “KUBECONFIG”.kubectl config get-contexts により、正しくkubeconfig.yamlが読み込まれていることを確認するMake sure kubeconfig.yaml is properly read by the system with the command “kubectl config get-contexts”.If successful, proceed to deploy Langfuse.Step 3: Deploying Langfuse to SAP BTP, Kyma RuntimeRefer to the official guide below to deploy langfuse-k8s:https://langfuse.com/self-hosting/kubernetes-helmClone the Source Code git clone https://github.com/langfuse/langfuse-k8s.git
cd langfuse-k8s/charts/langfuse Create a NamespaceCreate a namespace for testing on SAP BTP, Kyma runtime: kubectl create namespace langfuse-v3-preview Translation of texts in picture:JapaneseEnglishネームスペースを作成するCreate namespace.This procedure will create the namespace on SAP BTP, Kyma Runtime, where the Langfuse is going to be deployed.Deploy Using HelmUsing Helm, a Kubernetes package manager, update dependencies and deploy to the namespace created: helm dependency update
helm install langfuse . -n langfuse-v3-preview Step 4: Accessing the Langfuse UICheck Deployment StatusRun the following command to check the deployment status: kubectl get services -n langfuse-v3-preview According to this, the connection type of the langfuse-web application is set to ClusterIP, and no External-IP has been assigned, which makes it inaccessible from the outside. Therefore, we will execute the following command. kubectl edit service langfuse-web -n langfuse-v3-preview This command allows you to make changes to the configuration of langfuse-web. Let’s rewrite the settings as shown below. With this configuration, an External IP will be assigned after a short wait.Update Service TypeModify the settings as follows to allow for an External IP to be assigned: spec:
type: LoadBalancer ## Previously set as Cluster IP Confirm AccessConfirm access to the Langfuse UI by executing: kubectl get services -n langfuse-v3-preview Access using the following URL: http://<External-IP>:<Port> Sign Up and Log InThe above is the top screen when accessing the application. First, create an account by clicking on “Sign Up.”Even if redirected to http://localhost:3000 (a browser error page may appear) after account creation, this can be ignored because default login callback is set to redirect users there. The user account should be created successfully. Reaccess the http://<External IP>:<Port> URL and log in with the created account.Once you do so, you will successfully log in as shown. After that, you can create an Organization and a Project with any name, which will eventually lead you to the dashboard screen.ConclusionIn this blog, we introduced the procedure for deploying the OSS Langfuse on SAP BTP, Kyma runtime. By leveraging Langfuse’s support for Kubernetes deployment, we demonstrated how to build the environment with relative ease on the compatible SAP BTP, Kyma runtime.In the next blog, we plan to explain how to integrate Langfuse with LangChain for debugging and monitoring generative AI applications. Stay tuned!Next Blog:Developing, Running, and Monitoring LangChain Applications on SAP BTP Read More Technology Blogs by SAP articles
#SAP
#SAPTechnologyblog