Once SAP AI Core services are integrated into MDK (See Part 1), there will be numerous possibilities for MDK developers to implement AI-related use cases for end users. Let me introduce one such use case: generating structured data from multimodal inputs such as text, voice, and image using Generative AI. This just serves as a small step into a new universe of AI capabilities for MDK developers.
Part 1: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Setup
Part 2: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Business Use Cases (Current Blog)
Part 3: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement readings (Coming Soon)
Enable Multimodality AI in MDK: Incorporating multimodal AI in mobile development with MDK allows for integration of various input methods, such as free text input, photo capture, and audio and video recording. Following are a few potential scenarios:
Measurement readings: Capture and record correct measurements.Anomaly detection: Detect and report anomalies/irregularities in real time.Maintenance guidelines generation: Automatically generate maintenance guidelines based on gathered data.Work order and operation recording: Efficiently record and track work orders and operation.
To enable multimodality AI in app development using MDK, we can incorporate various input methods such as free text input, photo capture, audio and video recording. This allows users to interact with the app in a more natural and versatile manner. For instance, technicians can take measurement readings directly within the app, which can then be analyzed for anomaly detection. This helps in identifying potential issues before they escalate. Additionally, the app can generate maintenance guidelines based on the data collected, ensuring users have access to up-to-date and relevant information. Furthermore, the app can facilitate the recording of work orders and operations, streamlining the workflow and ensuring that all activities are documented accurately. By enabling these AI capabilities, we can create a comprehensive and efficient solution that leverages the power of multimodality AI to enhance app functionality and user experience.
Use Case – Meter Reading and Equipment Maintenance
1. Use camera to take meter reading.
2. AI to detect anomaly with camera and predict maintenance.
3. AI to generate service operation to-do list
Use Case – Record Work Order and Operation
1. Use voice or free text to input plain text.
2. Save the input as unstructured data
3. Conduct a similarity search in the vector database to construct a prompt.
4. Send the prompt to a foundational LLM model to generate structured information.
5. Populate the generated information into a form and confirm the data transaction.
The architectural diagram and workflow of these use cases are shown below.
Step-by-step flow in the above workflow:
Step 1: begins with the MDK developer to enable SAP AI core services, as described in the previous blog
Step 2 and 3: involve using a Retrieval Argument Generation (RAG) embedding model, such as text-embedding-ada-002, which is optional for general use, to embed and index domain-specific knowledge of your use case. This includes input text chunking, creation of vector embeddings, and storing and deletion of the vector embeddings in the Embedding Storage of the SAP HANA Cloud vector engine. These steps are optional; if the publicly available foundational LLM already meets the requirements of your use cases, you can skip the RAG process. I will publish a separated blog on the details of RAG.
Next, we will explain how everything works together from the end user’s perspective.
Step 4: the end user will interact with the MDK client using various multimodal inputs such as text, images, audio, and video. This interaction can include speaking to the mobile device, taking pictures, and recording audio or video. The MDK client will then use these inputs to construct a prompt.
Step 5: If your use case requires RAG, it will proceed to retrieve knowledge from the HANA Vector Database.
Step 6: The retrieved knowledge will be combined with the end user’s input to form a prompt and sent to the Multimodal AI. The Multimodal AI is a deployment of the SAP AI Core service you set up in the previous blog. Keep in mind that the prompt here can also be considered unstructured data such as a picture just taken, a voice recording, or text just keyed in.
Step 7: The Multimodal AI will generate structured data according to the specifications in your prompt. For our use cases, this will be a JSON file containing key-value pairs, which are transformed from your unstructured inputs such as images or audio. The structured data output will be populated into the MDK client and bound to the UI controls accordingly. This allows the user to interact with the UI controls to check the meter readings, detect any anomalies, view a list of recommended To-Do items from the AI, and complete the work order subsequently.
Step 8: Once the user confirms all this information via the UI controls, the transaction data will be sent to the backend.
So far, we have explained our use cases step by step. In the upcoming blogs, we will delve into the MDK project for these use cases and explain the implementation with code examples.
Once SAP AI Core services are integrated into MDK (See Part 1), there will be numerous possibilities for MDK developers to implement AI-related use cases for end users. Let me introduce one such use case: generating structured data from multimodal inputs such as text, voice, and image using Generative AI. This just serves as a small step into a new universe of AI capabilities for MDK developers.Part 1: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – SetupPart 2: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Business Use Cases (Current Blog)Part 3: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement readings (Coming Soon)Enable Multimodality AI in MDK: Incorporating multimodal AI in mobile development with MDK allows for integration of various input methods, such as free text input, photo capture, and audio and video recording. Following are a few potential scenarios:Measurement readings: Capture and record correct measurements.Anomaly detection: Detect and report anomalies/irregularities in real time.Maintenance guidelines generation: Automatically generate maintenance guidelines based on gathered data.Work order and operation recording: Efficiently record and track work orders and operation.To enable multimodality AI in app development using MDK, we can incorporate various input methods such as free text input, photo capture, audio and video recording. This allows users to interact with the app in a more natural and versatile manner. For instance, technicians can take measurement readings directly within the app, which can then be analyzed for anomaly detection. This helps in identifying potential issues before they escalate. Additionally, the app can generate maintenance guidelines based on the data collected, ensuring users have access to up-to-date and relevant information. Furthermore, the app can facilitate the recording of work orders and operations, streamlining the workflow and ensuring that all activities are documented accurately. By enabling these AI capabilities, we can create a comprehensive and efficient solution that leverages the power of multimodality AI to enhance app functionality and user experience. Use Case – Meter Reading and Equipment Maintenance1. Use camera to take meter reading.2. AI to detect anomaly with camera and predict maintenance.3. AI to generate service operation to-do list Use Case – Record Work Order and Operation1. Use voice or free text to input plain text.2. Save the input as unstructured data3. Conduct a similarity search in the vector database to construct a prompt. 4. Send the prompt to a foundational LLM model to generate structured information. 5. Populate the generated information into a form and confirm the data transaction. The architectural diagram and workflow of these use cases are shown below. Step-by-step flow in the above workflow:Step 1: begins with the MDK developer to enable SAP AI core services, as described in the previous blogStep 2 and 3: involve using a Retrieval Argument Generation (RAG) embedding model, such as text-embedding-ada-002, which is optional for general use, to embed and index domain-specific knowledge of your use case. This includes input text chunking, creation of vector embeddings, and storing and deletion of the vector embeddings in the Embedding Storage of the SAP HANA Cloud vector engine. These steps are optional; if the publicly available foundational LLM already meets the requirements of your use cases, you can skip the RAG process. I will publish a separated blog on the details of RAG.Next, we will explain how everything works together from the end user’s perspective.Step 4: the end user will interact with the MDK client using various multimodal inputs such as text, images, audio, and video. This interaction can include speaking to the mobile device, taking pictures, and recording audio or video. The MDK client will then use these inputs to construct a prompt.Step 5: If your use case requires RAG, it will proceed to retrieve knowledge from the HANA Vector Database.Step 6: The retrieved knowledge will be combined with the end user’s input to form a prompt and sent to the Multimodal AI. The Multimodal AI is a deployment of the SAP AI Core service you set up in the previous blog. Keep in mind that the prompt here can also be considered unstructured data such as a picture just taken, a voice recording, or text just keyed in.Step 7: The Multimodal AI will generate structured data according to the specifications in your prompt. For our use cases, this will be a JSON file containing key-value pairs, which are transformed from your unstructured inputs such as images or audio. The structured data output will be populated into the MDK client and bound to the UI controls accordingly. This allows the user to interact with the UI controls to check the meter readings, detect any anomalies, view a list of recommended To-Do items from the AI, and complete the work order subsequently.Step 8: Once the user confirms all this information via the UI controls, the transaction data will be sent to the backend.So far, we have explained our use cases step by step. In the upcoming blogs, we will delve into the MDK project for these use cases and explain the implementation with code examples. Read More Technology Blogs by SAP articles
#SAP
#SAPTechnologyblog
+ There are no comments
Add yours