SAP Mobile Development Kit (MDK) integration with SAP AI Core services: Part 3 – Measurement Reading

In this series of blogs, I will showcase how to integrate SAP AI Core services into SAP Mobile Development Kit (MDK) to develop mobile applications with AI capabilities.

Part 1: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – SetupPart 2: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Business Use CasesPart 3: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement Readings (Current Blog)

In the previous blogs, we described how to set up the environment for integrating SAP Mobile Development Kit (MDK) with SAP AI Core services and explored potential use cases. Starting with this blog, we will delve into the details of each use case one by one.

SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement Readings

The main task is to create an MDK rule for analyzing measurement readings, such as automatically analyzing water meter readings . It preprocesses the images, sends them to the SAP AI Core service, and handles the response to extract and store the readings.

Code snippet of SendMeterReadingRequest.js is as below

import BinaryToBase64 from ‘../Utils/BinaryToBase64’;

},
“required”: [“detectedReadings”],
}

export default function SendMeterReadingRequest(context) {
const prompt = `Analyze the image(s) provided to determine the reading displayed on the meter. It can be anolog meter or a digital one.
Identify and extract the reading with its corresponding unit (e.g., kWh for kilowatt-hours, m^3 for cubic meters). The value could be zero. For example: 0 psi.
Ensure the output clearly specifies both the numerical value and the unit of measurement for accurate automated meter reading.
`;

// Preprocess images
const platform = context.nativescript.platformModule;
attachments.forEach((attachment) => {
let base64String = BinaryToBase64(context, attachment.content);
const imageData = {
“type”: “image_url”,
“image_url”: {
“url”: `${IMAGE_URL_PREFIX}${base64String}`
}
}
body.messages[0].content.push(imageData);
});

A step-by-step breakdown:

Imports and Constants:BinaryToBase64 is imported from ../Utils/BinaryToBase64.IMAGE_URL_PREFIX is a constant string used to prefix the base64-encoded image data.schema defines the structure of the expected response from the AI service.

Function Definition:SendMeterReadingRequest is the main function that takes MDK app context as an argument.

Prompt and Request Body:The prompt is a string that describes the task for the AI service. In our case, we use plain language to ask the AI to analyze the image and extract the reading. We also request the AI to respond in the JSON data format specified in the schema.method is set to “POST” as the requirement of SAP AI Core servicebody is an object that contains the request payload, including the prompt and schema for the expected response.

Headers and Service Details:headers define the content type and resource group for the request.service and path specify the endpoint for the AI service, which should be aligned with what you defined in SAP BTP Cockpit and SAP Mobile Services.

Attachments and Preprocessing:pageProxy is used to get the attachments (images) from the context.The code iterates over each attachment, converts it to a base64 string using BinaryToBase64, and adds it to the request body

Sending the Request:The function logs a message indicating that the request is being sent.context.executeAction is called to send the request to the AI service.The response is parsed, and the detected readings are stored in the client data of the main page.

Handling the Response:The response from the AI service is parsed to extract the detected readings.These readings are stored in the client data of the main page for further use.

Key Functions and Variables

BinaryToBase64: Converts binary data to a base64-encoded string.IMAGE_URL_PREFIX: Prefix for base64-encoded image data URLs.schema: JSON schema defining the expected structure of the AI service response.SendMeterReadingRequest: Main function that processes images and sends a request to the AI service.prompt: Instruction for the AI service.body: Request payload containing the prompt and images.headers: HTTP headers for the request.service: AI service endpoint.path: Path to the AI service API.attachments: Images to be processed and sent to the AI service.

Code snippet of BinaryToBase64.js is as below

function Uint8ArrayToBase64(uint8Array) {
let binaryString = ”;
const CHUNK_SIZE = 0x8000; // 32768

for (let i = 0; i < uint8Array.length; i += CHUNK_SIZE) {
const chunk = uint8Array.subarray(i, i + CHUNK_SIZE);
binaryString += String.fromCharCode.apply(null, chunk);
}
return btoa(binaryString);
}

Uint8ArrayToBase64: Converts a Uint8Array to a Base64 string by processing it in chunks.

BinaryToBase64: Converts binary content to a Base64 string, handling different platforms (Android, iOS, Web) appropriately.

In one of the pages of your MDK app, you can specify the user interface to bind the meter readings result from AI. For example,

{
“_Type”: “Section.Type.SimplePropertyCollection”,
“_Name”: “MeterReadingResults”,
“Visible”: true,
“Header”: {
“Items”: [
{
“_Type”: “SectionHeaderItem.Type.Label”,
“Title”: “Meter Readings”,
“Position”: “Left”
}
]
},
“Layout”: {
“NumberOfColumns”: 1
},
“SimplePropertyCell”: {
“KeyName” : “{unit}”,
“Value” : “{reading}”
},
“Target”: “/MyApp/Rules/ImageAnalysis/GetMeterReadingResults.js”

Code snippet of GetMeterReadingResults.js

export default function GetMeterReadingResults(context) {
const pageProxy = context.evaluateTargetPathForAPI(“#Page:GeneratedServiceOrderPage”);
const cd = pageProxy.getClientData();
if (cd && cd.MeterReadingResults) {
return cd.MeterReadingResults[0]; // demo purpose to only get first reading read
} else {
return [];
}
}

With above code snippets, we show case how to preprocesses the images, transmits them to the SAP AI Core service for analysis, and manages the response to extract and store the meter readings.

In our next blog, we will explore how to detect anomalies using a similar approach.

In this series of blogs, I will showcase how to integrate SAP AI Core services into SAP Mobile Development Kit (MDK) to develop mobile applications with AI capabilities.Part 1: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – SetupPart 2: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Business Use CasesPart 3: SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement Readings (Current Blog)In the previous blogs, we described how to set up the environment for integrating SAP Mobile Development Kit (MDK) with SAP AI Core services and explored potential use cases. Starting with this blog, we will delve into the details of each use case one by one.SAP Mobile Development Kit (MDK) integration with SAP AI Core services – Measurement ReadingsThe main task is to create an MDK rule for analyzing measurement readings, such as automatically analyzing water meter readings . It preprocesses the images, sends them to the SAP AI Core service, and handles the response to extract and store the readings.Code snippet of SendMeterReadingRequest.js is as below import BinaryToBase64 from ‘../Utils/BinaryToBase64’;

},
“required”: [“detectedReadings”],
}

console.log(“Sending meter reading request…”);
return context.executeAction({
“Name”: “/MDKDevApp/Actions/SendRequest.action”,
“Properties”: {
“Target”: {
“Path”: path,
“RequestProperties”: {
“Method”: method,
“Headers”: headers,
“Body”: JSON.stringify(body)
},
“Service”: service
},
“ActivityIndicatorText”: “Sending meter reading request…”,
}
}).then(response => {
const results = JSON.parse(response.data.choices[0].message.function_call.arguments)
const mainPage = context.evaluateTargetPathForAPI(“#Page:GeneratedServiceOrderPage”);
const cd = mainPage.getClientData();
cd.MeterReadingResults = results.detectedReadings;
console.log(“Meter reading results:”, results.detectedReadings, “n”);
return true;
});
} A step-by-step breakdown:Imports and Constants:BinaryToBase64 is imported from ../Utils/BinaryToBase64.IMAGE_URL_PREFIX is a constant string used to prefix the base64-encoded image data.schema defines the structure of the expected response from the AI service. Function Definition:SendMeterReadingRequest is the main function that takes MDK app context as an argument. Prompt and Request Body:The prompt is a string that describes the task for the AI service. In our case, we use plain language to ask the AI to analyze the image and extract the reading. We also request the AI to respond in the JSON data format specified in the schema.method is set to “POST” as the requirement of SAP AI Core servicebody is an object that contains the request payload, including the prompt and schema for the expected response. Headers and Service Details:headers define the content type and resource group for the request.service and path specify the endpoint for the AI service, which should be aligned with what you defined in SAP BTP Cockpit and SAP Mobile Services. Attachments and Preprocessing:pageProxy is used to get the attachments (images) from the context.The code iterates over each attachment, converts it to a base64 string using BinaryToBase64, and adds it to the request body Sending the Request:The function logs a message indicating that the request is being sent.context.executeAction is called to send the request to the AI service.The response is parsed, and the detected readings are stored in the client data of the main page. Handling the Response:The response from the AI service is parsed to extract the detected readings.These readings are stored in the client data of the main page for further use. Key Functions and VariablesBinaryToBase64: Converts binary data to a base64-encoded string.IMAGE_URL_PREFIX: Prefix for base64-encoded image data URLs.schema: JSON schema defining the expected structure of the AI service response.SendMeterReadingRequest: Main function that processes images and sends a request to the AI service.prompt: Instruction for the AI service.body: Request payload containing the prompt and images.headers: HTTP headers for the request.service: AI service endpoint.path: Path to the AI service API.attachments: Images to be processed and sent to the AI service.Code snippet of BinaryToBase64.js is as below function Uint8ArrayToBase64(uint8Array) {
let binaryString = ”;
const CHUNK_SIZE = 0x8000; // 32768

for (let i = 0; i < uint8Array.length; i += CHUNK_SIZE) {
const chunk = uint8Array.subarray(i, i + CHUNK_SIZE);
binaryString += String.fromCharCode.apply(null, chunk);
}
return btoa(binaryString);
}

export default function BinaryToBase64(context, content) {
let base64String = “”;
let platform = context.nativescript.platformModule;
if (platform.isAndroid) {
base64String = android.util.Base64.encodeToString(content, android.util.DEFAULT);
} else if (platform.isIOS) {
base64String = content.base64EncodedStringWithOptions(0);
} else { // Assume MDK Web
base64String = Uint8ArrayToBase64(new Uint8Array(content));
}
return base64String;
} Uint8ArrayToBase64: Converts a Uint8Array to a Base64 string by processing it in chunks.BinaryToBase64: Converts binary content to a Base64 string, handling different platforms (Android, iOS, Web) appropriately.In one of the pages of your MDK app, you can specify the user interface to bind the meter readings result from AI. For example, {
“_Type”: “Section.Type.SimplePropertyCollection”,
“_Name”: “MeterReadingResults”,
“Visible”: true,
“Header”: {
“Items”: [
{
“_Type”: “SectionHeaderItem.Type.Label”,
“Title”: “Meter Readings”,
“Position”: “Left”
}
]
},
“Layout”: {
“NumberOfColumns”: 1
},
“SimplePropertyCell”: {
“KeyName” : “{unit}”,
“Value” : “{reading}”
},
“Target”: “/MyApp/Rules/ImageAnalysis/GetMeterReadingResults.js” Code snippet of GetMeterReadingResults.js export default function GetMeterReadingResults(context) {
const pageProxy = context.evaluateTargetPathForAPI(“#Page:GeneratedServiceOrderPage”);
const cd = pageProxy.getClientData();
if (cd && cd.MeterReadingResults) {
return cd.MeterReadingResults[0]; // demo purpose to only get first reading read
} else {
return [];
}
} With above code snippets, we show case how to preprocesses the images, transmits them to the SAP AI Core service for analysis, and manages the response to extract and store the meter readings.In our next blog, we will explore how to detect anomalies using a similar approach. Read More Technology Blogs by SAP articles