Skip to Content
Previous

Image Similarity Scoring ML service with SAPUI5

By Abdel DADOUCHE

Discover how to implement SAP Leonardo Machine Learning Functional Service in a SAPUI5 application

Details

You will learn

In this tutorial, you will learn how to leverage the Image Feature Extraction & Similarity Scoring SAP Leonardo Machine Learning Functional Services published from the SAP API Business Hub sandbox in a SAPUI5 application.

The Image Feature Extraction service allows you to extract a vector of features for any given image which can be used with Similarity Scoring service to compare vectors of features and compute a similarity score (cosine distance) ranging from -1 to 1.


Step 1: Get Your Sandbox URL

In order to consume the Image Feature Extraction & Similarity Scoring Machine Learning Functional Services, you will first need to get the service URI, your API Key and the request and response parameters.

Go to https://api.sap.com/ and click on the Browse tile.

SAP API Business Hub

Then you will be able to search for the SAP Leonardo Machine Learning - Functional Services, then click on the package found.

SAP API Business Hub

Click on Artifacts, then click on the Image Feature Extraction API.

SAP API Business Hub

On the Resource tab, you can notice the Image Feature Extraction API has only one resource (or service): /inference_sync.

If you expand the /inference_sync resource and look for the Parameters section, you will not that the service request require the following:

  • files (required) : the list of file(s) to be uploaded. Either:
    • one image file (image formats, such as .jpeg, .png, .tif, or .bmp)
    • one archive file containing multiple image files (format .zip, .tar.gz, or tar)
SAP API Business Hub

Now click on the Overview tab.

Note: the term inference refers to the application phase (scoring) an existing model (as opposed to the training or inception phase) and sync for synchronous.

SAP API Business Hub

As displayed on the screen, the sandbox URL for the Image Feature Extraction API where we need to append the API resource:

https://sandbox.api.sap.com/ml/featureextraction/inference_sync

Repeat the above instruction for the Similarity Scoring API.

the sandbox URL for the Similarity Scoring API where we need to append the API resource:

https://sandbox.api.sap.com/ml/similarityscoring/inference_sync

If you expand the /inference_sync resource and look for the Parameters section, you will not that the service request require the following:

  • options (required) : a JSON string with the following attributes:

    • numSimilarVectors (required): the number of most similar vectors to return in the response
    • algorithm (optional): the algorithm to use for calculation, one of [naive, matrix_mult, clustering]
    • example: {“numSimilarVectors”:5}

  • files (required) : the archive file without folder hierarchy containing files containing a vector of features (archive format zip, tar.gz, or tar)

    • Example for a vector of features:

    [0.012213259239223229, 0.06602939146591502, 0.2948209592491525, 0.3951995979880405]
    
Step 2: Get Your API key

When using any of the APIs outside of the SAP API Business Hub, an application key will be needed in every request header of your APIs calls.

To get to your API key, click on the key icon in the top right corner of the page. Click on the key icon.

The following pop-up should appear. Click on the Copy API Key button and save it in a text editor.

SAP API Business Hub

Now, let’s build a SAPUI5 application! But before doing so let’s first add the destination to connect to the SAP API Business Hub.

Step 3: Access the SAP Cloud Platform Cockpit

Go to your SAP Cloud Platform Cockpit account and access “Your Personal Developer Account”.

SAP Cloud Platform Cockpit
Step 4: Configure your destination

You will need to create a destination in your SAP Cloud Platform account that allow will your applications to connect to external APIs such as the SAP API Business Hub.

On the left side bar, you can navigate in Connectivity > Destinations.

Your Personal Developer Account

On the Destinations overview page, click on New Destination

Destinations

Enter the following information:

Field Name Value
Name sapui5ml-api
Type HTTP
Description SAP Leonardo Machine Learning APIs
URL https://sandbox.api.sap.com/ml
Proxy Type Internet
Authentication NoAuthentication

Then you will need to add the following properties to the destination:

Property Name Value
WebIDEEnabled true

Click on Save

New Destinations

You can use the Check Connectivity button HTML5 Applications next to the new Destination to validate that the URL can be accessed.

Step 5: Open the Web IDE

On the left side bar, you can navigate in Services, then using the search box enter Web IDE.

Web IDE

Click on the tile, then click on Open SAP Web IDE.

Web IDE

You will get access to the SAP Web IDE main page:

Web IDE
Step 6: Create your application using the SAPUI5 template

Click on New Project from Template in the Create Project section or use the File > New > Project from Template.

Project

Select the SAPUI5 Application tile, then click on Next

Project

Enter the following information, then click on Next

Field Name Value
Project Name sapui5ml-img-similarityscoring
Namespace demo
Project

Enter the following information, then click on Finish

Field Name Value
View Type XML
View Name demo
Project
Step 7: Extend the application resource roots

In order to ease the use of the provided code, we will add a new SAPUI5 resource roots. The main reason for this is that the rule used to generate the initial resource root by the project template has change many time over the time.

Edit the index.html file located under Workspace > sapui5ml > webapp and add the below element to the existing data-sap-ui-resourceroots property around line 15 (don’t forget the comma in between the existing element and the new one).

"sapui5ml": ""

It should eventually look something like this:

data-sap-ui-resourceroots='{"demosapui5ml-img-similarityscoring": "", "sapui5ml": ""}'

Click on the Save Button button (or press CTRL+S).

Step 8: Configure your SAPUI5 application

In order to use the previously configured destination, we need to add its declaration into the neo-app.json file along with the header white list configuration that will prevent HTTP header parameters to be filtered out.

Edit the neo-app.json file located under Workspace > sapui5ml-img-similarityscoring and replace the current content with the below code.

Then click on the Save Button button (or press CTRL+S).

{
	"welcomeFile": "/webapp/index.html",
	"routes": [{
		"path": "/resources",
		"target": {
			"type": "service",
			"name": "sapui5",
			"entryPath": "/resources"
		},
		"description": "SAPUI5 Resources"
	}, {
		"path": "/test-resources",
		"target": {
			"type": "service",
			"name": "sapui5",
			"entryPath": "/test-resources"
		},
		"description": "SAPUI5 Test Resources"
	}, {
		"path": "/ml",
		"target": {
			"type": "destination",
			"name": "sapui5ml-api"
		},
		"description": "ML API destination"
	}],
	"sendWelcomeFileRedirect": true,
	"headerWhiteList": [
		"APIKey"
	]
}

Note: headerWhiteList

By default, headers element like the APIKey will be blocked when used in a SAPUI5 control like the FileUploader.
This is the reason why we add it to the white list.

 

Step 9: Store your API setting in a JSON model

There are multiple options to achieve this goal. Here we will use a pre-loaded JSON model configured in the manifest.json file.

Create a new file named demo.json under Workspace > sapui5ml-img-similarityscoring > webapp > model, copy the below code and make sure you replace <<<<< COPY YOUR API KEY >>>>> by your the API key we retrieved in step 2.

Then click on the Save Button button (or press CTRL+S).

{
  "url_featureextraction" : "/ml/featureextraction/inference_sync",
	"url_similarityscoring" : "/ml/similarityscoring/inference_sync",
	"APIKey":"<<<<< COPY YOUR API KEY >>>>>"
}

Edit the manifest.json file located under Workspace > sapui5ml-img-similarityscoring > webapp and locate the models section (around line 55), and update the section like this:

Then click on the Save Button button (or press CTRL+S).

"models": {
  "i18n": {
    "type": "sap.ui.model.resource.ResourceModel",
    "settings": {
      "bundleName": "demosapui5ml-img-similarityscoring.i18n.i18n"
    }
  },
  "demo": {
    "type": "sap.ui.model.json.JSONModel",
    "preload": true,
    "uri": "model/demo.json"
  }
}
Step 10: Extend your SAPUI5 application With JSZip

JSZip is a JavaScript library for creating, reading and editing .zip files, with a lovely and simple API.

JSZip is dual licensed. You may use it under the MIT license or the GPLv3 license. Make sure o have a look at the LICENSE condition before continuing with the tutorial.

For detailed instructions about how to configure you SAPUI5 application with JSZip, you can refer to the following blog: Give the power of Zip to you SAPUI5 applications

For more details JSZip, you can refer to : https://stuk.github.io/jszip/

Step 11: Extend your SAPUI5 controller (1/4)

In this step, you will add a generic function to call the API. You can notice that this function allows you to use either Ajax or XHR depending on the mode parameter.

Edit the demo.controller.js file located under Workspace > sapui5ml-img-similarityscoring > webapp > controller and add the following function to the controller.

Then click on the Save Button button (or press CTRL+S).

callService: function(oController, service, url, type, mode, apiKey, formData, fnPrecessResult) {
	var ajaxSuccess = function(data, status, jqXHR) {
		// set the response as JSON in the demo model
		var fileName = formData.values().next().value.name;
		var file = formData.get("files");
		fnPrecessResult(oController, data, file, fileName);

		// close the busy indicator if all request have completed
		oController.requestCount--;
		if (oController.requestCount <= 0) {
			// close the busy indicator
			oController.oBusyIndicator.close();
		}
	};
	var ajaxError = function(jqXHR, status, message) {
		oController.getView().getModel("demo").setProperty("/resultVisible-" + service, false);
		MessageToast.show("Error for file : " + formData.values().next().value.name + " \n status: " + status + "\n message: " + message);
	};
	var xhrReadyStateChange = function() {
		if (this.readyState === this.DONE) {
			if (this.status === 200) {
				// set the response as JSON in the demo model
				var data = JSON.parse(this.response);
				var fileName = formData.values().next().value.name;
				var file = formData.get("files");
				fnPrecessResult(oController, data, file, fileName);
				// fnPrecessResult(oController, data, formData.values().next().value.name);
			} else {
				oController.getView().getModel("demo").setProperty("/resultVisible-" + service, false);
				MessageToast.show("Error for file : " + formData.values().next().value.name + " \n status: " + this.status + "\n message: " +
					this.response);
			}
			// close the busy indicator if all request have completed
			oController.requestCount--;
			if (oController.requestCount <= 0) {
				// close the busy indicator
				oController.oBusyIndicator.close();
			}
		}
	};

	if (mode === "ajax") {
		$.ajax({
			type: type,
			url: url,
			headers: {
				'Accept': 'application/json',
				'APIKey': apiKey
			},
			success: ajaxSuccess,
			error: ajaxError,
			contentType: false,
			async: false,
			data: formData,
			cache: false,
			processData: false
		});
	} else if (mode === "xhr") {
		var xhr = new XMLHttpRequest();
		xhr.withCredentials = false;
		xhr.addEventListener("readystatechange", xhrReadyStateChange);
		// setting request method & API endpoint, the last parameter is to set the calls as synchyronous
		xhr.open(type, url, false);
		// adding request headers
		xhr.setRequestHeader("Accept", "application/json");
		// API Key for API Sandbox
		xhr.setRequestHeader("APIKey", apiKey);
		// sending request
		xhr.send(formData);
	} else {
		oController.oBusyIndicator.close();
	}
}
Step 12: Extend your SAPUI5 controller (2/4)

In this step, you will add the code to process the Image Feature Extraction API call.

Edit the demo.controller.js file located under Workspace > sapui5ml-img-similarityscoring > webapp > controller and add the following function to the controller.

Then click on the Save Button button (or press CTRL+S).

onPressExtractFeatures: function(oControlEvent) {
  // get the current controller & view
  var oView = this.getView();

  // start the busy indicator
  this.oBusyIndicator = new sap.m.BusyDialog();
  this.oBusyIndicator.open();

  this.requestCount = 0;

  // clear previous results from the model
  oView.getModel("demo").setProperty("/result-featureextraction", null);
  oView.getModel("demo").setProperty("/resultVisible-featureextraction", null);
  oView.getModel("demo").setProperty("/resultVisible-similarityscoring", null);

  var srcFileIsImage = false;
  var result = oView.getModel("demo").getProperty("/result-featureextraction");
  if (!result) {
    result = [];
  }
  var processResult = function(oController, data, file, fileName) {
    if (!srcFileIsImage) {
      JSZip.loadAsync(file).then(function(zip) {
        Object.keys(zip.files).forEach(function(zipEntry) {
          zip.files[zipEntry].async("blob").then(function(zipEntryFile) {
            for (var i = 0; i < data.feature_vector_list.length; i++) {
              if (zipEntry === data.feature_vector_list[i].name) {
                // Set the URL and file name
                data.feature_vector_list[i].fileURL = URL.createObjectURL(zipEntryFile);
                data.feature_vector_list[i].name = fileName + " --- " + data.feature_vector_list[i].name;
                // push the result
                result.push(data.feature_vector_list[i]);
                // set the result back
                oController.getView().getModel("demo").setProperty("/result-featureextraction", result);
                // display the result table
                oController.getView().getModel("demo").setProperty("/resultVisible-featureextraction", true);
              }
            }
          });
        });
      });
    } else {
      // Set the URL
      data.feature_vector_list[0].fileURL = URL.createObjectURL(file);
      // push the result
      result.push(data.feature_vector_list[0]);
      // set the result back
      oController.getView().getModel("demo").setProperty("/result-featureextraction", result);
      // display the result table
      oController.getView().getModel("demo").setProperty("/resultVisible-featureextraction", true);
    }
  };

  // keep a reference of the uploaded files
  var mode = oControlEvent.getSource().data("mode");
  var url = oView.getModel("demo").getProperty("/url_featureextraction");
  var type = "POST";
  var apiKey = oView.getModel("demo").getProperty("/APIKey");
  for (var fileIndex = 0; fileIndex < oControlEvent.getParameters().files.length; fileIndex++) {
    var srcFile = oControlEvent.getParameters().files[fileIndex];
    if (srcFile.type.match('image.*')) {
      srcFileIsImage = true;
    } else {
      srcFileIsImage = false;
    }
    // create the form data to be sent in the request
    var formData = new window.FormData();
    formData.append("files", srcFile, srcFile.name);

    // increase request countor to close busy indicator
    this.requestCount++;

    // call the service
    this.callService(this, "featureextraction", url, type, mode, apiKey, formData, processResult);
  }
}

Note: JSZip library

Make sure to include the following piece of code at the very beginning of the controller code, else you will see validation errors in your code:

/* global JSZip:true */

 

Step 13: Extend your SAPUI5 controller (2/4)

In this step, you will add the code to process the Similarity Scoring API call.

Edit the demo.controller.js file located under Workspace > sapui5ml-img-similarityscoring > webapp > controller and add the following function to the controller.

Then click on the Save Button button (or press CTRL+S).

onPressScoreSimilarity: function(oControlEvent) {
  // get the current view
  var oView = this.getView();
  var oThis = this;

  // start the busy indicator
  this.oBusyIndicator = new sap.m.BusyDialog();
  this.oBusyIndicator.open();

  // clear previous results from the model
  oView.getModel("demo").setProperty("/resultVisible-similarityscoring", null);

  var zip = new JSZip();
  // create the files
  var result = oView.getModel("demo").getProperty("/result-featureextraction");
  for (var i = 0; i < result.length; i++) {
    zip.file(result[i].name, "[" + result[i].feature_vector + "]");
    result[i].result = [];
  }

  var url = oView.getModel("demo").getProperty("/url_similarityscoring");
  var type = "POST";
  var apiKey = oView.getModel("demo").getProperty("/APIKey");
  var mode = oControlEvent.getSource().data("mode");
  var options = "{\"numSimilarVectors\" : " + (result.length - 1) + "}";
  var processResult = function(oController, data, file, fileName) {
    for (var ii = 0; ii < data.similarityScoring.length; ii++) {
      for (var jj = 0; jj < result.length; jj++) {
        if (data.similarityScoring[ii].id === result[jj].name) {
          result[jj].result = data.similarityScoring[ii].similarVectors;
        }
      }
    }
    // set the result back
    oController.getView().getModel("demo").setProperty("/result-featureextraction", result);
    // display the result table
    oController.getView().getModel("demo").setProperty("/resultVisible-similarityscoring", true);
  };
  var processServiceCall = function(blob) {
    //saveAs(blob, "score.zip");
    var formData = new window.FormData();
    formData.append("files", blob, "score.zip");
    formData.append("options", options);

    oThis.callService(oThis, "similarityscoring", url, type, mode, apiKey, formData, processResult);
  };
  // call processServiceCall with the extracted features
  zip.generateAsync({
      type: "blob"
    })
    .then(processServiceCall);
Step 14: Extend your SAPUI5 controller (4/4)

In this step, you will add the Type Mismatch function that will be used when uploading the wrong type of files.

Edit the demo.controller.js file located under Workspace > sapui5ml-img-similarityscoring > webapp > controller and add the following function to the controller.

Then click on the Save Button button (or press CTRL+S).

fileTypeMissmatch: function(oControlEvent) {
  MessageToast.show("Wrong file type!");
}
Step 15: Extend your SAPUI5 view

The view will contain a carousel that will display the uploaded images along with a table to display the results along with a “File Upload” button to handle feature extraction call and a button for the similarity scoring.

Edit the demo.view.xml file located under Workspace > sapui5ml-img-similarityscoring > webapp > view and replace the existing code with the below code.

Then click on the Save Button button (or press CTRL+S).

<mvc:View xmlns:mvc="sap.ui.core.mvc" xmlns:table="sap.ui.table" xmlns:unified="sap.ui.unified" xmlns:layout="sap.ui.layout" xmlns="sap.m"
	xmlns:core="sap.ui.core" xmlns:custom="http://schemas.sap.com/sapui5/extension/sap.ui.core.CustomData/1"
	controllerName="sapui5ml.controller.demo" displayBlock="true">
	<App>
		<pages>
			<Page title="Image Similarity Scoring">
				<content>
					<Carousel pages="{path:'demo>/result-featureextraction'}" width="100%" visible="{= ${demo>/resultVisible-featureextraction} === true }">
						<pages>
							<VBox width="100%" direction="Column" alignItems="Center">
								<Image height="200px" class="sapUiLargeMargin" src="{demo>fileURL}"/>
								<Label text="File name: {demo>name}" class="sapUiLargeMargin"></Label>
								<table:Table rows="{demo>result}" enableBusyIndicator="true" selectionMode="Single" visibleRowCount="5" visible="{= ${demo>/resultVisible-similarityscoring} === true}">
									<table:columns>
										<table:Column sortProperty="id" filterProperty="label">
											<Label text="File"/>
											<table:template>
												<Text text="{demo>id}"/>
											</table:template>
										</table:Column>
										<table:Column sortProperty="score" filterProperty="score">
											<Label text="Score"/>
											<table:template>
												<Text text="{demo>score}"/>
											</table:template>
										</table:Column>
									</table:columns>
								</table:Table>
							</VBox>
						</pages>
					</Carousel>
				</content>
				<footer>
					<Toolbar width="100%">
						<content>
							<unified:FileUploader buttonOnly="true" sameFilenameAllowed="true" multiple="true" buttonText="Get Image Features" change="onPressExtractFeatures" custom:mode="ajax" fileType="zip,png,jpeg,jpg,bmp,tiff,tif" mimeType="application/x-zip-compressed,application/zip,application/octet-stream,image/png,image/jpg,image/jpeg,image/bmp,image/tiff" typeMissmatch="fileTypeMissmatch"></unified:FileUploader>
							<Button text="Score Similarity" press="onPressScoreSimilarity" custom:mode="ajax" visible="{= typeof ${demo>/resultVisible-featureextraction} !== 'undefined'}"/>
						</content>
					</Toolbar>
				</footer>
			</Page>
		</pages>
	</App>
</mvc:View>
Step 16: Test the application

Click on the Run icon Run Applications or press ALT+F5.

In the bar at the bottom, click on Get Image Features to pick a series of local pictures (at least 2).

The service will be called, and the images will be displayed in a carousel.

You can also combine image files with a zip that contains multiple images.

Then the Score Similarity will be made visible.

Click on Score Similarity to get the similarity score between the images.

Result
Solution: Project files

In case you are having problems when running the application, the complete project code can be found on the SAP Tutorial public GitHub repository.

However, this is not a repository you can clone and run the code.

You have to import the sapui5ml-img-similarityscoring directory content into your existing project directory.

Make sure you check the LICENSE before starting using its content.

Next Steps

Updated 01/11/2018

Time to Complete

40 Min

Intermediate

Next Steps

Next
Back to top