Image Labeling in Android with Fritz AI
Use Fritz AI’s pre-trained ML models to give your mobile application sight
Image labeling techniques can be used to obtain insights into the contents of an image. Some of the use cases for this are moderation of content as well as automatic metadata generation.
Using Fritz AI, we can do this on Android, identifying the contents of an image or even a video frame. In this piece, we’ll see how we can label images from our phone’s gallery.
Fritz AI makes it easy to teach your mobile apps to see, hear, sense, and think. Start building with a free Sandbox account.
Getting Started
The first step is to jump into your favorite editor and create a project. Once you do, copy the application ID. This can be found in your app’s build.gradle file. It will look something like this:
Next, you’ll need a Fritz AI account. For this project, you’ll want to select the “Pre-Trained Models” option. Just as a note, Fritz AI also has a model building platform (Fritz AI Studio) that allows you to build custom mobile-ready models from end-to-end.
Once you’re logged in, click on follow the 4 prompts to register your new application. First, select your target platform (here, Android).
Give your application a name and enter your application ID. Ensure you type the application ID in your app’s build.gradle file. Otherwise, the Fritz SDK won’t be able to communicate with your application.
The next step is to install the Fritz SDK. Jump over to your root-level Gradle file (build.gradle) and include the Maven repository for Fritz AI:
Next, add the dependency for the SDK to your app-level Gradle file (app/build.gradle):
Now, let’s register the FritzCustomModelService in our AndroidManifest:
The only thing remaining now is to initialize the SDK by calling Fritz.configure() with our API key:
With that in place, click next to verify that your application is able to communicate with Fritz AI.
Use Fritz Pre-trained Models
The pre-trained model we’ll use here has labels for more than 680 common objects. The model will give us the predicted label accompanied by the confidence interval.
According to the Fritz AI docs, we also need to specify aaptOptions in order to prevent compression of the tflite model. After adding it, your app/build.gradle file will look like this:
You can include the model in your app by adding this dependency in your app/build.gradle file. Note that this will make your app larger in size.
Ensure that you sync your project with your Gradle files so that the model can be downloaded.
The App Elements
This application contains just three key elements:
- a
TextViewthat will display the label of the image - a
Buttonthat when clicked prompts you to select an image - an
ImageViewfor displaying the picked image
Mobile machine learning is among the most cutting-edge and fastest-growing technologies. But how does it work, and what can you actually build with it? Our (free) ebook tackles these questions head on.
Obtaining the Image
The first thing we need to do is obtain the user’s permission so that the application can access their images. Once we get permission, we call the pickImage method, which will pick the image from the user’s phone. We do that by creating a listener on the button and overriding its onClick method:
Next, override the method used to request permission:
The pickImage will obtain the image using an Intent:
Obtain the Image
We can now obtain the picked image by overriding the onActivityResult method. The first thing we do upon obtaining the image is to set it on the imageView.
Create a Bitmap Image
Next, let’s create a bitmap image that we’ll pass on to Fritz AI in order to get the prediction.
Create a FritzVisionImage from an image
We now have a bitmap image. We can use it to create a FritzVisionImage:
Get a FritzVisionLabelPredictor
Since we’re using an on-device model, we can obtain a predictor immediately:
Label the image
We’re now ready to use this predictor to run predictions on the image. This is done using the predict function and passing it into the visionImage.
Display the Label
You can decide how you want to display the result. Here, I use a Toast as well as a TextView. The getResultString method from labelResult gives us the label and confidence score. After obtaining the result, we append it to the TextView and show it as a Toast as well.
Conclusion
Hopefully, this piece has shown you how easy it is to incorporate machine learning capabilities in your Android applications using Fritz AI. Check out the full source code in the repo below.
Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.
Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.
If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.
