Video Processing in Android with Fritz AI

Apply filters and segmentation to mobile videos with Fritz AI

Photo by Dilyara Garifullina on Unsplash

Fritz Vision includes a comprehensive API to process pre-recorded video. Some of the things we can do with it include running predictions on every frame, exporting the video as well as extracting individual frames. Fritz Vision also has the power to change old experiences into new ones using machine learning augmentations. In this article, we will see how we can use Fritz to apply filters and perform segmentation of people in a video. This is the result of such an operation.

Getting Started

The first step is to create an Android project. Once you do, note the application ID. Next, log into your Fritz account and register the application. This will set up communication between your application and Fritz AI.

Click on the next button and name your application. While you are there, don’t forget to enter your application ID. You can find the ID in your app’s build.gradle file.

With that out of the way, you can now install the Fritz AI SDK. In your root-level Gradle file (build.gradle) include the Maven repository for Fritz.

Now add the dependencies for the SDK in app/build.gradle. We add the Fritz Core, Style Transfer, People Segmentation, and Vision dependencies. Including the Style Transfer and People Segmentation models in your application will make your application larger in size. Now that you have changed the Gradle files, ensure that you sync that with your project. That will download all the necessary dependencies.

Before you close that file add renderscript support to improve image processing performance. Also, specify aaptOptions to prevent compression of TFLite models.

Now register the FritzCustomModelService in the AndroidManifest.

The next step is to initialize the SDK by calling Fritz.configure() with your API Key.

With that in place, click next to verify that your application is able to communicate with Fritz.

There’s a lot to consider when starting a mobile machine learning project. The Fritz AI Newsletter provides the in-depth resources and expertise you’ll need to get started.

Use Fritz Pre-trained Models

In this piece, we will use pre-built filters from Fritz. However, note that you can create your own filters. The model we will use will work on pre-recorded video, process it, and display the new video.

The App Elements

The application is made up of a video view that will be used to show the result after we have used Fritz to process the video.

Obtaining the Video

We shall use this video that is freely available from Pixaby. The first step is to create a videoView then parse the video link as a Uri.

Declare the Models

Let’s start by defining the pattern model from the Style Transfer module and the People Segmentation model from the segmentation module. We have already seen how to do this in a previous article. Since we are using an on-device model we can obtain the predictors immediately.

Define the Filters

The next step is to define the filter that we want to apply to the video. In this case, we declare the style and the people segmentation filter.

Create a FritzVisionVideo

With the filter ready, we now define FritzVisionVideo and pass those filters. Sounds easy, right? That’s because it is.

Display the Processed Video

The final step is to display the processed video. We do that using the videoView that we defined earlier. This is done in a couple of steps:

• First, we register a callback that will be invoked once the video is ready.
• Then, export the video file to a temporary location. You can store it on the user’s device if you so wish. Just note that you will have to get the user’s permission to write to their device.
• Finally, we display the video on the videoView once it’s ready.

Conclusion

Hopefully, this piece has shown you how easy it is to process videos in Android using Fritz AI. Check out the full source code in the repo below.

mwitiderrick/Video-Processing-in-Android-with-Fritz-AI

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.