Pose Estimation on Android with Fritz

By Jameson Toole

Developers using Fritz can now track the body positions of people in images and video with Pose Estimation.

Human pose estimation is the computer vision task of determining a person’s body position in a video or image. A picture goes in, and a machine learning model outputs the coordinates of detected body parts (e.g. shoulders, elbows, wrists, etc.) along with a confidence score indicating the certainty of the estimate. App developers can use pose estimation to build immersive AR experiences, AI-powered sports and fitness coaches, and gesture-based user interfaces.

With Fritz, you can now add pose estimation to your own app. Keep reading to see how.

Pose estimation running on live video.

You’ll need a free Fritz account if you don’t already have one. Sign up here. You’ll then need to follow these instructions to initialize and configure Fritz for your project.

First, add our repository in order to download the Vision API:

repositories {
maven { url “https://raw.github.com/fritzlabs/fritz-repository/master" }
}

Now include the dependencies in your app/build.gradle file

dependencies {
implementation ‘ai.fritz:core:3.0.1’
implementation ‘ai.fritz:vision:3.0.1’
implementation ‘ai.fritz.vision-pose-estimation-model:3.0.1’
}

Initialize the model and set some sensitivity parameters. A full list of options can be found here.

The pose estimation model takes a single image as input. Images can come from a camera, a photo roll, or live video. In video, each frame is passed to the model individually and the prediction result is turned into a pose object. We’ve added a convenient drawPose function to overlay skeletons on top of the original image or video.

The pose object also makes it easy for developers to access individual body joints, their positions, and how certain the model is in the estimate. For example, you can access the locations of arms detected by looping through the keypoints detected in the pose. Keypoints will often be detected even if they’re obscured or occluded by another object in the image. If the model didn’t find a body part because it was out of frame or the confidence score is too low, it will be excluded from keypoints.

That’s all it takes to add pose estimation to your app with Fritz! A free version capable of detecting a single person is available on our Basic plan. An advanced version capable of tracking multiple people is available on our Premium plan.

Get your free Fritz account at https://www.fritz.ai. For additional resources, dive into the documentation, or see a full demonstration in our open source Heartbeat app.

If you need some extra inspiration, this technology was featured in Apple’s Keynote address at the 2018 iPhone event. Homecourt demonstrated how basketball players could see advanced analytics during practice sessions and get coaching feedback right from their phones.

Editor’s Note: Ready to dive into some code? Check out Fritz on GitHub. You’ll find open source, mobile-friendly implementations of the popular machine and deep learning models along with training scripts, project templates, and tools for building your own ML-powered iOS and Android apps.

Join us on Slack for help with technical problems, to share what you’re working on, or just chat with us about mobile development and machine learning. And follow us on Twitter and LinkedIn for the all the latest content, news, and more from the mobile machine learning world.