AI Fitness Coach App with Pose Detection

Project overview

We built an AI fitness coach app for a wellness startup that wanted to bring personal-trainer style feedback to home workouts. Users prop up their phone, start a session and the app tracks their body position, counts repetitions and gives spoken cues when form slips, then logs meals and progress alongside the training plan.

The engagement covered product discovery, UX, the Flutter apps, on-device machine learning, a back end for plans and subscriptions, and a dashboard for the startup's coaches. It is part of our wider mobile app development work for health and wellness brands, and sits alongside our meditation app project.

The client and the challenge

The startup's early product was a video library. Users followed along without knowing whether they were doing movements correctly, dropped off after a few weeks and had to use a separate app to log food. Coaches had no view of what users actually did.

The new app needed real-time form feedback that worked in living rooms with poor lighting and cluttered backgrounds, on mid-range phones, without uploading camera video. It also needed plans that adapted to progress and a nutrition log simple enough that people kept using it. Users range from beginners to regular gym-goers, so feedback had to be encouraging without being vague.

Solution architecture & delivery

We chose Flutter so iOS and Android share the workout player, camera screens and plan logic. The camera stream is processed on the device by a TensorFlow Lite pose estimation model that returns body keypoints for each frame; video is never recorded or uploaded, which kept the privacy story simple and avoided network delay. A calibration step at the start of each session checks that the whole body is visible and asks the user to adjust the phone if needed.

Keypoints feed an exercise engine we wrote per movement. For a squat, for example, it tracks hip and knee angles to detect the lowest and highest point of each rep, and checks rules such as depth and knee alignment. Smoothing across frames avoids false counts from jitter, and when confidence drops because the user leaves the frame the app pauses counting and asks them to step back into view. Cues are spoken through audio so users do not need to look at the screen mid-set, and a form score for each set is saved so progress in technique is visible over weeks.

Workout plans are generated from goals, available equipment and a short assessment, then adjusted weekly from completed sessions, rep quality and user-reported effort. The nutrition log supports quick entries, favourites and barcode lookups, with daily targets tied to the plan rather than a generic calorie number. Missed sessions trigger a gentler plan the following week instead of simply stacking up overdue workouts.

Coaches see anonymised adherence and form trends in a web dashboard and can adjust plan templates. Health data is stored with explicit user consent, can be exported or deleted from settings, and is never shared for advertising. Subscriptions, reminders and progress streaks run through the back end with push notifications. Workout sessions work offline and sync when the phone reconnects.

Key features we built

• On-device pose detection with real-time form cues
• Automatic rep counting per exercise with pause when out of frame
• Adaptive workout plans based on goals, equipment and progress
• Nutrition logging with quick add, favourites and barcode lookup
• Progress charts for workouts, body measurements and nutrition
• Push reminders and streaks to support habit building
• In-app subscriptions with a free trial flow
• Coach dashboard for plan templates and anonymised trends

How we delivered it

1. Discovery. Workshops with the startup's coaches to choose launch exercises, define form rules and agree what data the app should and should not collect.
2. ML feasibility. Pose models benchmarked on a range of mid-range phones and recorded test sessions in different rooms before committing to the design. We also tested with different body types and clothing to make sure keypoints stayed reliable.
3. Build. Two-week sprints delivering the Flutter apps, exercise engine, plans, nutrition and dashboard, with coaches validating each new exercise.
4. Beta. Closed beta with existing subscribers, tuning rep detection thresholds and simplifying the nutrition flow from feedback.
5. Launch and support. Store release using our mobile app launch checklist, followed by new exercises and OS updates under a support agreement.

Technology choices

• Flutter — one codebase for iOS and Android with smooth camera overlays, workout players and shared plan logic.
• TensorFlow Lite — runs pose estimation on the device in real time, so form feedback works offline and camera video never leaves the phone.

Outcomes

• Users get form feedback during home workouts instead of following videos blindly
• Rep counting and session logging happen automatically
• Training and nutrition live in one app rather than two
• Camera video stays on the device, simplifying privacy and consent
• Coaches see how users actually train and refine plan templates accordingly
• New exercises are added by defining movement rules rather than retraining models

FAQ