Overview
Fitness trackers estimate calorie burn from heart rate and accelerometer data, which means they need a wearable and they have no idea what exercise you are actually doing. We wanted to see how far we could get with nothing but a camera.
The result was a web application where a user creates a profile with their height, weight, age, and sex, then uploads a video of their workout or streams it live from a webcam. A convolutional neural network classifies the exercise being performed, the app times how long it was performed for, and a physiological formula converts that into an estimated calorie expenditure that is logged to the user's workout history.
I worked on this as part of a four-person team. My contributions centred on the model — assembling and labelling the dataset, setting up the transfer learning pipeline, iterating on training runs, and integrating the trained model into the Flask application's inference path.
Try It
Pre-computed inferenceThe model is a 99 MB TensorFlow network and this site is a static export, so there is no server running inference here. Instead the original 2022 model was re-run offline over these clips and every frame's output was saved. Pressing play replays those real predictions in sync with the video — the numbers below are the model's actual output, not a simulation.
How It Works
1. Upload or stream
The user uploads an MP4 of their workout or streams a live webcam feed, and selects a perceived intensity level (light, moderate, or vigorous).
2. Frame extraction
OpenCV reads the video frame by frame, converts each frame to RGB, resizes it to 224×224, and applies ImageNet mean subtraction to match the preprocessing used during training.
3. Classification
Each frame is passed through the CNN, which outputs a softmax distribution over three exercise classes: deadlift, squat, and bench press.
4. Rolling average
Predictions are pushed into a fixed-length deque and averaged. This smooths out single-frame misclassifications caused by motion blur or awkward mid-rep angles, which was the single biggest accuracy win on real video.
5. Calorie estimation
The dominant class plus the elapsed duration is combined with the user's stored profile to produce a calorie estimate, which is written to their workout history.
Model Architecture
With only 150 labelled images we had no realistic chance of training a network from scratch, so the model is built on transfer learning. We took ResNet-50 pre-trained on ImageNet, discarded its classification head, froze every convolutional layer, and trained a small new head on top. The frozen backbone already knows how to detect edges, textures, limbs, and equipment; all the head has to learn is how those features map onto three exercises.
Training setup
- SGD, learning rate 1e-4, momentum 0.9
- Categorical cross-entropy loss
- Batch size 32, 1000 epochs
- 75 / 25 stratified train-test split
Augmentation
- Rotation ±30°, zoom ±15%
- Width / height shift ±20%
- Shear ±15%, horizontal flip
- Applied to training set only
Results
The model settled at roughly 88% validation accuracy on the held-out split. The training curves show the classic transfer learning signature: loss drops steeply within the first hundred epochs as the new head initialises, then both train and validation loss decline together for the remainder of the run. Training accuracy climbs slightly above validation accuracy, which is the mild overfitting you would expect from a dataset this small, but the two never diverge sharply — the frozen backbone and the dropout layer kept it in check.
From Classification to Calories
The CNN only answers "what exercise is this?". Turning that into a calorie figure takes two more pieces. The first is basal metabolic rate, calculated from the user's profile using the Harris-Benedict equation. The second is a MET value — a metabolic equivalent that expresses how demanding an activity is relative to sitting still — chosen from a lookup table keyed on the predicted exercise and the intensity the user selected.
Dividing BMR by 24 converts it to an hourly resting burn rate, multiplying by the MET scales it to the effort of the exercise, and multiplying by the elapsed duration gives the total. A deadlift at vigorous intensity carried a MET of 9, a squat 8, and a bench press 7, with lower values for moderate and light effort.
What I'd Do Differently
This was a fourth-year project built under a deadline, and there are several decisions in it I would not make again.
The dataset was far too small
150 labelled images across three classes is enough to demonstrate transfer learning, but nowhere near enough to generalise. The model was fragile on unusual camera angles, gym backgrounds it had not seen, and any exercise outside the three it knew.
Frame-level classification ignores motion
A still frame of someone at the bottom of a squat and someone racking a bar look similar to a 2D CNN. A model with a temporal component — a 3D CNN, an LSTM over frame embeddings, or a pose-estimation front end — would capture the actual movement rather than the pose.
The MET table was hardcoded
Intensity was self-reported and mapped to a small lookup table of MET values. Inferring intensity from rep tempo or estimated load would have made the calorie output far less dependent on the user guessing correctly.
Source Code
The project is split across two repositories: one for the training pipeline and one for the Flask application that serves the model.
The demo above replays predictions generated offline by the original 2022 model. The export script that produces that data lives in the website repository as scripts/export_calorie_demo.py.