Back to blog

Smart meal tracking on whatsapp

• Sanket Munot
twilio whatsapp ai generativeai gemini computer-vision healthtech nutrition api hackathon devchallenge opensource

This is a submission for Twilio Challenge v24.06.12

What I Built

I developed a proof-of-concept WhatsApp bot that accepts an image of the meal you’re having. In return, it provides the calorie intake along with detailed nutrition information such as carbohydrates, fats, and proteins.

This is just a proof of concept; we can significantly enhance it by incorporating a database to store user data and generate regular reports. Use background workers for image saving and calorie estimation from AI, and employing another worker to send the messages.

Read more about functionality and flow on github readme

Demo

I cannot provide a demo because I used ngrok to route WhatsApp webhook requests to localhost. But here are some screenshots ->

Positive case I send an image of Indian thali Indian thali

Positive case I send an image of burger meal Burger meal

Negative case Naruto, When he is kept away from Ramen (IYKYK) When you keep Naruto away from Ramen

Twilio and AI

  • Twilio’s API is used to receive images of food sent by users via WhatsApp.
  • Google’s Generative AI model - Gemeni analyzes the received images to estimate the caloric content and nutritional breakdown.
  • Twilio sends back the detailed nutritional information to the user on WhatsApp.
  • The integration of Twilio and AI enhances user interaction by providing instant and informative responses.
© 2026 Sanket Munot