Captur

An Android app that turns screenshots into Google Calendar events using on-device OCR and an AI model (Groq by default, Gemini as fallback).

How it works

1. Take a screenshot — when the floating preview appears in the corner, tap Share and select Captur (recommended). You can also share from the gallery later, or pick any image from the in-app gallery picker.
2. On-device OCR (Google ML Kit) extracts the text — no image ever leaves the device
3. Extracted text is sent to the AI model (Groq / Gemini) with the current date and timezone injected
4. The model returns structured event JSON (title, time, location, recurrence, reminders)
5. Review and edit the pre-filled form, then save to Google Calendar

Tech stack

• Framework: Expo 54 (bare workflow) + React Native 0.81 + TypeScript
• Navigation: Expo Router (file-based)
• OCR: Google ML Kit Text Recognition v2 via a custom Expo Module — fully on-device
• LLM: Groq (llama-3.3-70b-versatile) by default; Gemini (gemini-3.1-flash-lite) as fallback — switched via EXPO_PUBLIC_AI_PROVIDER at build time
• Calendar: Google Calendar API v3 via OAuth 2.0 (PKCE with expo-auth-session)
• State: Zustand (with expo-secure-store persistence for settings)
• Styling: NativeWind v4 (Tailwind CSS)
• Package manager: pnpm

Project structure

app/
  (tabs)/
    index.tsx           Home — how-to guide + gallery picker
    settings.tsx        Settings — Google auth, API key, screenshot deletion
  capture.tsx           OCR + AI pipeline (loading states)
  review.tsx            Editable event form + save to calendar

src/
  services/
    ocr.ts              Calls the native expo-ocr module
    ai.ts               Provider dispatcher — reads EXPO_PUBLIC_AI_PROVIDER
    groq.ts             Groq REST client (llama-3.3-70b-versatile)
    gemini.ts           Gemini REST client with Zod response validation
    calendar.ts         Google Calendar API + OAuth token management
    prompt.ts           Shared prompt builder (date/tz injection, few-shot examples)
  store/index.ts        Zustand store (auth, settings) with secure persistence
  types/index.ts        Shared TypeScript types
  utils/timezone.ts     Device timezone via expo-localization

modules/
  expo-ocr/             Custom Expo Module — ML Kit OCR + screenshot deletion (Kotlin)

android/                Native Android project (bare workflow)

Setup

Prerequisites

• Node.js 20+, pnpm
• Android Studio with an emulator or a physical Android device (API 24+)

1. Install dependencies

pnpm install

2. Google Cloud project

1. Go to Google Cloud Console and create a new project (or select an existing one).
2. Enable the Google Calendar API:
   • Go to APIs & Services → Enabled APIs & services
   • Click Enable APIs & services
   • Search for Google Calendar API and enable it

3. OAuth consent screen

Before creating OAuth credentials you need to configure the consent screen.

1. Go to APIs & Services → OAuth consent screen
2. Fill in the app name (e.g. Captur), user support email, and developer contact email, then save
3. Under Data Access → Add or remove scopes, add:

   https://www.googleapis.com/auth/calendar.events.owned
   

4. Under Audience → Test users, add the Google account(s) you'll use to test the app

4. OAuth credentials

1. Go to APIs & Services → Credentials → Create credentials → OAuth client ID
2. Create a client — Android type:
   • Package name: com.zawaer.captur
   • SHA-1 certificate fingerprint (run from the repo root):

     keytool -keystore android/app/debug.keystore -list -v

     Password: android. Copy the SHA-1 line from the output.
3. Create a second client — Web application type:
   • Under Authorized redirect URIs, add: http://localhost
4. Copy the Web client ID into a new .env.local file at the repo root:

   EXPO_PUBLIC_GOOGLE_WEB_CLIENT_ID=your_web_client_id.apps.googleusercontent.com
   

Both clients are required. The Android client ID is needed by the Google Sign-In SDK on the device — without it you'll get a "non-recoverable sign in failure". The Web client ID is used by expo-auth-session for the PKCE authorization code flow; the Android client ID alone does not support this flow.

5. Build and run

pnpm android

Triggers a full native build, compiles the expo-ocr ML Kit module, and installs the app.

6. Add your AI API key

The app defaults to Groq (llama-3.3-70b-versatile) for faster, more consistent inference. Gemini is available as a fallback. Set EXPO_PUBLIC_AI_PROVIDER to switch at build time.

Groq (default)

1. Go to console.groq.com/keys → Create API key
2. Make sure llama-3.3-70b-versatile is enabled at both the organization and project level in your Groq Cloud settings
3. Add the key to .env.local:

   EXPO_PUBLIC_AI_PROVIDER=groq
   EXPO_PUBLIC_GROQ_API_KEY=your_groq_api_key
   

Gemini (fallback)

1. Go to Google AI Studio → API Keys → Create API key
2. Add it to .env.local:

   EXPO_PUBLIC_AI_PROVIDER=gemini
   EXPO_PUBLIC_GEMINI_API_KEY=your_gemini_api_key
   

Android features

Share Sheet

The recommended workflow is to share directly from the screenshot preview that pops up in the corner right after taking a screenshot — tap it, hit Share, and select Captur. This is faster than going to the gallery later.

You can also open any image in your gallery → tap Share → select Captur.

Screenshot auto-deletion

After capturing, Captur can automatically delete the source screenshot from your gallery so you don't end up with duplicates.

• Delete after capturing — toggle on/off in Settings (on by default)
• Silent auto-delete — when enabled (requires granting the Manage Media permission), the system confirmation dialog is skipped and the screenshot is deleted immediately

Permissions

Permission	Why
INTERNET	Gemini API and Google Calendar API calls
READ_MEDIA_IMAGES	Accessing images shared from the gallery
MANAGE_MEDIA	Silent screenshot deletion without a system confirmation dialog
POST_NOTIFICATIONS	(reserved for future use)

TODO

• Clarifying questions — when the AI is uncertain about key details (e.g. ambiguous timezone, missing year, unclear start time), it could ask the user a short follow-up question before pre-filling the form rather than guessing or failing silently.

• Timezone inference — currently the device timezone is injected into the prompt and used when saving. This breaks when the screenshot contains times that are in a different timezone than the device (e.g. you're in the USA but the WhatsApp group is in Finland and the message says "meet at 18:00"). Some ideas, roughly ordered by effort:

  1. Gemini infers it — extend the Gemini response schema with an optional timeZone: string field (IANA, e.g. "Europe/Helsinki"). Gemini can often infer timezone from context clues: city/country names, language of the message, phone number prefixes, institution names, explicit UTC offset strings (+03:00). Fall back to device timezone when absent or unrecognised.
  2. Show & let user edit timezone — add a timezone row to the Review screen (already has the globe icon placeholder). If Gemini returned one, pre-fill it; otherwise default to device. The user can tap to change it before saving.
  3. Clarifying question — if the AI is uncertain about timezone (no explicit clue in the text), this falls under the clarifying-questions TODO above: ask "Which timezone is this event in?" before pre-filling the form.

Notes

• OCR runs entirely on-device via Google ML Kit — no image data is sent to any server
• AI provider and API keys are configured via EXPO_PUBLIC_AI_PROVIDER, EXPO_PUBLIC_GROQ_API_KEY, and EXPO_PUBLIC_GEMINI_API_KEY in .env.local
• OAuth refresh tokens are stored in the Android Keystore via expo-secure-store
• Settings (deletion toggles) are persisted across restarts via expo-secure-store
• When the AI infers a date from relative text ("next Friday", "tomorrow"), the Review screen shows an amber Inferred date badge so you can verify it before saving
• If the screenshot contains no recognisable event, the app shows a friendly message rather than an error