A conversational AI agent that responds to natural voice commands through browser-based speech recognition. It can search the web, check weather, perform calculations, tell time across timezones, and remember notes. Responses stream back as text and can be read aloud using OpenAI text-to-speech voices.
I built this voice assistant as a hands-on exploration of OpenAI tool calling and browser speech APIs. The goal was to create something that feels like a real assistant: you speak, it understands, it takes action, and it responds naturally.
The assistant supports five built-in tools: web search, weather lookup, math calculations, timezone-aware time queries, and a simple note-taking system. When you ask a question like "What is the weather in Austin?" or "Remember to call mom tomorrow", the model decides which tool to invoke, executes it, and weaves the result into a conversational response.
The frontend uses the Web Speech API (SpeechRecognition) to capture voice input directly in the browser. As you speak, interim transcripts appear in real time. Once the browser detects a pause, it finalizes the transcript and sends it to the backend.
The backend defines each tool as a JSON schema that describes its name, purpose, and parameters. When a user message arrives, I send it to GPT-4o-mini with the tool definitions attached. If the model decides a tool is needed, it returns a structured tool call instead of a plain response. I execute that tool, feed the result back into the conversation, and let the model generate the final answer.
The tools I implemented:
Once the assistant generates a response, it can optionally speak it aloud. I send the response text to OpenAI TTS endpoint and stream the audio back to the browser. Users can choose from six voice options (alloy, echo, fable, onyx, nova, shimmer) in the settings panel.
Like my RAG project, responses stream token-by-token into the chat. The UI creates a placeholder message before the stream begins so there is no layout jump. Tool execution happens mid-stream, with the model pausing to wait for tool results before continuing.
This project taught me how to orchestrate multi-step AI workflows where the model acts as a decision-maker. Tool calling is powerful but requires careful schema design so the model knows when and how to use each capability. The speech APIs were surprisingly smooth to integrate, though browser support varies. The hardest part was managing the async flow between speech input, API calls, tool execution, and audio playback without creating race conditions or janky UX.
This is the kind of agent architecture I would use for a production voice assistant or chatbot that needs to do more than just answer questions.
Interested in working together? I'm always open to discussing new projects and opportunities.
An open-source scheduling platform I built to replace Calendly for my own use. It connects to Google Calendar, merges availability across multiple calendars, and lets people book meetings through a clean public link. Includes AI features for generating event descriptions and follow-up drafts.
A streaming RAG powered knowledge search tool that lets users ingest their own documents, runs paragraph level embeddings, and returns grounded answers with source citations in real time. Built with OpenAI embeddings, GPT-4o-mini, a custom in-memory vector store, and a Next 16 API layer hardened with rate limits and basic abuse protections.
An interactive CTF-style playground for testing prompt injection attacks and defenses. Seven challenges with increasing difficulty teach how attackers try to extract secrets from LLMs and how defenders can build more robust systems. Each level adds new protections, from basic instructions to output filtering and multi-layer sandboxing.
From sound engineering to AI systems. Discover the journey that shaped how I build technology.