MYRAA
A JARVIS-style AI voice companion and desktop automation assistant built with the Google Gemini Live API. MYRAA combines real-time bidirectional voice conversations with 190 desktop control tools, persistent memory, and a holographic animated UI — all packaged as a cross-platform Electron desktop application.
Problem
Desktop AI assistants like JARVIS remain fictional — existing voice assistants are cloud-locked, lack deep OS-level control, have no persistent memory, and can't handle real-time interruption or screen-level multimodal understanding. Building a truly capable personal AI companion requires bridging voice AI, desktop automation, browser control, and persistent context into a single cohesive experience.
Approach
Designed a 3-tier architecture separating concerns cleanly: a React/Vite frontend for the holographic UI, a Node.js/WebSocket server bridging the Gemini Live API for real-time audio/video streaming, and a Python FastAPI agent exposing 190 desktop control tools. The system uses Gemini's function calling to dynamically route voice commands to the appropriate tool, with a persistent memory engine that extracts and consolidates facts from conversations across sessions.
Key Engineering Decisions
Gemini Live API over traditional STT→LLM→TTS pipeline
Chose Gemini's native bidirectional audio streaming to eliminate the latency of sequential speech-to-text, LLM inference, and text-to-speech stages. This enables natural interruption (user can cut off mid-sentence), real-time screen sharing with multimodal vision, and sub-second response latency — critical for a conversational companion feel.
Python FastAPI agent as a separate process
Separated the 190-tool desktop control layer into an independent Python/FastAPI process communicating over HTTP. This isolates OS-level automation (Playwright, pynput, OCR, system controls) from the Node server, allows independent scaling and crash recovery, enables PyInstaller freezing so end users don't need Python installed, and keeps tool modules cleanly organized across 31 categories.
WebSocket bridge for Gemini Live's streaming protocol
Built a dedicated WebSocket server on /live that bridges Gemini's bidirectional audio/video stream to the browser client. This enables real-time PCM audio forwarding (16kHz in, 24kHz out), compressed video frame streaming for screen sharing, function call dispatching to the Python agent, and memory synchronization — all over a single persistent connection.
Two-step confirmation for destructive power actions
Implemented a token-based safety system for shutdown/restart/sleep/lock: the model first requests permission verbally, then issues a tool call that returns a single-use 60-second confirmation token. The user must verbally confirm, and a second tool call with the token executes the action. This prevents accidental or adversarial power actions while keeping the voice interaction natural.
Persistent memory with Gemini-driven consolidation
Rather than simple chat history, built a 7-category memory system (Identity, Preferences, Goals, Projects, Relationships, Emotional, Behavior) where a background Gemini process continuously analyzes conversations and issues ADD/UPDATE/REMOVE transactions. Memories are injected into the system prompt as a 'Memory Core', giving the companion genuine long-term context about the user.
Electron + PyInstaller for zero-dependency distribution
Packaged the entire stack as an Electron desktop app with the Python agent frozen via PyInstaller. This delivers a single-installer experience (Windows NSIS/portable, Linux AppImage/.deb, macOS .dmg) where end users never need to install Node.js, Python, or any runtime — the companion just works out of the box.
Results
190
Desktop control tools across 31 categories
<1s
Voice response latency via Gemini Live streaming
50
Playwright browser automation tools for full web control
7
Persistent memory categories with auto-consolidation
3
Platform installers (Windows, Linux, macOS) with zero runtime dependencies
10+
Integrated services (email, calendar, RSS, ambient sounds, automations, meeting assistant)
Technologies