Documentation

OpenYabby is an open-source, voice-first AI assistant with multi-agent orchestration. It combines OpenAI's Realtime API (WebRTC) for bidirectional voice with Claude CLI spawned as child processes for autonomous task execution.

Say "Yabby" and watch a team of AI agents plan, code, design, and deploy projects with full system access to your Mac (terminal, files, browser, AppleScript, GUI). No buttons, no typing. Just your voice.

Tech Stack

Prerequisites

• Node.js 20+ (nodejs.org)
• PostgreSQL:create a database named yabby
• Redis running on localhost:6379
• Claude CLI:npm install -g @anthropic-ai/claude-code
• OpenAI API key with Realtime API access

Quick Start

# Clone the repository
git clone https://github.com/OpenYabby/OpenYabby.git
cd OpenYabby

# Install dependencies
npm install

# Configure environment
cp .env.example .env
# Edit .env with your API keys

# Start (migrations run automatically)
npm run dev

Environment Variables

Database Migrations

Migrations run automatically on startup. All migrations are idempotent (IF NOT EXISTS, ON CONFLICT). No manual migration step needed. Just start the server.

Docker (Partial)

# Start PostgreSQL + Redis via Docker
docker-compose up -d

# Then start Yabby natively (Claude CLI can't run in Docker)
npm run dev

The OpenYabby web interface is a single-page application served at http://localhost:3000. It provides a complete control center for managing agents, projects, tasks, and voice interaction.

Pages

Features

• Real-time updates via SSE (Server-Sent Events): tasks, heartbeats, notifications stream live
• Global search (Ctrl+/): search across projects, tasks, and agents
• Keyboard shortcuts:Ctrl+1 Dashboard, Ctrl+2 Tasks, Ctrl+3 Agents, Cmd+K toggle Yabby chat
• Collapsible sidebar with state persisted to localStorage
• Notification bell with dropdown for speaker notifications and plan reviews
• i18n:supports French, English, Spanish, German
• Dark/Light theme toggle
• Onboarding wizard on first visit

OpenYabby is designed voice-first. The entire system (apps, agents, projects, files, terminal) can be controlled hands-free through natural conversation. Voice is the primary way to interact with Yabby.

Getting Started: Activate Voice

Voice Pipeline: How It Works

From idea to deployed project. One voice command kicks off the full orchestration pipeline.

Full terminal access. File operations, package management, git, servers, scripts, and more.

Create agents, talk to them, switch context, check their status. All hands-free.

Create, monitor, and interact with full projects. Yabby handles the entire lifecycle.

Commands chain naturally, like a conversation. Here's a full project workflow done entirely by voice:

Pause, resume, interrupt. You're always in charge.

Full Mac access: AppleScript, bash, GUI control, screenshots, browser automation.

Speaker Verification (Optional)

An optional Python microservice (FastAPI + SpeechBrain ECAPA-TDNN) filters wake word detection to only your voice. This reduces false positives by 90%+ in multi-person environments.

• Go to Settings → Speaker Verification
• Record 3 voice samples saying "Yabby"
• Once enrolled, only your voice triggers wake word detection
• Embeddings are stored locally (never sent to cloud)
• If the service is down, detection continues normally (fail-open)

Voice Configuration

Text-Only Mode

You can also interact with Yabby by typing in the chat window (Cmd+K to toggle). Text messages are sent via DataChannel or trigger a text-only connection if the session is not active. Useful when you can't speak.

Noise Filter

Client-side regex filters block low-value utterances ("ok", "oui", "mmh", "non"... 44 patterns) from being sent to the voice model. Short audio clips (<0.2s or <3 words) are also discarded.

Agents are the core workforce of OpenYabby. Each agent is a Claude CLI process with its own session, system prompt, and working directory.

Agent Tiers

Standalone Agents

You can create agents outside of any project. These are standalone agents that persist across sessions and can be instructed directly via voice or chat.

• Standalone agents must have a valid human first name (e.g., "Marc", "Sofia", "Karim"). Technical names are rejected.
• They get their own working directory and conversation thread
• You can switch to an agent's context via voice: "Yabby, parle à Marc"
• Agents can be suspended, activated, or deleted from the Agents page

Agent Communication

Agents communicate via Redis pub/sub (yabby:agent-bus channel). When a sub-agent completes a task, the orchestrator automatically triggers a review task on the parent manager (with 60-second debounce to batch simultaneous completions).

Agent Voice Switching

You can talk directly to a specific agent. Yabby swaps the voice session instructions to that agent's context (same WebRTC connection persists). Say "Back to Yabby" to return.

Projects are containers for multi-agent work. Each project gets an isolated sandbox directory at ~/Desktop/Yabby Projects/{name}-{id}/ (configurable), auto-initialized with src/, docs/, README, .gitignore, and git init.

Project Lifecycle (5 Phases)

Creating a Project

There are several ways to create a project:

• Voice: "Yabby, crée-moi un portfolio dark mode avec une section blog"
• Dashboard: Click "New Project" on the Projects page
• Channel: Send a project request via WhatsApp, Telegram, or any connected channel

A task is a Claude CLI child process spawned to execute a specific instruction. Each task runs with full system access (bash, AppleScript, GUI, file system, browser).

Task Lifecycle

Task Runners

OpenYabby supports multiple CLI runners for task execution:

Logging

Every task produces two log files:

• logs/{taskId}-activity.log: Structured activity log (tool calls, status changes)
• logs/{taskId}-raw.log: Raw CLI output

Background Tasks (v0.1.3+)

When an agent calls Bash(run_in_background=true) for a long-running job (batch script, dev server, scraper), OpenYabby tracks it as a background task, separate from the parent CLI task. The bg process survives the CLI's exit and is monitored at the OS level via PID polling — the agent gets notified asynchronously when it ends, even minutes or hours later.

How tracking works

1. A PreToolUse hook intercepts each Bash(run_in_background=true) call and writes a per-call bookkeeper script that captures the host PID and exit code via wait $C; rc=$?.
2. The bookkeeper detaches via nohup so the bg child survives the CLI exit.
3. A central bg-watcher polls kill -0 <pid> every 30 seconds. When the PID disappears, it reads the exit code and routes the agent's next-turn notification:

Service tag

For permanent services (Node/PHP/Streamlit dev servers, watchers), the agent appends [bg:service] to the tool's description:

Bash(
  command="node server.js",
  description="dev preview server :3000 [bg:service]",
  run_in_background=true
)

The hook drops a marker file so the watcher uses the [BG_SERVICE_DIED] wording instead of treating an exit as a "completion".

Introspection & control tools

Agents have 5 dedicated tools to inspect and control their bg tasks (callable via POST /api/tools/execute):

Web UI

The Activity page shows a Background Tasks panel below the CLI events stream, with running-first sort, status badges, elapsed timer, service marker, and a 15-second auto-refresh. Both sections scroll independently.

Resilience

On server restart, the startup sweep doesn't blindly orphan rows in running state — it checks each row's PID with kill -0 first. Live bg jobs survive Yabby restarts (the watcher re-attaches automatically). Only truly-gone PIDs are marked orphaned.

Retry Detection

OpenYabby monitors task activity for infinite retry loops. When a pattern of repeated tool calls is detected (30 calls scanned), the system intervenes to unblock the stuck agent.

GUI Lock

Tasks that need to interact with the GUI (screenshots, mouse clicks) must acquire a lock (yabby:gui_lock in Redis). Only one task can control the GUI at a time. Lock auto-expires after 5 minutes or when the task finishes.

OpenYabby connects directly to WhatsApp Web via the Baileys protocol. No Meta Business API, no approval process, no cost. Just scan a QR code and you're live. Everything runs inside isolated groups so your personal chats are never touched.

Setup (2 minutes)

Getting WhatsApp connected is dead simple:

1. Open the dashboard → Channels → Config
2. Toggle WhatsApp to enabled → Save
3. A QR code appears on screen
4. On your phone: WhatsApp → Settings → Linked Devices → Link a Device
5. Scan the QR code. That's it.

What Happens After Connection

The moment the QR code is scanned:

1. Yabby automatically creates a group called “🤖 Yabby Assistant”
2. A welcome message is sent to confirm the connection
3. All Yabby interactions happen exclusively in this group
4. Your personal chats, other groups, and DMs are completely ignored

If the server restarts, the existing group is recovered automatically from the database — no duplicates.

Per-Agent Groups

This is where it gets powerful. When you create an agent (via voice, API, or dashboard), Yabby automatically creates a dedicated WhatsApp group for that agent:

• Group named “💬 [Role] [Name]” — e.g., “💬 Frontend Dev [Lucia]”
• Messages in that group are routed directly to that specific agent
• Each agent has its own isolated conversation thread

Workspace Switching

You can change an agent's working directory on the fly:

POST /api/agents/:id/change-workspace
{
  "workspace_path": "/Users/me/Projects/my-app",
  "reason": "Switch to my-app repo"
}

When the workspace changes:

• The agent's running task is gracefully stopped
• A new session starts in the new directory
• Previous conversation context is injected so the agent remembers what it was doing
• All future tasks run in the new workspace

Voice Messages

Send a voice note in the WhatsApp group and Yabby will auto-transcribe it via Whisper, then process it as a text instruction. The full voice pipeline works: your audio becomes an action.

Features

• Group isolation: Yabby only responds in its dedicated groups, never in personal chats
• Voice notes: Auto-transcribed via Whisper, treated as instructions
• Message chunking: Long responses auto-split at 4096 characters
• Deduplication: Redis-based tracking prevents double-processing
• Spam filter: Short messages (“ok”, “oui”, emojis) debounced over 2 seconds
• Notifications: Task completions, errors, and milestones broadcast to the group
• Auto-reconnect: Exponential backoff (5s → 60s max, 10 attempts) on disconnect
• Session persistence: Credentials survive restarts, no re-scanning

Slash Commands

Tips

Connect a Telegram Bot to interact with Yabby from anywhere. Supports both text and voice messages.

Setup

1. Create a bot via @BotFather on Telegram
2. Copy the Bot Token
3. Go to Channels → Config in the dashboard
4. Enable Telegram and paste the bot token
5. Start chatting with your bot on Telegram

Features

• Text messages:Send instructions naturally, auto-chunked at 4096 chars
• Voice messages:Send a voice note, Yabby transcribes it via Whisper and responds
• Voice replies:Yabby can respond with voice notes (TTS → OGG/Opus format)
• Audio file support:Attach audio files for transcription
• Same tools as voice:Full function-calling loop with max 5 tool iterations
• Slash commands:/status, /new, /reset, /help

Beyond WhatsApp and Telegram, OpenYabby supports Discord, Slack, and Signal as messaging channels.

Channel Configuration

All channels are configured from the Channels → Config tab:

• Enable/disable each channel independently
• DM policy: open (anyone can talk) or closed (whitelist only)
• Mention gating: in groups, Yabby only responds when @mentioned
• Allowed users: whitelist with recent user suggestions

Unified Conversation

All channels share the same conversation context as voice. A fact mentioned via WhatsApp is remembered when you speak to Yabby by voice, and vice versa.

Connectors let agents interact with external services. v0.1.3 ships with a growing catalog of 37 connectors organized by category, plus support for any MCP (Model Context Protocol) server.

Connector Categories

How Connectors Work

1. Browse the catalog from Connectors page
2. Click a connector → enter credentials (API key, OAuth, etc.)
3. Test the connection. Yabby validates credentials before saving
4. Once connected, the connector's tools become available to all agents
5. Connectors can be scoped to specific projects or kept global

MCP Servers

MCP (Model Context Protocol) allows agents to use external tool servers. OpenYabby can:

• Auto-configure MCP servers from the catalog (command, args, env variables)
• Custom MCP: Add any MCP server by specifying command and arguments
• Bridge tools: MCP tool schemas are automatically converted to OpenAI function-calling format
• .mcp.json generation: When a task spawns, Yabby generates a .mcp.json in the working directory so the CLI runner can access all connected MCP servers

Credential Security

All connector credentials are encrypted at rest using AES-256-GCM. The encryption key is derived from YABBY_SECRET (or auto-derived from OPENAI_API_KEY).

Schedule recurring tasks that Yabby executes automatically. The scheduler ticks every 30 seconds and supports three schedule types.

Schedule Types

Creating a Scheduled Task

1. Go to Scheduling page
2. Click New Scheduled Task
3. Enter: name, description, task prompt/template
4. Choose schedule type and configure timing
5. Optionally assign to a project or standalone agent
6. Set max retries and retry delay

Features

• Run history:View all past executions with status, task ID, and results
• Manual trigger:Run any scheduled task on demand
• Retry logic:Configurable max retries (default 3) with delay between attempts
• Pause/Resume:Temporarily disable without deleting
• Orphan recovery:Missed runs are detected and recovered on startup
• Agent queue integration:For standalone agents, tasks are queued to preserve session state

All configuration is managed through the Settings page, organized in 5 tabs. Changes are validated in real-time and hot-reloaded without server restart.

General

• UI Language: French, English, Spanish, German
• Speech Language: Language for voice recognition
• Voice settings: Model, voice selection (6 voices), noise reduction, VAD type, mic toggle
• Memory: Extraction model, embedder, extraction frequency (every N turns)
• TTS Provider: Edge TTS (free), ElevenLabs, OpenAI, System
• Task Runner: Claude, Codex, Aider, Goose, Cline, Continue, or custom binary
• LLM providers: API keys for OpenAI, Anthropic, Google, Groq, Mistral, Ollama, OpenRouter

Speaker Verification

• Enrollment status and calibration UI
• Record 3 voice samples using browser microphone + Silero VAD
• Delete enrollment to reset

Projects

• Sandbox root: Where project files are stored (default: ~/Desktop/Yabby Projects)
• Clean on archive: Delete files when a project is archived

Authentication

• Enable/disable gateway auth
• Gateway password for web access
• Session TTL (days)
• API token generation for programmatic access

Usage

• 30-day cost summary across all LLM providers
• Breakdown by provider: calls, input/output tokens, cost in USD
• Daily usage chart

System Overview

Voice Pipeline

Key Design Patterns

• Dual-write cache: All writes go to PostgreSQL + Redis simultaneously. Reads check Redis first (24h TTL), fallback to PG.
• Soft delete: Nothing is actually deleted. Status is set to archived. All queries filter WHERE status != 'archived'.
• Name resolution: Tools accept ID or name. Resolution: exact ID → exact name → ILIKE contains → fuzzy match → role match.
• SSE + WebSocket: Both channels emit identical events. Frontend uses SSE; WebSocket for presence/typing.
• Config hot-reload: Settings changes propagate via Redis pub/sub. No server restart needed.
• Fail-open: Optional services (speaker verification, tunnel) fail gracefully. The system continues working.

21 Voice Tools

These are the base tools available to Yabby during voice sessions (agents and connectors add more):

Relay Tunnel

OpenYabby includes a WebSocket tunnel to relay.openyabby.com for mobile access. A tunnel code is assigned and persisted to .env. Proxies HTTP + WebSocket traffic to localhost with auto-reconnect (exponential backoff: 2s → 30s max). Disable with DISABLE_TUNNEL=true.