OpenMuse

Alpha, for self-hosting and building on. Open-ended reasoning, live Google accounts, and CopilotKit Rich Threads require their own configuration. See what is verified and the roadmap.

On iPhone, ask OpenMuse to find interesting stories on Hacker News and summarize CopilotKit. On desktop, ask it to check the school-trip email, open the message, and research exhibits at Monterey Bay Aquarium. The agent shows email and browser results inline. Take control opens that same browser session when you need it.

The 38-second iPhone and 42-second desktop web demos show the current interface, framed in 16:9. The send arrow becomes a stop square inside the input pill while the agent replies, then switches back. Stopping keeps your draft intact. See the recording notes for the model setup and reproduction steps.

Mobile MP4 · Web MP4 · Recording details and reproduction

OpenMuse is a personal-agent application inspired by Meta Muse and patterned after OpenBot's emphasis on an agent's computer, visible work, and rich results. It runs its own server, task worker, and browser worker. You can inspect and change the source under the MIT license.

The computer combines persistent Chromium and an optional Linux workspace. The agent can browse public pages, run commands in its own container, work with files, and move PDFs between the computer and the app. You can open its browser or terminal and continue the work. Graphical desktops and autonomous checkout remain future work.

The feature inventory maps the Muse references to the implementation. Health/bank/social connectors, device push, voice, generated executable tools, and automatic reservations/payments are on the roadmap.

Requirements: Node 24 LTS and pnpm 11.19.0. The local app needs no model, Google account, Docker, or Intelligence subscription.

git clone https://github.com/CopilotKit/OpenMuse.git openmuse
cd openmuse
pnpm install --frozen-lockfile
cp .env.example .env
pnpm dev

In another terminal:

pnpm dev:web

Open localhost:8081. The API runs at localhost:8787/api/health.

1. In Chat, send “Complete the permission slip”. Open the task, supply fictional form values, inspect the saved PDF, and review the prepared reply. This writes only to the local mailbox.
2. In Goals → Track, create a built-in availability watch, then change the built-in test page to trigger an alert.
3. In Menu → Delegate task → Finance, use Try example transactions to create an interactive spending tracker.
4. Start the browser worker and configure a model, then ask “Check out Hacker News for cool stuff” or “Summarize copilotkit.ai”. Follow the browser inline and use Take control to open its session. For a key-free version of this flow, follow the AI Mock demo setup.

For iOS or Android, use pnpm --dir apps/mobile ios or pnpm --dir apps/mobile android. Xcode or Android tooling is required. The PDF reader needs an Expo development build; use native setup.

Copy the commented settings in .env.example into your private .env:

1. Set AGENT_BACKEND=model, MODEL=provider/model-id, and the matching provider key. CopilotKit supports the configured OpenAI, Anthropic or Google provider. Fictional data can still be used with a real model. Provider keys stay on the server.
2. For personal mail/calendar, set WORKSPACE_MODE=live, a random OPENMUSE_ACCESS_KEY of at least 24 characters, and TOKEN_ENCRYPTION_KEY containing 32 random bytes encoded as base64. Restart the API.
3. Configure a Google OAuth web client with Gmail and Calendar APIs enabled. Set GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET; register ${PUBLIC_API_URL}/api/google/callback as its redirect URI. Configure consent/test-user access in your Google project.
4. Open Apps → Gmail (or Google Calendar), connect read access, and grant write access when needed. Every send or calendar change still requires its own stored review. Changing/disconnecting the account invalidates pending connection-bound work.

Google credentials are encrypted at rest. File URLs and browser consoles use short-lived signatures. This deployment uses one owner protected by a shared access key; it is not a multi-tenant authentication system. Use HTTPS and restricted network access for a remote host. Keep the default local-data mode on loopback.

Set BROWSER_WORKER_URL=http://127.0.0.1:8790 and a random WORKER_TOKEN of at least 32 characters in .env.

pnpm --dir apps/worker exec playwright install chromium
pnpm dev:browser

Or use docker compose --env-file .env -f infra/compose.yaml up --build -d. The same token must reach the API and worker. Sessions have persistent Chromium profiles; the app can open a live screenshot console and import PDF downloads. Agent tools can read public pages and hand interactive work to the person. Worker setup and boundaries.

Build the computer image, enable it on the API, then open Computer → Terminal → Start computer:

docker build -t openmuse-computer:local apps/computer
COMPUTER_ENABLED=true pnpm dev

The API needs access to the Docker CLI and engine. Commands run in a nonroot container with no host-directory mounts or credentials. A named /workspace volume retains files when stopped. Terminal networking is disabled; public web access uses the browser worker. Commands have a 30-second limit and saved output/exit receipts. Files supports folders, text editing, and PDF transfer to/from Documents. This is a Linux container, not Meta Secure VM. Setup, Colima option, and boundaries.

By default, embedded PGlite, documents and the signing key live in .openmuse/; browser profiles live in .openmuse/browser-profiles/. Keep that directory private and back it up. The API hosts the task worker. The host must remain running for background work.

For a separate task worker, configure the same DATABASE_URL, secrets and shared DATA_DIR for both processes, then set TASK_WORKER_ENABLED=false on the API and run pnpm dev:worker. PGlite cannot be opened by separate processes. Production commands are pnpm build:server, pnpm start and pnpm start:worker. Run one API instance; task workers coordinate through SQL leases.

No hidden retry occurs after an uncertain external write. Review its provider outcome before creating a replacement. Pausing/cancelling prevents subsequent task steps; an already approved in-flight provider request may finish.

Set CPK_INTELLIGENCE_API_KEY on the server and restart it to use CopilotKit Intelligence for conversation persistence and replay. The native menu uses useThreads; rich tool results link back to saved tasks, documents, and browser sessions. The default keeps one conversation in your local database.

Intelligence is a separate service and is not included in this repository's MIT license. No project key is shipped. Configuration and validation boundaries.

flowchart TD
  Client[Expo / React Native / Web] -->|AG-UI and authenticated API| API[Hono + CopilotKit runtime]
  API --> Tasks[Durable task worker]
  API --> Threads[Optional CopilotKit Intelligence]
  API --> Store[(PGlite or PostgreSQL)]
  Tasks --> Store
  Tasks --> Review[Stored action review]
  Review --> Google[Gmail / Calendar adapters]
  Tasks --> Browser[Chromium worker + persistent profiles]
  API --> Browser
  API --> Computer[Optional Docker Linux computer]
  Tasks --> Computer
  Computer --> Volume[(Persistent workspace volume)]
  Tasks --> Files[PDF files + structured artifacts]
  API -. future adapter .-> OpenBot[OpenBot]

OpenMuse's native client and personal-agent workflows are independent of OpenBot. The disabled OpenBot adapter is pinned and contract-tested against upstream interfaces. Live user/session bridging, routine mapping, and computer backend wiring remain future work. OpenBot's Intelligence runtime is not a raw AG-UI endpoint. Integration contract.

pnpm lint
pnpm typecheck
pnpm test
pnpm build:server
pnpm build:web
pnpm build:ios
pnpm build:android
pnpm --dir apps/worker typecheck
pnpm test:browser
pnpm test:computer

Platform build scripts export JavaScript/Hermes bundles; they do not produce signed app binaries. Browser checks require installed Chromium and public fixture access. CI also exercises the browser and Linux computer containers. See contribution guidance and verification results.

Issues and pull requests are welcome. Start with CONTRIBUTING.md, ROADMAP.md, and the security policy.

MIT licensed. OpenMuse is independent of Meta and is not an official CopilotKit product. Its original interface and fictional assets are included; Meta's screenshots and mascot are not redistributed. Website, email, and document content supplies evidence, not permission to act.