Getting Started with Rabbithole

github.com/ajbt200128/rabbithole — Open-source Rust tool for LLM-driven on-the-fly website generation — live demo: isarabbithole.com

Contents
  1. Prerequisites
  2. Installation & Build
  3. First Run
  4. Using --seed-file
  5. SQLite Persistence (--db)
  6. Setting a Cost Budget
  7. Web Tools
  8. Visiting the Site & Loading Pages
  9. Debug Console
  10. Next Steps

Rabbithole is an open-source Rust server that generates entire websites on demand using the Anthropic Claude API. You provide a seed prompt describing your homepage; every other page is generated lazily the first time it is visited and cached permanently. Pages are isolated — each is produced by a separate LLM call with no shared state beyond what you encode in link prompts. Web search and fetch tools are enabled by default, letting Claude pull real-time content into generated pages.

This guide walks through everything you need to get from zero to a running site in a few minutes.

1. Prerequisites

Rust Toolchain

Rabbithole is built with Rust stable. Install the toolchain via rustup if you haven't already: 

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, make sure cargo and rustc are on your PATH. Verify with: 

rustc --version
cargo --version

Anthropic API Key

Rabbithole calls the Claude API for every page generation. You need an Anthropic account with API access. Obtain a key from the Anthropic console and export it: 

export ANTHROPIC_API_KEY="sk-ant-..."
Warning: Every page visit that has not been cached triggers an API call. API costs can accumulate quickly, especially with web tools enabled. Always set --max-cost before running publicly.

Optional: SQLite

If you plan to use --db for persistent storage, SQLite is required. It is available on most systems; on Debian/Ubuntu: 

sudo apt install libsqlite3-dev

2. Installation & Build

Clone the repository from GitHub:

git clone https://github.com/ajbt200128/rabbithole
cd rabbithole

Build the release binary (recommended for production and better performance):

cargo build --release

The compiled binary will be at target/release/rabbithole. Build times vary; on first compile, Rust will download and compile all dependencies. Expect 1–3 minutes on a modern machine.

Alternatively, use cargo run --release -- [flags] to build and run in one step. During development, omit --release for faster (but slower-running) builds: 

cargo run -- --seed "My website about space exploration"

3. First Run

The minimum required argument is --seed (or --seed-file). The seed is a plain-text prompt describing what the homepage of your generated site should contain. The server starts on port 8080 by default. 

cargo run --release -- --seed "A homepage about space exploration"

Or, using the compiled binary directly: 

./target/release/rabbithole --seed "A homepage about space exploration"

You should see output similar to:

[INFO] Starting Rabbithole server on http://0.0.0.0:8080
[INFO] Seed prompt loaded (42 chars)
[INFO] Web tools: enabled
[INFO] Depth limit: 5
[INFO] Storage: in-memory

Open http://localhost:8080/ in your browser. The homepage generation begins immediately on first visit. See §8 Visiting the Site for how the loading screen works.

Tip: Write descriptive seed prompts. The quality and coherence of the entire generated site depends on the detail you provide here. Include the aesthetic, tone, subject matter, any recurring terminology, and navigation structure you want. See Examples for seed prompt ideas.

Changing the Port

Use --port to run on a different port:

./target/release/rabbithole --seed "My site" --port 3000

4. Using --seed-file

For longer or more complex seed prompts, store the prompt in a plain-text file and pass it with --seed-file. This is strongly recommended for any prompt longer than a sentence or two — shell quoting and escaping become awkward quickly. 

./target/release/rabbithole --seed-file ./prompts/space-homepage.txt

Example space-homepage.txt:

A richly detailed homepage for "Cosmos Explorer" — a website dedicated to space
exploration news, mission archives, and educational content. The site has a dark
theme (#0a0a1a background, white text, cyan accent #00e5ff). Navigation includes:
Home, Missions, News, Solar System, Deep Space, About. Use a dense, information-
rich layout similar to NASA's website circa 2010. All section headings in uppercase.
Feature a "Mission of the Week" section and a "Latest Launches" feed. Link to at
least 8 subpages. No hero banners wider than 600px.
Note: --seed and --seed-file are mutually exclusive. Using both will produce an error.

5. SQLite Persistence (--db)

By default, Rabbithole stores all generated pages in memory. This means every time you restart the server, all cached pages are lost and will be regenerated on next visit (incurring API cost again).

Use --db with a file path to enable SQLite-backed persistence. Pages generated in this session survive restarts: 

./target/release/rabbithole \
  --seed "A homepage about space exploration" \
  --db ./site.db

On startup, Rabbithole will create the SQLite database file if it does not exist, or re-use it if it does. All previously generated pages will be served immediately from the database without any API calls.

Storage Mode Flag Survives Restart? Best For
In-memory (default) none No Quick experiments, ephemeral demos
SQLite --db <path> Yes Development, long-running sites, cost control
Tip: For the live demo at isarabbithole.com, SQLite persistence is used so the site's page graph accumulates over time without re-generating pages on each server restart.

6. Setting a Cost Budget (--max-cost)

Each page generation calls the Claude API and consumes tokens. Costs can add up unexpectedly, especially on sites with many pages or deep recursion. The --max-cost flag sets a hard USD spending cap; once reached, new (uncached) page requests will return an error rather than triggering an API call. 

./target/release/rabbithole \
  --seed "A homepage about space exploration" \
  --db ./site.db \
  --max-cost 5.00
Strongly recommended: Always set --max-cost before exposing Rabbithole to any external traffic or leaving it running unattended. Without it, there is no upper bound on API spend.

The cost counter is tracked in-memory per process. It resets on restart unless you combine it with --db and a persistent cost log (see Configuration reference for details).

Typical Cost Estimates

Scenario Approx. Pages Est. Cost (USD)
Quick demo, no web tools 5–15 $0.10–$0.40
Medium site, web tools on 20–50 $0.50–$2.00
Large site, deep recursion 50–200+ $2.00–$10.00+

Estimates based on Claude 3.5 Sonnet pricing as of early 2025. Actual costs vary by prompt length, model, and web tool usage.

7. Web Tools

Rabbithole passes two tools to Claude when generating each page:

These tools are enabled by default. When active, Claude can search for and incorporate real, up-to-date information into generated pages — news articles, documentation, reference data, images (hotlinked), and more. This significantly increases the quality and accuracy of content-heavy pages.

Disable web tools with the --no-web-tools flag: 

./target/release/rabbithole \
  --seed "A homepage about space exploration" \
  --no-web-tools

Reasons you might disable web tools:

See Web Tools documentation for more detail on how search results are injected into the prompt, rate limiting, and tool call behavior.

8. Visiting the Site & the Loading Page

When you visit any URL that has not yet been generated, Rabbithole immediately serves a loading placeholder page while generation runs in the background. The placeholder page polls the /__ready endpoint using JavaScript: 

GET /__ready?path=/your/page/path

The /__ready endpoint returns HTTP 200 with {"ready": true} once the page has been generated and cached, or {"ready": false} if generation is still in progress. The loading page polls this every 1–2 seconds and automatically reloads when the page becomes ready.

Generation Flow

  1. Browser requests GET /some/page.html
  2. No cache hit → Rabbithole serves loading page immediately (HTTP 200)
  3. Background task spawned: calls Claude API with the mapped prompt for this URL
  4. Loading page polls GET /__ready?path=/some/page.html every ~1.5s
  5. Claude returns HTML + ---MAPPINGS--- delimiter + JSON link map
  6. HTML stored in cache; link mappings stored for future page visits
  7. /__ready returns {"ready": true}
  8. Browser reloads; full generated page served from cache

Depth Limiting

Each page has an associated depth — the number of link hops from the seed homepage. The default depth limit is 5. When a page at depth 5 would generate links to further pages, those links are still rendered as <a href> in the HTML, but clicking them will return a static "maximum depth reached" page rather than triggering a new generation. This prevents infinite recursive site expansion.

Override the depth limit with --max-depth:

./target/release/rabbithole --seed "..." --max-depth 3

9. Debug Console

Every generated page embeds debug metadata in the HTML source (visible in browser DevTools). Open DevTools with F12 and check the <head> for a <!-- rabbithole-debug --> comment block, or look at the Console tab where Rabbithole logs structured metadata as a JavaScript object on page load.

The debug output includes:

Field Description
prompt The full prompt string used to generate this page
depth This page's depth from the seed root (0 = homepage)
input_tokens Number of tokens in the prompt sent to Claude
output_tokens Number of tokens in Claude's response
cost_usd Estimated USD cost of this page generation
generation_ms Wall-clock time in milliseconds for generation
cached true if served from cache, false if freshly generated
web_tools Whether web_search/web_fetch were enabled for this generation

Example console output:

rabbithole {
  prompt: "A page about the Apollo 11 mission. Site: Cosmos Explorer...",
  depth: 2,
  input_tokens: 4821,
  output_tokens: 3104,
  cost_usd: 0.0412,
  generation_ms: 8341,
  cached: false,
  web_tools: true
}

10. Next Steps

You're up and running. Here are some directions to explore:

Quick Reference

Flag Type Default Description
--seed <TEXT> string required* Inline seed prompt for the homepage
--seed-file <PATH> path required* Path to file containing the seed prompt
--db <PATH> path in-memory SQLite database file for persistent page cache
--max-cost <USD> float unlimited Hard cap on total API spend in USD
--max-depth <N> int 5 Maximum link depth from homepage
--no-web-tools flag off Disable web_search and web_fetch tools
--port <N> int 8080 HTTP server port

* One of --seed or --seed-file is required; they are mutually exclusive.

---MAPPINGS--- [{"url": "/index.html", "prompt": "Homepage / index for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that dynamically generates entire websites on the fly using LLMs (Claude API). Each page is generated in isolation when first visited, cached permanently. The ---MAPPINGS--- delimiter separates HTML from JSON link mappings. Depth tracking prevents infinite recursion (default limit: 5). Web tools (web_search, web_fetch) are on by default.\n\nThis homepage should serve as a project overview and documentation index: what Rabbithole is, why it's interesting, key features (lazy per-URL generation, permanent caching, isolation, MAPPINGS delimiter, depth limiting, SQLite persistence, web tools), quick install snippet (git clone + cargo build --release), and links to all doc sections.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), default system fonts (Arial/Helvetica), no gradients, no shadows, no rounded corners, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Simple pipe-separated nav bar at top: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About — linking to /index.html, /docs/getting-started.html, /docs/architecture.html, /docs/configuration.html, /docs/web-tools.html, /docs/deployment.html, /examples.html, /about.html. Pre/code blocks with light gray background (#f4f4f4) and 1px solid #ccc border. Dense, information-rich layout. H2 headings with bottom border. Footer with border-top."}, {"url": "/docs/architecture.html", "prompt": "Architecture deep-dive page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that dynamically generates entire websites on the fly using LLMs (Claude API). Each page is generated in isolation when first visited and cached permanently.\n\nThis page covers: (1) The core concept: each URL maps to a prompt, Claude generates full HTML + ---MAPPINGS--- delimiter + JSON array of {url, prompt} pairs. (2) Page isolation — no shared state between page generations, all context must be encoded in the prompt string. (3) The MAPPINGS delimiter format in detail: how the AI splits HTML from the JSON link map after . (4) The caching layer: in-memory HashMap vs SQLite backend, cache key is URL path. (5) The generation pipeline: request comes in → cache miss → serve loading page → spawn async task → call Claude API → parse HTML and MAPPINGS → store in cache → /__ready endpoint returns true. (6) Depth tracking: how depth is incremented per hop, stored alongside each cached page, enforced at link-mapping insertion time. (7) The /__ready polling mechanism in detail. (8) Concurrency: what happens if two requests for the same uncached URL arrive simultaneously (deduplication). (9) The system prompt structure passed to Claude.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), default system fonts (Arial/Helvetica), no gradients, no shadows, no rounded corners, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About. Pre/code blocks with #f4f4f4 background and 1px solid #ccc. Dense layout, H2 with bottom border, footer with border-top."}, {"url": "/docs/configuration.html", "prompt": "Full Configuration Reference page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that generates websites on the fly via Claude API.\n\nThis page is a comprehensive reference for all CLI flags and environment variables. Cover: --seed, --seed-file (mutually exclusive), --db (SQLite path), --max-cost (USD float, hard cap), --max-depth (default 5), --no-web-tools (disable web_search + web_fetch), --port (default 8080). Environment variables: ANTHROPIC_API_KEY (required), any others. For each flag: type, default value, description, example usage, interaction with other flags. Also cover: how the cost counter works (in-memory, resets on restart unless tracked in DB), how depth is stored, what happens when max-cost is exceeded (error page vs silent failure), how to combine flags for common scenarios (production deployment, local dev, ephemeral demo).\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), default system fonts (Arial/Helvetica), no gradients, no shadows, no rounded corners, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About — all correct paths. Pre/code blocks with #f4f4f4 and 1px solid #ccc. Dense tables for flag reference. H2 with bottom border, footer with border-top."}, {"url": "/docs/web-tools.html", "prompt": "Web Tools documentation page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that generates websites on the fly using Claude API, with web_search and web_fetch tools enabled by default.\n\nThis page covers: (1) Overview of the two tools: web_search (takes a query string, returns snippets + URLs from search results) and web_fetch (takes a URL, returns page body as plain text with HTML stripped). (2) How tools are passed to Claude in the API request (Anthropic tool_use format). (3) How tool calls work during generation: Claude can make multiple tool calls per page, results are injected back as tool_result messages. (4) Impact on page quality: real-time data, accurate facts, hotlinked images. (5) Impact on cost: each tool call adds tokens; search results can be large. (6) Rate limiting and error handling: what happens if a search fails or times out. (7) The --no-web-tools flag to disable both tools. (8) Copyright and content considerations when web tools pull in external content. (9) Examples of good vs bad tool usage patterns in generated pages. (10) How web tools interact with depth limiting and the prompt propagation system.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), default system fonts (Arial/Helvetica), no gradients, no rounded corners, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About. Pre/code with #f4f4f4 and 1px solid #ccc border. Dense, technical content. H2 with bottom border. Footer with border-top."}, {"url": "/docs/deployment.html", "prompt": "Deployment guide page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that generates websites on the fly using Claude API (Rust, cargo build --release, port 8080 default, SQLite optional).\n\nThis page covers: (1) Building a release binary: cargo build --release, binary at target/release/rabbithole. (2) Running behind nginx as a reverse proxy: sample nginx config block with proxy_pass to localhost:8080, setting Host header, timeouts (generation can take 10-30s so proxy_read_timeout should be high). (3) Running behind Caddy: sample Caddyfile. (4) Systemd service file example for running as a daemon with environment variable for ANTHROPIC_API_KEY. (5) Docker: basic Dockerfile example (FROM rust, COPY, cargo build --release, EXPOSE 8080, CMD). (6) Environment variable management: never hardcode ANTHROPIC_API_KEY, use .env files or secrets managers. (7) Security considerations: don't expose without --max-cost, consider rate limiting at the proxy layer, generated content is untrusted HTML. (8) The live demo at isarabbithole.com as a reference deployment. (9) Monitoring and log output format.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), Arial/Helvetica fonts, no gradients or shadows. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About. Pre/code with #f4f4f4 background and 1px solid #ccc. H2 with bottom border. Footer with border-top."}, {"url": "/examples.html", "prompt": "Examples gallery page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that generates entire websites on the fly using Claude API. You provide a seed prompt; every page is generated lazily on first visit and cached permanently.\n\nThis page showcases a variety of seed prompts and the kinds of sites they produce. Include at least 8-10 diverse examples with: the seed prompt text, a description of what the generated site looks like, notable features of the output, and any flags recommended (e.g. --no-web-tools for fictional sites). Examples should span: a space exploration encyclopedia, a vintage 1990s-style personal homepage, a fictional town newspaper, a documentation site for a made-up programming language, a recipe site with ethnic cuisine focus, a retro video game fan wiki, a fake corporate intranet portal, a travel blog about a fictional country, a local government services site. Also include a note about this very site (the Rabbithole docs) being generated by Rabbithole itself as a meta-example.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), Arial/Helvetica fonts, no gradients, no shadows, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About. Pre/code with #f4f4f4 and 1px solid #ccc. Dense layout. H2 with bottom border. Footer with border-top."}, {"url": "/about.html", "prompt": "About page for the Rabbithole documentation site. Rabbithole is an open-source Rust tool (github.com/ajbt200128/rabbithole, live at isarabbithole.com) that generates entire websites on the fly using Claude API (Anthropic). Created by ajbt200128 on GitHub.\n\nThis page covers: what Rabbithole is and why it was built (exploring the idea of LLM-generated navigable web content, lazily rendered on demand), the philosophical angle (every page is a one-shot LLM generation with no memory of other pages), the technology stack (Rust, Axum or similar web framework, Anthropic Claude API, SQLite via rusqlite), license (check GitHub but likely MIT), how to contribute (GitHub issues and PRs), acknowledgments. Also include a section on the meta-nature of this documentation site itself being generated by Rabbithole. Link to the GitHub repo and the live demo.\n\nDESIGN STYLE: Plain minimal HTML like gcc.gnu.org or Craigslist. White background (#ffffff), Arial/Helvetica fonts, no gradients, no shadows, no animations. Blue underlined links (#0000cc), visited purple (#551a8b). Pipe-separated nav: Home | Getting Started | Architecture | Configuration | Web Tools | Deployment | Examples | About. Pre/code blocks with #f4f4f4 background and 1px solid #ccc border. Dense layout. H2 with bottom border. Footer with border-top."}]