Transcript Search

GoClaw indexes all conversations into a searchable database, giving your agent persistent memory that survives context compaction.

Overview

Transcript search solves a fundamental problem with LLM agents: context windows are finite, but conversations are forever.

When your context window fills up, GoClaw compacts old messages into summaries. Without transcript search, the details of those conversations are lost to the agent. With transcript search, your agent can query the full history and recover context on demand.

Key Features

Feature	Description
Hybrid Search	Combines semantic embeddings with BM25 keyword matching
Automatic Indexing	New messages indexed every 30 seconds
Embedding Backfill	Historical chunks get embeddings added automatically
OpenClaw Import	Merges OpenClaw conversation history into the index
Real-time Sync	New OpenClaw messages indexed while running side-by-side
Configurable Chunking	Control how messages are grouped into searchable units

How It Compares

	GoClaw Transcripts	Claude Insights	ChatGPT Memory
Storage	Local (SQLite)	Cloud	Cloud
Privacy	Your machine	Anthropic servers	OpenAI servers
Persistence	Permanent	Unknown	Limited
Cross-platform	Merges OpenClaw + GoClaw	Single platform	Single platform
Search Type	Semantic + Keyword	Unknown	Keyword?
Offline	Yes	No	No

Setup

1. Configure Embedding Provider

Transcript search requires embeddings. Any OpenAI-compatible API works:

Option A: LM Studio (Recommended for local)

{
  "llm": {
    "providers": {
      "lmstudio": {
        "driver": "openai",
        "baseURL": "http://localhost:1234"
      }
    },
    "embeddings": {
      "models": ["lmstudio/text-embedding-nomic-embed-text-v1.5"]
    }
  }
}

Option B: Ollama

{
  "llm": {
    "providers": {
      "ollama": {
        "driver": "ollama",
        "url": "http://localhost:11434"
      }
    },
    "embeddings": {
      "models": ["ollama/nomic-embed-text"]
    }
  }
}

2. Enable Transcript Indexing

{
  "transcript": {
    "enabled": true,
    "indexIntervalSeconds": 30,
    "batchSize": 100,
    "backfillBatchSize": 20
  }
}

3. Verify It’s Working

After starting GoClaw, you should see:

openai: embedding ready name=lmstudio dimensions=768
memory: provider upgraded from=none to=lmstudio
transcript: starting indexer

And periodically:

transcript: sync completed messagesProcessed=5 chunksCreated=2 progress="500/500 (100%)"
transcript: backfill progress processed=20 remaining=150 elapsed=1.2s

Configuration Reference

{
  "transcript": {
    "enabled": true,
    "indexIntervalSeconds": 30,
    "batchSize": 100,
    "backfillBatchSize": 20,
    "maxGroupGapSeconds": 300,
    "maxMessagesPerChunk": 8,
    "maxEmbeddingContentLen": 16000,
    "query": {
      "maxResults": 10,
      "minScore": 0.3,
      "vectorWeight": 0.7,
      "keywordWeight": 0.3
    }
  }
}

Indexing Options

Field	Type	Default	Description
`enabled`	bool	`true`	Enable transcript indexing
`indexIntervalSeconds`	int	`30`	How often to check for new messages
`batchSize`	int	`100`	Max messages to process per sync cycle
`backfillBatchSize`	int	`10`	Max chunks to add embeddings to per cycle
`maxGroupGapSeconds`	int	`300`	Max time gap (5 min) before starting new chunk
`maxMessagesPerChunk`	int	`8`	Max messages per conversation chunk
`maxEmbeddingContentLen`	int	`16000`	Max chars to send to embedding model

Search Options

Field	Type	Default	Description
`query.maxResults`	int	`10`	Maximum results per search
`query.minScore`	float	`0.3`	Minimum similarity score (0-1)
`query.vectorWeight`	float	`0.7`	Weight for semantic search
`query.keywordWeight`	float	`0.3`	Weight for keyword search

How It Works

Message → Chunk → Embedding

Messages arrive
    ↓
Group by session + time gap (≤5 min)
    ↓
Create conversation chunks (≤8 messages each)
    ↓
Generate embedding via LM Studio/Ollama
    ↓
Store in SQLite with vector index

Chunking Strategy

Messages are grouped into “conversation chunks” based on:

Same session — Messages from the same conversation
Time proximity — Within maxGroupGapSeconds of each other
Size limit — At most maxMessagesPerChunk messages

This creates semantically coherent units that are:

Small enough for accurate embeddings
Large enough for context (not single messages)
Temporally grouped (related discussion stays together)

Hybrid Search

When searching, GoClaw combines two approaches:

Vector Search (70% weight by default)
- Query embedded via same model
- Cosine similarity against all chunks
- Finds semantically similar content
Keyword Search (30% weight by default)
- BM25 full-text search
- Catches exact matches vector might miss
- Handles names, IDs, specific terms

Final score: vector * 0.7 + keyword * 0.3

Agent Tools