Telegram Archive Full-Text Search

TgWitness / Documentation · Investigation

Archive analysis · Local OCR and transcription · Wiki

Preview. In Journal, choose Best match · word forms next to the search field. Results are ranked with BM25 and match other forms of the same word: перевод finds переводом and переводы, transfer finds transfers, a Chinese phrase is found without spaces. Newest first · exact text keeps the previous substring search over the whole record.

The index is derived data. It is rebuilt from the signed journal, is never signed itself and never enters an export. The original records do not change.

On this page

• Search syntax
• Languages
• Where the index lives
• Results and the signed journal
• Recognized text
• Access
• Ask the archive and MCP
• Dependencies
• Benchmark

Search syntax#

Prepositions and other function words are ignored when other words are present. Look-alike letters mixed into a word (a Latin a inside a Cyrillic word), fullwidth and decorative Unicode letters and invisible characters are normalized.

Languages#

The language is detected per record from its script and letters:

The Ukrainian dictionary (pymorphy3-dicts-uk) is distributed under GPL-3.0, so it is not installed by default. Install it with pip install "tgwitness[uk-dictionary]"; the index notices the new analyzer and rebuilds itself.

Where the index lives#

The main administrator chooses the mode under Journal → Filters → Search index for each workspace. TGWITNESS_FULLTEXT=disk|memory|off sets the default for workspaces without their own choice. Switching to memory or off deletes the file. The index file has the same protection as the journal: it sits in the private data directory. It is not encrypted, and is not part of backups or off-site copies.

New records are indexed in the background within seconds; a search also indexes a short backlog first, so a record saved a moment ago is found. An existing journal is indexed after the update without blocking the panel; until then, results cover the indexed records and the journal shows the progress. Rebuild index starts again from the first record.

python -m tgwitness index status --data-dir data/live
python -m tgwitness index rebuild --data-dir data/live --json

rebuild needs the panel to be stopped (exit code 3 otherwise). With --json, standard output carries only the result; progress goes to standard error.

Results and the signed journal#

The index stores the sequence number and hash of the last journal record it covers and checks them against the journal when it opens. A replaced, restored or edited journal causes a rebuild. Every response reports this position, and every hit is read back from the signed journal by its sequence number: the index decides which records to show and in what order, never their content. Chat access is applied to the journal's own chat field, not to index content.

Recognized text#

OCR and transcripts stay in their encrypted artifacts. Their index exists only in memory and is rebuilt when a recognition job completes. A hit appears on its media.saved record, labelled as machine recognition with the engine, model and an excerpt; it is never presented as text sent in Telegram. The same index serves Search recognized text, which still also matches plain substrings.

Access#

Chat-scoped access finds only records from its chats, including recognized text of their attachments. chat: and other filters can only narrow that set. Index status shown to scoped access omits journal totals. Only the main administrator can change the mode or rebuild the index.

Ask the archive and MCP#

Ask the archive retrieves its sources from this index when it is complete: the best matches across the selected chats, with a saved text counted once. Up to 24 sources are sent; long texts are cut around their match and marked as truncated. Questions about recent changes, saved versions of one message, and searches while the index is still building use the previous chronological retrieval. The preview shows which method was used.

MCP clients can call search_messages with ranked=true and page with next_offset. Scoped connections receive only their chats.

Ranked pagination covers at most the first 1,000 results, regardless of page size. Narrow the query, chat or period to inspect further matches. Newest-first searches order messages, OCR and transcription results before selecting each page.

Dependencies#

SQLite FTS5 with the unicode61 tokenizer is required; it is part of the SQLite in current Python builds. Without it, best-match search reports that it is unavailable.

Benchmark#

python tools/bench_fulltext.py --messages 1000000 --work <empty directory> creates a deterministic signed journal of mixed Russian, Ukrainian, English, Chinese and Japanese messages with links and handles in 200 chats, builds the index and times typical searches (50 results, median of 7 runs).

Measured on Windows 11, Intel Core i9-13900K, 128 GB RAM, Python 3.12.10, SQLite 3.49.1, data on an exFAT drive, analyzer fulltext-v1+pymorphy3-ru+light-uk+snowball-en:

An exact phrase is checked literally on candidates in rank order until the page is full, so its total is not counted. Times grow with the number of matching records; narrowing the chat or period makes broad searches faster. The synthetic corpus repeats a small vocabulary, so common words match far more records than in real chats.