Tree-sitter

With this integration your indexing pipeline splits source files along functions, classes and other syntax boundaries instead of at a fixed length, and adds code-aware metadata to every chunk. Tree-sitter is a parser library with a grammar per language. Swiftide ships the grammars and a set of transformers that use them.

Feature flag

cargo add swiftide@0.34 --features tree-sitter

The tree-sitter feature flag adds the grammars and the transformers ChunkCode, MetadataQACode, MetadataRefsDefsCode, OutlineCodeTreeSitter and CompressCodeOutline. They are re-exported at swiftide::indexing::transformers, next to the other transformers. SupportedLanguages sits at swiftide::integrations::treesitter.

The tree-sitter-perl feature flag adds Perl and turns on tree-sitter too.

Supported languages

// Case-insensitive: "python", "Python" and "PYTHON" all parse
let language: SupportedLanguages = "python".parse()?;

indexing::Pipeline::from_loader(
    FileLoader::new("src").with_extensions(language.file_extensions()),
)
.then_chunk(ChunkCode::try_for_language(language)?)
.then_store_with(MemoryStorage::default())
.run()
.await?;

Source: snippets/src/integrations/tree_sitter.rs#L20-L29

SupportedLanguages implements FromStr and ignores case. ChunkCode, OutlineCodeTreeSitter and MetadataRefsDefsCode take a SupportedLanguages value or a string such as "rust", and return an error that says Treesitter language not supported for a name they do not know. MetadataQACode and CompressCodeOutline take no language, only an LLM client. file_extensions() gives the extensions to pass to FileLoader.

Language	Names	Extensions	Outline	References and definitions
Rust	`rust`	rs	yes	yes
TypeScript	`typescript`	ts, tsx, js, jsx	yes	yes
JavaScript	`javascript`	js, jsx	yes	yes
Python	`python`	py	yes	yes
Ruby	`ruby`	rb	yes	yes
Java	`java`	java	yes	yes
C#	`csharp`, `c-sharp`, `c#`	cs, csx	yes	yes
Go	`go`	go	no	yes
Solidity	`solidity`	sol	no	yes
C	`c`	c, h, o	no	no
C++	`cpp`, `c++`	c, h, o, cc, cpp	no	no
Elixir	`elixir`	ex, exs	no	no
HTML	`html`	html, htm, xhtml	no	no
PHP	`php`	php	no	no
Perl	`perl`	pm, pl, t	no	no

Every language works with ChunkCode. The last two columns say which languages OutlineCodeTreeSitter and MetadataRefsDefsCode handle.

Chunk size and splitting

indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
    .filter_cached(Redis::try_from_url(redis_url, "swiftide-examples")?)
    .then(MetadataQACode::new(openai_client.clone()))
    .then_chunk(ChunkCode::try_for_language_and_chunk_size(
        "rust",
        10..2048,
    )?)
    .then_in_batch(Embed::new(openai_client.clone()).with_batch_size(10))
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(1536)
            .collection_name("swiftide-examples")
            .build()?,
    )
    .run()
    .await?;

Source: examples/index_codebase.rs#L52-L68

The example also uses the redis, qdrant and openai feature flags, and needs a running Redis server for the node cache. See feature flags.

ChunkCode is a chunker. It parses each file and walks the syntax tree, adding top-level items to a chunk. It closes the chunk when the next item would bring it to the maximum size or past it. An item larger than the maximum is split along its own children, one level down at a time.

The chunk size is a ChunkSize. A number, such as 1000, is a maximum in bytes. A range, such as 10..2048, sets a minimum and a maximum: a chunk no larger than the minimum is dropped. ChunkCode::try_for_language uses a maximum of 1500 bytes. with_concurrency sets how many files are chunked at once.

Several languages in one pipeline

ChunkCode handles one language. To index several, pick a ChunkCode per file extension, as the ask-your-codebase showcase does with its ChunkByLanguage chunker (see Chunking). The first language that lists an extension wins, so the order of the list matters where extensions overlap.

Questions per chunk

let stats = indexing::Pipeline::from_loader(
    FileLoader::new(codebase.root()).with_extensions(&languages.extensions()),
)
.filter(test_code::not_test_files(codebase.clone()))
// Drop unit tests before the cache, which keys on path and content. Changing the content
// after it would make the cache miss on every run.
.then(test_code::without_unit_tests)
.then(files_found.count())
// Skip files whose path and content are unchanged since the last successful run
.filter_cached(codebase.node_cache()?)
.then(files_changed.count())
.then(record_relative_path(codebase.clone()))
// Split along syntax boundaries (functions, impls, ...) with tree-sitter
.then_chunk(ChunkByLanguage::try_new(languages, CHUNK_SIZE)?)
// Ask the LLM which questions each chunk answers; stored as metadata and embedded
.then(
    MetadataQACode::from_client(llm)
        .prompt_template(QUESTIONS_PROMPT)
        .num_questions(QUESTIONS_PER_CHUNK)
        .concurrency(concurrent_requests)
        .build()?,
)
// Embed chunk + metadata locally, in batches
.then_in_batch(Embed::new(embedding::local_model()?).with_batch_size(embedding::BATCH_SIZE))
.then_store_with(codebase.vector_store()?)
.run()
.await?;

Source: examples/ask-your-codebase/src/index.rs#L89-L115

The showcase builds swiftide with the tree-sitter, fastembed, lancedb, redb and openai feature flags.

MetadataQACode asks a large language model (LLM) which questions the code answers and stores the reply under the metadata key Questions and Answers (code). A question at query time then finds the chunk more easily, because the questions are embedded with the code. Run it before chunking to send one prompt per file, which every chunk of that file then inherits. Run it after chunking to get questions per chunk, at one prompt per chunk. It writes 5 questions by default. MetadataQACode::new(client) builds it with the defaults, and from_client(client) returns a builder for prompt_template, num_questions and concurrency.

The showcase runs it after chunking, so each chunk gets its own questions. It also replaces the default prompt with one that asks about behaviour, defined at the top of the same file. When a node has Outline metadata, the default prompt includes the outline.

The showcase’s run, with claude-haiku-5-5 writing the questions and FastEmbed embedding on the CPU, indexed the swiftide-agents crate like this:

Indexed 34 chunks from 14 new or changed files (14 files found, 0 unchanged) in 91.9s

Outlines, references and definitions

OutlineCodeTreeSitter, CompressCodeOutline and MetadataRefsDefsCode give each chunk an outline of its file and the symbols it defines and uses. For pipelines that add them to every chunk, see Code-aware indexing.

Gotchas

OutlineCodeTreeSitter and MetadataRefsDefsCode build for every language, but panic on a node in a language they do not handle. OutlineCodeTreeSitter skips nodes below its minimum file size first. The panic happens inside a spawned task, so run() fails with a join error instead of crashing the process. Check the table before you use them.
When a file has a syntax error, ChunkCode logs a warning and returns the whole file as one chunk, whatever its size. OutlineCodeTreeSitter returns an error for that node, and the first error stops the whole pipeline: run() fails. Add filter_errors() after it when your codebase may contain files that do not parse. See Streams and concurrency.
SupportedLanguages::Typescript lists tsx, js and jsx in file_extensions(), but uses the plain TypeScript grammar, which cannot parse JSX. A .tsx or .jsx file that contains JSX has a syntax error, with the consequences above. Put JavaScript before TypeScript when you pick a language by extension, so js and jsx go to the JavaScript grammar.
The C and C++ lists both include c, h and o. o is the extension of compiled object files, so a FileLoader built from them picks up build output.
MetadataQACode and CompressCodeOutline send one prompt per node. On a large codebase that is many requests; lower the concurrency or wrap the client in LanguageModelWithBackOff to stay within rate limits.

Reference

Ask your codebase