Skip to content
swiftide

Changelog

Changelog

All notable changes to this project will be documented in this file.

0.16.0 - 2025-01-02

New features

  • 52e341e (lancedb) Public method for opening table (#514)

  • 3254bd3 (query) Generic templates with document rendering (#520)

Reworks `PromptTemplate` to a more generic `Template`, such that they
  can also be used elsewhere. This deprecates `PromptTemplate`.

  As an example, an optional `Template` in the `Simple` answer
  transformer, which can be used to customize the output of retrieved
  documents. This has excellent synergy with the metadata changes in #504.
  • 235780b (query) Documents as first class citizens (#504)
For simple RAG, just adding the content of a retrieved document might be
  enough. However, in more complex use cases, you might want to add
  metadata as well, as is or for conditional formatting.

  For instance, when dealing with large amounts of chunked code, providing
  the path goes a long way. If generated metadata is good enough, could be
  useful as well.

  With this retrieved Documents are treated as first class citizens,
  including any metadata as well. Additionally, this also paves the way
  for multi retrieval (and multi modal).
  • 584695e (query) Add custom SQL query generation for pgvector search (#478)
Adds support for custom retrieval queries with the sqlx query builder for PGVector. Puts down the fundamentals for custom query building for any retriever.

  ---------

  • b55bf0b (redb) Public database and table definition (#510)

  • 176378f Implement traits for all Arc dynamic dispatch (#513)

If you use i.e. a `Persist` or a `NodeCache` outside swiftide as well, and you already have it Arc'ed, now it just works.
  • dc9881e Allow opt out of pipeline debug truncation

Bug fixes

  • 2831101 (lancedb) Metadata should be nullable in lancedb (#515)

  • c35df55 (macros) Explicit box dyn cast fixing Rust Analyzer troubles (#523)

Miscellaneous

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.15.0…0.16.0

0.15.0 - 2024-12-23

New features

  • a1b9a2d (query) Ensure concrete names for transformations are used when debugging (#496)

  • 7779c44 (query) Ensure query pipeline consistently debug logs in all other stages too

  • 55dde88 (query) Debug full retrieved documents when debug mode is enabled (#495)

  • 66031ba (query) Log query pipeline answer on debug (#497)

Miscellaneous

  • d255772 (agents) Rename CommandError::FailedWithOutput to CommandError::NonZeroExit (#484)
Better describes what is going on. I.e. `rg` exits with 1 if nothing is
  found, tests generally do the same if they fail.
  • 0000000 Update Cargo.toml dependencies

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.14.4…0.15.0

0.14.4 - 2024-12-11

New features

  • 7211559 (agents) EXPERIMENTAL Agents in Swiftide (#463)
Agents are coming to Swiftide! We are still ironing out all the kinks,
  while we make it ready for a proper release. You can already experiment
  with agents, see the rustdocs for documentation, and an example in
  `/examples`, and feel free to contact us via github or discord. Better
  documentation, examples, and tutorials are coming soon.

  Run completions in a loop, define tools with two handy macros, customize
  the agent by hooking in on lifecycle events, and much more.

  Besides documentation, expect a big release for what we build this for
  soon! 🎉
  • 3751f49 (query) Add support for single embedding retrieval with PGVector (#406)

Miscellaneous

  • 5ce4d21 Clippy and deps fixes for 1.83 (#467)

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.14.3…0.14.4

0.14.3 - 2024-11-20

New features

  • 1774b84 (integrations) Add PGVector support for indexing (#392)

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.14.2…0.14.3

0.14.2 - 2024-11-08

Bug fixes

  • 3924322 (querying) Query pipeline is now properly send and sync when possible (#425)

Miscellaneous

- **Tokio on rt-multi-thread only**
  - **Remove manual checks from lancedb integration test**
  - **Ensure all deps in workspace manifest**
  - **Remove unused deps**
  - **Remove examples and benchmarks from default members**

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.14.1…0.14.2

0.14.1 - 2024-10-27

Bug fixes

  • 5bbcd55 Revert 0.14 release as mistralrs is unpublished (#417)
Revert the 0.14 release as `mistralrs` is unpublished and unfortunately
  cannot be released.

Miscellaneous

- **Revert "fix: Revert 0.14 release as mistralrs is unpublished
  ([#417](https://github.com/bosun-ai/swiftide/pull/417))"**
  - **Fix changelog**

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.14.0…0.14.1

0.14.0 - 2024-10-27

Bug fixes

  • 551a9cb (indexing) [breaking] Node ID no longer memoized (#414)
As @shamb0 pointed out in [#392](https://github.com/bosun-ai/swiftide/pull/392), there is a potential issue where Node
  ids are get cached before chunking or other transformations, breaking
  upserts and potentially resulting in data loss.

BREAKING CHANGE: This PR reworks Nodes with a builder API and a private id. Hence, manually creating nodes no longer works. In the future, all the fields are likely to follow the same pattern, so that we can decouple the inner fields from the Node’s implementation.

  • c091ffa (indexing) Use atomics for key generation in memory storage (#415)

Miscellaneous

  • 0000000 Update Cargo.toml dependencies

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.13.4…0.14.0

0.13.4 - 2024-10-21

Bug fixes

  • 47455fb (indexing) Visibility of ChunkMarkdown builder should be public

  • 2b3b401 (indexing) Improve splitters consistency and provide defaults (#403)

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.13.3…0.13.4

Changelog

All notable changes to this project will be documented in this file.

0.13.3 - 2024-10-11

Bug fixes

  • 2647f16 (deps) Update rust crate text-splitter to 0.17 (#366)

  • d74d85b (indexing) Add missing Embed::batch_size implementation (#378)

  • 95f78d3 (tree-sitter) Compile regex only once (#371)

Regex compilation is not cheap, use a static with a oncelock instead.

Miscellaneous

  • 0000000 Update Cargo.toml dependencies

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.13.2…0.13.3

0.13.2 - 2024-10-05

New features

  • 4b13aa7 (core) Add support for cloning all trait objects (#355)
For instance, if you have a `Box<dyn SimplePrompt>`, you can now clone
  into an owned copy and more effectively use the available generics. This
  also works for borrowed trait objects.
  • ed3da52 (indexing) Support Redb as embedable nodecache (#346)
Adds support for Redb as an embeddable node cache, allowing full local
  app development without needing external services.

Bug fixes

  • 06f8336 (indexing) Ensure name() returns concrete name on trait objects (#351)

Miscellaneous

  • 8237c28 (core) Mock traits for testing should not have their docs hidden

  • 0000000 Update Cargo.toml dependencies

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.13.1…0.13.2

0.13.1 - 2024-10-02

Bug fixes

  • e6d9ec2 (lancedb) Should not error if table exists (#349)

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.13.0…0.13.1

0.13.0 - 2024-09-26

New features

  • 7d8a57f (indexing) [breaking] Removed duplication of batch_size (#336)

BREAKING CHANGE: The batch size of batch transformers when indexing is now configured on the batch transformer. If no batch size or default is configured, a configurable default is used from the pipeline. The default batch size is 256.

Bug fixes

  • 23b96e0 (tree-sitter) [breaking] SupportedLanguages are now non-exhaustive (#331)

BREAKING CHANGE: SupportedLanguages are now non-exhaustive. This means that matching on SupportedLanguages will now require a catch-all arm. This change was made to allow for future languages to be added without breaking changes.

Miscellaneous

  • 923a8f0 (fastembed,qdrant) Better batching defaults (#334)
Qdrant and FastEmbed now have a default batch size, removing the need to set it manually. The default batch size is 50 and 256 respectively.

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.12.3…0.13.0

0.12.3 - 2024-09-23

New features

  • da5df22 (tree-sitter) Implement Serialize and Deserialize for SupportedLanguages (#314)

Bug fixes

  • a756148 (tree-sitter) Fix javascript and improve tests (#313)
As learned from [#309](https://github.com/bosun-ai/swiftide/pull/309), test coverage for the refs defs transformer was
  not great. There _are_ more tests in code_tree. Turns out, with the
  latest treesitter update, javascript broke as it was the only language
  not covered at all.

Miscellaneous

  • e8e9d80 (docs) Add documentation to query module (#276)

Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.12.2…0.12.3

v0.12.2 - 2024-09-20

Docs

  • d84814e Fix broken documentation links and other cargo doc warnings (#304) by @tinco
Running `cargo doc --all-features` resulted in a lot of warnings.

Full Changelog: https://github.com/bosun-ai/swiftide/compare/v0.12.1…v0.12.2

v0.12.1 - 2024-09-16

New features

  • ec227d2 (indexing,query) Add concise info log with transformation name by @timonv

  • 01cf579 (query) Add query_mut for reusable query pipelines by @timonv

  • 081a248 (query) Improve query performance similar to indexing in 0.12 by @timonv

  • 8029926 (query,indexing) Add duration in log output on pipeline completion by @timonv

Bug fixes

  • 39b6ecb (core) Truncate long strings safely when printing debug logs by @timonv

  • 8b8ceb9 (deps) Update redis by @timonv

  • 16e9c74 (openai) Reduce debug verbosity by @timonv

  • 6914d60 (qdrant) Reduce debug verbosity when storing nodes by @timonv

  • 3d13889 (query) Reduce and improve debugging verbosity by @timonv

  • 133cf1d (query) Remove verbose debug and skip self in instrumentation by @timonv

  • ce17981 Clippy by @timonv

  • a871c61 Fmt by @timonv

Miscellaneous

  • d62b047 (ci) Update testcontainer images and fix tests by @timonv

Full Changelog: https://github.com/bosun-ai/swiftide/compare/v0.12.0…v0.12.1

v0.12.0 - 2024-09-13

New features

  • e902cb7 (query) Add support for filters in SimilaritySingleEmbedding (#298) by @timonv
Adds support for filters for Qdrant and Lancedb in
  SimilaritySingleEmbedding. Also fixes several small bugs and brings
  improved tests.
  • f158960 Major performance improvements (#291) by @timonv
Futures that do not yield were not run in parallel properly. With this
  futures are spawned on a tokio worker thread by default.

  When embedding (fastembed) and storing a 85k row dataset, there's a
  ~1.35x performance improvement:
  <img width="621" alt="image"
  src="https://github.com/user-attachments/assets/ba2d4d96-8d4a-44f1-b02d-6ac2af0cedb7">

  ~~Need to do one more test with IO bound futures as well. Pretty huge,
  not that it was slow.~~

  With IO bound openai it's 1.5x.

Bug fixes

  • f8314cc (indexing) Limit logged chunk to max 100 chars (#292) by @timonv

  • f95f806 (indexing) Debugging nodes should respect utf8 char boundaries by @timonv

  • 8595553 Implement into_stream_boxed for all loaders by @timonv

  • 9464ca1 Bad embed error propagation (#293) by @timonv

- **fix(indexing): Limit logged chunk to max 100 chars**
  - **fix: Embed transformers must correctly propagate errors**

Miscellaneous

  • 45d8a57 (ci) Use llm-cov preview via nightly and improve test coverage (#289) by @timonv
Fix test coverage in CI. Simplified the trait bounds on the query
  pipeline for now to make it all work and fit together, and added more
  tests to assert boxed versions of trait objects work in tests.

  • 408f30a (deps) Update testcontainers (#295) by @timonv

  • 37c4bd9 (deps) Update treesitter (#296) by @timonv

  • 8d9e954 Cargo update by @timonv

Full Changelog: https://github.com/bosun-ai/swiftide/compare/v0.11.1…v0.12.0

v0.11.1 - 2024-09-10

New features

  • 3c9491b Implemtent traits T for Box