Changelog
Changelog
All notable changes to this project will be documented in this file.
0.11.0 - 2024-09-08
Added
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.10.0…0.11.0
Changelog
All notable changes to this project will be documented in this file.
0.10.0 - 2024-09-06
BREAKING CHANGE: Indexing nodes now have their ID calculated using UUIDv3 via MD5 as the previous algorithm was unreliable and broke in 1.81. Added benefit that collision chance is even smaller. This means that when indexing again, nodes will have different IDs and upsert will not work. Backwards compatibility is non-trivial. If this is a huge issue, ping us on discord and we will look into it.
Added
Fixed
Other
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.9.2…0.10.0
0.9.2 - 2024-09-04
Added
84e9bae (indexing) Add chunker for text with text_splitter (#270)
387fbf2 (query) Hybrid search for qdrant in query pipeline (#260)
Fixed
Other
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.9.1…0.9.2
0.9.1 - 2024-09-01
Added
Fixed
Other
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.9.0…0.9.1
0.9.0 - 2024-08-15
Added
2443933 (qdrant) Add access to inner client for custom operations (#242)
4fff613 (query) Add concurrency on query pipeline and add query_all
Fixed
- 8a1cc69 (query) After retrieval current transormation should be empty
Other
3d213b4 (readme) Add link to 0.8 release
e9d0016 (indexing,integrations) Move tree-sitter dependencies to integrations (#235)
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.8.0…0.9.0
0.8.0 - 2024-08-12
Added
67336f1 (indexing) Sparse vector support with Splade and Qdrant (#222)
2e25ad4 (indexing) Default LLM for indexing pipeline and boilerplate Transformer macro (#227)
e728a7c (uncategorized) Code outlines in chunk metadata (#137)
Fixed
Other
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.7.1…0.8.0
Changelog
All notable changes to this project will be documented in this file.
0.7.1 - 2024-08-04
Features
Documentation
f7accde (readme) Add 0.7 announcement
ba07ab9 (readme) Readme improvements
1539393 (readme) Update README.md
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.7.0…0.7.1
0.7.0 - 2024-07-28
Features
Documentation
- 2114aa4 (readme) Add copy on the query pipeline
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.7…0.7.0
0.6.7 - 2024-07-22
Features
beea449 (prompt) Add Into for strings to PromptTemplate (#193)
f3091f7 (transformers) References and definitions from code (#186)
Documentation
504fe26 (pipeline) Add note that closures can also be used as transformers
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.6…0.6.7
0.6.6 - 2024-07-16
Features
Documentation
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.5…0.6.6
0.6.5 - 2024-07-15
Features
Documentation
Miscellaneous Tasks
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.4…0.7.0
0.6.4 - 2024-07-14
Bug Fixes
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.3…0.6.4
0.6.3 - 2024-07-14
Bug Fixes
- 47418b5 (prompts) Fix breaking issue with prompts not found
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.1…0.6.3
0.6.1 - 2024-07-12
Documentation
- 085709f (docsrs) Disable unstable and rustdoc scraping
Full Changelog: https://github.com/bosun-ai/swiftide/compare/0.6.0…0.6.1
0.6.0 - 2024-07-12
Features
Adds Prompts as first class citizens. This is a breaking change as
SimplePrompt with just a a `&str` is no longer allowed.
This introduces `Prompt` and `PromptTemplate`. A template uses jinja
style templating build on tera. Templates can be converted into prompts,
and have context added. A prompt is then send to something that prompts,
i.e. openai or bedrock.
Additional prompts can be added either compiled or as one-offs.
Additionally, it's perfectly fine to prompt with just a string as well,
just provide an `.into()`.
For future development, some LLMs really benefit from system prompts,
which this would enable. For the query pipeline we can also take a much
more structured approach with composed templates and conditionals.
Added named vector support to qdrant. A pipeline can now have its embed
mode configured, either per field, chunk and metadata combined (default)
or both. Vectors need to be configured on the qdrant client side.
See `examples/store_multiple_vectors.rs` for an example.
Shoutout to @pwalski for the contribution. Closes #62.
---------
Bug Fixes
9334934 (chunkcode) Use correct chunksizes (#122) by @timonv in #122
c5bf796 (ci) Add clippy back to ci (#147) by @timonv in #147
4c9ed77 (ci) Properly check if contributors are present by @timonv
5de6af4 (ci) Only add contributors if present by @timonv
eb8364e (ci) Try overriding the github repo for git cliff by @timonv
c3aee48 (deps) Update rust crate spider to v1.98.9 (#146) by @renovate[bot] in #146
6d75f14 (deps) Update rust crate htmd to v0.1.6 (#144) by @renovate[bot] in #144
a691d61 (deps) Update rust crate async-openai to v0.23.4 (#136) by @renovate[bot] in #136
8e22937 (deps) Update rust crate aws-sdk-bedrockruntime to v1.39.0 (#143) by @renovate[bot] in #143
bf3b677 (deps) Update rust crate fastembed to v3.9.0 (#141) by @renovate[bot] in #141
2b13523 (deps) Update rust crate fastembed to v3.7.1 (#135) by @renovate[bot] in #135
dd32ef3 (deps) Update rust crate async-trait to v0.1.81 (#134) by @renovate[bot] in #134
adc4bf7 (deps) Update aws-sdk-rust monorepo (#125) by @renovate[bot] in #125
7af97b5 (deps) Update rust crate spider to v1.98.7 (#124) by @renovate[bot] in #124
ff92abd (deps) Update rust crate tree-sitter-javascript to v0.21.4 (#126) by @renovate[bot] in #126
9c261b8 (deps) Update rust crate text-splitter to v0.14.1 (#127) by @renovate[bot] in #127
28f5b04 (deps) Update rust crate tree-sitter-typescript to v0.21.2 (#128) by @renovate[bot] in #128
dfc76dd (deps) Update rust crate serde to v1.0.204 (#129) by @renovate[bot] in #129
3b98334 (deps) Update rust crate serde_json to v1.0.120 (#115) by @renovate[bot] in #115
7357fea (deps) Update rust crate spider to v1.98.6 (#119) by @renovate[bot] in #119
353cd9e (qdrant) Upgrade and better defaults (#118) by @timonv in #118
- **fix(deps): update rust crate qdrant-client to v1.10.1**
- **fix(qdrant): upgrade to new qdrant with sensible defaults**
- **feat(qdrant): safe to clone with internal arc**
---------
- b53636c (uncategorized) Inability to store only some of
EmbeddedFields (#139) by @pwalski in #139 Fixes:#138
Documentation
8405c9e (contributing) Add guidelines on code design (#113) by @timonv in #113
5691ac9 (readme) Add preproduction warning by @timonv
4c40e27 (readme) Add back coverage badge by @timonv
3e447fe (readme) Link to CONTRIBUTING (#114) by @timonv in #114
37af322 (rustdocs) Rewrite the initial landing page (#149) by @timonv in #149
- **Add homepage and badges to cargo toml**
- **documentation landing page improvements**
- 7686c2d (uncategorized) Templated prompts are now a major feature by @timonv
Performance
- **tune cargo for faster builds**
- **perf(swiftide): increase local build performance**
Miscellaneous Tasks
Loosen up dependencies so swiftide is a bit more flexible to add to
existing projects
3d235dd (uncategorized) Release
d2a9ea1 (uncategorized) Enable clippy pedantic (#132) by @timonv in #132
51c114c (uncategorized) Various tooling & community improvements (#131) by @timonv in #131
- **fix(ci): ensure clippy runs with all features**
- **chore(ci): coverage using llvm-cov**
- **chore: drastically improve changelog generation**
- **chore(ci): add sanity checks for pull requests**
- **chore(ci): split jobs and add typos**
Swiftide is not an ingestion pipeline (loading data), but an indexing
pipeline (prepping for search).
There is now a temporary, deprecated re-export to match the previous api.
New Contributors
- @pwalski made their first contribution in #139
Full Changelog: https://github.com/bosun-ai/swiftide/compare/swiftide-v0.5.0…0.6.0
swiftide-v0.5.0 - 2024-07-01
Features
8812fbf (ingestion_pipeline) Build a pipeline from a stream by @timonv
5aeb3a7 (ingestion_pipeline) Splitting and merging streams by @timonv
6a88651 (ingestion_pipeline) Implement filter (#109) by @timonv in #109
6101bed (uncategorized) AWS bedrock support (#92) by @timonv in #92
Adds an integration with AWS Bedrock, implementing SimplePrompt for
Anthropic and Titan models. More can be added if there is a need. Same
for the embedding models.
Bug Fixes
17a2be1 (changelog) Add scope by @timonv
46752db (ci) Add concurrency configuration by @timonv
b155de6 (ci) Fix naming of github actions by @timonv
2dbf14c (ci) Fix benchmarks in ci by @timonv
3cc2e06 (ci) Fix release-plz changelog parsing by @timonv
2650605 (deps) Update rust crate serde_json to v1.0.119 (#110) by @renovate[bot] in #110
5c16c8e (deps) Update rust crate strum to v0.26.3 (#101) by @renovate[bot] in #101
52cf37b (deps) Update rust crate fastembed to v3.7.0 (#104) by @renovate[bot] in #104
2401414 (deps) Update rust crate text-splitter to 0.14.0 (#105) by @renovate[bot] in #105
4c019eb (deps) Update rust crate htmd to v0.1.5 (#96) by @renovate[bot] in #96
8e15004 (deps) Update rust crate serde_json to v1.0.118 (#99) by @renovate[bot] in #99
9b4ef81 (deps) Update rust crate spider to v1.98.3 (#100) by @renovate[bot] in #100
963919b (transformers) Fix too small chunks being retained and api by @timonv [breaking]
cba981a (uncategorized) Replace unwrap with expect and add comment on panic by @timonv in #111
6430af7 (uncategorized) Use native cargo bench format and only run benchmarks crate by @timonv
2c31513 (uncategorized) Just use keepachangelog by @timonv
e8198d8 (uncategorized) Use git cliff manually for changelog generation by @timonv
5e8da00 (uncategorized) Fix oversight in ingestion pipeline tests by @timonv
Documentation
929410c (readme) Add diagram to the readme (#107) by @timonv in #107
b014f43 (uncategorized) Improve documentation across the project (#112) by @timonv in #112
Miscellaneous Tasks
206e432 (ci) Add support for merge queues by @timonv
e243212 (ci) Enable continous benchmarking and improve benchmarks (#98) by @timonv in #98
a8b02a3 (uncategorized) Release v0.5.0 (#103) by @github-actions[bot] in #103
162c6ef (uncategorized) Ensure feat is always in Added by @timonv
5f09c11 (uncategorized) Add initial benchmarks by @timonv
b953638 (uncategorized) Configure Renovate (#94) by @renovate[bot] in #94
Full Changelog: https://github.com/bosun-ai/swiftide/compare/swiftide-v0.4.3…swiftide-v0.5.0
swiftide-v0.4.3 - 2024-06-28
Bug Fixes
- ab3dc86 (memory_storage) Fallback to incremental counter when missing id by @timonv
Documentation
4076092 (readme) Clean up and consistent badge styles by @timonv in #93
dad3e02 (readme) Add ci badge by @timonv
Miscellaneous Tasks
1ebbc2f (uncategorized) Manual release-plz update by @timonv
bdebc24 (uncategorized) Clippy by @timonv
Full Changelog: https://github.com/bosun-ai/swiftide/compare/swiftide-v0.4.2…swiftide-v0.4.3
swiftide-v0.4.2 - 2024-06-26
Features
- 926cc0c (ingestion_stream) Implement into for Result