Storing the results

After processing nodes in the pipeline you probably want to store the results. Pipelines support multiple storage steps, but need at least one. A storage implements the Persist trait.

The Persist trait

Which is defined as follows:

pub trait Persist: Debug + Send + Sync {
    async fn setup(&self) -> Result<()>;
    async fn store(&self, node: IngestionNode) -> Result<IngestionNode>;
    async fn batch_store(&self, nodes: Vec<IngestionNode>) -> IngestionStream;
    fn batch_size(&self) -> Option<usize> {
        None
    }
}

Setup functions are run right away, asynchronously when the pipeline starts. This could include setting up collections, tables, connections etcetera. Because more might happen after storing, both store and batch_store are expected to return the nodes they processed.

If batch_size is implemented for the storage, the stream will always prefer batch_store.

Built in storage

Name	Description	Feature Flag
Redis	Persists nodes by default as json	redis
Qdrant	Persists nodes in qdrant; expects a vector to be set	qdrant
MemoryStorage	Persists nodes in memory; great for debugging