Caching and filtering nodes

When nodes have already been processed by the pipeline, they can often be skipped, speeding up the pipeline and saving costs. A node cache implements the NodeCache trait.

The NodeCache trait

Which is defined as follows:

pub trait NodeCache: Send + Sync + Debug {
    async fn get(&self, node: &IngestionNode) -> bool;
    async fn set(&self, node: &IngestionNode);
}

Or in human language: “Given a Node, provide methods to set and get from the cache”.

Built in chunkers

Name	Description	Feature Flag
Redis	Can get and set nodes using multiplexed connections	redis