How to ingest from with multiple sources in one flow with flow builder

[petrarca]

Thank you so far for this great work.
I have, based on the examples, tried to ingest data from more than one source.
Like in this code:

@cocoindex.flow_def(name="CodeEmbedding")
def code_embedding_flow(
    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -> None:
    """ Ingest from multiple folders and create embeddings """
    code_embeddings = data_scope.add_collector()

    # First folder
    data_scope["files1"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(
            path="/some/valid/path1",
            included_patterns=_INCLUDED_PATTERN,
            excluded_patterns=_EXCLUDED_PATTERN)
    )

    with data_scope["files1"].row() as file:
        file["extension"] = file["filename"].transform(extract_extension)
        file["chunks"] = file["content"].transform(
            cocoindex.functions.SplitRecursively(),
            language=file["extension"],
            chunk_size=1000,
            min_chunk_size=300,
            chunk_overlap=300,
        )
        with file["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].call(code_to_embedding)
            code_embeddings.collect(
                filename=file["filename"],
                location=chunk["location"],
                code=chunk["text"],
                embedding=chunk["embedding"],
                start=chunk["start"],
                end=chunk["end"],
            )

     # Seoncd folder -> Adding that will crash the app
    data_scope["files2"] = flow_builder.add_source(
        cocoindex.sources.LocalFile(
            path="/some/valid/path2",
            included_patterns=_INCLUDED_PATTERN,
            excluded_patterns=_EXCLUDED_PATTERN)
    )

    with data_scope["files2"].row() as file:
        file["extension"] = file["filename"].transform(extract_extension)
        file["chunks"] = file["content"].transform(
            cocoindex.functions.SplitRecursively(),
            language=file["extension"],
            chunk_size=1000,
            min_chunk_size=300,
            chunk_overlap=300,
        )
        with file["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].call(code_to_embedding)
            code_embeddings.collect(
                filename=file["filename"],
                location=chunk["location"],
                code=chunk["text"],
                embedding=chunk["embedding"],
                start=chunk["start"],
                end=chunk["end"],
            )

    code_embeddings.export(
        "coco_rag",
        cocoindex.targets.Postgres(),
        primary_key_fields=["filename", "location"],
        vector_indexes=[
            cocoindex.VectorIndexDef(
                field_name="embedding",
                metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
            )
        ],
    )

With one source it works without any problem. As soon as I add one more (as final solution, this is loaded from a configuration) I run into the following exception when performing an update of the index (db was already updated with cocoindex setup ...):

...
thread 'tokio-runtime-worker' panicked at src/execution/evaluator.rs:233:14:
called `Option::unwrap()` on a `None` value
[2025-08-02T13:21:51Z ERROR cocoindex_engine::execution::source_indexer] JoinError::Panic(Id(50), "called `Option::unwrap()` on a `None` value", ...)

thread 'tokio-runtime-worker' panicked at src/execution/evaluator.rs:233:14:
called `Option::unwrap()` on a `None` value
[2025-08-02T13:21:51Z ERROR cocoindex_engine::execution::source_indexer] JoinError::Panic(Id(42), "called `Option::unwrap()` on a `None` value", ...)

thread 'tokio-runtime-worker' panicked at src/execution/evaluator.rs:233:14:
called `Option::unwrap()` on a `None` value
[2025-08-02T13:21:51Z ERROR cocoindex_engine::execution::source_indexer] JoinError::Panic(Id(63), "called `Option::unwrap()` on a `None` value", ...)
...

Any idea how to solve that?

[georgeh0]

It's a bug, we'll fix it soon. Note that cocoindex is designed to support multi source, but the support is not officially ready yet. We'll finalize this part soon and add a related example.

[georgeh0]

This problem has been fixed in the latest v0.1.77 version (release)

[petrarca]

Wow, that was fast. Confirmed, it works. Thank you!