Unhandled Vector Store Save Failures

[HarshitR2004]

File: embedding_pipeline.py

The embed_and_store_documents function lacks error handling for store.save_local(), causing crashes or silent failures. Issues like full disk, permission errors, or network problems can result in lost embeddings with no user notification, making the process unreliable.

[ashvin2005]

Hi, I’d like to work on this issue. Could you please assign it to me?

[Heisenberg-Vader]

Hi, I am willing to attempt a fix for this issue.
Here is my approach, and please let me know any improvements for this approach.
I’ll add error handling around store.save_local() in the embed_and_store_documents() function. For this, I'll wrap the call in a try except block to catch issues such as full disk errors, permission errors, or IO/network issues as mentioned in the description. Make logs for meaningful error messages with context. Try to save the partial progress for the already embedded documents, in case the saving fails.

[HarshitR2004]

Hi, I am willing to attempt a fix for this issue. Here is my approach, and please let me know any improvements for this approach. I’ll add error handling around store.save_local() in the embed_and_store_documents() function. For this, I'll wrap the call in a try except block to catch issues such as full disk errors, permission errors, or IO/network issues as mentioned in the description. Make logs for meaningful error messages with context. Try to save the partial progress for the already embedded documents, in case the saving fails.

Thank you for your intrest, but I have already submited a PR regarding this