Skip to content

Conversation

@rayshrey
Copy link
Contributor

Description

  • Fix for native memory leak issue identified during indexing benchmarking (closing ParquetWriter and it's internal classes properly)
  • Fixes slow indexing due to addition of same entry multiple time in DocumentWriterPool's ConcurrentQueue
  • Added additional jvm arguments and dependencies for better native memory tracking and debugging of Arrow Allocators

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

❌ Gradle check result for aa2ee07: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: rayshrey <rayshrey@amazon.com>
@github-actions
Copy link
Contributor

❌ Gradle check result for 6c4b215: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@bharath-techie bharath-techie merged commit 22ae1ec into opensearch-project:feature/datafusion Oct 31, 2025
6 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants