feat: bounded size auto sharding via zarr.config

[ilan-gold]

closes #3568

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[codecov]

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.90%. Comparing base (182504c) to head (7f71875).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/zarr/core/chunk_grids.py	86.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3574      +/-   ##
==========================================
+ Coverage   61.86%   61.90%   +0.03%     
==========================================
  Files          85       85              
  Lines       10137    10150      +13     
==========================================
+ Hits         6271     6283      +12     
- Misses       3866     3867       +1     

Files with missing lines	Coverage Δ
src/zarr/core/config.py	38.88% <ø> (ø)
src/zarr/core/chunk_grids.py	63.33% <86.66%> (+3.52%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ilan-gold]

Not sure I fully grasp what is going on with codecov, it looks like the tests run and assert the things they should and use the funcionality promised.

[melonora]

In SpatialData we could also have a use case for this. Many users are not as experienced with the whole concept of sharding and chunking, so having the option to provide a maximum size with some explanation in documentation would be very beneficial.

[ilan-gold]

I was curious how xarray handles sharding and from what I can tell it is also manually done there:

https://github.com/pydata/xarray/blob/6da8c2c92326d05b03bffa1a022d9f87f4f6278d/xarray/tests/test_backends.py#L2891-L2907

@jhamman Is this true? Would xarray benefit from a feature like this or are we thinking about the problem wrong and people should be doing this manually?

[melonora]

@ilan-gold I think that is correct at the moment. See also pydata/xarray#9947. In dask I am working on having the interface just provide a shard_factor argument. I think the interface mostly just depends on the user. Some users like to think more in sizes, particularly when they are lessed experienced with chunking / sharding as they can relate it easier to the capability of their hardware.

[d-v-b]

can we find a denser name for this parameter? Something like max_shard_size_bytes?

[ilan-gold]

Also perhaps something more accurate? Like target_shard_size_bytes? It's not meant as a bound but a target

[d-v-b]

2 small requests but otherwise this looks great, thanks @ilan-gold

[ilan-gold]

@d-v-b What is the take on the fact that this is pre-compression? Something I just realized messing around with this. Should we document that more clearly?

[d-v-b]

@d-v-b What is the take on the fact that this is pre-compression? Something I just realized messing around with this. Should we document that more clearly?

Maybe just indicate in the docs that this setting governs the in-memory size?

Also, how will this handle data types that don't have a computable size per element (like variable-length bytes)?