Skip to content

copy_raster / _ImageryUploaderAGOL.upload_file fails for raster uploads around 50–60 MB #2406

@mirouml

Description

@mirouml

Describe the bug
When publishing hosted imagery layers using copy_raster() (ArcGIS API for Python 2.4.2), raster uploads around 50–60 MB enter an infinite retry loop inside _ImageryUploaderAGOL.upload_file, never completing. Smaller and larger files upload successfully.

To Reproduce
Steps to reproduce the behavior:

from arcgis.gis import GIS
from arcgis.raster import copy_raster

gis = GIS("home")
service_item = copy_raster(
    input_raster=r"C:\path\to\raster_51mb.tif",
    output_name="test_raster",
    gis=gis
)

error:

# Process stalls indefinitely repeating upload attempts
# No explicit exception is raised

Expected behavior
Upload should complete successfully regardless of raster size.

Platform (please complete the following information):

  • OS: Windows 11 / AWS Lambda (AL2)
  • Python API Version: 2.4.2
  • azure-storage-blob: 12.17.x
  • Python: 3.11

Additional context
Cause: upload_blob() performs a single PUT for < 64 MiB files; this often times out and loops indefinitely.
Workaround: Replace the call with manual block upload:

import base64
from azure.storage.blob import BlobBlock
blob = self.container.get_blob_client(blobname)
block_ids = []
with open(file_name, "rb") as f:
    i = 0
    while chunk := f.read(4*1024*1024):
        block_id = base64.b64encode(f"{i:08d}".encode()).decode()
        blob.stage_block(block_id, data=chunk)
        block_ids.append(BlobBlock(block_id=block_id))
        i += 1
blob.commit_block_list(block_ids)

This resolves the issue for 50–60 MB rasters.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions