failed to get trace with id: xxx Status: 499 status code 499 Body: context canceled

[alvargut]

Describe the bug
From Grafana (where I configured to pull data from Tempo), whenever I want to see details for a trace, quite often I get this error

failed to get trace with id: 7ec467384213425e8401600fcaff68b0 Status: 499 status code 499 Body: context canceled

On Grafana Tempo host, I can see these messages whenever Grafana queries extra info from Tempo using "journalctl -f -u tempo":

sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692547646Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=97f5f7d8-5a7e-44ed-9caa-c96c4adc1b85 found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692553799Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=653fbec3-e156-405e-8558-dd7f7c1993cd found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692565691Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=98da67d6-307a-4e88-98ca-c617339cbb7f found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.69263318Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=9657d425-d03e-4ed6-8e6e-a3ba5c1c32d7 found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692646832Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=63c08a1e-efe6-4b00-922b-1d197314f78a found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692566013Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=67d088e0-a2e9-49c3-bc20-9b94e8f4408c found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.692666195Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=65e1339d-708c-4d62-96a7-80b02588677c found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-17T09:08:50.692640194Z caller=server.go:2136 traceID=0e61b005e8f56bd5 msg="GET /querier/api/traces/7ec467384213425e8401600fcaff68b0?blockEnd=9cbc14e5e0a72f050000000000000000&blockStart=97829cbc14e5e0a70000000000000000&mode=blocks (500) 927.502µs Response: \"error finding trace by id, blockID: 9a7e8e4f-2536-41a8-bfbf-3de7f9cfc9d8: error retrieving bloom bloom-0 (single-tenant, 9a7e8e4f-2536-41a8-bfbf-3de7f9cfc9d8): does not exist; error finding trace by id, blockID: 9c2f3cbf-b078-43e8-a377-baa81d688970: error retrieving bloom bloom-0 (single-tenant, 9c2f3cbf-b078-43e8-a377-baa81d688970): does not exist; error finding trace by id, blockID: 98bda52a-16eb-41ae-b101-12b3bd0a7d92: error retrieving bloom bloom-0 (single-tenant, 98bda52a-16eb-41ae-b101-12b3bd0a7d92): does not exist; error finding trace by id, blockID: 9b13a52d-1eef-483f-848f-73db1942cb12: error retrieving bloom bloom-0 (single-tenant, 9b13a52d-1eef-483f-848f-73db1942cb12): does not exist; error finding trace by id, blockID: 9c4d888e-e6c5-4a6e-88ea-7f8d821dfc88: error retrieving bloom bloom-0 (single-tenant, 9c4d888e-e6c5-4a6e-88ea-7f8d821dfc88): does not exist; error finding trace by id, blockID: 9bcf90a1-508e-4775-a516-42327bc940b7: error retrieving bloom bloom-0 (single-tenant, 9bcf90a1-508e-4775-a516-42327bc940b7): does not exist; error finding trace by id, blockID: 9b1bca18-37d9-4552-92f1-e97e386ecc58: error retrieving bloom bloom-0 (single-tenant, 9b1bca18-37d9-4552-92f1-e97e386ecc58): does not exist; error finding trace by id, blockID: 98ff20ac-1088-4c57-a761-38bc0c57a2c2: error retrieving bloom bloom-0 (single-tenant, 98ff20ac-1088-4c57-a761-38bc0c57a2c2): does not exist; error finding trace by id, blockID: 9855467a-05ad-4e36-a47b-b2270d4a9a8c: error retrieving bloom bloom-0 (single-tenant, 9855467a-05ad-4e36-a47b-b2270d4a9a8c): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-6f905678-634d-4b19-8606-a03415405515-0; X-Scope-Orgid: single-tenant; uber-trace-id: 0e61b005e8f56bd5:6eca591372f4ad81:6a45d486057c6900:0; "
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-17T09:08:50.692651796Z caller=server.go:2136 traceID=0e61b005e8f56bd5 msg="GET /querier/api/traces/7ec467384213425e8401600fcaff68b0?blockEnd=1f58d0fac687d6350000000000000000&blockStart=1a1f58d0fac687d70000000000000000&mode=blocks (500) 39.868447ms Response: \"error finding trace by id, blockID: 1a8c4407-0550-4fcd-b420-5fb044f81c52: error retrieving bloom bloom-0 (single-tenant, 1a8c4407-0550-4fcd-b420-5fb044f81c52): does not exist; error finding trace by id, blockID: 1e0dc0d8-88a4-4a29-9a82-eac839c06bec: error retrieving bloom bloom-0 (single-tenant, 1e0dc0d8-88a4-4a29-9a82-eac839c06bec): does not exist; error finding trace by id, blockID: 1dca1283-f907-4351-9e2e-ab7888aee29e: error retrieving bloom bloom-0 (single-tenant, 1dca1283-f907-4351-9e2e-ab7888aee29e): does not exist; error finding trace by id, blockID: 1d632c89-c287-4437-b0f7-a7372f00d812: error retrieving bloom bloom-0 (single-tenant, 1d632c89-c287-4437-b0f7-a7372f00d812): does not exist; error finding trace by id, blockID: 1aff424a-5b48-4e23-9a65-464bbf4d2c8e: error retrieving bloom bloom-0 (single-tenant, 1aff424a-5b48-4e23-9a65-464bbf4d2c8e): does not exist; error finding trace by id, blockID: 1f044c48-090b-4b8f-bfd4-cb68c77c6aaf: error retrieving bloom bloom-0 (single-tenant, 1f044c48-090b-4b8f-bfd4-cb68c77c6aaf): does not exist; error finding trace by id, blockID: 1c999cf9-acfd-42c6-87d9-b8dc9066cf38: error retrieving bloom bloom-0 (single-tenant, 1c999cf9-acfd-42c6-87d9-b8dc9066cf38): does not exist; error finding trace by id, blockID: 1c51b6ee-5821-4a03-bbaf-a77177e11cfe: error retrieving bloom bloom-0 (single-tenant, 1c51b6ee-5821-4a03-bbaf-a77177e11cfe): does not exist; error finding trace by id, blockID: 1a8ea617-0e1b-4750-8d8c-b3ef6ca689c6: error retrieving bloom bloom-0 (single-tenant, 1a8ea617-0e1b-4750-8d8c-b3ef6ca689c6): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-6f905678-634d-4b19-8606-a03415405515-0; X-Scope-Orgid: single-tenant; uber-trace-id: 0e61b005e8f56bd5:62e11e5a6cc02f66:6a45d486057c6900:0; "
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.693098867Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=fded93d2-e45f-4d5f-b7f8-65ed21485758 found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-17T09:08:50.693191906Z caller=server.go:2136 traceID=0e61b005e8f56bd5 msg="GET /querier/api/traces/7ec467384213425e8401600fcaff68b0?blockEnd=ffffffffffffffffffffffffffffffff&blockStart=fac687d6343eb1a10000000000000000&mode=blocks (500) 28.305371ms Response: \"error finding trace by id, blockID: fefaefdd-74e8-411d-97d4-67edeaabfdc4: error retrieving bloom bloom-0 (single-tenant, fefaefdd-74e8-411d-97d4-67edeaabfdc4): does not exist; error finding trace by id, blockID: ff35ecae-54cb-4414-bd46-95a390b31bc8: error retrieving bloom bloom-0 (single-tenant, ff35ecae-54cb-4414-bd46-95a390b31bc8): does not exist; error finding trace by id, blockID: fe7753dc-89cc-4689-ba7e-0919f3970655: error retrieving bloom bloom-0 (single-tenant, fe7753dc-89cc-4689-ba7e-0919f3970655): does not exist; error finding trace by id, blockID: fed35f57-3dec-496e-a5b2-cda2bb86a21d: error retrieving bloom bloom-0 (single-tenant, fed35f57-3dec-496e-a5b2-cda2bb86a21d): does not exist; error finding trace by id, blockID: fbff4046-af2e-4029-82b4-b29a72eb502e: error retrieving bloom bloom-0 (single-tenant, fbff4046-af2e-4029-82b4-b29a72eb502e): does not exist; error finding trace by id, blockID: fb244778-9ad9-4b34-851c-f482d693d944: error retrieving bloom bloom-0 (single-tenant, fb244778-9ad9-4b34-851c-f482d693d944): does not exist; error finding trace by id, blockID: ff95edb8-0846-4f9c-91f4-caf25fff200a: error retrieving bloom bloom-0 (single-tenant, ff95edb8-0846-4f9c-91f4-caf25fff200a): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-6f905678-634d-4b19-8606-a03415405515-0; X-Scope-Orgid: single-tenant; uber-trace-id: 0e61b005e8f56bd5:27850d303f6ac05d:6a45d486057c6900:0; "
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.69324394Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=8a15d43b-41e5-4005-b104-12b86461c411 found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=info ts=2025-09-17T09:08:50.693272612Z caller=tempodb.go:335 org_id=single-tenant msg="searching for trace in block" findTraceID=7ec467384213425e8401600fcaff68b0 block=66b33ae1-d188-4bf2-a873-4e55bacc9a15 found=false
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-17T09:08:50.693320214Z caller=server.go:2136 traceID=0e61b005e8f56bd5 msg="GET /querier/api/traces/7ec467384213425e8401600fcaff68b0?blockEnd=8d0fac687d6343eb0000000000000000&blockStart=87d6343eb1a1f58d0000000000000000&mode=blocks (500) 40.135696ms Response: \"error finding trace by id, blockID: 895a5081-7218-4b4a-ad55-8bdf72ba4d97: error retrieving bloom bloom-0 (single-tenant, 895a5081-7218-4b4a-ad55-8bdf72ba4d97): does not exist; error finding trace by id, blockID: 8ad9a0e1-eb5f-4f76-9682-84aee1ebb517: error retrieving bloom bloom-0 (single-tenant, 8ad9a0e1-eb5f-4f76-9682-84aee1ebb517): does not exist; error finding trace by id, blockID: 8c151326-20e0-4f09-9423-3f00cc12a1f4: error retrieving bloom bloom-0 (single-tenant, 8c151326-20e0-4f09-9423-3f00cc12a1f4): does not exist; error finding trace by id, blockID: 89b64d6b-88a8-4f81-9709-496898b460e2: error retrieving bloom bloom-0 (single-tenant, 89b64d6b-88a8-4f81-9709-496898b460e2): does not exist; error finding trace by id, blockID: 8b56048f-8196-4baa-9939-be22d072abc1: error retrieving bloom bloom-0 (single-tenant, 8b56048f-8196-4baa-9939-be22d072abc1): does not exist; error finding trace by id, blockID: 88ef18a8-a31b-46d9-b4f2-30862881041e: error retrieving bloom bloom-0 (single-tenant, 88ef18a8-a31b-46d9-b4f2-30862881041e): does not exist; error finding trace by id, blockID: 88f8d9c7-0383-4b46-84c4-185e7dbd7f7f: error retrieving bloom bloom-0 (single-tenant, 88f8d9c7-0383-4b46-84c4-185e7dbd7f7f): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-6f905678-634d-4b19-8606-a03415405515-0; X-Scope-Orgid: single-tenant; uber-trace-id: 0e61b005e8f56bd5:79515ecc9825a315:6a45d486057c6900:0; "
sep 17 11:08:50 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-17T09:08:50.693356643Z caller=server.go:2136 traceID=0e61b005e8f56bd5 msg="GET /querier/api/traces/7ec467384213425e8401600fcaff68b0?blockEnd=687d6343eb1a1f590000000000000000&blockStart=6343eb1a1f58d0fb0000000000000000&mode=blocks (500) 1.386592ms Response: \"error finding trace by id, blockID: 65514576-2929-4e6b-9eba-b8eceb2c92aa: error retrieving bloom bloom-0 (single-tenant, 65514576-2929-4e6b-9eba-b8eceb2c92aa): does not exist; error finding trace by id, blockID: 6649fce0-8dc5-43bf-93a3-850a9f38697f: error retrieving bloom bloom-0 (single-tenant, 6649fce0-8dc5-43bf-93a3-850a9f38697f): does not exist; error finding trace by id, blockID: 636ff885-4dec-445a-ae9b-7a2f6832942a: error retrieving bloom bloom-0 (single-tenant, 636ff885-4dec-445a-ae9b-7a2f6832942a): does not exist; error finding trace by id, blockID: 65aec92c-3573-43d7-a136-bd819083aa53: error retrieving bloom bloom-0 (single-tenant, 65aec92c-3573-43d7-a136-bd819083aa53): does not exist; error finding trace by id, blockID: 66ceb0d7-d47d-4391-952e-ca2538e570f6: error retrieving bloom bloom-0 (single-tenant, 66ceb0d7-d47d-4391-952e-ca2538e570f6): does not exist; error finding trace by id, blockID: 63f1bec4-b989-4339-a04a-11cf62f71c1c: error retrieving bloom bloom-0 (single-tenant, 63f1bec4-b989-4339-a04a-11cf62f71c1c): does not exist; error finding trace by id, blockID: 684787dc-aca0-4e7c-9215-5afa2152bc9f: error retrieving bloom bloom-0 (single-tenant, 684787dc-aca0-4e7c-9215-5afa2152bc9f): does not exist; error finding trace by id, blockID: 66bbf6be-caac-4b9b-93ca-d1e0a5ed470c: error retrieving bloom bloom-0 (single-tenant, 66bbf6be-caac-4b9b-93ca-d1e0a5ed470c): does not exist; error finding trace by id, blockID: 64c8e17f-d5e2-4027-85d6-05c063b94325: error retrieving bloom bloom-0 (single-tenant, 64c8e17f-d5e2-4027-85d6-05c063b94325): does not exist; error finding trace by id, blockID: 67674cb3-24c1-43ec-929a-3df14b8791d4: error retrieving bloom bloom-0 (single-tenant, 67674cb3-24c1-43ec-929a-3df14b8791d4): does not exist; error finding trace by id, blockID: 64fe8896-3753-45a4-8550-e3d60e1d762d: error retrieving bloom bloom-0 (single-tenant, 64fe8896-3753-45a4-8550-e3d60e1d762d): does not exist; error finding trace by id, blockID: 64cceac9-9682-421e-bd23-8e8f34cad520: error retrieving bloom bloom-0 (single-tenant, 64cceac9-9682-421e-bd23-8e8f34cad520): does not exist; error finding trace by id, blockID: 661c6889-40f5-4c69-9f3a-ba746d162e0f: error retrieving bloom bloom-0 (single-tenant, 661c6889-40f5-4c69-9f3a-ba746d162e0f): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-6f905678-634d-4b19-8606-a03415405515-0; X-Scope-Orgid: single-tenant; uber-trace-id: 0e61b005e8f56bd5:7f8d4a1f9da61ad6:6a45d486057c6900:0; "

The connection between Grafana->Tempo is OK, as it can get data, problem is when querying more details

If I click on any of the above traces for gathering more details, I get this error right away:

Environment:

• Infrastructure: VM on bare-metal
• Deployment tool: local install on RHELv.8.4
• Version:
• [root@ ~]# tempo --version tempo, version 2.4.2 (branch: HEAD, revision: 2225623e7) build user: build date: go version: go1.21.3 platform: linux/amd64 tags: unknown [root@ ~]#

Additional Context
TEmpo config:

`stream_over_http_enabled: true
server:
http_listen_port: 3200
log_level: info

query_frontend:
search:
#max_duration: 6h
duration_slo: 5s
throughput_bytes_slo: 1.073741824e+09
trace_by_id:
duration_slo: 5s

distributor:
receivers: # this configuration will listen on all ports and protocols that tempo is capable of.
jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can
protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
thrift_http: #
grpc: # for a production deployment you should only enable the receivers you need!
thrift_binary:
thrift_compact:
otlp:
protocols:
http:
grpc:

ingester:
max_block_duration: 1h # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
compaction:
block_retention: 720h # overall Tempo trace retention. 30 days

metrics_generator:
registry:
external_labels:
source: tempo
cluster: docker-compose
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true

storage:
trace:
backend: local # backend configuration to use
wal:
path: /tmp/tempo/wal # where to store the the wal locally
local:
path: /tmp/tempo/blocks

overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics] # enables metrics generator`

[joe-elliott]

I'm surprised you're getting a 499 on this. It should be a 500, but these errors suggest you have corrupt blocks:

"error finding trace by id, blockID: 9a7e8e4f-2536-41a8-bfbf-3de7f9cfc9d8: error retrieving bloom bloom-0 

when looking up blocks by trace id tempo will check bloom filters first and it seems like this (and some other) blocks are missing bloom filters?

[alvargut]

Thanks @joe-elliott for your help on this issue. However, I don't get what you mean by this

when looking up blocks by trace id tempo will check bloom filters first and it seems like this (and some other) blocks are missing bloom filters?

What can I do to fix that or check what you are saying it's missing? Thank you!

[joe-elliott]

Tempo will store blocks in folders named:

s3://<your bucket>/<tenant id>/<block id>

The blocks themselves contain files like: https://grafana.com/docs/tempo/latest/operations/architecture/#ingester

Check the blocks in your error message to see if they have a bloom-0 object. If not they are corrupt. If you have a lot of corrupt blocks check your ingester logs for clues and confirm that your ingesters are stable.

[alvargut]

Ok, so if I do this query on Grafana

I get this ID= be68be3f48e345e892ff3c9ad9520186

Then on the host running Tempo (it's a RHEL 8.4), I see this (using journalctl -fu tempo):

sep 23 19:20:03 mxfuw-nso01-api01 tempo[2998145]: level=warn ts=2025-09-23T17:20:03.121793344Z caller=server.go:2136 traceID=66181ce293adfeb4 msg="GET /querier/api/traces/be68be3f48e345e892ff3c9ad9520186?blockEnd=3eb1a1f58d0fac690000000000000000&blockStart=397829cbc14e5e0b0000000000000000&mode=blocks (500) 60.040154ms Response: \"error finding trace by id, blockID: 3d350dc4-70a3-4554-8f45-e975466ad28d: error retrieving bloom bloom-0 (single-tenant, 3d350dc4-70a3-4554-8f45-e975466ad28d): does not exist; error finding trace by id, blockID: 3cee9a12-e526-493c-ac04-d0b9fbcb6379: error retrieving bloom bloom-0 (single-tenant, 3cee9a12-e526-493c-ac04-d0b9fbcb6379): does not exist; error finding trace by id, blockID: 3be9183d-80de-4eb1-96dc-f959010ae3bb: error retrieving bloom bloom-0 (single-tenant, 3be9183d-80de-4eb1-96dc-f959010ae3bb): does not exist; error finding trace by id, blockID: 3c437c3d-5778-4908-8b50-a1dd1147568c: error retrieving bloom bloom-0 (single-tenant, 3c437c3d-5778-4908-8b50-a1dd1147568c): does not exist; error finding trace by id, blockID: 3bdb4114-6054-4d01-99d0-62562500f12b: error retrieving bloom bloom-0 (single-tenant, 3bdb4114-6054-4d01-99d0-62562500f12b): does not exist; error finding trace by id, blockID: 3e11bbd1-2b4a-44bb-9c79-e7713a4a4dd9: error retrieving bloom bloom-0 (single-tenant, 3e11bbd1-2b4a-44bb-9c79-e7713a4a4dd9): does not exist; error finding trace by id, blockID: 3b890f03-2f3b-4e5f-a2ef-c80c64108e07: error retrieving bloom bloom-0 (single-tenant, 3b890f03-2f3b-4e5f-a2ef-c80c64108e07): does not exist; error finding trace by id, blockID: 3b424f62-8ff2-46aa-8498-6647a06dd04b: error retrieving bloom bloom-0 (single-tenant, 3b424f62-8ff2-46aa-8498-6647a06dd04b): does not exist\\n\" ws: false; Accept: application/protobuf; Accept-Encoding: gzip; User-Agent: Grafana/9.4.7; X-Datasource-Uid: JPZ7fmIHk; X-Grafana-Org-Id: 1; X-Panel-Id: Q-62d92a64-0073-4d91-8475-1e6718a764d9-0; X-Scope-Orgid: single-tenant; uber-trace-id: 66181ce293adfeb4:2a808df2d1841d8f:3636a1890bdee2d0:0; "

Saying "error retrieving bloom bloom-0 (single-tenant, 3b424f62-8ff2-46aa-8498-6647a06dd04b): does not exist"

On the file system where Tempo stores the data, folder is empty:

[root@mxfuw-nso01-api01 single-tenant]# pwd
/tmp/tempo/blocks/single-tenant
[root@mxfuw-nso01-api01 single-tenant]# ls -la 3b424f62-8ff2-46aa-8498-6647a06dd04b
total 60
drwxr-xr-x   2 tempo tempo    23 sep  5 10:01 .
drwxr-xr-x 723 tempo tempo 40960 sep 23 19:22 ..
-rw-r--r--   1 tempo tempo   407 sep  5 10:01 meta.json
[root@mxfuw-nso01-api01 single-tenant]# 

The contents of "meta.json is:

[root@mxfuw-nso01-api01 single-tenant]# cat 3b424f62-8ff2-46aa-8498-6647a06dd04b/meta.json 
{"format":"vParquet3","blockID":"3b424f62-8ff2-46aa-8498-6647a06dd04b","minID":"AA2KKVQsS9+me+5tVC0Xig==","maxID":"/+yb9RSWQCC3+3ZBza5+8w==","tenantID":"single-tenant","startTime":"2025-09-04T09:00:00+02:00","endTime":"2025-09-04T09:55:41+02:00","totalObjects":5216,"size":5351804,"compactionLevel":4,"encoding":"none","indexPageSize":0,"totalRecords":1,"dataEncoding":"","bloomShards":1,"footerSize":12876}
[root@mxfuw-nso01-api01 single-tenant]# 

Folder nor partition doesn't seem full

[root@mxfuw-nso01-api01 single-tenant]# du -sh /tmp/tempo/*
1,9G    /tmp/tempo/blocks
34M     /tmp/tempo/generator
5,2M    /tmp/tempo/wal
[root@mxfuw-nso01-api01 single-tenant]# 
[root@mxfuw-nso01-api01 single-tenant]# df -hTP
S.ficheros                     Tipo     Tamaño Usados  Disp Uso% Montado en
devtmpfs                       devtmpfs    32G      0   32G   0% /dev
tmpfs                          tmpfs       32G      0   32G   0% /dev/shm
tmpfs                          tmpfs       32G   3,3G   29G  11% /run
tmpfs                          tmpfs       32G      0   32G   0% /sys/fs/cgroup
/dev/mapper/rhel-root          xfs        348G    39G  310G  12% /
/dev/mapper/rhel-home          xfs         70G   1,2G   69G   2% /home
/dev/mapper/rhel-var_lib_mysql xfs         50G   1,4G   49G   3% /var/lib/mysql
/dev/sda1                      xfs       1014M   201M  814M  20% /boot
tmpfs                          tmpfs      6,3G      0  6,3G   0% /run/user/1025
tmpfs                          tmpfs      6,3G      0  6,3G   0% /run/user/1022
[root@mxfuw-nso01-api01 single-tenant]

[joe-elliott]

If this is a one time thing, I would just delete the block and ignore it. If it happens regularly we would need to figure it out.