Speed up batou age interaction #496

PhilTaken · 2025-02-17T10:32:00Z

The most relevant commit is 8d569b2.

Due to the previous implementation working with generous amounts of temporary files (especially so when using age-diffable), the IO overhead from reading from and writing to these files is large enough to severely slow down secret handling operations.

This PR implements in-memory handling of deployment secrets via bindings to the rage library, a rust implementation of the age spec. This leads to massively improved secret handling performance.

While this PR does introduce some added requirements on external libraries, it also relaxes the dependence on the (r)age cli.

When implementing this PR, I discovered a few bugs in the test implementation such as improper isolation of calls to git and gpg having issues creating an agent socket in nested folder structures as well as a dependence on a very specific terminal width where a bunch of tests would fail due to very strict string comparison. I have added these changes to this PR.

N.B.: due to limitations around handling of RSA keys in the cryptography library that rage depends on, RSA keys with moduli > 4096 are (currently) not supported via the bindings. In these cases, the implementation attempts to fall back to calling the age binary - see commit 000b98a. Since rage is incompatible with these RSA keys anyways, the previous command discovery implementation could be simplified.

elikoga · 2025-02-17T11:54:34Z

Calling out to the rage library is probably the right call for performance here. I distinctly remember being very frustrated during development of the age binary integration.

PhilTaken · 2025-02-17T14:06:08Z

The largest performance hog now is the call out to op to read the password, at around 300-600ms each the impact is pretty significant if you want to use batou secrets decrypttostdout in git diffs. It currently sits around 1 second total for each side of the diff for each commit.

Certainly usable but could be significantly improved imho.

zagy · 2025-02-27T06:14:57Z

CHANGES.d/20250217_112817_ph_speedup_batou_age_interaction.md

This needs to note that 3.7 is no longer supported.

Added in eb1bd5f

zagy · 2025-02-27T06:30:51Z

src/batou/tests/test_deploy.py

        """\
 batou/2... (cpython 3...)
-================================== Preparing ===================================
+... Preparing ...


It's not quite clear why this changes when changing the encryption?

This is not directly related to the encryption but was required so I can run the test suite on my machine to test all other changes. The specific commit is d4cacca. I can move these out to a separate PR now if you want.

zagy · 2025-02-27T07:02:54Z

I also get an error in our standard deploy container, meaning there is another dependency which might not be available everywhere (think switches). While a remote system does not need do decrypt (since that happens locally) the dependency would be required. We need to re-think this, I suppose.

  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

PhilTaken · 2025-02-27T08:59:28Z

Yes, the added dependency on pyrage does require a rust compiler if there is no wheel available for that platform.

We could gate the depedency on pyrage behind a feature flag that is enabled by default so it can be turned off for some systems 🤔

leona-ya · 2025-06-05T08:58:20Z

src/batou/secrets/encryption.py

+                key_content = f.read()
+            try:
+                priv_key = serialization.load_ssh_private_key(key_content, None)
+            except ValueError:


This actually should be

Suggested change

except ValueError:

except (ValueError, TypeError):

I ran into an issue where batou secrets edit staging failed with the error message thrown inside cryptography

Key is password-protected, but password was not provided.

that is a TypeError

Will add, thanks for the review. I did not consider that case.

If only it was clear what kinds of exceptions could be thrown when calling a function...

PhilTaken · 2025-06-12T11:50:11Z

I will attempt to loosen the dependency on pyrage by adding a bit of yes/no logic in the encryption code such that systems that either don't need pyrage at all or don't have a pre-built pyrage wheels available are not deprecated with this PR

Massively improve secret handling performance by avoiding large amounts of writing to / reading from temporary files. This solves a lot of complexity around the age call because it does not require piping the ssh key's password into a subprocess' stdin conditionally. Instead, this can be handled using `cryptography` to handle the password-protected ssh key and `pyrage`, python bindings to the rage library which supports de/encrypting in-memory. Another benefit of this implementation is a relaxed requirement (now optional) on the `age` CLI since the bindings to the rage library provide a full implementation out of the box.

The test suite provides some git configuration, additional custom git configuration should not cause test cases to fail. GPG has a maximum length for the path to the GPG agent socket, Copying the entire gpg directory to a temporary directory ensures that the path is not too long when the repo is cloned into a somewhat nested directory structure.

recipient errors can occur when using rsa keys with module > 4096 as recipients with rage, age can handle those

PhilTaken self-assigned this Feb 17, 2025

PhilTaken requested a review from zagy as a code owner February 17, 2025 10:32

PhilTaken requested a review from ctheune February 17, 2025 10:32

zagy requested changes Feb 27, 2025

View reviewed changes

PhilTaken force-pushed the phil/speedup-batou-age-interaction branch from 7b7b6f5 to ba432a8 Compare February 27, 2025 08:41

leona-ya reviewed Jun 5, 2025

View reviewed changes

PhilTaken force-pushed the phil/speedup-batou-age-interaction branch from eb1bd5f to 844647f Compare June 12, 2025 11:46

PhilTaken force-pushed the phil/speedup-batou-age-interaction branch from 54e8738 to f7d4c98 Compare August 15, 2025 09:03

PhilTaken added 15 commits August 22, 2025 11:07

update example lockfiles

483854f

fix tests: remove assert on fixed terminal width

da8b856

add fallback behaviour for age encryption

7e96241

recipient errors can occur when using rsa keys with module > 4096 as recipients with rage, age can handle those

improve devshell: update + add uv, gpg, mercurial, svn

4a11a7e

fix nix build

3d696c7

add changelog

66afb18

fix type hinting: python < 3.9 cannot subscript type

d090640

breaking: drop python 3.7 compatibility

1533c79

update requirement lockfiles after dropping python 3.7

140ad67

update lockfiles with python3.8

4f2e15a

add note about dropping python 3.7

b0222b1

catch more errors during age encryption

5088ff3

fix test in CI

664b79c

PhilTaken force-pushed the phil/speedup-batou-age-interaction branch from 679e1cb to 664b79c Compare August 22, 2025 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Speed up batou age interaction #496

Speed up batou age interaction #496

Uh oh!

PhilTaken commented Feb 17, 2025

Uh oh!

elikoga commented Feb 17, 2025

Uh oh!

PhilTaken commented Feb 17, 2025

Uh oh!

zagy Feb 27, 2025

Uh oh!

PhilTaken Feb 27, 2025

Uh oh!

zagy Feb 27, 2025

Uh oh!

PhilTaken Feb 27, 2025

Uh oh!

zagy commented Feb 27, 2025

Uh oh!

PhilTaken commented Feb 27, 2025

Uh oh!

leona-ya Jun 5, 2025

Uh oh!

PhilTaken Jun 12, 2025

Uh oh!

PhilTaken commented Jun 12, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Speed up batou age interaction #496

Are you sure you want to change the base?

Speed up batou age interaction #496

Uh oh!

Conversation

PhilTaken commented Feb 17, 2025

Uh oh!

elikoga commented Feb 17, 2025

Uh oh!

PhilTaken commented Feb 17, 2025

Uh oh!

zagy Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

PhilTaken Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

zagy Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

PhilTaken Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

zagy commented Feb 27, 2025

Uh oh!

PhilTaken commented Feb 27, 2025

Uh oh!

leona-ya Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

PhilTaken Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

PhilTaken commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PhilTaken commented Jun 12, 2025 •

edited

Loading