New workspace flow

[tjfulle]

This PR implements an updated workflow as described in https://github.com/sandialabs/canary/blob/tjfulle/repo-layout/workspace-cmd-spec.md

The problem

The current behavior of Canary is to create a TestResults directory and in it, a .canary directory. All tests are run in the TestResults directory and configurations, options, metadata stored in TestResults/.canary. The test results directory is mutable: future test runs simply updated the previous results. To rerun tests, create reports from previous results, etc., canary must be invoked inside the test directory. In this case, canary can recover the previous settings and test results.

Three helper flags aid with this workflow: run -d PATH causes canary to write test results to PATH instead of TestResults; run -w causes canary to remove the test results directory before execution; and -C PATH causes canary to navigate to PATH before processing the command line.

More concretely:

$ canary run [options] DIR  # run the tests in DIR
...
$ canary run [options] OTHER_DIR
==> Error:: TestResults already exists
$ canary run -d OtherTestResults [options] OTHER_DIR
...
$ canary report cdash create  # must be run in the results directory
==> Error: not a test session
$ canary -C OtherTestResults report cdash create
...
$ canary run -w [options] DIR  # Create new session and run tests in DIR
...

Feedback from Sierra developers indicated that this was confusing. Particularly the interaction between the -C, -d, and -w flags.

Introduction the Canary Workspace

This PR refactors canary away from the above workflow. Rather than describe the workflow, we show it

NOTE: The visible behavior of canary run [options] [PATH] is unchanged - with the exception of the -d flag mentioned above. -d is now hidden from help pages and still "works" but is not recommended.

The new workflow:

$ canary init  # create a Canary workspace in .canary
$ canary add PATH  # Add test generators found in PATH to the workspace
$ canary run  # Generate test cases from generators in workspace and run them
$ canary status
...

This demonstrates a very simple workflow where generators are added, test cases generated, and run. The new workflow allows for far more flexibility:

$ canary init 
$ canary add PATH1 PATH2
$ canary select -k fast --tag fast  # generate test cases with the keyword filter fast and tag the selection for later use
$ canary select -k spam --tag spam
$ canary run fast  # run only the tests in the 'fast' selection above
$ canary run spam
$ canary run [default]  # run all the unfiltered tests in the workspace
$ canary status  # show status of all tests in the workspace
$ canary gc  # clean out generated test files in tests that finished successfully
$ canary report cdash create  # no need to navigate anywhere

Some implementation details

• canary init creates the Canary "workspace" in $PWD/.canary.
• Each invocation of canary run creates a new Session whose results are kept in .canary/sessions/SESSION_ID
• canary keeps a database of all of the latest test results across all test sessions.
• Test sessions are never "resumed". Each new run creates a new session.
• Test case dependencies are tracked across sessions.

To do

• Update documentation
• Feedback from @mdmosby
• Verify canary-sierra works
• Verify canary-distributed works

[tjfulle]

Diagram, courtesy of ChatGPT

┌────────────────────────────┐
│         Workspace          │
│ (.canary root, owns state) │
├────────────────────────────┤
│ • index.json               │  ← all known cases
│ • latest.json              │  ← latest results across sessions
│ • refs/{HEAD,latest}       │
│ • sessions/                │  ← all session dirs
│ • tags/                    │  ← tag symlinks
│ • view/                    │  ← workspace-wide current view
└──────────┬─────────────────┘
           │ creates
           ▼
┌────────────────────────────┐
│         Session            │
│ (isolated run directory)   │
├────────────────────────────┤
│ • index.json               │  ← case dependency graph
│ • latest.json              │  ← results in this session
│ • config                   │  ← snapshot of workspace config
│ • work/                    │  ← test case locks
│ • .TAG                     │
└──────────┬─────────────────┘
           │ initialized from
           ▼
┌────────────────────────────┐
│      CaseSelection         │
│ (immutable list of cases)  │
├────────────────────────────┤
│ • ids[]                    │  ← case identifiers
│ • deps{}                   │  ← dependency map
│ • hash                     │  ← SHA-256 fingerprint
│ • created_on               │
└──────────┬─────────────────┘
           │ enumerates
           ▼
┌────────────────────────────┐
│         TestCase           │
│ (atomic runnable unit)     │
├────────────────────────────┤
│ • id                       │
│ • deps[]                   │
│ • status / result / meta   │
│ • testcase.lock (per case) │
└────────────────────────────┘

[tjfulle]

on disk layout

.canary/
├── index.json              # all known case IDs
├── latest.json             # latest result of each case
├── refs/
│   ├── HEAD → sessions/2025-11-06_14-01-22
│   └── latest → sessions/2025-11-06_14-01-22
├── sessions/
│   ├── 2025-11-06_14-01-22/
│   │   ├── index.json
│   │   ├── latest.json
│   │   ├── config
│   │   ├── work/
│   │   │   ├── ab/cd1234/testcase.lock
│   │   │   └── ...
│   │   └── .TAG
│   └── ...
├── view/                   # fast access symlinks
│   ├── ab/cd1234 → ../sessions/.../work/ab/cd1234
│   └── ...
├── tags/
│   ├── main → ../cache/tags/main
│   └── v1.0 → ../cache/tags/v1.0
├── cache/
│   └── tags/
│       ├── main/
│       │   ├── selection.json
│       │   └── meta.json
│       └── ...
└── objects/
    └── generators/
        ├── 12/34abcd.json
        └── ...

[tjfulle]

Currently running Alegra's nightly tests.

[tjfulle]

Workflow for running night tests:

$ source setup-env.sh  # project specific settings
$ canary init
$ canary add $PROJECT_DIR/tests  # PROJECT_DIR set in setup-env.sh
$ canary select --tag=nightly -o dakota -k 'not long' 
$ canary info
Workspace:   /gpfs/alegra/work/2025.11/.canary
Version:     25.11.6+74b91053-dirty
Generators:  1717
Sessions:    2
Latest:      None
Tags:        default, nightly
$ canary run [batch options] nightly

Some notes:

• canary info should include more info? Number of tests in a tagged selection? The options associated with the tag?
• I have a LOT of batch options and other options. Would be nice to have those set for future use too?

[tjfulle]

I just re-read @mdmosby document above and noticed that I used the name canary select that he was not too keen on. We can/should bike shed all of the names in the new workflow.

[tjfulle]

So the nightly tests ran great. About 5000 passed fine. 80 failed. The 80 that failed are at the end of a complicated dependency chain. I'll have to do a bit more investigating to determine the root cause.

If all of Alegra's tests can pass with the new workflow I'd say that constitutes completion of the technical implementation.

[tjfulle]

Superseded by #67