feat: adds a parameter object for the action object

[baywet]

Introducing the Overlay Parameters proposal

fixes #33
fixes #136

Summary

This pull-request introduces new parameters for the action object. The values of those parameters can be read from the environment variables, or when missing from the default values field. It also introduces a string interpolation syntax that needs to avoid any conflicts (see the issues linked above) so that values can be inserted anywhere in the action.

Vision

This proposal has multiple goals:

1. Make OpenAPI descriptions more dynamic via environment variables. This has been a long-requested ability.
2. Make Overlay actions more scalable through reducing repetitive actions.
3. Leverage Overlay documents to avoid adding complexity to OpenAPI and to be consistent with the ecosystem.
4. Avoid the use of pre-processing scripts/tooling as it’s highly inefficient for everyone using overlays to have to roll out their own scripts.
5. Allow for better interoperability between implementers through standardization (instead of custom scripts or custom extensions).

The short-term use of this would be to enable goals 1 & 2. In the long term there might be an additional opportunity to come up with “libraries of overlays” which would distribute nice additions for OpenAPI descriptions (standard paging patterns, etc…) but I’m not trying to solve for that.

Execution

Example scenarios

Matrix

See this comment #238 (comment) or that one #33 (comment)

Environment variables

See the initial issue, which has tons of great examples. I think a key recap could be “I want to use my OpenAPI description as a template for multiple hosts deployments, and I don’t know the hosts names in advances” (think about cloud deployments with tenants, etc…). This also helps enable a set of CI/CD scenarios (like deploying a temporary API on a host to run some tests, and deploying the updated documentation for the API alongside)

1.1 Release

This should not hold the 1.1 release. I’m happy to punt it to a 1.2 release if we think it’s going to get the 1.1 release out of the door faster. And at the same time, I’d like to continue making progress on this by getting actionable feedback from the reviewers.

Implementation

It might not be clearly called out in the current proposal, but the intended implementation guideline is the following in order:

1. Parse the Overlay document (implementers already have most of that)

2. Foreach action that uses parameters

   1. Resolve the values of each parameter
   2. Project the values of each parameter into a “projected action” (one of the parameter values combinations + the action)
   3. Foreach “projected action” execute the string interpolation of the parameter values
   4. Insert all the projected actions in place of the original parameterized action

3. Apply all the actions, projected or regular. (implementers already have that)

This is important to enable separation of concerns, get the matrix projection right, and enable maximum reusability with the code that’s already in place for v1.0. I’m happy to rework the wording to make this clear in the specification.

Remaining questions

Do we have the right interpolation syntax?

I’m happy with any syntax really, if it meets the constraints we have. I’ve forked the discussion here #136 (comment)

Should we always read Environment variables?

Since it’s in the same “parametrization domain”, I thought it’d be nice to bundle changes together. In the initial draft of this proposal, we had the ability to have constants/avoid reading the environment variables, which we’ve lost with the latest revisions. I think we should restore that ability somehow. What about a field “ignoreEnvironmentVariable” (Boolean, default false)? When set to true, the implementer would only read the defaultValues.

Do we have the right matrix/projection syntax?

I am especially interested in the projection, matrix, and scaling aspects, as my team has been dealing with these challenges and is eager to find a solution. But it also adds great value. And users of the specification are not required to use it as it is defined.
I know this is a contentious point, and there have been suggestions to cut that aspect from the proposal. I will not put efforts behind a version of that proposal without the matrix aspect in one form or another. I’m happy to take actionable suggestions to make it better/safer/simpler/…

Do we need parameters references?

Ralf made a pertinent comment about the fact that if multiple actions had the same parameter name, they are effectively the same parameter since they’d read from the same environment variable. And he suggested hoisting them at the document level instead of the action. While this would reduce the repetitive definitions further, we’d still need to flag any given action is using parameters, so implementers can do the pre-processing. This would require defining parameter references at the action level.
I think Lorna made a pertinent comment in another context: we want to keep the complexity low to bolster adoption. I think the “global parameters” comment goes against that, with minimal gains. And this is something we can review in v2 if people are tired of re-defining the same parameters.

I hope this rewrite of the initial comment brings a lot of clarity, I spent a lot of time on it :)

[baywet]

Another note: I threw copilot at implementing this proposal in Clio and it made an interesting "suggestion". Use the values field for a parameter defined as sourced from environment when the environment variable is not found.

This also begs the question. Of what should happen when the environment variable is not available? Error? Fallback when available and error if not? Tool specific?

[ralfhandl]

I like the idea of adding a default value if the source is environment. I'd probably name it default or defaultValue and not just value.

[ralfhandl]

Why local to an action and not global to allow use across multiple related actions, for example when applying traits?

[mikekistler]

I like @ralfhandl 's suggestion to make these global rather than within the action.

[baywet]

The reason why I didn't make the parameters global is because having the parameters on the action makes it easy for any implementer to detect parameters are used for any given action and expansion is required.

If we want to make global parameters a thing, I think we then need to introduce a parameter reference mechanism. Where we'd mandate that the action has a reference entry for a global parameter to be used and for the expansion + interpolation to happen.

Thoughts?

[ralfhandl]

I like that: parameters are global, and need to be referenced by actions that use them.

[ralfhandl]

Are we going to make parameters global and use/reference/import them in the actions?

If the values are provided in env variables, their names are global anyway, and same name means same value(s).

[baywet]

I'm open to it. What do we want the references to look like? we could use the $ref syntax so it feels familiar, and it opens the door to having more kinds of references, external references, etc... in the future if we need to. But the downside is that it's a bit verbose and might bring in much more than we need (again, external references, etc...)

[ralfhandl]

I wouldn’t use $ref and simply reference by name:

• global parameters is an array of parameter definition objects
• action-local parameters is an array of parameter names

Similar to the tags keyword in OpenAPI.

Or use different keywords, for example usedParameters in actions.

[baywet]

I like the idea of adding a default value if the source is environment. I'd probably name it default or defaultValue and not just value.

Differentiating the field would surely make it clearer to the users.
We could also easily define the behavior of zero values for a given parameter means the action "does not get applied at all" this way.

[ralfhandl]

I really want to restrict parameters to a single value and avoid the complexity of matrix processing.

That can be done in a wrapper script.

[baywet]

I'll try to provide more context about the intention of this proposal. It was put together mainly for the matrix feature because it's the only use case we have for parameters.
I added support for environment variables as a nice to have to try to solve a similar need in a consistent manner.
Yes, templating of actions can be done outside of the overlay, but it's added complexity and custom infrastructure that needs to be built for everyone who needs it.
Removing the matrix aspect makes inline values definitions moot as well.

I don't think I'll push this proposal to completion without matrix in one form or another.

So if you instead have suggestions to mitigate the risks you outlined without removing the feature all together, I suggest we follow that path instead.

[ralfhandl]

Removing the matrix aspect makes inline values definitions moot as well.

That would be another nice simplification:

• parameter values always come from the environment
• optionally allow a default value inline in case the environment variable is undefined

[baywet]

@ralfhandl thanks for the default values suggestion, I think it simplifies things a great deal. Updated to incorporate that.

As for the matrix, can you please articulate the risk vector more? And maybe suggest other solutions to mitigate it than "remove matrix entirely"?

There are tons of precedents in the wild for that, most broadly adopted (that I know of) being the GitHub actions workflow matrix

So I'm not sure why doing the same here would constitute a riskier scenario or an anti-pattern.

[ralfhandl]

To me it seems extremely unlikely that the target expression and the update value will be parameterized with the same single string value, which would be necessary for the current implicit loops over multi-valued parameters.

If you have three or more realistic examples where this is the case, I may reconsider.

Until then I believe that most use cases for loops over parameterized actions will require groups of parameter values.

Assume an API has paths for several "thingies" and we need to insert systematic descriptions for each operation on each thingy.

API structure is something like

paths:
  /foo:
    get: ...
    patch: ...
  /bar:
    get: ...
    patch: ...

The descriptions we want to add are along the lines of

• Read Foobaroo thingies
• Update Foobaroo thingies
• Read Barberix thingies
• Update Barberix thingies

We would need an array of (thingy path segment, thingy name)

• foo, Foobaroo
• bar, Barberix

and an array of (method, operation purpose)

• get, Read
• patch, Update

Then two nested loops over these arrays can produce the necessary matrix of actions.

Which means I need arrays of "structured" parameters.

[mikekistler]

I'm only half paying attention here, as this week is a crazy week for me at work. But I wanted to mention something that occurred to me while I was working on some Spectral rules recently.

Spectral allows the "given" (JSON Path of the things to check) to be either a string (single JSON path), or an array of strings (JSON Paths). This can be quite handy.

I wonder if that has anything to do with the discussion on "looping" here?

[baywet]

Thank you for the additional information.

Side car pattern on existing schemas

One usage of the matrix feature would be to implement a sidecar pattern at scale.

Note: I'm not employing the actual component types/names in the example below due to embargo on upcoming features.

In our case (which motivated my work on this issue) we get a description from OpenAI, which contains dozens of "ToolResponse" component schemas. Azure OpenAI has an additional layer that allows you to define a "ToolResponsePolicy" using additional APIs, I'd like to avoid having to redefining the dozens of side cars. Using an overlay like this one would solve for that.

overlay: 1.1.0
actions:
  - description: define the side cars
    target: $.components.schemas["${toolName}ToolResponsePolicy"]
    update:
      type: "object"
      description: "The ${toolName} Tool ResponsePolicy"
      properties:
        tool_name:
          const: ${toolName}
    parameters:
      - name: toolName
        defaultValues:
         - "MCP"
         - "WebSearch"
         - "SemanticSearch"
         - "..."

Combination with source

Note: this example is mangled as well by intention due to embargo constraints, don't try to correlate with public API references.

Another pattern related to this side car is to create additional path segments when specialized operations are required, without having to re-define everything. Which is why I used a syntax which doesn't collide with path parameters in OAI.

overlay: 1.1.0
actions:
  - description: Duplicate the eval_run operation for the tool definition
    target: $.paths[".../models/evals/{evalId}/${toolName}/run
    source: $.paths[".../models/evals/{evalId}/run
    parameters:
    - name: toolName
      defaultValues:
         - "MCP"
         - "WebSearch"
         - "SemanticSearch"
         - "..."
   # potentially another update action to add additional properties for each of the tool eval runs
   # this way we don't have to re-define all the properties
   # and no, we can't use a base type because most of the types are shipped by OpenAI and we're only re-using them

Hopefully this illustrates the need pretty well. I kept it as one dimension examples to keep it simple, but you could also have another dimension per version and a third one per environment (public, national, etc...), where each dimension would add additional properties.

[ralfhandl]

@baywet Thanks for the detailed example. It can also be slightly modified to support my point:

• Suppose my company has different naming conventions for path segments and tool names, established before tool names where showing up in path segments.
• This means I need pairs of (tool-name-for-path, tool-name-for-description)

Again array of structured parameter.

[baywet]

@ralfhandl I feel like we're having two separate conversation here. I'm arguing we should have a matrix feature, and thought you were arguing against? But now seem to want structure data support?

Could you please clarify? And also provide examples?

[ralfhandl]

I think the matrix feature without structured parameter arrays only works for your current special cases: you are extremely lucky that targets and update objects can be parameterized with the same strings.

Slightly different naming conventions would break that, and the obvious fix are arrays of value tuples or flat objects, and I have provided such examples.

Adding the easily breakable variant to the spec adds complexity with little potential for use.

For me it is either add the "full" solution which has similar complexity, or don't add the array feature at all and place the loops around the Overlay processor.

[baywet]

Thank you for the additional information.

If I'm following correctly, you'd essentially like to define the defaultValues as

string | Record<string, string>

correct?

I'm open to that, what does that mean for environment variables parsing though?

[ralfhandl]

Correct.

In an environment variable I'd pass a stringified JSON array of flat objects.

[baywet]

I'm guessing in that case we'd do away entirely with the separator, people could simply provide a JSON array of strings for the "simple format" ?

[ralfhandl]

Yes, that would be a nice simplification!

[baywet]

@ralfhandl Thank you for the additional information.
Updated things to match our conversation, let me know what you think!

[mikekistler]

Seems like there is no way to say "don't look in the environment, just use these values". Is that intentional?

[baywet]

That was one reservation I had as well which is why I initially had the source enum in place.
Happy to restore it. Or maybe we can have a boolean instead to ignore the environment variable.

[baywet]

@ralfhandl I'd like your input on this one please since you helped me refactor into a simpler syntax, we kind of lost a capability in the process.

[lornajane]

I am not in favour of the matrix feature. I would like us to consider simple parameters though.

I'm very confused about how we're using environment variables here, or it might be a terminology problem! Environment variables: the values in the environment when the tool executes? Or something else in this context?

My previous proposal for supporting environment variables was rejected because we received feedback about the difficulty of parsing dollar signs in value expressions where there might already be dollar signs in the OpenAPI content for other reasons. I notice however that we have again used dollar signs in the syntax and would like to make sure that we reach out to the wider community for their feedback before pushing through a change like this. /cc @ThomasRooney

[baywet]

@lornajane why are you not in favour of matrix parameters?

As for the syntax, happy to change it to something else if we have a better suggestion

[baywet]

Comment from @lornajane from the call: we might be able to cut some things to make it simpler to review/implement: matrix essentially, and using the string replace in the target should be prohibited for now.

[baywet]

@mikekistler @mikeschinkel @lornajane @ralfhandl I just rewrote the initial comment of this pull request to help clarify the vision, approach, remaining questions, etc... Please have a read, hopefully this helps us unlock further progress.

[ralfhandl]

Add note that same name in different actions means same environment variable value in all actions.

Which also makes different default values questionable.