repo structure options for multi-language publishing and testing..

[mighdoll]

I guess there are many options to support multiple shader languages and tests in Lygia. Here's a few:

• single-repo - all shaders and all tests in one repo.
• multi-repo - each language w/tests in its own repo.
• monorepo - all shaders and all tests in one repo, with subdirectories containing their own package.json.
• submodules - all shaders in one repo, per language tests in separate repos with submodules including their main repo.

single-repo:

lygia:
  wesl-test/
  glsl-test/
  package.json
  color/
  ...

multi-repo:

lygia_wgsl:
  package.json
  test/
  color/  #only .wgsl files
  ...

monorepo:

lygia:
  packages/
    lygia_wgsl/
      package.json
      test/
    lygia_glsl/
      package.json
      test/
  package.json  # lygia npm pkg has multiple exports, users import as `lygia/glsl`, or `lygia/wgsl`, etc.
  color/
  ...

submodule:

lygia:
  color/
  ...
lygia_wgsl:   
  lygia -- submodule -- 
  tests/
  package.json. # lygia_wgsl npm package has a single export, users import as 'lygia_wgsl'

We've some experience with these in the wgsl-tooling-wg repos. Happy to help you with any of them. I'd recommend single-repo if you can, it's by far the easiest. monorepo is the next simplest if you want to keep the shader language implementations together in the same directory as now.

Any solution with multiple repos raises hassles of keeping the repos in sync. And PRs don't work across multiple repos. We use submodules too, but I'd avoid that if you can. Submodules add complexity and hassles for educating/configuring every new contributor, plus all the problems with multiple repos.

[patriciogonzalezvivo]

Thank you so much @mighdoll for this breakdown! You are right! single, multi and mono repo works great for npm packages. That would satisfy glsl and wesl greatly and allow for per language tests all through npm tooling.
That said
At the same time, this approach is not compatible with HLSL (Unity), CUDA, Metal, WGSL (cargo) and GLSL (throught Visual Studio Code, Minecraft, ComfyUI, TouchDesigner and CMake projects). All of those require different and custom strategies.

That's the reason I keep leaning towards the sub-module approach.

On that spirit, I was wondering if it's ok if I create a lygia_wesl repo and add you as the main contributor, and start exploring that idea

[mighdoll]

WGSL (cargo)

We haven't talked about it yet, but note the wgsl-tooling-wg tooling has support for cargo and rust users as well. When you're ready for that, we can engage with some of my more rusty colleagues. bevy has added support (and is internally is starting to transition to the standard wgsl-tooling-wg extensions from their previous custom #import approach). It'll be nice for bevy users to have easy access to lygia!

re: ongoing maintenance, I'm game to be one of the maintainers. but not the only one, you've got to stay on it too :-). I'm sure there'll be ongoing improvements in tooling and tuning for contributors in using the extensions like the import or condition syntax (or multi-methods when we get to that..) I can help on that for sure. I won't likely have time to port glsl contributions to wgsl/wesl ongoing. But I think best would be to expect contributors to submit new shader functions with at least a basic test to keep the maintenance burden down. With two repos, contributors will have to submit two PRs, but I suppose there's not so many contributors to educate..

re: publishing to npm, do you prefer lygia/wesl or lygia_wesl? Actually we've been generally encouraging people to publish with _wgsl or /wgsl, see Publishing Packages - naming conventions. So maybe lygia/wgsl or lygia_wgsl. (The wgsl-tooling-wg stuff supports vanilla WGSL. e.g. lygia/math/inside.wesl is vanilla WGSL. We're just trying standardize the WGSL extensions like import)

But either way, do you prefer the lygia/ (separate export within the same npm package) or the separate npm package with the _? I haven't done the / with separate repos, but I imagine we can find a way.. I guess the / is more idiomatic for npm: e.g. we do wesl-plugin/vite and wesl-plugin/webpack. But is it better to have separate npm packages for lygia? Can you educate me? Which kind of users use lygia from npm and which use lygia via submodules or some other way?

[mighdoll]

re: publishing to npm, do you prefer lygia/wesl or lygia_wesl?

now realizing I think maybe the idea is to npm publish as lygia - just as is done currently. lygia_wgsl would be a repo just for tests (and maybe examples). Is that right? lygia_wgsl wouldn't be published to npm at all, just lygia.

[vade]

I brought some of this up in the Metal issue #165

Im personally a fan of either a submodule or mono repo approach where each language is self contained in a parent folder.

As a user of lygia and a dev who makes use of Lygia in 3rd party tools, I'd really prefer to be able to target a specific shader language and not get extra stuff. I have zero use for server driven NPM packaging, and a language specific repo would allow for language specific affordances, like a swift package manager declaration which requires subtly different repo layout assumptinons.

[mighdoll]

As a user of lygia and a dev who makes use of Lygia in 3rd party tools, I'd really prefer to be able to target a specific shader language and not get extra stuff.

How do you get the metal shaders now? Some kind of git fetch and then filter for the files you want? And you want to filter more easily by just selecting directories?

As a user of lygia and a dev who makes use of Lygia in 3rd party tools, I'd really prefer to be able to target a specific shader language and not get extra stuff. I have zero use for server driven NPM packaging,

For the npm web user, I think it's a bit different. Including the extra languages doesn't help, but doesn't really hurt. It's a little space on the developers machine, but the developer's application will anyway only include the selected lygia functions in the selected language.

(I'm game to support whatever direction and my 2c is to keep it simple unless there's a strong need to complicate.)

[vade]

Apologies, missed your response

Today, i package everything and only parse the metal shaders via extension filtering. Its stupid as fuck, but it works. Its not elegant, and packaging the other shaders isnt a huge deal, but its cumbersome and unnecessary and inelegant.

Also, the specific directory structure required for a swift package cant be met with the current layout, which stops some easy integrations, automatic metal library precompilation, and stuff™

Part of this is frankly an ergonomics issue. It isnt a show stopper, it just makes app bundle inclusion awkward.