Toolchain incorrectly overwrites header search path variables set by modules

[bedroge]

While building Clp-1.17.10-foss-2024a.eb in EESSI, we ran into an issue with finding the CoinUtils header files:
EESSI/software-layer#1294 (comment)

ClpDualRowDantzig.cpp:6:10: fatal error: CoinPragma.hpp: No such file or directory

This file is actually in EBROOTCOINUTILS/include/coin, and that directory should have been added to the relevant environment variable by the module file, see https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/c/CoinUtils/CoinUtils-2.11.12-GCC-13.3.0.eb#L32.

In EESSI we use

module-search-path-headers               (E) = include_paths
search-path-cpp-headers                  (E) = include_paths

and when debugging it, I noticed that the module does indeed have:

prepend_path("CPLUS_INCLUDE_PATH", pathJoin(root, "include"))
prepend_path("CPLUS_INCLUDE_PATH", pathJoin(root, "include", "coin"))

But the interactive EB shell showed that the include/coin directory was not part of C[PLUS]_INCLUDE_PATH anymore during the configure/build step?!

I think the cause is very similar to what I wrote for a similar issue in EESSI/software-layer#1216, in particular this comment: EESSI/software-layer#1216 (comment).
In short, the module does provide the right information, but EB's toolchain.py overrides these environment variables (here: https://github.com/easybuilders/easybuild-framework/blob/easybuild-framework-v5.1.2/easybuild/tools/toolchain/toolchain.py#L1187). The values for C[PLUS]_INCLUDE_PATH are determined in https://github.com/easybuilders/easybuild-framework/blob/easybuild-framework-v5.1.2/easybuild/tools/toolchain/toolchain.py#L1159, and this seems to be hardcoded to include only for regular dependencies (toolchain components could have an extra_dirs), so it does not take any additional subdirs into account.

Previously, this probably wasn't an issue because of what I wrote in that aforementioned comment: older versions used CPATH from the module file, but CPPFLAGS in toolchain.py, so they wouldn't interfere with each other. Now they're both using C[PLUS]_INCLUDE_PATH, and they start overwriting each other's value.

[bedroge]

Example that reproduces the issue:

$ cat foo.h 
#define FOO "foo"

$ cat bar.c 
#include <foo.h>
#include <stdio.h>

int
main(int argc, char **argv)
{
  printf(FOO);
}

$ cat foo-1.0-GCCcore-13.3.0.eb 
easyblock = 'MakeCp'

name = 'foo'
version = '1.0'

homepage = 'foo'
description = 'foo'

toolchain = {'name': 'GCC', 'version': '13.3.0'}

sources = [{
'filename': 'foo.h', 'extract_cmd': 'cp %s %(builddir)s'}
]

skipsteps = ['build']
files_to_copy = [(['foo.h'], 'include/foo')]

sanity_check_paths = {
    'files': ['include/foo/foo.h'],
    'dirs': [],
}

modextrapaths = {MODULE_LOAD_ENV_HEADERS: 'include/foo'}

$ cat bar-1.0-GCCcore-13.3.0.eb 
easyblock = 'CmdCp'

name = 'bar'
version = '1.0'

homepage = 'bar'
description = 'bar'

toolchain = {'name': 'GCC', 'version': '13.3.0'}

sources = [
    {'filename': 'bar.c', 'extract_cmd': 'cp %s %(builddir)s'}
]

dependencies = [('foo', '1.0')]

cmds_map = [('.*', "$CC $CFLAGS -o bar bar.c")]

files_to_copy = [(['bar'], 'bin')]

sanity_check_paths = {
    'files': ['bin/bar'],
    'dirs': [],
}

With the default EB configuration, this works fine, because the foo module will get:

prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("CPATH", pathJoin(root, "include", "foo"))

And, even though the bar build then (incorrectly) does:

CPPFLAGS set to -I/data/apps/software/foo/1.0-GCC-13.3.0/include

it still works, because CPATH does still contain the right paths.

But by using eb --module-search-path-headers=include_paths --search-path-cpp-headers=include_paths, the foo module will contain:

prepend_path("CPLUS_INCLUDE_PATH", pathJoin(root, "include"))
prepend_path("CPLUS_INCLUDE_PATH", pathJoin(root, "include", "foo"))
prepend_path("C_INCLUDE_PATH", pathJoin(root, "include"))
prepend_path("C_INCLUDE_PATH", pathJoin(root, "include", "foo"))

and the bar build will override these to incorrect values, leading to:

bar.c:1:10: fatal error: foo.h: No such file or directory
    1 | #include <foo.h>
      |          ^~~~~~~
compilation terminated.

[bedroge]

Note that the defaults for both options are:

--module-search-path-headers -> cpath
--search-path-cpp-headers -> flags

So, the same issue also occurs when using cpath for the latter, as it will then again overwrite the correct $CPATH coming from the module by an incomplete one:

eb --module-search-path-headers=cpath --search-path-cpp-headers=cpath ./foo-1.0-GCCcore-13.3.0.eb
eb --module-search-path-headers=cpath --search-path-cpp-headers=cpath ./bar-1.0-GCCcore-13.3.0.eb

bar.c:1:10: fatal error: foo.h: No such file or directory
    1 | #include <foo.h>
      |          ^~~~~~~
compilation terminated.

[bedroge]

I was thinking of a solution where we would basically let the toolchain append to existing environment variables (or take existing environment variables and add their values to self.variables): that would probably solve the issue in many (all?) cases, but with for instance --module-search-path-headers=cpath --search-path-cpp-headers=flags it would still not know how to populate $CPPFLAGS correctly, because the module uses CPATH. Things will probably still work then, because $CPATH is also set, but it still feels wrong to have an incomplete value for $CPPFLAGS.

(edit: maybe we could use the environment variable that corresponds to the method used for the module search path to populate the toolchain's search path option, i.e. use CPATH to populate CPPFLAGS somehow?)

Solving it properly seems tricky though, as there seems to be something inherently wrong in how this environment is set up. To do it properly, the toolchain would have to know which extra subdirs have to be added in _add_dependency_cpp_headers , but how do we obtain that information there?