iso: Enable BTF debug symbols to support eBPF tooling #21800

blu3r4y · 2025-10-25T18:19:25Z

This PR sets kernel options CONFIG_DEBUG_INFO_BTF so that BTF type information is generated, which is necessary for eBPF-based tooling such as Tetragon to work. Additionally, for x86_64 only, it adds the missing, but recommended option CONFIG_BPF_KPROBE_OVERRIDE to also support policy enforcement (https://tetragon.io/docs/installation/faq/#what-are-the-linux-kernel-configuration-options-needed-to-run-tetragon).

I already built the image locally and started a cluster like so:

minikube start --driver=hyperv --iso-url=file://C:/path/to/your/minikube-amd64.iso

I temporarily uploaded the ISO files here; I will delete them again in a few weeks:

Tetragon can now successfully discover BTF type information.

level=info msg="BTF discovery: default kernel btf file found" btf-file=/sys/kernel/btf/vmlinux
level=error msg="unable to read /sys/kernel/security directory" error="open /sys/kernel/security: no such file or directory"
level=info msg="BPF detected features: override_return: true, buildid: true, kprobe_multi: false, uprobe_multi true, fmodret: true, fmodret_syscall: true, signal: true, large: true, link_pin: true, lsm: false, missed_stats_kprobe_multi: false, missed_stats_kprobe: true, batch_update: true"

Fixes #21788. Similar intention as closed PR #21622.

I am not sure what other steps are necessary to bump the ISO for minikube.

linux-foundation-easycla · 2025-10-25T18:19:33Z

The committers listed above are authorized under a signed CLA.

✅ login: blu3r4y / name: Mario Kahlhofer (7387cd3, 987f175)
✅ login: minikube-bot / name: minikube-bot (bb40a8e)

k8s-ci-robot · 2025-10-25T18:19:34Z

Welcome @blu3r4y!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-10-25T18:19:35Z

Hi @blu3r4y. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

minikube-bot · 2025-10-25T18:21:03Z

Can one of the admins verify this patch?

nirs · 2025-10-25T19:09:37Z

@blu3r4y thanks! do we have a reason to split this to 2 commits? The commits message do not explain what each commit does and do not provide any justification for the changes.

If there is no reason to split the commits, please squash them. The commit message should explain why every config is needed, since we cannot keep comments in the config files. (they are removed by menuconfig).

It seems that the pr message describes all the details, but this info should be available in git log.

When the change is ready we will build the iso using the iso build job, which will build an iso and add a commit updating minikube to consume this iso.

nirs · 2025-10-25T19:13:21Z

/cc @nirs

nirs · 2025-10-25T19:13:41Z

/cc @medyagh

blu3r4y · 2025-10-25T19:38:47Z

@nirs Squashed the two commits into one and added the full description to the commit message.

nirs · 2025-10-25T19:54:26Z

@blu3r4y thank!

I tested that running linux-menuconfig-* targets does not change the configs and the changes looks good to me.

Can you share instructions on how to test this functionality? Ideally we can add a simple integration tests so future kernel upgrade will not break this functionality again.

/lgtm

nirs · 2025-10-25T19:54:45Z

/ok-to-build-iso

blu3r4y · 2025-10-25T20:34:54Z

@nirs I think a simple check if file /sys/kernel/btf/vmlinux is present in the image would suffice. I quickly glanced over the existing integration tests and it seems reasonable to add a check to test/integration/guest_env_test.go, which is already checking that certain binaries and mounts are available. Something like this could work:

t.Run("Files", func(t *testing.T) {
	// /sys/kernel/btf/vmlinux -- Check if BTF types are present for eBPF tooling
	for _, file := range []string{"/sys/kernel/btf/vmlinux"} {
		file := file
		t.Run(file, func(t *testing.T) {
			t.Parallel()
			rr, err := Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", fmt.Sprintf("test -f %s && echo 'OK' || echo 'NOT FOUND'", file)))
			if err != nil {
				t.Errorf("failed to verify existence of %q file : args %q: %v", file, rr.Command(), err)
			}

			if !strings.Contains(rr.Stdout.String(), "OK") {
				t.Errorf("expected file %q to exist, but it does not", file)
			}
		})
	}
})

I can add another commit to this PR adding a test.

Do you prefer having a test in guest_env_test.go, or do you rather prefer having a standalone test?

nirs · 2025-10-25T21:00:04Z

@nirs I think a simple check if file /sys/kernel/btf/vmlinux is present in the image would suffice. I quickly glanced over the existing integration tests and it seems reasonable to add a check to test/integration/guest_env_test.go, which is already checking that certain binaries and mounts are available. Something like this could work:
t.Run("Files", func(t *testing.T) {
	// /sys/kernel/btf/vmlinux -- Check if BTF types are present for eBPF tooling
	for _, file := range []string{"/sys/kernel/btf/vmlinux"} {
		file := file
		t.Run(file, func(t *testing.T) {
			t.Parallel()
			rr, err := Run(t, exec.CommandContext(ctx, Target(), "-p", profile, "ssh", fmt.Sprintf("test -f %s && echo 'OK' || echo 'NOT FOUND'", file)))
			if err != nil {
				t.Errorf("failed to verify existence of %q file : args %q: %v", file, rr.Command(), err)
			}

			if !strings.Contains(rr.Stdout.String(), "OK") {
				t.Errorf("expected file %q to exist, but it does not", file)
			}
		})
	}
})
I can add another commit to this PR adding a test.

Do you prefer having a test in guest_env_test.go, or do you rather prefer having a standalone test?

guest_env_test.go looks perfect for this test.

A generic Files test that we can extend later looks good, but it will help to add a description for every file. Maybe add a more specific BTF test instead? This will make it easy to document the test and add more checks later if we want to test more details.

A link to the issue in the test will help to understand why the test is needed.

The test used `go:build iso` so it was not included in the integration tests. Change to `go:build integration` so we test in the CI. Rename the file and the test name to make it more clear that this test is about the iso image. Skip the test for non vm-driver, since this tests is about the iso image. We may need to add a similar test or adapt this test so it can be used also with the kicbase image. This test will be useful to validate kubernetes#21800, avoiding regressions such as kubernetes#21788.

nirs · 2025-10-27T21:29:26Z

@blu3r4y #21801 was merged today. Can you rebase on master and resolve the conflict?

This change sets kernel option CONFIG_DEBUG_INFO_BTF so that BTF type information is generated, which is necessary for eBPF-based tooling such as Tetragon to work. Additionally, for x86_64 only, it adds the recommended option CONFIG_BPF_KPROBE_OVERRIDE to also support policy enforcement. Setting CONFIG_DEBUG_INFO_BTF also requires setting CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT. Setting CONFIG_BPF_KPROBE_OVERRIDE also requires setting CONFIG_FUNCTION_ERROR_INJECTION.

blu3r4y · 2025-10-28T09:58:00Z

@nirs Rebased. If I am not mistaken, I believe the ISO needs to be built again by the bot as well.

nirs · 2025-10-28T10:11:08Z

@nirs Rebased. If I am not mistaken, I believe the ISO needs to be built again by the bot as well.

If you rebased on master and there was no conflict in this commit:
6104756

It means that the ISO did not update and the one we built in this PR is still valid. If there was a conflict, it means that the ISO was updated by another PR, and we need to rebuild to build with the latest source.

If we need to build again, delete the ISO commit and we can start a new build.

nirs · 2025-10-28T11:20:43Z

/ok-to-build-iso

blu3r4y · 2025-10-28T11:21:17Z

Removed the ISO commit so that it can be rebuilt with the latest source after rebasing (there was a conflict, yes).

minikube-bot · 2025-10-28T17:10:25Z

Hi @blu3r4y, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

medyagh · 2025-10-28T17:40:26Z

/ok-to-test

test/integration/iso_test.go

BTF types are required for CO-RE eBPF programs. Check if the kernel was compiled with CONFIG_DEBUG_INFO_BTF and created the /sys/kernel/btf/vmlinux file.

k8s-ci-robot · 2025-10-29T08:02:41Z

@blu3r4y: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
integration-kvm-crio-linux-x86-64	`bb40a8e`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-kvm-containerd-linux-x86-64	`bb40a8e`	link	true	`/test integration-kvm-containerd-linux-x86-64`
integration-docker-containerd-linux-x86-64	`bb40a8e`	link	true	`/test integration-docker-containerd-linux-x86-64`
integration-kvm-docker-linux-x86-64	`bb40a8e`	link	true	`/test integration-kvm-docker-linux-x86-64`
integration-docker-docker-linux-x86-64	`bb40a8e`	link	true	`/test integration-docker-docker-linux-x86-64`
integration-none-docker-linux-x86-64	`bb40a8e`	link	true	`/test integration-none-docker-linux-x86-64`
integration-docker-crio-linux-x86-64	`bb40a8e`	link	true	`/test integration-docker-crio-linux-x86-64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

minikube-pr-bot · 2025-10-29T08:48:08Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21800 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 44.3s    │ 43.4s                  │
│ enable ingress │ 17.6s    │ 17.6s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 45.9s 45.0s 43.3s 42.7s 44.8s
Times for minikube (PR 21800) start: 43.1s 41.7s 42.6s 44.9s 44.6s

Times for minikube ingress: 19.8s 16.9s 19.8s 15.8s 15.8s
Times for minikube (PR 21800) ingress: 15.3s 17.4s 16.9s 16.9s 21.4s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21800 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 23.9s    │ 24.2s                  │
│ enable ingress │ 12.5s    │ 12.3s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.3s 22.4s 23.4s 28.0s 24.2s
Times for minikube (PR 21800) start: 24.2s 24.3s 22.1s 24.8s 25.7s

Times for minikube ingress: 12.7s 12.7s 12.7s 13.7s 10.7s
Times for minikube (PR 21800) ingress: 10.7s 13.8s 13.7s 12.7s 10.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21800 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.0s    │ 20.5s                  │
│ enable ingress │ 21.4s    │ 21.4s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 21.6s 22.8s 20.8s 21.6s 23.3s
Times for minikube (PR 21800) start: 20.1s 19.0s 22.7s 20.3s 20.3s

Times for minikube ingress: 20.2s 23.2s 23.2s 20.2s 20.2s
Times for minikube (PR 21800) ingress: 21.2s 22.2s 20.2s 21.2s 22.2s

minikube-pr-bot · 2025-10-29T09:47:19Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate

Besides the following environments also have failed tests:

Docker_Linux_crio: 37 failed (gopogh)
Docker_Linux_crio_arm64: 41 failed (gopogh)
KVM_Linux_crio: 3 failed (gopogh)

To see the flake rates of all tests by environment, click here.

blu3r4y · 2025-10-29T14:06:47Z

I built the Windows binary locally again with the new ISO and can confirm that Minikube and Tetragon are still working for me.

medyagh · 2025-10-29T18:07:18Z

thank you @blu3r4y for fixing this regression

medyagh · 2025-10-29T18:07:22Z

/lgtm

k8s-ci-robot · 2025-10-29T18:07:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: blu3r4y, medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

blu3r4y · 2025-10-29T18:10:01Z

sure thing, thanks @nirs for the amazing guidance and support 👏

k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Oct 25, 2025

k8s-ci-robot requested review from ComradeProgrammer and prezha October 25, 2025 18:19

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 25, 2025

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 25, 2025

blu3r4y changed the title ~~iso: Enable BTF debug symbols to support eBPF tooling (fixes #21788)~~ iso: Enable BTF debug symbols to support eBPF tooling Oct 25, 2025

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Oct 25, 2025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 25, 2025

blu3r4y mentioned this pull request Oct 25, 2025

BTF type information is missing since recent kernel update 6.6.95, making eBPF tools such as Tetragon crash on start-up #21788

Closed

k8s-ci-robot requested a review from nirs October 25, 2025 19:13

k8s-ci-robot requested a review from medyagh October 25, 2025 19:13

blu3r4y force-pushed the iso-linux-6.6-btf branch 2 times, most recently from 0fb0436 to d7d91bd Compare October 25, 2025 19:37

k8s-ci-robot assigned nirs Oct 25, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 25, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 27, 2025

blu3r4y force-pushed the iso-linux-6.6-btf branch from 0c0c90c to 68d34e6 Compare October 28, 2025 09:56

k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 28, 2025

blu3r4y force-pushed the iso-linux-6.6-btf branch from 68d34e6 to 3f251d9 Compare October 28, 2025 11:19

nirs reviewed Oct 28, 2025

View reviewed changes

test/integration/iso_test.go Show resolved Hide resolved

blu3r4y and others added 2 commits October 29, 2025 09:01

test: Ensure that BTF type information is available

7387cd3

BTF types are required for CO-RE eBPF programs. Check if the kernel was compiled with CONFIG_DEBUG_INFO_BTF and created the /sys/kernel/btf/vmlinux file.

Updating ISO to v1.37.0-1761658712-21800

bb40a8e

blu3r4y force-pushed the iso-linux-6.6-btf branch from 51d28e8 to bb40a8e Compare October 29, 2025 08:01

medyagh merged commit 37b032e into kubernetes:master Oct 29, 2025
30 of 48 checks passed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 29, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2025

blu3r4y deleted the iso-linux-6.6-btf branch October 30, 2025 09:58

iso: Enable BTF debug symbols to support eBPF tooling #21800

iso: Enable BTF debug symbols to support eBPF tooling #21800

Uh oh!

Conversation

blu3r4y commented Oct 25, 2025

Uh oh!

linux-foundation-easycla bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 25, 2025

Uh oh!

k8s-ci-robot commented Oct 25, 2025

Uh oh!

minikube-bot commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

blu3r4y commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

blu3r4y commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

nirs commented Oct 27, 2025

Uh oh!

blu3r4y commented Oct 28, 2025

Uh oh!

nirs commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirs commented Oct 28, 2025

Uh oh!

blu3r4y commented Oct 28, 2025

Uh oh!

minikube-bot commented Oct 28, 2025

Uh oh!

medyagh commented Oct 28, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 29, 2025

Uh oh!

minikube-pr-bot commented Oct 29, 2025

Uh oh!

minikube-pr-bot commented Oct 29, 2025

Uh oh!

blu3r4y commented Oct 29, 2025

Uh oh!

medyagh commented Oct 29, 2025

Uh oh!

medyagh commented Oct 29, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 29, 2025

Uh oh!

blu3r4y commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

linux-foundation-easycla bot commented Oct 25, 2025 •

edited

Loading

nirs commented Oct 28, 2025 •

edited

Loading