diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 8be70ac46adea..ce18075f83b33 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5380,11 +5380,6 @@ menu: parent: code_coverage identifier: code_coverage_data_collected weight: 2 - - name: Upload Reports - url: code_coverage/upload/ - parent: code_coverage - identifier: code_coverage_upload - weight: 3 - name: PR Gates url: pr_gates/ pre: ci diff --git a/content/en/code_coverage/setup.md b/content/en/code_coverage/setup.md index 47c01cb7bf6b3..ac66348c0602a 100644 --- a/content/en/code_coverage/setup.md +++ b/content/en/code_coverage/setup.md @@ -92,6 +92,346 @@ If you wish to gate on PR coverage, configure PR Gates rules in Datadog. Navigate to [PR Gates rule creation][5] and configure a rule to gate on total or patch coverage. +## Upload code coverage reports + +Update your CI pipeline to upload code coverage report files to Datadog. This involves installing and running the `datadog-ci` CLI in your CI environment. + +See [Data Collected][14] for details on what data is collected during code coverage report upload. + +### Supported coverage report formats + +Datadog supports the following coverage data formats—expand for examples: + +{{% collapse-content title="LCOV" level="h4" expanded=false id="lcov" %}} +{{< code-block lang="text" >}} +TN: +SF:src/example.c +FN:3,add +FNDA:5,add +FNF:1 +FNH:1 +DA:3,5 +DA:4,5 +DA:5,5 +DA:8,0 +DA:9,0 +LF:5 +LH:3 +BRDA:4,0,0,5 +BRDA:4,0,1,0 +BRF:2 +BRH:1 +end_of_record +{{< /code-block >}} +{{% /collapse-content %}} + +{{% collapse-content title="Cobertura XML" level="h4" expanded=false id="cobertura-xml" %}} +{{< code-block lang="xml" >}} + + + + + src + + + + + + + + + + + + + + + + + + + + + + + + + + +{{< /code-block >}} +{{% /collapse-content %}} + +{{% collapse-content title="Jacoco XML" level="h4" expanded=false id="jacoco-xml" %}} +{{< code-block lang="xml" >}} + + + + + + + + + + + + + +{{< /code-block >}} +{{% /collapse-content %}} + +{{% collapse-content title="Clover XML" level="h4" expanded=false id="clover-xml" %}} +{{< code-block lang="xml" >}} + + + + + + + + + + + + + + + + + + +{{< /code-block >}} +{{% /collapse-content %}} + +{{% collapse-content title="OpenCover XML" level="h4" expanded=false id="opencover-xml" %}} +{{< code-block lang="xml" >}} + + + + + Example.dll + + + + + + + + + + + + + + + + + + + + + + + + + +{{< /code-block >}} +{{% /collapse-content %}} + +{{% collapse-content title="Simplecov JSON" level="h4" expanded=false id="simplecov-json" %}} +{{< code-block lang="json" >}} +{ + "meta": { + "simplecov_version": "0.21.2" + }, + "coverage": { + "/path/to/file1.rb": { + "lines": [ + null, + 1, + 2, + 0, + null, + 1, + null, + null, + null, + "ignored", + "ignored", + "ignored", + null + ], + "branches": [] + }, + "/path/to/file2.rb": { + "lines": [1, 1, null, 0, 1], + "branches": [] + } + } +} +{{< /code-block >}} +{{% /collapse-content %}} + +### Install the datadog-ci CLI + +Install the [`datadog-ci`][7] CLI globally using `npm`: + +{{< code-block lang="shell" >}} +npm install -g @datadog/datadog-ci +{{< /code-block >}} + +#### Standalone binary + +If installing Node.js in the CI is an issue, standalone binaries are provided with [Datadog CI releases][8]. Only _linux-x64_, _linux-arm64_, _darwin-x64_, _darwin-arm64_ (MacOS) and _win-x64_ (Windows) are supported. To install, run the following from your terminal: + +{{< tabs >}} +{{% tab "Linux" %}} +{{< code-block lang="shell" >}} +curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci" && chmod +x /usr/local/bin/datadog-ci +{{< /code-block >}} + +Then run any command with `datadog-ci`: +{{< code-block lang="shell" >}} +datadog-ci version +{{< /code-block >}} +{{% /tab %}} + +{{% tab "MacOS" %}} +{{< code-block lang="shell" >}} +curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_darwin-x64" --output "/usr/local/bin/datadog-ci" && chmod +x /usr/local/bin/datadog-ci +{{< /code-block >}} + +Then run any command with `datadog-ci`: +{{< code-block lang="shell" >}} +datadog-ci version +{{< /code-block >}} +{{% /tab %}} + +{{% tab "Windows" %}} +{{< code-block lang="powershell" >}} +Invoke-WebRequest -Uri "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_win-x64" -OutFile "datadog-ci.exe" +{{< /code-block >}} + +Then run any command with `Start-Process -FilePath "datadog-ci.exe"`: +{{< code-block lang="powershell" >}} +Start-Process -FilePath "./datadog-ci.exe" -ArgumentList version +{{< /code-block >}} +{{% /tab %}} +{{< /tabs >}} + +#### Docker image + +Alternatively, you can update your CI job to run in a container based on the [Datadog CI Docker image][13]. +The image comes with `datadog-ci` preinstalled and ready to use. + +### Uploading coverage reports + +
+Datadog automatically aggregates all reports for the same commit on the backend. You don't need to merge coverage reports before uploading them. +
+ +To upload your code coverage reports to Datadog, run the following command. Provide a valid [Datadog API key][9] (`DD_API_KEY`), and one or more file paths to either the coverage report files directly or directories containing them: + +{{< tabs >}} +{{% tab "GitHub Actions" %}} +
+
+steps:
+- name: Upload coverage reports to Datadog
+  run: datadog-ci coverage upload .
+  env:
+    DD_API_KEY: ${{ secrets.DD_API_KEY }}
+    DD_SITE: {{< region-param key="dd_site" >}}
+
+
+{{% /tab %}} +{{% tab "Gitlab" %}} +
+
+test:
+  stage: test
+  script:
+    - ... # run your tests and generate coverage reports
+    - datadog-ci coverage upload . # make sure to add the DD_API_KEY CI/CD variable
+
+
+{{% /tab %}} +{{< /tabs >}} + +The command recursively searches the specified directories for supported coverage report files, so specifying the current directory (`.`) is usually sufficient. +See the [`datadog-ci` documentation][10] for more details on the `datadog-ci coverage upload` command. + +Shortly after the code coverage report upload is finished, Datadog adds a PR comment with code coverage percentage values. +You can also view your coverage data aggregated by pull request in the [Code Coverage page][11] in Datadog, with the ability to examine individual files and lines of code. + +{{< img src="/code_coverage/pr_details.png" text="Code Coverage PR details page in Datadog" style="width:100%" >}} + +## Troubleshooting + +### Coverage upload command does not detect coverage report files + +The `datadog-ci coverage upload` command automatically detects supported coverage report files in the specified directories using heuristics, such as file names and extensions. +If your coverage report files do not match expected patterns, the command might not detect them automatically. In this case, specify the report format and provide the file paths as positional arguments. For example: + +{{< code-block lang="shell" >}} +datadog-ci coverage upload --format=lcov \ + src/coverage-reports/unit-tests/coverage.info \ + src/coverage-reports/e2e-tests/coverage.info +{{< /code-block >}} + +### Coverage upload fails with "Format could not be detected" error + +The `datadog-ci coverage upload` command automatically detects the format of the coverage report files based on their content and file extension. +If the command fails with the following error: +``` +Invalid coverage report file [...]: format could not be detected +``` +specify the format explicitly using the `--format` option, like this: + +{{< code-block lang="shell" >}} +datadog-ci coverage upload --format=cobertura reports/cobertura.xml +{{< /code-block >}} + +### Coverage upload outputs "Could not sync git metadata" error + +Git metadata upload is only required if you can't integrate your CI provider directly with Datadog. +If you are using a [source code provider integration][12], such as Datadog GitHub app or Gitlab integration, you can disable the git metadata upload by passing the `--skip-git-metadata-upload=1` flag to the `datadog-ci coverage upload` command, like this: + +{{< code-block lang="shell" >}} +datadog-ci coverage upload --skip-git-metadata-upload=1 . +{{< /code-block >}} + +### Datadog UI does not show changed files in the PR view + +By default, the "Changed files" table only contains executable source code files that are present in the uploaded coverage reports. +Select **Non-executable files** or **All** in the table header to display all files that were changed in the PR, regardless of whether they are executable or not. + +{{< img src="/code_coverage/non_executable_files.png" text="In Changed files, you have the option to select Non-executable on the table header" style="width:100%" >}} + +If a source code file is mistakenly marked as non-executable, it is probably missing from your uploaded coverage reports. +Make sure that you are uploading all of your relevant reports, and double-check your coverage tool configuration to verify that coverage data is collected for all applicable files. + +Test sources are not considered executable files as they are not part of the production codebase being measured for coverage. + +### Datadog UI shows incorrect file paths + +Code Coverage relies on the file paths in coverage reports to be either absolute or relative to the repository root. +If the paths in your report are relative to a different directory in your repository, specify the correct base path (relative to the repo root) with the `--base-path` option when running the `datadog-ci coverage upload` command, like this: + +{{< code-block lang="shell" >}} +datadog-ci coverage upload --base-path=frontend/src . +{{< /code-block >}} + +### Discrepancy between Datadog UI and coverage report values + +Datadog automatically merges coverage reports for the same commit. +As a result, the coverage percentage displayed in the Datadog UI may differ from the values in your individual coverage reports, especially if those reports contain overlapping or duplicate source code file entries. + +If you use an external tool (such as [ReportGenerator](https://reportgenerator.io/)) to merge coverage reports before uploading to Datadog, +ensure your merged reports do not contain duplicate source code file entries. +Datadog deduplicates overlapping files across reports, which can result in differences between your original coverage values and the merged values displayed in the Datadog UI. + + + ## Further reading {{< partial name="whats-next/whats-next.html" >}} @@ -102,3 +442,11 @@ Navigate to [PR Gates rule creation][5] and configure a rule to gate on total or [4]: https://app.datadoghq.com/organization-settings/roles [5]: https://app.datadoghq.com/ci/pr-gates/rule/create [6]: /code_coverage/upload/ +[7]: https://www.npmjs.com/package/@datadog/datadog-ci +[8]: https://github.com/DataDog/datadog-ci/releases +[9]: https://app.datadoghq.com/organization-settings/api-keys +[10]: https://github.com/DataDog/datadog-ci/blob/master/packages/datadog-ci/src/commands/coverage/README.md +[11]: https://app.datadoghq.com/ci/code-coverage +[12]: /code_coverage/setup/#integrate-with-source-code-provider +[13]: https://hub.docker.com/r/datadog/ci +[14]: /code_coverage/data_collected/#code-coverage-report-upload diff --git a/content/en/code_coverage/upload.md b/content/en/code_coverage/upload.md deleted file mode 100644 index ac2c74594b4d4..0000000000000 --- a/content/en/code_coverage/upload.md +++ /dev/null @@ -1,368 +0,0 @@ ---- -title: Upload Code Coverage Reports -description: "Learn how to upload code coverage reports to Datadog using the datadog-ci CLI." -further_reading: - - link: "/code_coverage" - tag: "Documentation" - text: "Code Coverage" - - link: "/code_coverage/setup" - tag: "Documentation" - text: "Set Up Code Coverage" - - link: "/code_coverage/data_collected" - tag: "Documentation" - text: "Learn what data is collected for Code Coverage" ---- - -{{< callout url="http://datadoghq.com/product-preview/code-coverage/" >}} -Code Coverage is in Preview. This product replaces Test Optimization's code coverage feature, which is being deprecated. Complete the form to request access for the new Code Coverage product. -{{< /callout >}} - -Update your CI pipeline to upload code coverage report files to Datadog. This involves installing and running the `datadog-ci` CLI in your CI environment. - -See [Data Collected][6] for details on what data is collected during code coverage report upload. - -## Supported coverage report formats - -Datadog supports the following coverage data formats—expand for examples: - -{{% collapse-content title="LCOV" level="h4" expanded=false id="lcov" %}} -{{< code-block lang="text" >}} -TN: -SF:src/example.c -FN:3,add -FNDA:5,add -FNF:1 -FNH:1 -DA:3,5 -DA:4,5 -DA:5,5 -DA:8,0 -DA:9,0 -LF:5 -LH:3 -BRDA:4,0,0,5 -BRDA:4,0,1,0 -BRF:2 -BRH:1 -end_of_record -{{< /code-block >}} -{{% /collapse-content %}} - -{{% collapse-content title="Cobertura XML" level="h4" expanded=false id="cobertura-xml" %}} -{{< code-block lang="xml" >}} - - - - - src - - - - - - - - - - - - - - - - - - - - - - - - - - -{{< /code-block >}} -{{% /collapse-content %}} - -{{% collapse-content title="Jacoco XML" level="h4" expanded=false id="jacoco-xml" %}} -{{< code-block lang="xml" >}} - - - - - - - - - - - - - -{{< /code-block >}} -{{% /collapse-content %}} - -{{% collapse-content title="Clover XML" level="h4" expanded=false id="clover-xml" %}} -{{< code-block lang="xml" >}} - - - - - - - - - - - - - - - - - - -{{< /code-block >}} -{{% /collapse-content %}} - -{{% collapse-content title="OpenCover XML" level="h4" expanded=false id="opencover-xml" %}} -{{< code-block lang="xml" >}} - - - - - Example.dll - - - - - - - - - - - - - - - - - - - - - - - - - -{{< /code-block >}} -{{% /collapse-content %}} - -{{% collapse-content title="Simplecov JSON" level="h4" expanded=false id="simplecov-json" %}} -{{< code-block lang="json" >}} -{ - "meta": { - "simplecov_version": "0.21.2" - }, - "coverage": { - "/path/to/file1.rb": { - "lines": [ - null, - 1, - 2, - 0, - null, - 1, - null, - null, - null, - "ignored", - "ignored", - "ignored", - null - ], - "branches": [] - }, - "/path/to/file2.rb": { - "lines": [1, 1, null, 0, 1], - "branches": [] - } - } -} -{{< /code-block >}} -{{% /collapse-content %}} - -## Install the datadog-ci CLI - -Install the [`datadog-ci`][7] CLI globally using `npm`: - -{{< code-block lang="shell" >}} -npm install -g @datadog/datadog-ci -{{< /code-block >}} - -### Standalone binary - -If installing Node.js in the CI is an issue, standalone binaries are provided with [Datadog CI releases][8]. Only _linux-x64_, _linux-arm64_, _darwin-x64_, _darwin-arm64_ (MacOS) and _win-x64_ (Windows) are supported. To install, run the following from your terminal: - -{{< tabs >}} -{{% tab "Linux" %}} -{{< code-block lang="shell" >}} -curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci" && chmod +x /usr/local/bin/datadog-ci -{{< /code-block >}} - -Then run any command with `datadog-ci`: -{{< code-block lang="shell" >}} -datadog-ci version -{{< /code-block >}} -{{% /tab %}} - -{{% tab "MacOS" %}} -{{< code-block lang="shell" >}} -curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_darwin-x64" --output "/usr/local/bin/datadog-ci" && chmod +x /usr/local/bin/datadog-ci -{{< /code-block >}} - -Then run any command with `datadog-ci`: -{{< code-block lang="shell" >}} -datadog-ci version -{{< /code-block >}} -{{% /tab %}} - -{{% tab "Windows" %}} -{{< code-block lang="powershell" >}} -Invoke-WebRequest -Uri "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_win-x64" -OutFile "datadog-ci.exe" -{{< /code-block >}} - -Then run any command with `Start-Process -FilePath "datadog-ci.exe"`: -{{< code-block lang="powershell" >}} -Start-Process -FilePath "./datadog-ci.exe" -ArgumentList version -{{< /code-block >}} -{{% /tab %}} -{{< /tabs >}} - -### Docker image - -Alternatively, you can update your CI job to run in a container based on the [Datadog CI Docker image][13]. -The image comes with `datadog-ci` preinstalled and ready to use. - -## Uploading coverage reports - -
-Datadog automatically aggregates all reports for the same commit on the backend. You don't need to merge coverage reports before uploading them. -
- -To upload your code coverage reports to Datadog, run the following command. Provide a valid [Datadog API key][9] (`DD_API_KEY`), and one or more file paths to either the coverage report files directly or directories containing them: - -{{< tabs >}} -{{% tab "GitHub Actions" %}} -
-
-steps:
-- name: Upload coverage reports to Datadog
-  run: datadog-ci coverage upload .
-  env:
-    DD_API_KEY: ${{ secrets.DD_API_KEY }}
-    DD_SITE: {{< region-param key="dd_site" >}}
-
-
-{{% /tab %}} -{{% tab "Gitlab" %}} -
-
-test:
-  stage: test
-  script:
-    - ... # run your tests and generate coverage reports
-    - datadog-ci coverage upload . # make sure to add the DD_API_KEY CI/CD variable
-
-
-{{% /tab %}} -{{< /tabs >}} - -The command recursively searches the specified directories for supported coverage report files, so specifying the current directory (`.`) is usually sufficient. -See the [`datadog-ci` documentation][10] for more details on the `datadog-ci coverage upload` command. - -Shortly after the code coverage report upload is finished, Datadog adds a PR comment with code coverage percentage values. -You can also view your coverage data aggregated by pull request in the [Code Coverage page][11] in Datadog, with the ability to examine individual files and lines of code. - -{{< img src="/code_coverage/pr_details.png" text="Code Coverage PR details page in Datadog" style="width:100%" >}} - -## Troubleshooting - -### Coverage upload command does not detect coverage report files - -The `datadog-ci coverage upload` command automatically detects supported coverage report files in the specified directories using heuristics, such as file names and extensions. -If your coverage report files do not match expected patterns, the command might not detect them automatically. In this case, specify the report format and provide the file paths as positional arguments. For example: - -{{< code-block lang="shell" >}} -datadog-ci coverage upload --format=lcov \ - src/coverage-reports/unit-tests/coverage.info \ - src/coverage-reports/e2e-tests/coverage.info -{{< /code-block >}} - -### Coverage upload fails with "Format could not be detected" error - -The `datadog-ci coverage upload` command automatically detects the format of the coverage report files based on their content and file extension. -If the command fails with the following error: -``` -Invalid coverage report file [...]: format could not be detected -``` -specify the format explicitly using the `--format` option, like this: - -{{< code-block lang="shell" >}} -datadog-ci coverage upload --format=cobertura reports/cobertura.xml -{{< /code-block >}} - -### Coverage upload outputs "Could not sync git metadata" error - -Git metadata upload is only required if you can't integrate your CI provider directly with Datadog. -If you are using a [source code provider integration][12], such as Datadog GitHub app or Gitlab integration, you can disable the git metadata upload by passing the `--skip-git-metadata-upload=1` flag to the `datadog-ci coverage upload` command, like this: - -{{< code-block lang="shell" >}} -datadog-ci coverage upload --skip-git-metadata-upload=1 . -{{< /code-block >}} - -### Datadog UI does not show changed files in the PR view - -By default, the "Changed files" table only contains executable source code files that are present in the uploaded coverage reports. -Select **Non-executable files** or **All** in the table header to display all files that were changed in the PR, regardless of whether they are executable or not. - -{{< img src="/code_coverage/non_executable_files.png" text="In Changed files, you have the option to select Non-executable on the table header" style="width:100%" >}} - -If a source code file is mistakenly marked as non-executable, it is probably missing from your uploaded coverage reports. -Make sure that you are uploading all of your relevant reports, and double-check your coverage tool configuration to verify that coverage data is collected for all applicable files. - -Test sources are not considered executable files as they are not part of the production codebase being measured for coverage. - -### Datadog UI shows incorrect file paths - -Code Coverage relies on the file paths in coverage reports to be either absolute or relative to the repository root. -If the paths in your report are relative to a different directory in your repository, specify the correct base path (relative to the repo root) with the `--base-path` option when running the `datadog-ci coverage upload` command, like this: - -{{< code-block lang="shell" >}} -datadog-ci coverage upload --base-path=frontend/src . -{{< /code-block >}} - -### Discrepancy between Datadog UI and coverage report values - -Datadog automatically merges coverage reports for the same commit. -As a result, the coverage percentage displayed in the Datadog UI may differ from the values in your individual coverage reports, especially if those reports contain overlapping or duplicate source code file entries. - -If you use an external tool (such as [ReportGenerator](https://reportgenerator.io/)) to merge coverage reports before uploading to Datadog, -ensure your merged reports do not contain duplicate source code file entries. -Datadog deduplicates overlapping files across reports, which can result in differences between your original coverage values and the merged values displayed in the Datadog UI. - -## Further reading - -{{< partial name="whats-next/whats-next.html" >}} - -[6]: /code_coverage/data_collected/#code-coverage-report-upload -[7]: https://www.npmjs.com/package/@datadog/datadog-ci -[8]: https://github.com/DataDog/datadog-ci/releases -[9]: https://app.datadoghq.com/organization-settings/api-keys -[10]: https://github.com/DataDog/datadog-ci/blob/master/packages/datadog-ci/src/commands/coverage/README.md -[11]: https://app.datadoghq.com/ci/code-coverage -[12]: /code_coverage/setup/#integrate-with-source-code-provider -[13]: https://hub.docker.com/r/datadog/ci -