Code Coverage report in Bitbucket with JaCoCo

[white hornet]

I want to configure a bitbucket pipeline for Ballerina code PRs which runs bal test --test-report --code-coverage command and if the code coverage is less than a configured number, the pipeline needs to fail. How can I achieve this? Also is it possible to show the JaCoCo style report(generated with --coverage-format=xml) with the Bitbucket pipeline?

[fervent sable]

Hi @white hornet

You can use an external tool like CodeCov to publish the Code Coverage reports. (I guess you have to have a paid plan for private repositories, for public repositories, they provide the service for free).
You can use a CodeCov.yaml file to configure the code coverage threshold for a given repository. Then the pull request checks will be added to the repository and CodeCov will notify if the coverage drops from the threshold.

When you configure CodeCov, it will check for all the generated coverage xml files and generated the report for the repository.

[gaunt willow]

We generate a JaCoCo style report when executing bal test --test-report --code-coverage --coverage-format=xml. Like Thisaru mentioned, it's better to use a tool like Codecov which will handle the requirement for you.

[white hornet]

However still my requirement of failing the pipeline if the required coverage percentage is not met is not achieved that. I believe internally we are using the java jacoco plugin. If that is the case, then I guess it is just adding the support to pass a parameter read from bal test command so that it will be applied to fail Jacoco build(minimum coverage not met).

ex: bal test --test-report --code-coverage --min-coverage=20.0

However I have achieved my use case with a workaround:

pipelines:
  pull-requests:
    "**":
      - step:
          name: Build and Test My API
          image: ballerina/ballerina:2201.10.4
          script:
            - cd my-data-api
            - bal build
            - bal test --test-report --code-coverage --coverage-format=xml
            - cd ../
          artifacts:
            - my-data-api/target/report/**
      - step:
          name: "Validate Coverage"
          image: alpine:latest
          user: root
          script:
            - apk update && apk add jq bc
            - MINIMUM_COVERAGE=50.0
            - COVERAGE=$(jq '.coveragePercentage' my-data-api/target/report/test_results.json)
            - |
                echo "Coverage Percentage: $COVERAGE"
                if (( $(echo "$COVERAGE < $MINIMUM_COVERAGE" | bc -l) )); then
                  echo "Coverage ($COVERAGE%) is below the threshold of $MINIMUM_COVERAGE%. Failing pipeline."
                  exit 1
                else
                  echo "Coverage ($COVERAGE%) meets the threshold."
                fi

However there was an issue faced when using jq as the generated test_results.json file was corrupted due to this issue: https://github.com/ballerina-platform/ballerina-lang/pull/44012

So to fix this temporary in our side, updated the pipeline as below to read the first occurrence of coveragePercentage

pipelines:
  pull-requests:
    "**":
      - step:
          name: Build and Test My Data API
          image: ballerina/ballerina:2201.10.4
          script:
            - cd my-data-api
            - bal build
            - bal test --test-report --code-coverage --coverage-format=xml
            - cd ../
          artifacts:
            - my-data-api/target/report/**
      - step:
          name: "Validate Coverage"
          image: alpine:latest
          user: root
          script:
            - apk update && apk add sed
            - MINIMUM_COVERAGE=35.0
            - COVERAGE=$(grep -o '"coveragePercentage":[[:space:]]*[0-9]\+\.[0-9]\+' my-data-api/target/report/test_results.json | head -n 1 | sed -E 's/.*:[[:space:]]*//')
            - echo "Coverage Percentage:"$COVERAGE
            - |
              if ! awk "BEGIN { exit (($COVERAGE < $MINIMUM_COVERAGE) ? 1 : 0) }"; then
                echo "Coverage ($COVERAGE%) is below the required threshold ($MINIMUM_COVERAGE%). Failing pipeline."
                exit 1
              else
                echo "Coverage ($COVERAGE%) meets the required threshold."
              fi