Summary

post-pr-comment.sh gates the pipeline on the dd-trace integration matrix, but a configuration that produced no result counted as neither a pass nor a failure — so a broken cell could leave a PR green.

Found while investigating why PR #790 was red. There the flaky musl-x64-openj9-jdk21 cell did write a VALIDATION_FAILED log, so it gated correctly. Had the same cell hit its 10-minute timeout instead, the PR would have reported success.

The hole

The collection loop classifies each configuration, and unknown increments neither counter:

  if [ "${s1_status}" = "pass" ] && [ "${s2_status}" = "pass" ]; then
    RESULTS["${config_name}"]="pass";  TOTAL_PASS=$((TOTAL_PASS + 1))
  elif [ "${s1_status}" = "fail" ] || [ "${s2_status}" = "fail" ]; then
    RESULTS["${config_name}"]="fail";  TOTAL_FAIL=$((TOTAL_FAIL + 1))
  else
    RESULTS["${config_name}"]="unknown"        # ← counted nowhere
  fi
done

TOTAL=$((TOTAL_PASS + TOTAL_FAIL))             # ← excludes it entirely

Two shapes reach that branch, both from a matrix job dying before it writes its validation logs — the timeout firing mid-run, a lost runner, or a failure during prerequisite setup (jbang / JDK download):

• artifact directory present but unreadable → classified unknown
• artifact directory absent → the loop never sees the configuration at all

With 39 passes and one such cell, TOTAL_FAIL stays 0, the overall status is success, and the only trace is a :grey_question: in the matrix that nothing requires anyone to read. The existing TOTAL == 0 guard catches only the case where every cell dies; partial is the likely case.

Change

• The expected matrix (8 platforms × 5 JDKs) is declared once, before collection, and drives both the completeness check and the table rendering, so the two cannot drift. Previously the platform list was inlined in the rendering branch only.
• Collection now walks that expected matrix rather than only what turned up on disk, so a missing directory is caught alongside an empty one.
• Any expected configuration that produced neither a pass nor a fail is counted and gates.
• The comment names them under a ### No result produced heading instead of leaving a grey cell to be interpreted.
• The exit log distinguishes the two conditions, since they now differ.

Verified

Fixtures of 39 passing configurations plus one in each state, run against this script and against the one on main:

Note for anyone testing locally: the script uses declare -A, so macOS's bash 3.2 cannot run it and fails with exit 2 before reaching any of this logic. Use bash 4+.

Deliberately not included

Quarantining a known-flaky configuration remains the intended way to keep CI green — see #777, which introduces that policy for the GitHub Actions JUnit suite. It does not cover this path: these integration tests emit no JUnit XML (they are shell plus JFR conformance validation, reported by grepping validation logs), so a quarantine here needs the same policy — a list keyed by configuration name with a review_by expiry — against a different signal.

This change only ensures quarantine becomes the only way a red or missing cell leaves CI green.

🤖 Generated with Claude Code