mirror of
https://github.com/mattermost/mattermost-mobile.git
synced 2026-11-09 20:50:12 +00:00
65b8a408ed
* ci(e2e): triage CMT failures per test and add run breakdown tool
CMT had no per-test judging: its Detox template summary jobs fail on
FAILURES > 0, so 17/642 iOS and 12/650 Android failures (plus one iOS
shard whose simulator hit NSPOSIXErrorDomain in simctl privacy) made
the latest CMT run red, as in all 9 CMT runs in the last 30 days.
Wire the existing e2e-triage composite into compatibility-matrix-
testing.yml (report-only unless E2E_TRIAGE_MODE=enforce, same gate as
PR/main), with a new triage-lanes input so CMT summarises its per-shard
contexts instead of the PR lanes.
Also add detox/utils/e2e-run-breakdown.js: groups a gh run's jobs by
framework (Detox/Maestro) and platform (iOS/iPad/Android) so failure
rates per lane no longer need TSIO or signed-in log access.
* ci(e2e): serialize CMT triage status writes and count timed-out jobs as failures
The toolkit's status publisher has no compare-and-swap, and its
status.ts requires every writer of the shared e2e-test/triage context
to hold the same non-cancelling, SHA-scoped lock. The CMT tsio-triage
matrix wrote that context in parallel with no lock. Add max-parallel: 1
so matrix entries run one at a time without discarding queued entries,
plus a job concurrency group keyed on the CMT commit SHA with
cancel-in-progress: false to serialize against triage writers from
other runs of the same commit.
Also count GitHub job conclusion timed_out as a failure in
e2e-run-breakdown; it previously fell through to other, so a timed-out
job showed as zero failures and was omitted from failingJobs.
* ci(e2e): judge CMT channel-message failures as flaky, infra, known, or suspected app bug
The CMT channel rollup only posted totals, so every red run needed a
human to open TSIO and decide whether the app broke. The rollup already
fetches each leg's TSIO consolidated payload for per-leg counts; it now
also extracts this commit's failing tests with their error text and
earlier history, and the channel message groups them: infra/environment
(server-disabled feature, simctl, network, hook timeout, setup crash, or
a sibling cascade in the same suite), flaky (passed on retry or mixed
pass/fail history), known failing (failed before this run), and
suspected app bug (first seen, no infra cause — labelled as needing a
human confirm). When TSIO has detail for fewer tests than the failing
total, the message says how many were judged.
* ci(e2e): require a valid job and suite key for CMT cascade judgements
A failure with no job and a title that reduces to an empty suite key
produced the cascade key '|', which could match an infra failure with
the same empty key and reclassify a suspected app bug as a cascade.
Build the cascade key only when both the job and the suite are
non-empty, exclude invalid keys from the infra set, and require a
valid key before reclassifying.
* ci(e2e): require corroboration before the CMT message says suspected app bug
Suspected app bug was the fallback for any first-seen failure with no
infra marker, so ordinary visibility/existence timeouts (MM-T3199,
MM-T3206, MM-70693_2 on the 833 run) were labelled app bugs on a single
observation. Now that label needs a direct content/state assertion —
never a wait expiring — plus corroboration: the same test failing on
another leg of this run, or earlier passes in TSIO history now failing
that assertion. Everything weaker lands in a new Needs a closer look
bucket that names why the evidence is insufficient.
* ci(e2e): tighten CMT judgement corroboration (legs, run attempts, matcher text)
Three review fixes in cmt-channel-notify:
- alsoFailedElsewhere counted an empty job name as a second leg, so one
real leg plus a missing job name could promote a failure to suspected
app bug. Only non-empty job names are tracked now.
- The consolidated endpoint ignores gh_run_attempt (verified live:
gh_run_attempt=99 returns full data, run_attempt=99 returns none), so
fetchConsolidated now sends run_attempt. failuresFromConsolidated also
excludes every attempt of this GitHub run from previousStatuses, so a
pass in attempt 1 cannot corroborate an app-bug call in attempt 2.
- Removed 'doesn't match the selected view' from the timing patterns so
a matcher mismatch that carries a direct assertion (toHaveText and
friends) can reach the hard-assertion gate instead of being dismissed
as a timing wait.
---------
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| artifacts.js | ||
| build-tsio-job-config.js | ||
| build-tsio-job-config.test.js | ||
| cli-args.js | ||
| cmt-channel-notify.js | ||
| cmt-channel-notify.test.js | ||
| constants.js | ||
| disable_ios_autofill.js | ||
| e2e-run-breakdown.js | ||
| e2e-run-breakdown.test.js | ||
| failed-jest-specs.js | ||
| failed-jest-specs.test.js | ||
| generate_detox_config_ci.js | ||
| jest_stare.js | ||
| maestro_report.js | ||
| maestro_report.test.js | ||
| maestro_test_cases.js | ||
| merge-jest-results-for-tsio.js | ||
| merge-jest-results-for-tsio.test.js | ||
| post_message_as.js | ||
| report.js | ||
| report.test.js | ||
| resolve-tsio-job-config.sh | ||
| retriable_cloud_error.js | ||
| retriable_cloud_error.test.js | ||
| spec-durations.js | ||
| test_cases.js | ||
| tsio-channel-notify-rollup.js | ||
| tsio-channel-notify-rollup.test.js | ||
| tsio-report-status.js | ||
| tsio-report-status.test.js | ||
| webhook_utils.js | ||
| write-tsio-failure-stub.mjs | ||
| xml_escape.test.js | ||