mirror of
https://github.com/mattermost/mattermost-mobile.git
synced 2026-11-05 10:28:54 +00:00
65b8a408ed
* ci(e2e): triage CMT failures per test and add run breakdown tool
CMT had no per-test judging: its Detox template summary jobs fail on
FAILURES > 0, so 17/642 iOS and 12/650 Android failures (plus one iOS
shard whose simulator hit NSPOSIXErrorDomain in simctl privacy) made
the latest CMT run red, as in all 9 CMT runs in the last 30 days.
Wire the existing e2e-triage composite into compatibility-matrix-
testing.yml (report-only unless E2E_TRIAGE_MODE=enforce, same gate as
PR/main), with a new triage-lanes input so CMT summarises its per-shard
contexts instead of the PR lanes.
Also add detox/utils/e2e-run-breakdown.js: groups a gh run's jobs by
framework (Detox/Maestro) and platform (iOS/iPad/Android) so failure
rates per lane no longer need TSIO or signed-in log access.
* ci(e2e): serialize CMT triage status writes and count timed-out jobs as failures
The toolkit's status publisher has no compare-and-swap, and its
status.ts requires every writer of the shared e2e-test/triage context
to hold the same non-cancelling, SHA-scoped lock. The CMT tsio-triage
matrix wrote that context in parallel with no lock. Add max-parallel: 1
so matrix entries run one at a time without discarding queued entries,
plus a job concurrency group keyed on the CMT commit SHA with
cancel-in-progress: false to serialize against triage writers from
other runs of the same commit.
Also count GitHub job conclusion timed_out as a failure in
e2e-run-breakdown; it previously fell through to other, so a timed-out
job showed as zero failures and was omitted from failingJobs.
* ci(e2e): judge CMT channel-message failures as flaky, infra, known, or suspected app bug
The CMT channel rollup only posted totals, so every red run needed a
human to open TSIO and decide whether the app broke. The rollup already
fetches each leg's TSIO consolidated payload for per-leg counts; it now
also extracts this commit's failing tests with their error text and
earlier history, and the channel message groups them: infra/environment
(server-disabled feature, simctl, network, hook timeout, setup crash, or
a sibling cascade in the same suite), flaky (passed on retry or mixed
pass/fail history), known failing (failed before this run), and
suspected app bug (first seen, no infra cause — labelled as needing a
human confirm). When TSIO has detail for fewer tests than the failing
total, the message says how many were judged.
* ci(e2e): require a valid job and suite key for CMT cascade judgements
A failure with no job and a title that reduces to an empty suite key
produced the cascade key '|', which could match an infra failure with
the same empty key and reclassify a suspected app bug as a cascade.
Build the cascade key only when both the job and the suite are
non-empty, exclude invalid keys from the infra set, and require a
valid key before reclassifying.
* ci(e2e): require corroboration before the CMT message says suspected app bug
Suspected app bug was the fallback for any first-seen failure with no
infra marker, so ordinary visibility/existence timeouts (MM-T3199,
MM-T3206, MM-70693_2 on the 833 run) were labelled app bugs on a single
observation. Now that label needs a direct content/state assertion —
never a wait expiring — plus corroboration: the same test failing on
another leg of this run, or earlier passes in TSIO history now failing
that assertion. Everything weaker lands in a new Needs a closer look
bucket that names why the evidence is insufficient.
* ci(e2e): tighten CMT judgement corroboration (legs, run attempts, matcher text)
Three review fixes in cmt-channel-notify:
- alsoFailedElsewhere counted an empty job name as a second leg, so one
real leg plus a missing job name could promote a failure to suspected
app bug. Only non-empty job names are tracked now.
- The consolidated endpoint ignores gh_run_attempt (verified live:
gh_run_attempt=99 returns full data, run_attempt=99 returns none), so
fetchConsolidated now sends run_attempt. failuresFromConsolidated also
excludes every attempt of this GitHub run from previousStatuses, so a
pass in attempt 1 cannot corroborate an app-bug call in attempt 2.
- Removed 'doesn't match the selected view' from the timing patterns so
a matcher mismatch that carries a direct assertion (toHaveText and
friends) can reach the hard-assertion gate instead of being dismissed
as a timing wait.
---------
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| bandwidth-throttling | ||
| build-android-maestro-apk | ||
| compute-e2e-build-fingerprint | ||
| e2e-override-label | ||
| e2e-override-status | ||
| e2e-triage | ||
| e2e-verified-status | ||
| generate-specs | ||
| prepare-android-build | ||
| prepare-ios-build | ||
| prepare-low-bandwidth | ||
| prepare-mobile-build | ||
| prepare-node-deps | ||
| provision-e2e-servers | ||
| s3-build-cache | ||
| s3-spec-durations | ||
| setup-intune | ||
| setup-intune-ssh-key | ||
| setup-ssh-key | ||
| start-proxy | ||
| test | ||
| test-coverage | ||
| update-commit-status | ||