* ci(e2e): triage CMT failures per test and add run breakdown tool
CMT had no per-test judging: its Detox template summary jobs fail on
FAILURES > 0, so 17/642 iOS and 12/650 Android failures (plus one iOS
shard whose simulator hit NSPOSIXErrorDomain in simctl privacy) made
the latest CMT run red, as in all 9 CMT runs in the last 30 days.
Wire the existing e2e-triage composite into compatibility-matrix-
testing.yml (report-only unless E2E_TRIAGE_MODE=enforce, same gate as
PR/main), with a new triage-lanes input so CMT summarises its per-shard
contexts instead of the PR lanes.
Also add detox/utils/e2e-run-breakdown.js: groups a gh run's jobs by
framework (Detox/Maestro) and platform (iOS/iPad/Android) so failure
rates per lane no longer need TSIO or signed-in log access.
* ci(e2e): serialize CMT triage status writes and count timed-out jobs as failures
The toolkit's status publisher has no compare-and-swap, and its
status.ts requires every writer of the shared e2e-test/triage context
to hold the same non-cancelling, SHA-scoped lock. The CMT tsio-triage
matrix wrote that context in parallel with no lock. Add max-parallel: 1
so matrix entries run one at a time without discarding queued entries,
plus a job concurrency group keyed on the CMT commit SHA with
cancel-in-progress: false to serialize against triage writers from
other runs of the same commit.
Also count GitHub job conclusion timed_out as a failure in
e2e-run-breakdown; it previously fell through to other, so a timed-out
job showed as zero failures and was omitted from failingJobs.
* ci(e2e): judge CMT channel-message failures as flaky, infra, known, or suspected app bug
The CMT channel rollup only posted totals, so every red run needed a
human to open TSIO and decide whether the app broke. The rollup already
fetches each leg's TSIO consolidated payload for per-leg counts; it now
also extracts this commit's failing tests with their error text and
earlier history, and the channel message groups them: infra/environment
(server-disabled feature, simctl, network, hook timeout, setup crash, or
a sibling cascade in the same suite), flaky (passed on retry or mixed
pass/fail history), known failing (failed before this run), and
suspected app bug (first seen, no infra cause — labelled as needing a
human confirm). When TSIO has detail for fewer tests than the failing
total, the message says how many were judged.
* ci(e2e): require a valid job and suite key for CMT cascade judgements
A failure with no job and a title that reduces to an empty suite key
produced the cascade key '|', which could match an infra failure with
the same empty key and reclassify a suspected app bug as a cascade.
Build the cascade key only when both the job and the suite are
non-empty, exclude invalid keys from the infra set, and require a
valid key before reclassifying.
* ci(e2e): require corroboration before the CMT message says suspected app bug
Suspected app bug was the fallback for any first-seen failure with no
infra marker, so ordinary visibility/existence timeouts (MM-T3199,
MM-T3206, MM-70693_2 on the 833 run) were labelled app bugs on a single
observation. Now that label needs a direct content/state assertion —
never a wait expiring — plus corroboration: the same test failing on
another leg of this run, or earlier passes in TSIO history now failing
that assertion. Everything weaker lands in a new Needs a closer look
bucket that names why the evidence is insufficient.
* ci(e2e): tighten CMT judgement corroboration (legs, run attempts, matcher text)
Three review fixes in cmt-channel-notify:
- alsoFailedElsewhere counted an empty job name as a second leg, so one
real leg plus a missing job name could promote a failure to suspected
app bug. Only non-empty job names are tracked now.
- The consolidated endpoint ignores gh_run_attempt (verified live:
gh_run_attempt=99 returns full data, run_attempt=99 returns none), so
fetchConsolidated now sends run_attempt. failuresFromConsolidated also
excludes every attempt of this GitHub run from previousStatuses, so a
pass in attempt 1 cannot corroborate an app-bug call in attempt 2.
- Removed 'doesn't match the selected view' from the timing patterns so
a matcher mismatch that carries a direct assertion (toHaveText and
friends) can reach the hard-assertion gate instead of being dismissed
as a timing wait.
---------
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| android_emulator | ||
| e2e | ||
| maestro | ||
| provision | ||
| scripts | ||
| shared | ||
| utils | ||
| .babelrc | ||
| .detoxrc.json | ||
| CLAUDE.md | ||
| create_android_emulator.sh | ||
| generate_unified_report.js | ||
| inject-detox-settings.js | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| save_maestro_zephyr.js | ||
| save_report.js | ||
| tsconfig.json | ||
| webhook_server.js | ||
How to Run Detox Tests
This guide will help you set up and run Detox tests for your project.
Install Dependencies
First, navigate to the root directory of your project and install the necessary dependencies by running:
npm install
navigate to the detox folder and run npm install
Configure your server details
detox/e2e/support/test_config.ts defaults to one local server on port 8065.
For iOS, both the test runner and simulator use http://localhost:8065. For
Android, the test runner uses http://localhost:8065 while the emulator uses
http://10.0.2.2:8065.
If you use a remote server, set the logical site variable to a URL reachable from both the host test runner and the device:
export SITE_1_URL="https://your-server.example.com"
export ADMIN_USERNAME="your-username"
export ADMIN_PASSWORD="your-password"
export ADMIN_EMAIL="your-email"
You can instead create a detox/.env file:
# detox/.env
SITE_1_URL=https://your-server.example.com
ADMIN_USERNAME=your-username
ADMIN_PASSWORD=your-password
ADMIN_EMAIL=your-email@example.com
Note: no need to provide all variables, only the ones that differ.
SITE_2_URL and SITE_3_URL are only needed for tests that add distinct
Mattermost servers, such as server_login/server_list.e2e.ts. If they are
unset, their URL helpers alias site 1 and tests that require distinct servers
are skipped. The five-server ANDROID_SITE_* and IOS_SITE_* topology is
assigned by CI and is not required for ordinary local runs.
Android
Build Detox Android App
To build the Detox Android app, navigate to the detox folder and run:
npm run e2e:android-build
The debug apk will be built and available at android/app/build/outputs/apk/debug/app-debug.apk
Run Detox Android Tests
Create emulator
./create_android_emulator.sh SDK_VERSION AVD_NAME
# example ./create_android_emulator.sh 34 pixel_5a_avd
# example ./create_android_emulator.sh 34 pixel_5a_avd --headless
# If we want to see the emulator logs. Run it in debug mode example ./create_android_emulator.sh 34 pixel_5a_avd --debug
To execute the Detox tests on Android, navigate to the detox folder and run:
npm run e2e:android-test
# To run a particular tests
npm run e2e:android-test <path to test file>
Local runs now default to a lower-memory setup so they coexist better with other tools:
- Detox uses
1worker unless you setDETOX_MAX_WORKERS. - Detox caps Node heap at
4096MBunless you already setNODE_OPTIONS. - Android Detox builds cap Gradle heap at
3072MBand use at most2Gradle workers unless you overrideMM_GRADLE_XMX_MBorMM_GRADLE_WORKERS_MAX. - Android emulator RAM defaults to
3072MBwhen created or refreshed; override withMM_ANDROID_EMULATOR_RAM_MB.
iOS
Build iOS Simulator
To build the iOS simulator for Detox, from the project folder run:
npm run e2e:ios-build
This will build the Simulator .zip file at the root folder.
Create a folder named mobile-artifacts at the project root. Move the zip file under mobile-artifacts and unzip it there.
# From project root
mkdir mobile-artifacts
Configure your local
Update detox/.detoxrc.json to specify which device you want to run it. Change:
"device": {
"type": "__DEVICE_NAME__",
"os": "__DEVICE_OS_VERSION__"
},
to something like:
"device": {
"type": "iPhone 17 Pro",
"os": "iOS 26.4"
},
To get your values run xcrun simctl list devices to find both the device and iOS version.
Run iOS Tests
To execute the Detox tests on iOS, navigate to the detox folder and run:
npm run e2e:ios-test
# To run a particular tests
npm run e2e:android-test path to test file.
Disabling Password Autofill
from iOS v26 password autofill can interfere with login tests by automatically filling credentials. To disable this feature on your simulator:
# Interactive mode - select simulator from list
npm run e2e:ios-disable-autofill
# Or specify simulator ID directly
npm run e2e:ios-disable-autofill -- --simulator-id SIMULATOR_UDID
When to use this:
- if using iOS 26 or later
- Before running iOS E2E tests if you notice password fields being auto-filled
- When login tests fail unexpectedly due to autofill interference
- After creating a new iOS simulator for testing
Note: CI environments automatically disable this setting, so this is only needed for local development.
Finding your simulator ID:
xcrun simctl list devices | grep Booted
TIP : For iOS, you can download the simulator from ~Mobile: Test build or ~Release: Mobile Apps channel in the community.
Results
The Local Runs generate artifacts under detox/artifacts/ios-debug-** or detox/artifacts/android-debug-**.
You can see the html report, failure screenshot under that folder.
Webhook sidecar (mm_blocks / interactive dialog specs)
The mm_blocks_* and interactive-dialog specs register integrations that the
Mattermost server calls back into. detox/webhook_server.js serves those
callbacks on :3000, and the server must be able to reach it:
-
Local server (
SITE_1_URL=http://localhost:8065): no tunnel needed, the server reaches127.0.0.1:3000directly.cd detox && npm run start:webhook -
Remote/cloud test server: the server cannot reach your machine, so the sidecar needs a public HTTPS origin.
detox/scripts/start_webhook_sidecar.shstartswebhook_server.jsand resolves that origin, in priority order:Env Behaviour WEBHOOK_PUBLIC_BASE_URLUse an already-routable HTTPS origin as-is (named tunnel, ngrok, reverse proxy). Preferred. WEBHOOK_PUBLIC_BASE_URL+CLOUDFLARED_TUNNEL_TOKENRun a named Cloudflare tunnel for that hostname. The token is passed via TUNNEL_TOKENbecause argv is world-readable.neither Fall back to a trycloudflare.comquick tunnel. Time-boxed and DNS-flaky, so treat it as a last resort.cd detox SITE_1_URL="https://your-test-server" \ WEBHOOK_PUBLIC_BASE_URL="https://your-tunnel-hostname" \ bash scripts/start_webhook_sidecar.shPin a different
cloudflaredwithCLOUDFLARED_VERSION; the script defaults to a known-good release rather thanlatest.
The script always exits 0 — it exports WEBHOOK_SIDECAR_READY and
WEBHOOK_CALLBACKS_REACHABLE instead of failing, and specs that need a webhook
fail fast on their own health check. A quick tunnel passes outbound health checks
but does not reliably deliver server-to-sidecar callbacks, so it reports
WEBHOOK_CALLBACKS_REACHABLE=false.