mattermost-mobile-upstream/detox
yasser khan 3e1331248c
Some checks are pending
ci / lint-typecheck (push) Waiting to run
ci / detox-lint-typecheck (push) Waiting to run
ci / test (push) Waiting to run
ci(e2e): say whether each main-run failure is new, still failing, flaky or a setup problem (#10284)
* ci(e2e): say whether each main-run failure is new, still failing, flaky or a setup problem, from the triage verdict

The channel message judged failures from their error text, beside the E2E triage
jobs that judge the same failures from TSIO history, and the two disagreed: on
main run 40764686027 it listed tests that passed on retry as needing a look. The
message now waits for triage and reads each run's findings. On main, a test that
passed every earlier main run and fails now is new, with the changes merged since
the last main run named (TSIO's previous run, GitHub's compare); a run where most
tests failed at once is called out first. Runs with no triage verdict keep the
error-text judgement.

* ci(e2e): judge from the error text each failed test triage named no verdict for, and say how many were not checked

* ci(e2e): count each test once per run, so a test that failed and then passed its retry is flaky, not a failure

* ci(e2e): a main-run failure triage calls flaky or a test problem is not blamed on new changes; a same-commit re-run says nothing was merged; a long PR list says it is partial

* Fix TSIO current report selection

Co-authored-by: yasserfaraazkhan <15979783+yasserfaraazkhan@users.noreply.github.com>

* Fix lint comment spacing

Co-authored-by: yasserfaraazkhan <15979783+yasserfaraazkhan@users.noreply.github.com>

* ci(e2e): update the TSIO upload action, so a failed screenshot upload no longer drops a shard's results

* ci(e2e): an earlier attempt of the same main run is not the previous run, so a re-run still lists the changes merged since the last run

* ci(e2e): drop the note from the TSIO upload action pin comment

* ci(e2e): a re-run replaces the first attempt's triage findings, so the channel message never posts stale verdicts

* ci(e2e): the main-run message says in a few lines which failures may be product bugs and which are not, and posts nothing when none may be

* ci(e2e): main and CMT call a failure a product bug only when it fails again on a fresh device, and clear what the error text proves is not the app

* ci(e2e): a re-run hands Android Detox and Maestro results to the channel message too

* ci(e2e): CMT re-runs start after a failed lane; a spec that differs from main, or a feature the server has off, is not a product bug

* ci(e2e): the CMT message plans the re-run as the planner job does, leaving out tests that differ from main

* ci(e2e): one channel post per run, and a test's name can never mention the channel

* ci(e2e): a re-run keeps the run's own report and runs only the failed flows

* ci(e2e): a PR run posts to the channel only when something failed

* ci(e2e): an E2E/Override PR keeps its statuses on a new push; a few simpler lines in the channel notify

* ci(e2e): exclude isolated report-problem flows from reruns and prevent label/action concurrency collisions

<!-- coderabbit-ai-generated:v1 -->

* ci(e2e): pass support paths to E2E triage

* ci(e2e): pin TSIO report-upload to release-0.16.0 (main@3212731)

Brings the TSIO action pin up to the latest release, which includes
mattermost-test-system-io#129, #130 and #124. The report-upload action itself
is unchanged between the two pins.

* ci(e2e): re-run E2E jobs once when a lane is red only on infrastructure

Triage (rerun-attempts: 1, with actions: read to read the failed jobs) asks for
one re-run when a lane is red only because jobs died or reports never arrived;
a build or test-server failure holds it. e2e-tests-rerun.yml carries the re-run
out once the run finishes. CMT keeps its own re-run of failed specs.

---------
2026-11-03 07:45:59 +05:30
..
android_emulator test(detox): re-submit RF→Detox migration without unintended UI changes (#9847) 2026-06-19 22:44:02 +05:30
e2e Mm 70564 channel labels use in channel creation (#10146) 2026-10-30 15:03:55 +01:00
maestro ci(e2e): triage red Detox and Maestro runs against test history (#10172) 2026-10-26 10:26:45 +05:30
provision E2E: run shared channels tests on CI, demo plugin v0.12.2 on all full-suite servers (#10267) 2026-10-27 11:42:07 +05:30
scripts ci(e2e): triage red Detox and Maestro runs against test history (#10172) 2026-10-26 10:26:45 +05:30
shared E2E: run shared channels tests on CI, demo plugin v0.12.2 on all full-suite servers (#10267) 2026-10-27 11:42:07 +05:30
utils ci(e2e): say whether each main-run failure is new, still failing, flaky or a setup problem (#10284) 2026-11-03 07:45:59 +05:30
.babelrc Barebones code for v2 2020-11-17 14:20:58 -03:00
.detoxrc.json test(detox): re-submit RF→Detox migration without unintended UI changes (#9847) 2026-06-19 22:44:02 +05:30
CLAUDE.md test(e2e): close manual release-test gaps (#10125) 2026-09-11 09:48:02 +05:30
create_android_emulator.sh E2E: CI flake fixes; bump Maestro 2.11.0 and Detox 20.51.4 (#10257) 2026-10-19 22:58:41 +05:30
generate_unified_report.js test(detox): re-submit RF→Detox migration without unintended UI changes (#9847) 2026-06-19 22:44:02 +05:30
inject-detox-settings.js Revert "e2e(detox): Rainforest to Detox migration (#9790)" (#9834) 2026-06-15 17:57:01 +02:00
package-lock.json E2E: CI flake fixes; bump Maestro 2.11.0 and Detox 20.51.4 (#10257) 2026-10-19 22:58:41 +05:30
package.json Mm 70564 channel labels use in channel creation (#10146) 2026-10-30 15:03:55 +01:00
README.md test(e2e): close manual release-test gaps (#10125) 2026-09-11 09:48:02 +05:30
save_maestro_zephyr.js feat(e2e): Run e2e against different server version when rc branch is pushed (#9800) 2026-07-07 18:02:13 +05:30
save_report.js fix: stabilize server switch, saved posts, favorites, and related runtime races (#10122) 2026-09-09 00:35:15 +05:30
tsconfig.json Fix/detox hittest cluster (#10023) 2026-08-27 08:58:03 +05:30
webhook_server.js FIX: Correct Android minute interval and UTC+offset date display in datetime fields (#10202) 2026-10-22 15:12:27 -05:00

How to Run Detox Tests

This guide will help you set up and run Detox tests for your project.

Install Dependencies

First, navigate to the root directory of your project and install the necessary dependencies by running:

npm install

navigate to the detox folder and run npm install

Configure your server details

detox/e2e/support/test_config.ts defaults to one local server on port 8065. For iOS, both the test runner and simulator use http://localhost:8065. For Android, the test runner uses http://localhost:8065 while the emulator uses http://10.0.2.2:8065.

If you use a remote server, set the logical site variable to a URL reachable from both the host test runner and the device:

  export SITE_1_URL="https://your-server.example.com"
  export ADMIN_USERNAME="your-username"
  export ADMIN_PASSWORD="your-password"
  export ADMIN_EMAIL="your-email"

You can instead create a detox/.env file:

  # detox/.env
  SITE_1_URL=https://your-server.example.com
  ADMIN_USERNAME=your-username
  ADMIN_PASSWORD=your-password
  ADMIN_EMAIL=your-email@example.com

Note: no need to provide all variables, only the ones that differ.

SITE_2_URL and SITE_3_URL are only needed for tests that add distinct Mattermost servers, such as server_login/server_list.e2e.ts. If they are unset, their URL helpers alias site 1 and tests that require distinct servers are skipped. The five-server ANDROID_SITE_* and IOS_SITE_* topology is assigned by CI and is not required for ordinary local runs.

Android

Build Detox Android App

To build the Detox Android app, navigate to the detox folder and run:

npm run e2e:android-build

The debug apk will be built and available at android/app/build/outputs/apk/debug/app-debug.apk

Run Detox Android Tests

Create emulator

./create_android_emulator.sh SDK_VERSION AVD_NAME

# example ./create_android_emulator.sh 34 pixel_5a_avd
# example ./create_android_emulator.sh 34 pixel_5a_avd --headless
# If we want to see the emulator logs. Run it in debug mode example ./create_android_emulator.sh 34 pixel_5a_avd --debug

To execute the Detox tests on Android, navigate to the detox folder and run:

npm run e2e:android-test

# To run a particular tests

npm run e2e:android-test <path to test file>

Local runs now default to a lower-memory setup so they coexist better with other tools:

  • Detox uses 1 worker unless you set DETOX_MAX_WORKERS.
  • Detox caps Node heap at 4096MB unless you already set NODE_OPTIONS.
  • Android Detox builds cap Gradle heap at 3072MB and use at most 2 Gradle workers unless you override MM_GRADLE_XMX_MB or MM_GRADLE_WORKERS_MAX.
  • Android emulator RAM defaults to 3072MB when created or refreshed; override with MM_ANDROID_EMULATOR_RAM_MB.

iOS

Build iOS Simulator

To build the iOS simulator for Detox, from the project folder run:

npm run e2e:ios-build

This will build the Simulator .zip file at the root folder.

Create a folder named mobile-artifacts at the project root. Move the zip file under mobile-artifacts and unzip it there.

# From project root
mkdir mobile-artifacts

Configure your local

Update detox/.detoxrc.json to specify which device you want to run it. Change:

 "device": {
     "type": "__DEVICE_NAME__",
     "os": "__DEVICE_OS_VERSION__"
 },

to something like:

 "device": {
     "type": "iPhone 17 Pro",
     "os": "iOS 26.4"
 },

To get your values run xcrun simctl list devices to find both the device and iOS version.

Run iOS Tests

To execute the Detox tests on iOS, navigate to the detox folder and run:

npm run e2e:ios-test

# To run a particular tests

npm run e2e:android-test path to test file.

Disabling Password Autofill

from iOS v26 password autofill can interfere with login tests by automatically filling credentials. To disable this feature on your simulator:

# Interactive mode - select simulator from list
npm run e2e:ios-disable-autofill

# Or specify simulator ID directly
npm run e2e:ios-disable-autofill -- --simulator-id SIMULATOR_UDID

When to use this:

  • if using iOS 26 or later
  • Before running iOS E2E tests if you notice password fields being auto-filled
  • When login tests fail unexpectedly due to autofill interference
  • After creating a new iOS simulator for testing

Note: CI environments automatically disable this setting, so this is only needed for local development.

Finding your simulator ID:

xcrun simctl list devices | grep Booted

TIP : For iOS, you can download the simulator from ~Mobile: Test build or ~Release: Mobile Apps channel in the community.

Results

The Local Runs generate artifacts under detox/artifacts/ios-debug-** or detox/artifacts/android-debug-**. You can see the html report, failure screenshot under that folder.

Webhook sidecar (mm_blocks / interactive dialog specs)

The mm_blocks_* and interactive-dialog specs register integrations that the Mattermost server calls back into. detox/webhook_server.js serves those callbacks on :3000, and the server must be able to reach it:

  • Local server (SITE_1_URL=http://localhost:8065): no tunnel needed, the server reaches 127.0.0.1:3000 directly.

    cd detox && npm run start:webhook
    
  • Remote/cloud test server: the server cannot reach your machine, so the sidecar needs a public HTTPS origin. detox/scripts/start_webhook_sidecar.sh starts webhook_server.js and resolves that origin, in priority order:

    Env Behaviour
    WEBHOOK_PUBLIC_BASE_URL Use an already-routable HTTPS origin as-is (named tunnel, ngrok, reverse proxy). Preferred.
    WEBHOOK_PUBLIC_BASE_URL + CLOUDFLARED_TUNNEL_TOKEN Run a named Cloudflare tunnel for that hostname. The token is passed via TUNNEL_TOKEN because argv is world-readable.
    neither Fall back to a trycloudflare.com quick tunnel. Time-boxed and DNS-flaky, so treat it as a last resort.
    cd detox
    SITE_1_URL="https://your-test-server" \
    WEBHOOK_PUBLIC_BASE_URL="https://your-tunnel-hostname" \
    bash scripts/start_webhook_sidecar.sh
    

    Pin a different cloudflared with CLOUDFLARED_VERSION; the script defaults to a known-good release rather than latest.

The script always exits 0 — it exports WEBHOOK_SIDECAR_READY and WEBHOOK_CALLBACKS_REACHABLE instead of failing, and specs that need a webhook fail fast on their own health check. A quick tunnel passes outbound health checks but does not reliably deliver server-to-sidecar callbacks, so it reports WEBHOOK_CALLBACKS_REACHABLE=false.