1. All projects
  2. chromium
  3. ToTAndroidOfficial
  4. 8857535481487202768
Status: FAILED
Triggered by: -
Duration: 5 hours
Actions:

Definition (rev 36bc28844ffd5e030af6e41e2e0e4ddc5c92b699) 

buildbucket: <
  server: "cr-buildbucket.appspot.com"
  bucket: "ci"
  builder: "ToTAndroidOfficial"
>

Incoming triggers

Incoming properties

{
  "branch": "refs/heads/main",
  "repository": "https://chromium.googlesource.com/chromium/src",
  "revision": "b162147d88c14a8921bc25369981c2c66259db92"
}

Incoming tags

  • buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/b162147d88c14a8921bc25369981c2c66259db92
  • gitiles_ref:refs/heads/main

Debug log (UTC)

[01:35:26.861] New invocation is queued and will start shortly
[01:35:27.934] Starting the invocation (attempt 1)
[01:35:27.969] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/b162147d88c14a8921bc25369981c2c66259db92"
[01:35:27.969] Popped gitiles commit info from properties and tags
[01:35:27.969] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:35:27.970] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:35:27.970] Buildbucket request:
{
  "requestId": "8857535481487202768",
  "builder": {
    "project": "chromium",
    "bucket": "ci",
    "builder": "ToTAndroidOfficial"
  },
  "properties": {
    },
  "gitilesCommit": {
    "host": "chromium.googlesource.com",
    "project": "chromium/src",
    "id": "b162147d88c14a8921bc25369981c2c66259db92",
    "ref": "refs/heads/main"
  },
  "tags": [
    {
      "key": "scheduler_invocation_id",
      "value": "8857535481487202768"
    },
    {
      "key": "scheduler_job_id",
      "value": "chromium/ToTAndroidOfficial"
    },
    {
      "key": "user_agent",
      "value": "luci-scheduler"
    }
  ],
  "notify": {
    "pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
  }
}
[01:35:28.893] Scheduled build:
{
  "id": "8692105419522175825",
  "builder": {
    "project": "chromium",
    "bucket": "ci",
    "builder": "ToTAndroidOfficial"
  },
  "number": 25832,
  "createdBy": "project:chromium",
  "createTime": "2026-01-21T01:35:28.114809408Z",
  "updateTime": "2026-01-21T01:35:28.114809408Z",
  "status": "SCHEDULED",
  "input": {
    "gitilesCommit": {
      "host": "chromium.googlesource.com",
      "project": "chromium/src",
      "id": "b162147d88c14a8921bc25369981c2c66259db92",
      "ref": "refs/heads/main"
    }
  }
}
[01:35:28.893] Task URL: https://cr-buildbucket.appspot.com/build/8692105419522175825
[01:35:28.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:2:0) after 4m50s
[01:35:29.904] Received PubSub notification, asking Buildbucket for the build status
[01:35:29.929] Build status: SCHEDULED
[01:36:40.724] Received PubSub notification, asking Buildbucket for the build status
[01:36:40.747] Build status: STARTED
[01:40:18.921] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:2:0)
[01:40:18.947] Build status: STARTED
[01:40:18.947] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:5:0) after 9m46s
[01:50:05.001] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:5:0)
[01:50:05.375] Build status: STARTED
[01:50:05.375] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:6:0) after 9m40s
[01:59:45.399] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:6:0)
[01:59:45.416] Build status: STARTED
[01:59:45.416] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:7:0) after 1m42s
[02:01:27.419] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:7:0)
[02:01:27.465] Build status: STARTED
[02:01:27.465] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:8:0) after 1m33s
[02:03:00.485] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:8:0)
[02:03:00.502] Build status: STARTED
[02:03:00.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:9:0) after 6m1s
[02:09:01.569] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:9:0)
[02:09:01.588] Build status: STARTED
[02:09:01.588] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:10:0) after 5m56s
[02:14:57.609] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:10:0)
[02:14:57.631] Build status: STARTED
[02:14:57.631] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:11:0) after 4m27s
[02:19:24.656] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:11:0)
[02:19:24.668] Build status: STARTED
[02:19:24.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:12:0) after 3m16s
[02:22:40.755] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:12:0)
[02:22:40.772] Build status: STARTED
[02:22:40.772] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:13:0) after 4m57s
[02:27:37.836] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:13:0)
[02:27:37.856] Build status: STARTED
[02:27:37.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:14:0) after 2m21s
[02:29:58.879] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:14:0)
[02:29:58.914] Build status: STARTED
[02:29:58.914] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:15:0) after 8m35s
[02:38:33.954] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:15:0)
[02:38:33.971] Build status: STARTED
[02:38:33.971] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:16:0) after 6m54s
[02:45:28.037] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:16:0)
[02:45:28.084] Build status: STARTED
[02:45:28.084] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:17:0) after 6m27s
[02:51:55.195] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:17:0)
[02:51:55.209] Build status: STARTED
[02:51:55.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:18:0) after 1m20s
[02:53:15.244] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:18:0)
[02:53:15.267] Build status: STARTED
[02:53:15.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:19:0) after 3m10s
[02:56:25.306] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:19:0)
[02:56:25.320] Build status: STARTED
[02:56:25.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:20:0) after 3m17s
[02:59:42.440] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:20:0)
[02:59:42.525] Build status: STARTED
[02:59:42.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:21:0) after 4m14s
[03:03:56.602] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:21:0)
[03:03:56.618] Build status: STARTED
[03:03:56.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:22:0) after 6m12s
[03:10:08.696] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:22:0)
[03:10:08.716] Build status: STARTED
[03:10:08.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:23:0) after 2m0s
[03:12:08.772] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:23:0)
[03:12:08.789] Build status: STARTED
[03:12:08.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:24:0) after 9m3s
[03:21:11.775] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:24:0)
[03:21:11.810] Build status: STARTED
[03:21:11.810] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:25:0) after 2m46s
[03:23:57.834] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:25:0)
[03:23:57.849] Build status: STARTED
[03:23:57.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:26:0) after 2m6s
[03:26:04.055] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:26:0)
[03:26:04.092] Build status: STARTED
[03:26:04.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:27:0) after 3m1s
[03:29:05.251] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:27:0)
[03:29:05.399] Build status: STARTED
[03:29:05.399] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:28:0) after 8m58s
[03:38:03.419] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:28:0)
[03:38:03.432] Build status: STARTED
[03:38:03.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:29:0) after 8m12s
[03:46:15.456] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:29:0)
[03:46:15.476] Build status: STARTED
[03:46:15.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:30:0) after 9m27s
[03:55:42.545] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:30:0)
[03:55:42.562] Build status: STARTED
[03:55:42.562] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:31:0) after 4m21s
[04:00:03.579] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:31:0)
[04:00:03.595] Build status: STARTED
[04:00:03.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:32:0) after 5m7s
[04:05:10.642] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:32:0)
[04:05:10.656] Build status: STARTED
[04:05:10.656] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:33:0) after 5m12s
[04:10:22.669] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:33:0)
[04:10:22.709] Build status: STARTED
[04:10:22.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:34:0) after 2m42s
[04:13:04.732] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:34:0)
[04:13:04.749] Build status: STARTED
[04:13:04.749] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:35:0) after 5m40s
[04:18:45.069] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:35:0)
[04:18:45.093] Build status: STARTED
[04:18:45.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:36:0) after 5m55s
[04:24:40.117] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:36:0)
[04:24:40.144] Build status: STARTED
[04:24:40.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:37:0) after 9m23s
[04:34:03.194] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:37:0)
[04:34:03.223] Build status: STARTED
[04:34:03.223] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:38:0) after 3m16s
[04:37:19.552] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:38:0)
[04:37:19.566] Build status: STARTED
[04:37:19.566] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:39:0) after 5m23s
[04:42:42.601] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:39:0)
[04:42:42.628] Build status: STARTED
[04:42:42.628] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:40:0) after 9m49s
[04:52:31.647] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:40:0)
[04:52:31.664] Build status: STARTED
[04:52:31.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:41:0) after 1m46s
[04:54:17.684] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:41:0)
[04:54:17.701] Build status: STARTED
[04:54:17.701] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:42:0) after 8m51s
[05:03:08.733] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:42:0)
[05:03:08.839] Build status: STARTED
[05:03:08.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:43:0) after 9m5s
[05:12:13.953] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:43:0)
[05:12:13.972] Build status: STARTED
[05:12:13.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:44:0) after 4m57s
[05:17:10.970] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:44:0)
[05:17:10.987] Build status: STARTED
[05:17:10.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:45:0) after 6m19s
[05:23:29.989] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:45:0)
[05:23:30.011] Build status: STARTED
[05:23:30.011] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:46:0) after 4m5s
[05:27:35.329] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:46:0)
[05:27:35.341] Build status: STARTED
[05:27:35.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:47:0) after 1m37s
[05:29:12.864] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:47:0)
[05:29:12.925] Build status: STARTED
[05:29:12.925] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:48:0) after 9m34s
[05:38:46.995] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:48:0)
[05:38:47.017] Build status: STARTED
[05:38:47.017] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:49:0) after 1m15s
[05:40:02.045] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:49:0)
[05:40:02.068] Build status: STARTED
[05:40:02.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:50:0) after 1m46s
[05:41:48.086] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:50:0)
[05:41:48.098] Build status: STARTED
[05:41:48.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:51:0) after 9m30s
[05:51:18.167] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:51:0)
[05:51:18.182] Build status: STARTED
[05:51:18.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:52:0) after 4m23s
[05:55:41.216] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:52:0)
[05:55:41.233] Build status: STARTED
[05:55:41.233] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:53:0) after 6m27s
[06:02:08.252] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:53:0)
[06:02:08.281] Build status: STARTED
[06:02:08.281] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:54:0) after 5m12s
[06:07:20.425] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:54:0)
[06:07:20.446] Build status: STARTED
[06:07:20.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:55:0) after 3m11s
[06:10:31.531] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:55:0)
[06:10:31.588] Build status: STARTED
[06:10:31.588] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:56:0) after 7m5s
[06:17:36.666] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:56:0)
[06:17:36.732] Build status: STARTED
[06:17:36.732] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:57:0) after 5m12s
[06:22:48.799] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:57:0)
[06:22:48.815] Build status: STARTED
[06:22:48.815] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:58:0) after 1m33s
[06:24:21.844] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:58:0)
[06:24:22.134] Build status: STARTED
[06:24:22.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:59:0) after 6m10s
[06:30:32.186] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:59:0)
[06:30:32.200] Build status: STARTED
[06:30:32.201] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:60:0) after 2m8s
[06:32:40.262] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:60:0)
[06:32:40.276] Build status: STARTED
[06:32:40.276] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:61:0) after 4m28s
[06:37:08.312] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:61:0)
[06:37:08.342] Build status: STARTED
[06:37:08.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:62:0) after 6m17s
[06:43:25.358] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:62:0)
[06:43:25.373] Build status: STARTED
[06:43:25.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:63:0) after 6m20s
[06:49:45.416] Handling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:63:0)
[06:49:45.436] Build status: STARTED
[06:49:45.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTAndroidOfficial:8857535481487202768:64:0) after 9m14s
[06:58:48.404] Received PubSub notification, asking Buildbucket for the build status
[06:58:48.419] Build:
{
  "id": "8692105419522175825",
  "builder": {
    "project": "chromium",
    "bucket": "ci",
    "builder": "ToTAndroidOfficial"
  },
  "number": 25832,
  "createdBy": "project:chromium",
  "createTime": "2026-01-21T01:35:28.114809408Z",
  "startTime": "2026-01-21T01:36:40.412521911Z",
  "endTime": "2026-01-21T06:58:47.764258162Z",
  "updateTime": "2026-01-21T06:58:47.764258162Z",
  "status": "FAILURE",
  "input": {
    "gitilesCommit": {
      "host": "chromium.googlesource.com",
      "project": "chromium/src",
      "id": "b162147d88c14a8921bc25369981c2c66259db92",
      "ref": "refs/heads/main"
    }
  }
}
[06:58:48.419] Invocation finished in 5h23m21.563176489s with status FAILED