[10:34:33.225] New invocation is queued and will start shortly
[10:34:34.330] Starting the invocation (attempt 1)
[10:34:34.398] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/c0c92e9526f921ccb3ced4e1904ebe71ffccb0ce"
[10:34:34.398] Popped gitiles commit info from properties and tags
[10:34:34.398] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[10:34:34.400] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[10:34:34.400] Buildbucket request:
{
"requestId": "8981528810877296720",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxASan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0c92e9526f921ccb3ced4e1904ebe71ffccb0ce",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8981528810877296720"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxASan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[10:34:34.927] Scheduled build:
{
"id": "8816098748902884401",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxASan"
},
"number": 15979,
"createdBy": "project:chromium",
"createTime": "2022-04-23T10:34:34.478161241Z",
"updateTime": "2022-04-23T10:34:34.478161241Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0c92e9526f921ccb3ced4e1904ebe71ffccb0ce",
"ref": "refs/heads/main"
}
},
"canary": true
}
[10:34:34.927] Task URL: https://cr-buildbucket.appspot.com/build/8816098748902884401
[10:34:34.927] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:2:0) after 9m31s
[10:36:05.445] Received PubSub notification, asking Buildbucket for the build status
[10:36:05.475] Build status: STARTED
[10:44:06.582] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:2:0)
[10:44:06.601] Build status: STARTED
[10:44:06.601] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:4:0) after 5m57s
[10:50:03.852] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:4:0)
[10:50:03.880] Build status: STARTED
[10:50:03.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:5:0) after 5m39s
[10:55:43.055] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:5:0)
[10:55:43.094] Build status: STARTED
[10:55:43.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:6:0) after 8m11s
[11:03:54.119] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:6:0)
[11:03:54.138] Build status: STARTED
[11:03:54.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:7:0) after 1m15s
[11:05:09.159] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:7:0)
[11:05:09.217] Build status: STARTED
[11:05:09.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:8:0) after 2m0s
[11:07:09.382] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:8:0)
[11:07:09.401] Build status: STARTED
[11:07:09.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:9:0) after 1m57s
[11:09:06.416] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:9:0)
[11:09:06.448] Build status: STARTED
[11:09:06.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:10:0) after 3m34s
[11:12:40.481] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:10:0)
[11:12:40.510] Build status: STARTED
[11:12:40.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:11:0) after 4m16s
[11:16:57.010] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:11:0)
[11:16:57.054] Build status: STARTED
[11:16:57.054] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:12:0) after 9m42s
[11:26:39.071] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:12:0)
[11:26:39.094] Build status: STARTED
[11:26:39.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:13:0) after 7m22s
[11:34:01.112] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:13:0)
[11:34:01.155] Build status: STARTED
[11:34:01.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:14:0) after 3m1s
[11:37:02.176] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:14:0)
[11:37:02.209] Build status: STARTED
[11:37:02.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:15:0) after 5m38s
[11:42:40.227] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:15:0)
[11:42:40.256] Build status: STARTED
[11:42:40.256] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:16:0) after 8m51s
[11:51:31.282] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:16:0)
[11:51:31.312] Build status: STARTED
[11:51:31.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:17:0) after 5m19s
[11:56:50.537] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:17:0)
[11:56:50.568] Build status: STARTED
[11:56:50.569] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:18:0) after 4m41s
[12:01:31.975] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:18:0)
[12:01:32.006] Build status: STARTED
[12:01:32.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:19:0) after 2m43s
[12:04:15.047] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:19:0)
[12:04:15.089] Build status: STARTED
[12:04:15.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:20:0) after 4m46s
[12:09:01.532] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:20:0)
[12:09:01.561] Build status: STARTED
[12:09:01.561] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:21:0) after 4m46s
[12:13:47.587] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:21:0)
[12:13:47.610] Build status: STARTED
[12:13:47.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:22:0) after 4m37s
[12:18:25.155] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:22:0)
[12:18:25.183] Build status: STARTED
[12:18:25.183] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:23:0) after 6m43s
[12:25:08.246] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:23:0)
[12:25:08.297] Build status: STARTED
[12:25:08.297] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:24:0) after 1m12s
[12:26:20.313] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:24:0)
[12:26:20.334] Build status: STARTED
[12:26:20.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:25:0) after 9m46s
[12:36:06.357] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:25:0)
[12:36:06.390] Build status: STARTED
[12:36:06.390] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:26:0) after 2m7s
[12:38:13.405] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:26:0)
[12:38:13.438] Build status: STARTED
[12:38:13.438] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:27:0) after 4m44s
[12:42:57.615] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:27:0)
[12:42:57.649] Build status: STARTED
[12:42:57.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:28:0) after 7m33s
[12:50:30.701] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:28:0)
[12:50:30.784] Build status: STARTED
[12:50:30.784] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:29:0) after 4m19s
[12:54:49.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:29:0)
[12:54:49.823] Build status: STARTED
[12:54:49.823] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:30:0) after 1m5s
[12:55:54.840] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:30:0)
[12:55:54.876] Build status: STARTED
[12:55:54.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:31:0) after 3m56s
[12:59:50.893] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:31:0)
[12:59:50.922] Build status: STARTED
[12:59:50.922] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:32:0) after 7m44s
[13:07:34.942] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:32:0)
[13:07:34.962] Build status: STARTED
[13:07:34.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:33:0) after 5m21s
[13:12:56.269] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:33:0)
[13:12:56.302] Build status: STARTED
[13:12:56.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:34:0) after 6m7s
[13:19:03.424] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:34:0)
[13:19:03.454] Build status: STARTED
[13:19:03.454] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:35:0) after 4m16s
[13:23:19.473] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:35:0)
[13:23:19.509] Build status: STARTED
[13:23:19.509] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:36:0) after 6m49s
[13:30:08.741] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:36:0)
[13:30:08.762] Build status: STARTED
[13:30:08.762] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:37:0) after 3m41s
[13:33:49.780] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:37:0)
[13:33:49.811] Build status: STARTED
[13:33:49.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:38:0) after 6m26s
[13:40:15.957] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:38:0)
[13:40:15.998] Build status: STARTED
[13:40:15.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:39:0) after 5m33s
[13:45:49.067] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:39:0)
[13:45:49.102] Build status: STARTED
[13:45:49.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:40:0) after 8m29s
[13:54:18.122] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:40:0)
[13:54:18.145] Build status: STARTED
[13:54:18.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:41:0) after 6m13s
[14:00:31.344] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:41:0)
[14:00:31.391] Build status: STARTED
[14:00:31.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:42:0) after 9m20s
[14:09:51.407] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:42:0)
[14:09:51.481] Build status: STARTED
[14:09:51.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:43:0) after 4m2s
[14:13:53.481] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:43:0)
[14:13:53.539] Build status: STARTED
[14:13:53.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:44:0) after 4m5s
[14:17:58.558] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:44:0)
[14:17:58.594] Build status: STARTED
[14:17:58.594] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:45:0) after 7m7s
[14:25:05.612] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:45:0)
[14:25:05.648] Build status: STARTED
[14:25:05.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:46:0) after 5m25s
[14:30:31.037] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:46:0)
[14:30:31.073] Build status: STARTED
[14:30:31.073] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:47:0) after 5m23s
[14:35:54.332] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:47:0)
[14:35:54.369] Build status: STARTED
[14:35:54.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:48:0) after 2m54s
[14:38:48.388] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:48:0)
[14:38:48.420] Build status: STARTED
[14:38:48.420] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:49:0) after 1m50s
[14:40:38.436] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:49:0)
[14:40:38.462] Build status: STARTED
[14:40:38.462] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:50:0) after 6m20s
[14:46:58.620] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:50:0)
[14:46:58.649] Build status: STARTED
[14:46:58.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:51:0) after 3m58s
[14:50:56.668] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:51:0)
[14:50:56.715] Build status: STARTED
[14:50:56.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:52:0) after 3m44s
[14:54:40.807] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:52:0)
[14:54:40.851] Build status: STARTED
[14:54:40.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:53:0) after 1m6s
[14:55:46.874] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:53:0)
[14:55:46.915] Build status: STARTED
[14:55:46.915] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:54:0) after 2m20s
[14:58:06.934] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:54:0)
[14:58:06.990] Build status: STARTED
[14:58:06.990] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:55:0) after 4m39s
[15:02:46.013] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:55:0)
[15:02:46.073] Build status: STARTED
[15:02:46.073] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:56:0) after 7m34s
[15:10:20.235] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:56:0)
[15:10:20.261] Build status: STARTED
[15:10:20.261] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:57:0) after 6m37s
[15:16:57.351] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:57:0)
[15:16:57.446] Build status: STARTED
[15:16:57.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:58:0) after 4m44s
[15:21:41.707] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:58:0)
[15:21:41.732] Build status: STARTED
[15:21:41.732] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:59:0) after 9m14s
[15:30:55.756] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:59:0)
[15:30:55.799] Build status: STARTED
[15:30:55.799] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:60:0) after 1m58s
[15:32:53.873] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:60:0)
[15:32:53.911] Build status: STARTED
[15:32:53.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:61:0) after 5m41s
[15:38:34.994] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:61:0)
[15:38:35.020] Build status: STARTED
[15:38:35.020] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:62:0) after 5m28s
[15:44:03.157] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:62:0)
[15:44:03.190] Build status: STARTED
[15:44:03.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:63:0) after 7m22s
[15:51:25.231] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:63:0)
[15:51:25.262] Build status: STARTED
[15:51:25.262] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:64:0) after 3m32s
[15:54:57.286] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:64:0)
[15:54:57.344] Build status: STARTED
[15:54:57.344] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:65:0) after 7m23s
[16:02:20.464] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:65:0)
[16:02:20.490] Build status: STARTED
[16:02:20.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:66:0) after 4m31s
[16:06:51.572] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:66:0)
[16:06:51.607] Build status: STARTED
[16:06:51.607] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:67:0) after 6m4s
[16:12:55.648] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:67:0)
[16:12:55.706] Build status: STARTED
[16:12:55.706] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxASan:8981528810877296720:68:0) after 8m21s
[16:17:22.218] Received PubSub notification, asking Buildbucket for the build status
[16:17:22.238] Build:
{
"id": "8816098748902884401",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxASan"
},
"number": 15979,
"createdBy": "project:chromium",
"createTime": "2022-04-23T10:34:34.478161241Z",
"startTime": "2022-04-23T10:36:04.951328Z",
"endTime": "2022-04-23T16:17:21.952186671Z",
"updateTime": "2022-04-23T16:17:21.952186671Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0c92e9526f921ccb3ced4e1904ebe71ffccb0ce",
"ref": "refs/heads/main"
}
},
"canary": true
}
[16:17:22.238] Invocation finished in 5h42m49.027262164s with status FAILED