[08:02:25.293] New invocation is queued and will start shortly
[08:02:26.392] Starting the invocation (attempt 1)
[08:02:26.430] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/791db24354146975d991a84a5391e49ba241ea4e"
[08:02:26.430] Popped gitiles commit info from properties and tags
[08:02:26.430] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[08:02:26.430] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[08:02:26.431] Buildbucket request:
{
"requestId": "8905165139525866768",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "791db24354146975d991a84a5391e49ba241ea4e",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8905165139525866768"
},
{
"key": "scheduler_job_id",
"value": "chromium/Windows Cross deterministic"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[08:02:27.097] Scheduled build:
{
"id": "8739735077593776641",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"number": 2509,
"createdBy": "project:chromium",
"createTime": "2024-08-13T08:02:26.512267937Z",
"updateTime": "2024-08-13T08:02:26.512267937Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "791db24354146975d991a84a5391e49ba241ea4e",
"ref": "refs/heads/main"
}
}
}
[08:02:27.097] Task URL: https://cr-buildbucket.appspot.com/build/8739735077593776641
[08:02:27.097] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:2:0) after 5m40s
[08:02:28.165] Received PubSub notification, asking Buildbucket for the build status
[08:02:28.181] Build status: SCHEDULED
[08:08:07.119] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:2:0)
[08:08:07.153] Build status: SCHEDULED
[08:08:07.153] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:4:0) after 4m4s
[08:12:11.185] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:4:0)
[08:12:11.203] Build status: SCHEDULED
[08:12:11.204] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:5:0) after 1m43s
[08:13:54.336] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:5:0)
[08:13:54.363] Build status: SCHEDULED
[08:13:54.363] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:6:0) after 6m57s
[08:20:51.570] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:6:0)
[08:20:51.586] Build status: SCHEDULED
[08:20:51.586] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:7:0) after 2m10s
[08:23:04.249] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:7:0)
[08:23:04.284] Build status: SCHEDULED
[08:23:04.284] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:8:0) after 4m59s
[08:28:03.307] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:8:0)
[08:28:03.329] Build status: SCHEDULED
[08:28:03.329] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:9:0) after 9m15s
[08:34:31.202] Received PubSub notification, asking Buildbucket for the build status
[08:34:31.221] Build status: STARTED
[08:37:18.358] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:9:0)
[08:37:18.384] Build status: STARTED
[08:37:18.384] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:11:0) after 3m57s
[08:41:15.420] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:11:0)
[08:41:15.442] Build status: STARTED
[08:41:15.442] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:12:0) after 4m56s
[08:46:11.472] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:12:0)
[08:46:11.550] Build status: STARTED
[08:46:11.550] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:13:0) after 8m58s
[08:55:09.540] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:13:0)
[08:55:09.566] Build status: STARTED
[08:55:09.566] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:14:0) after 5m39s
[09:00:48.661] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:14:0)
[09:00:48.681] Build status: STARTED
[09:00:48.681] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:15:0) after 3m41s
[09:04:29.718] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:15:0)
[09:04:29.737] Build status: STARTED
[09:04:29.737] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:16:0) after 3m30s
[09:08:00.253] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:16:0)
[09:08:00.274] Build status: STARTED
[09:08:00.274] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:17:0) after 9m5s
[09:17:05.302] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:17:0)
[09:17:05.351] Build status: STARTED
[09:17:05.351] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:18:0) after 7m41s
[09:24:46.466] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:18:0)
[09:24:46.504] Build status: STARTED
[09:24:46.504] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:19:0) after 8m24s
[09:33:10.536] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:19:0)
[09:33:10.559] Build status: STARTED
[09:33:10.559] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:20:0) after 3m36s
[09:36:46.912] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:20:0)
[09:36:46.960] Build status: STARTED
[09:36:46.960] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:21:0) after 2m24s
[09:39:11.067] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:21:0)
[09:39:11.092] Build status: STARTED
[09:39:11.092] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:22:0) after 1m4s
[09:40:15.114] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:22:0)
[09:40:15.133] Build status: STARTED
[09:40:15.133] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:23:0) after 5m58s
[09:46:13.162] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:23:0)
[09:46:13.182] Build status: STARTED
[09:46:13.182] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:24:0) after 1m33s
[09:47:46.258] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:24:0)
[09:47:46.322] Build status: STARTED
[09:47:46.322] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:25:0) after 6m56s
[09:54:42.359] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:25:0)
[09:54:42.380] Build status: STARTED
[09:54:42.380] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:26:0) after 5m29s
[10:00:11.568] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:26:0)
[10:00:11.588] Build status: STARTED
[10:00:11.588] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:27:0) after 3m47s
[10:03:59.130] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:27:0)
[10:03:59.149] Build status: STARTED
[10:03:59.149] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:28:0) after 5m47s
[10:09:46.244] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:28:0)
[10:09:46.272] Build status: STARTED
[10:09:46.272] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:29:0) after 6m1s
[10:15:47.270] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:29:0)
[10:15:47.286] Build status: STARTED
[10:15:47.286] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:30:0) after 7m23s
[10:23:10.399] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:30:0)
[10:23:10.416] Build status: STARTED
[10:23:10.416] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:31:0) after 8m47s
[10:31:57.405] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:31:0)
[10:31:57.495] Build status: STARTED
[10:31:57.495] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:32:0) after 3m16s
[10:35:13.514] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:32:0)
[10:35:13.542] Build status: STARTED
[10:35:13.542] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:33:0) after 6m18s
[10:41:31.555] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:33:0)
[10:41:31.630] Build status: STARTED
[10:41:31.630] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:34:0) after 8m20s
[10:49:51.700] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:34:0)
[10:49:51.717] Build status: STARTED
[10:49:51.717] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:35:0) after 2m56s
[10:52:47.744] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:35:0)
[10:52:47.797] Build status: STARTED
[10:52:47.797] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:36:0) after 1m30s
[10:54:17.917] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:36:0)
[10:54:17.944] Build status: STARTED
[10:54:17.944] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:37:0) after 1m39s
[10:55:57.130] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:37:0)
[10:55:57.211] Build status: STARTED
[10:55:57.211] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:38:0) after 6m3s
[11:02:00.398] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:38:0)
[11:02:00.416] Build status: STARTED
[11:02:00.416] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:39:0) after 4m38s
[11:06:38.441] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:39:0)
[11:06:38.472] Build status: STARTED
[11:06:38.472] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:40:0) after 3m25s
[11:10:03.862] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:40:0)
[11:10:03.878] Build status: STARTED
[11:10:03.878] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:41:0) after 3m46s
[11:13:50.115] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:41:0)
[11:13:50.143] Build status: STARTED
[11:13:50.143] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:42:0) after 8m7s
[11:21:57.508] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:42:0)
[11:21:57.524] Build status: STARTED
[11:21:57.524] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:43:0) after 7m32s
[11:29:29.546] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:43:0)
[11:29:29.563] Build status: STARTED
[11:29:29.563] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:44:0) after 9m50s
[11:39:19.965] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:44:0)
[11:39:19.984] Build status: STARTED
[11:39:19.984] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:45:0) after 1m58s
[11:41:18.134] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:45:0)
[11:41:18.164] Build status: STARTED
[11:41:18.164] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:46:0) after 7m6s
[11:48:24.487] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:46:0)
[11:48:24.524] Build status: STARTED
[11:48:24.524] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:47:0) after 8m30s
[11:56:54.677] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:47:0)
[11:56:54.705] Build status: STARTED
[11:56:54.705] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:48:0) after 1m48s
[11:58:42.891] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:48:0)
[11:58:42.984] Build status: STARTED
[11:58:42.984] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:49:0) after 3m59s
[12:02:42.315] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:49:0)
[12:02:42.339] Build status: STARTED
[12:02:42.339] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:50:0) after 6m23s
[12:09:05.379] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:50:0)
[12:09:05.411] Build status: STARTED
[12:09:05.411] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:51:0) after 9m56s
[12:19:01.431] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:51:0)
[12:19:01.460] Build status: STARTED
[12:19:01.460] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:52:0) after 7m52s
[12:26:53.485] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:52:0)
[12:26:53.548] Build status: STARTED
[12:26:53.548] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:53:0) after 7m30s
[12:34:23.586] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:53:0)
[12:34:23.618] Build status: STARTED
[12:34:23.618] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:54:0) after 5m50s
[12:40:13.683] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:54:0)
[12:40:13.737] Build status: STARTED
[12:40:13.737] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:55:0) after 5m25s
[12:45:38.935] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:55:0)
[12:45:38.963] Build status: STARTED
[12:45:38.963] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:56:0) after 3m15s
[12:48:54.171] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:56:0)
[12:48:54.194] Build status: STARTED
[12:48:54.194] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:57:0) after 3m54s
[12:52:48.247] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:57:0)
[12:52:48.269] Build status: STARTED
[12:52:48.269] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:58:0) after 1m38s
[12:54:26.298] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:58:0)
[12:54:26.392] Build status: STARTED
[12:54:26.392] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:59:0) after 8m9s
[13:02:35.502] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:59:0)
[13:02:35.522] Build status: STARTED
[13:02:35.522] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:60:0) after 4m50s
[13:07:25.589] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:60:0)
[13:07:25.616] Build status: STARTED
[13:07:25.616] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:61:0) after 6m19s
[13:13:44.906] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:61:0)
[13:13:44.930] Build status: STARTED
[13:13:44.930] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:62:0) after 8m30s
[13:22:14.957] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:62:0)
[13:22:15.132] Build status: STARTED
[13:22:15.132] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:63:0) after 9m30s
[13:31:45.166] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:63:0)
[13:31:45.187] Build status: STARTED
[13:31:45.187] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:64:0) after 4m35s
[13:36:20.238] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:64:0)
[13:36:20.257] Build status: STARTED
[13:36:20.257] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:65:0) after 7m42s
[13:44:02.332] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:65:0)
[13:44:02.420] Build status: STARTED
[13:44:02.420] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:66:0) after 5m49s
[13:49:51.759] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:66:0)
[13:49:51.811] Build status: STARTED
[13:49:51.811] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:67:0) after 9m28s
[13:59:19.857] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:67:0)
[13:59:19.916] Build status: STARTED
[13:59:19.917] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:68:0) after 1m5s
[14:00:24.966] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:68:0)
[14:00:25.054] Build status: STARTED
[14:00:25.054] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:69:0) after 7m39s
[14:08:04.218] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:69:0)
[14:08:04.237] Build status: STARTED
[14:08:04.237] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:70:0) after 2m22s
[14:10:26.283] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:70:0)
[14:10:26.315] Build status: STARTED
[14:10:26.315] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:71:0) after 4m37s
[14:15:03.354] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:71:0)
[14:15:03.373] Build status: STARTED
[14:15:03.373] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:72:0) after 7m33s
[14:22:36.453] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:72:0)
[14:22:36.474] Build status: STARTED
[14:22:36.474] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:73:0) after 9m56s
[14:32:33.116] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:73:0)
[14:32:33.152] Build status: STARTED
[14:32:33.152] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:74:0) after 5m7s
[14:37:40.176] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:74:0)
[14:37:40.243] Build status: STARTED
[14:37:40.243] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:75:0) after 8m30s
[14:46:10.379] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:75:0)
[14:46:10.399] Build status: STARTED
[14:46:10.399] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:76:0) after 3m57s
[14:50:07.450] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:76:0)
[14:50:07.469] Build status: STARTED
[14:50:07.469] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:77:0) after 5m25s
[14:55:32.523] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:77:0)
[14:55:32.560] Build status: STARTED
[14:55:32.560] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:78:0) after 1m22s
[14:56:54.687] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:78:0)
[14:56:54.793] Build status: STARTED
[14:56:54.793] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:79:0) after 5m12s
[15:02:06.884] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:79:0)
[15:02:06.912] Build status: STARTED
[15:02:06.912] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:80:0) after 5m19s
[15:07:25.946] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:80:0)
[15:07:25.967] Build status: STARTED
[15:07:25.967] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:81:0) after 1m50s
[15:09:15.992] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:81:0)
[15:09:16.009] Build status: STARTED
[15:09:16.009] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:82:0) after 2m33s
[15:11:49.086] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:82:0)
[15:11:49.256] Build status: STARTED
[15:11:49.256] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:83:0) after 6m8s
[15:17:57.284] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:83:0)
[15:17:57.308] Build status: STARTED
[15:17:57.308] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:84:0) after 5m16s
[15:23:13.443] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:84:0)
[15:23:13.498] Build status: STARTED
[15:23:13.498] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:85:0) after 4m27s
[15:27:40.538] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:85:0)
[15:27:40.557] Build status: STARTED
[15:27:40.557] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:86:0) after 1m36s
[15:29:16.589] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:86:0)
[15:29:16.604] Build status: STARTED
[15:29:16.605] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:87:0) after 2m19s
[15:31:35.689] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:87:0)
[15:31:35.709] Build status: STARTED
[15:31:35.709] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:88:0) after 7m29s
[15:39:04.722] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:88:0)
[15:39:04.750] Build status: STARTED
[15:39:04.750] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:89:0) after 7m22s
[15:46:26.776] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:89:0)
[15:46:26.796] Build status: STARTED
[15:46:26.796] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:90:0) after 3m2s
[15:49:28.822] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:90:0)
[15:49:28.884] Build status: STARTED
[15:49:28.885] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:91:0) after 9m52s
[15:59:20.933] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:91:0)
[15:59:20.952] Build status: STARTED
[15:59:20.952] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:92:0) after 3m45s
[16:03:05.981] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:92:0)
[16:03:06.005] Build status: STARTED
[16:03:06.005] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:93:0) after 7m18s
[16:10:23.989] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:93:0)
[16:10:24.022] Build status: STARTED
[16:10:24.022] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:94:0) after 7m50s
[16:18:14.063] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:94:0)
[16:18:14.178] Build status: STARTED
[16:18:14.178] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:95:0) after 1m18s
[16:19:32.247] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:95:0)
[16:19:32.280] Build status: STARTED
[16:19:32.280] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:96:0) after 1m54s
[16:21:26.313] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:96:0)
[16:21:26.469] Build status: STARTED
[16:21:26.469] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:97:0) after 7m19s
[16:28:45.570] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:97:0)
[16:28:45.590] Build status: STARTED
[16:28:45.590] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:98:0) after 7m16s
[16:36:01.628] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:98:0)
[16:36:01.660] Build status: STARTED
[16:36:01.660] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:99:0) after 8m44s
[16:44:45.685] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:99:0)
[16:44:45.711] Build status: STARTED
[16:44:45.711] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:100:0) after 9m59s
[16:54:44.744] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:100:0)
[16:54:44.762] Build status: STARTED
[16:54:44.762] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:101:0) after 2m47s
[16:57:31.802] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:101:0)
[16:57:31.871] Build status: STARTED
[16:57:31.871] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:102:0) after 5m34s
[17:03:05.908] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:102:0)
[17:03:05.931] Build status: STARTED
[17:03:05.931] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:103:0) after 1m0s
[17:04:05.956] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:103:0)
[17:04:06.058] Build status: STARTED
[17:04:06.058] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:104:0) after 9m41s
[17:13:47.104] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:104:0)
[17:13:47.168] Build status: STARTED
[17:13:47.168] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:105:0) after 7m12s
[17:20:59.254] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:105:0)
[17:20:59.274] Build status: STARTED
[17:20:59.274] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:106:0) after 2m15s
[17:23:14.326] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:106:0)
[17:23:14.393] Build status: STARTED
[17:23:14.393] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:107:0) after 3m43s
[17:26:57.547] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:107:0)
[17:26:57.570] Build status: STARTED
[17:26:57.570] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:108:0) after 5m46s
[17:32:43.599] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:108:0)
[17:32:43.633] Build status: STARTED
[17:32:43.633] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:109:0) after 7m20s
[17:40:03.713] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:109:0)
[17:40:03.735] Build status: STARTED
[17:40:03.735] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:110:0) after 5m0s
[17:45:03.775] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:110:0)
[17:45:03.817] Build status: STARTED
[17:45:03.818] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:111:0) after 8m10s
[17:53:13.896] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:111:0)
[17:53:13.917] Build status: STARTED
[17:53:13.917] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:112:0) after 8m34s
[18:01:47.944] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:112:0)
[18:01:47.966] Build status: STARTED
[18:01:47.966] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:113:0) after 3m6s
[18:04:54.098] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:113:0)
[18:04:54.136] Build status: STARTED
[18:04:54.136] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:114:0) after 8m41s
[18:13:35.172] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:114:0)
[18:13:35.240] Build status: STARTED
[18:13:35.241] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:115:0) after 2m30s
[18:16:05.365] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:115:0)
[18:16:05.383] Build status: STARTED
[18:16:05.383] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:116:0) after 5m13s
[18:21:18.458] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:116:0)
[18:21:18.529] Build status: STARTED
[18:21:18.529] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:117:0) after 8m6s
[18:29:24.596] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:117:0)
[18:29:24.621] Build status: STARTED
[18:29:24.621] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:118:0) after 5m52s
[18:35:16.685] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:118:0)
[18:35:16.712] Build status: STARTED
[18:35:16.712] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:119:0) after 6m32s
[18:41:48.837] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:119:0)
[18:41:48.854] Build status: STARTED
[18:41:48.855] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:120:0) after 1m50s
[18:43:38.860] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:120:0)
[18:43:38.888] Build status: STARTED
[18:43:38.888] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:121:0) after 5m5s
[18:48:43.902] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:121:0)
[18:48:43.924] Build status: STARTED
[18:48:43.924] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:122:0) after 3m42s
[18:52:25.980] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:122:0)
[18:52:25.998] Build status: STARTED
[18:52:25.998] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:123:0) after 3m31s
[18:55:57.025] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:123:0)
[18:55:57.045] Build status: STARTED
[18:55:57.045] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:124:0) after 6m5s
[19:02:02.059] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:124:0)
[19:02:02.081] Build status: STARTED
[19:02:02.081] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:125:0) after 4m57s
[19:06:59.116] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:125:0)
[19:06:59.137] Build status: STARTED
[19:06:59.137] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:126:0) after 6m14s
[19:13:13.173] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:126:0)
[19:13:13.208] Build status: STARTED
[19:13:13.208] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:127:0) after 3m12s
[19:16:25.252] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:127:0)
[19:16:25.270] Build status: STARTED
[19:16:25.270] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:128:0) after 9m45s
[19:26:10.300] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:128:0)
[19:26:10.318] Build status: STARTED
[19:26:10.318] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:129:0) after 2m18s
[19:28:28.365] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:129:0)
[19:28:28.409] Build status: STARTED
[19:28:28.409] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:130:0) after 9m29s
[19:37:57.441] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:130:0)
[19:37:57.459] Build status: STARTED
[19:37:57.459] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:131:0) after 2m32s
[19:40:29.488] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:131:0)
[19:40:29.523] Build status: STARTED
[19:40:29.523] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:132:0) after 2m40s
[19:43:09.536] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:132:0)
[19:43:09.553] Build status: STARTED
[19:43:09.553] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:133:0) after 2m2s
[19:45:11.581] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:133:0)
[19:45:11.600] Build status: STARTED
[19:45:11.600] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:134:0) after 9m5s
[19:54:16.700] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:134:0)
[19:54:16.718] Build status: STARTED
[19:54:16.718] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:135:0) after 3m30s
[19:57:46.835] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:135:0)
[19:57:46.853] Build status: STARTED
[19:57:46.853] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:136:0) after 7m18s
[20:05:04.896] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:136:0)
[20:05:04.931] Build status: STARTED
[20:05:04.931] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:137:0) after 4m58s
[20:10:02.954] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:137:0)
[20:10:03.012] Build status: STARTED
[20:10:03.012] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:138:0) after 2m39s
[20:12:42.044] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:138:0)
[20:12:42.060] Build status: STARTED
[20:12:42.060] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:139:0) after 3m17s
[20:15:59.133] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:139:0)
[20:15:59.165] Build status: STARTED
[20:15:59.165] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:140:0) after 8m24s
[20:24:23.251] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:140:0)
[20:24:23.269] Build status: STARTED
[20:24:23.269] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:141:0) after 9m49s
[20:34:12.350] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:141:0)
[20:34:12.607] Build status: STARTED
[20:34:12.607] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8905165139525866768:142:0) after 9m19s
[20:40:26.505] Received PubSub notification, asking Buildbucket for the build status
[20:40:26.522] Build:
{
"id": "8739735077593776641",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"number": 2509,
"createdBy": "project:chromium",
"createTime": "2024-08-13T08:02:26.512267937Z",
"startTime": "2024-08-13T08:34:27.955668058Z",
"endTime": "2024-08-13T20:40:26.231208137Z",
"updateTime": "2024-08-13T20:40:26.231208137Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "791db24354146975d991a84a5391e49ba241ea4e",
"ref": "refs/heads/main"
}
}
}
[20:40:26.522] Invocation finished in 12h38m1.23687322s with status FAILED