[14:01:29.249] New invocation is queued and will start shortly
[14:01:30.385] Starting the invocation (attempt 1)
[14:01:30.418] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/786fc458f2835e20dbe9af8d2710e9239dd822cb"
[14:01:30.418] Popped gitiles commit info from properties and tags
[14:01:30.418] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[14:01:30.418] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[14:01:30.418] Buildbucket request:
{
"requestId": "8893002555555938640",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Win x64 Cross Builder (reclient compare)"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "786fc458f2835e20dbe9af8d2710e9239dd822cb",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8893002555555938640"
},
{
"key": "scheduler_job_id",
"value": "chromium/Win x64 Cross Builder (reclient compare)"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[14:01:30.933] Scheduled build:
{
"id": "8727572493591628881",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Win x64 Cross Builder (reclient compare)"
},
"number": 2936,
"createdBy": "project:chromium",
"createTime": "2024-12-25T14:01:30.496809739Z",
"updateTime": "2024-12-25T14:01:30.496809739Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "786fc458f2835e20dbe9af8d2710e9239dd822cb",
"ref": "refs/heads/main"
}
}
}
[14:01:30.933] Task URL: https://cr-buildbucket.appspot.com/build/8727572493591628881
[14:01:30.933] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:2:0) after 3m16s
[14:01:31.946] Received PubSub notification, asking Buildbucket for the build status
[14:01:31.973] Build status: SCHEDULED
[14:04:47.042] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:2:0)
[14:04:47.143] Build status: SCHEDULED
[14:04:47.143] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:4:0) after 6m53s
[14:06:42.385] Received PubSub notification, asking Buildbucket for the build status
[14:06:42.412] Build status: STARTED
[14:11:40.267] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:4:0)
[14:11:40.291] Build status: STARTED
[14:11:40.291] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:6:0) after 7m7s
[14:18:47.327] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:6:0)
[14:18:47.356] Build status: STARTED
[14:18:47.356] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:7:0) after 2m5s
[14:20:52.498] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:7:0)
[14:20:52.613] Build status: STARTED
[14:20:52.613] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:8:0) after 9m58s
[14:30:50.641] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:8:0)
[14:30:50.663] Build status: STARTED
[14:30:50.663] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:9:0) after 3m55s
[14:34:45.690] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:9:0)
[14:34:45.717] Build status: STARTED
[14:34:45.717] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:10:0) after 9m27s
[14:44:13.659] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:10:0)
[14:44:13.709] Build status: STARTED
[14:44:13.709] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:11:0) after 5m54s
[14:50:08.035] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:11:0)
[14:50:08.056] Build status: STARTED
[14:50:08.056] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:12:0) after 5m55s
[14:56:03.174] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:12:0)
[14:56:03.276] Build status: STARTED
[14:56:03.276] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:13:0) after 7m38s
[15:03:41.303] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:13:0)
[15:03:41.331] Build status: STARTED
[15:03:41.331] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:14:0) after 4m7s
[15:07:48.447] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:14:0)
[15:07:48.472] Build status: STARTED
[15:07:48.472] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:15:0) after 3m1s
[15:10:49.584] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:15:0)
[15:10:49.602] Build status: STARTED
[15:10:49.602] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:16:0) after 1m57s
[15:12:46.628] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:16:0)
[15:12:46.657] Build status: STARTED
[15:12:46.657] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:17:0) after 1m12s
[15:13:58.693] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:17:0)
[15:13:58.721] Build status: STARTED
[15:13:58.721] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:18:0) after 4m17s
[15:18:15.747] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:18:0)
[15:18:15.833] Build status: STARTED
[15:18:15.833] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:19:0) after 7m58s
[15:26:13.874] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:19:0)
[15:26:13.913] Build status: STARTED
[15:26:13.913] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:20:0) after 4m2s
[15:30:16.046] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:20:0)
[15:30:16.077] Build status: STARTED
[15:30:16.077] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:21:0) after 6m22s
[15:36:38.204] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:21:0)
[15:36:38.237] Build status: STARTED
[15:36:38.237] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:22:0) after 2m16s
[15:38:54.315] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:22:0)
[15:38:54.336] Build status: STARTED
[15:38:54.336] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:23:0) after 3m36s
[15:42:30.358] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:23:0)
[15:42:30.381] Build status: STARTED
[15:42:30.381] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:24:0) after 8m19s
[15:50:49.413] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:24:0)
[15:50:49.442] Build status: STARTED
[15:50:49.442] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:25:0) after 3m19s
[15:54:08.612] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:25:0)
[15:54:08.632] Build status: STARTED
[15:54:08.632] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:26:0) after 8m10s
[16:02:18.739] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:26:0)
[16:02:18.764] Build status: STARTED
[16:02:18.764] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:27:0) after 8m40s
[16:10:59.049] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:27:0)
[16:10:59.067] Build status: STARTED
[16:10:59.067] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:28:0) after 9m20s
[16:20:19.089] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:28:0)
[16:20:19.110] Build status: STARTED
[16:20:19.110] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:29:0) after 8m2s
[16:28:21.225] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:29:0)
[16:28:21.252] Build status: STARTED
[16:28:21.252] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:30:0) after 6m42s
[16:35:03.287] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:30:0)
[16:35:03.385] Build status: STARTED
[16:35:03.385] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:31:0) after 3m58s
[16:39:01.765] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:31:0)
[16:39:01.792] Build status: STARTED
[16:39:01.792] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:32:0) after 4m47s
[16:43:48.907] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:32:0)
[16:43:48.936] Build status: STARTED
[16:43:48.936] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:33:0) after 2m2s
[16:45:51.042] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:33:0)
[16:45:51.059] Build status: STARTED
[16:45:51.060] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:34:0) after 7m45s
[16:53:36.180] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:34:0)
[16:53:36.282] Build status: STARTED
[16:53:36.282] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:35:0) after 8m1s
[17:01:37.298] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:35:0)
[17:01:37.333] Build status: STARTED
[17:01:37.333] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:36:0) after 3m28s
[17:05:05.355] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:36:0)
[17:05:05.379] Build status: STARTED
[17:05:05.379] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:37:0) after 2m12s
[17:07:17.399] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:37:0)
[17:07:17.421] Build status: STARTED
[17:07:17.421] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:38:0) after 5m3s
[17:12:20.517] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:38:0)
[17:12:20.542] Build status: STARTED
[17:12:20.542] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:39:0) after 7m54s
[17:20:14.666] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:39:0)
[17:20:14.682] Build status: STARTED
[17:20:14.682] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:40:0) after 6m52s
[17:27:06.743] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:40:0)
[17:27:06.759] Build status: STARTED
[17:27:06.760] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:41:0) after 9m57s
[17:37:04.338] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:41:0)
[17:37:04.355] Build status: STARTED
[17:37:04.355] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:42:0) after 8m19s
[17:45:23.391] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:42:0)
[17:45:23.410] Build status: STARTED
[17:45:23.410] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:43:0) after 6m1s
[17:51:24.482] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:43:0)
[17:51:24.509] Build status: STARTED
[17:51:24.509] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:44:0) after 1m26s
[17:52:50.532] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:44:0)
[17:52:50.551] Build status: STARTED
[17:52:50.552] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:45:0) after 8m25s
[18:01:15.541] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:45:0)
[18:01:15.570] Build status: STARTED
[18:01:15.570] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:46:0) after 7m9s
[18:08:24.810] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:46:0)
[18:08:24.827] Build status: STARTED
[18:08:24.827] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:47:0) after 5m58s
[18:14:23.002] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:47:0)
[18:14:23.068] Build status: STARTED
[18:14:23.068] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:48:0) after 5m52s
[18:20:15.079] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:48:0)
[18:20:15.096] Build status: STARTED
[18:20:15.096] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:49:0) after 8m11s
[18:28:26.116] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:49:0)
[18:28:26.140] Build status: STARTED
[18:28:26.140] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:50:0) after 3m29s
[18:31:55.222] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:50:0)
[18:31:55.239] Build status: STARTED
[18:31:55.239] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:51:0) after 3m18s
[18:35:13.309] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:51:0)
[18:35:13.335] Build status: STARTED
[18:35:13.335] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:52:0) after 6m31s
[18:41:44.355] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:52:0)
[18:41:44.396] Build status: STARTED
[18:41:44.396] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:53:0) after 5m5s
[18:46:49.573] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:53:0)
[18:46:49.602] Build status: STARTED
[18:46:49.602] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:54:0) after 4m20s
[18:51:10.205] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:54:0)
[18:51:10.235] Build status: STARTED
[18:51:10.235] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:55:0) after 1m21s
[18:52:31.796] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:55:0)
[18:52:31.813] Build status: STARTED
[18:52:31.813] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:56:0) after 2m47s
[18:55:18.835] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:56:0)
[18:55:18.849] Build status: STARTED
[18:55:18.849] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:57:0) after 9m13s
[19:04:32.129] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:57:0)
[19:04:32.143] Build status: STARTED
[19:04:32.143] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:58:0) after 1m23s
[19:05:55.249] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:58:0)
[19:05:55.267] Build status: STARTED
[19:05:55.267] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:59:0) after 7m52s
[19:13:47.286] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:59:0)
[19:13:47.307] Build status: STARTED
[19:13:47.307] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:60:0) after 6m29s
[19:20:16.364] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:60:0)
[19:20:16.383] Build status: STARTED
[19:20:16.383] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:61:0) after 8m6s
[19:28:22.419] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:61:0)
[19:28:22.452] Build status: STARTED
[19:28:22.452] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:62:0) after 1m53s
[19:30:15.475] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:62:0)
[19:30:15.492] Build status: STARTED
[19:30:15.492] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:63:0) after 6m50s
[19:37:05.523] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:63:0)
[19:37:05.602] Build status: STARTED
[19:37:05.602] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:64:0) after 5m40s
[19:42:45.621] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:64:0)
[19:42:45.642] Build status: STARTED
[19:42:45.642] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:65:0) after 2m27s
[19:45:12.711] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:65:0)
[19:45:12.732] Build status: STARTED
[19:45:12.732] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:66:0) after 4m16s
[19:49:28.759] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:66:0)
[19:49:28.839] Build status: STARTED
[19:49:28.839] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:67:0) after 5m35s
[19:55:03.859] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:67:0)
[19:55:03.876] Build status: STARTED
[19:55:03.876] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:68:0) after 3m58s
[19:59:01.891] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:68:0)
[19:59:01.910] Build status: STARTED
[19:59:01.910] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:69:0) after 9m42s
[20:08:43.931] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:69:0)
[20:08:43.959] Build status: STARTED
[20:08:43.959] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:70:0) after 7m43s
[20:16:26.984] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:70:0)
[20:16:27.006] Build status: STARTED
[20:16:27.006] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:71:0) after 8m46s
[20:25:13.040] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:71:0)
[20:25:13.055] Build status: STARTED
[20:25:13.055] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:72:0) after 8m4s
[20:33:17.076] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:72:0)
[20:33:17.183] Build status: STARTED
[20:33:17.183] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:73:0) after 2m52s
[20:36:09.209] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:73:0)
[20:36:09.225] Build status: STARTED
[20:36:09.226] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:74:0) after 3m42s
[20:39:51.255] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:74:0)
[20:39:51.315] Build status: STARTED
[20:39:51.315] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:75:0) after 6m46s
[20:46:37.292] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:75:0)
[20:46:37.308] Build status: STARTED
[20:46:37.308] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:76:0) after 4m21s
[20:50:58.348] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:76:0)
[20:50:58.516] Build status: STARTED
[20:50:58.516] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:77:0) after 6m56s
[20:57:54.659] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:77:0)
[20:57:54.677] Build status: STARTED
[20:57:54.677] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:78:0) after 2m25s
[21:00:19.763] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:78:0)
[21:00:19.778] Build status: STARTED
[21:00:19.778] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:79:0) after 5m1s
[21:05:20.871] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:79:0)
[21:05:20.888] Build status: STARTED
[21:05:20.888] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:80:0) after 6m44s
[21:12:04.958] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:80:0)
[21:12:04.984] Build status: STARTED
[21:12:04.984] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:81:0) after 1m21s
[21:13:26.058] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:81:0)
[21:13:26.073] Build status: STARTED
[21:13:26.073] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:82:0) after 4m1s
[21:17:27.145] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:82:0)
[21:17:27.162] Build status: STARTED
[21:17:27.162] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:83:0) after 9m44s
[21:27:11.180] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:83:0)
[21:27:11.244] Build status: STARTED
[21:27:11.244] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:84:0) after 5m49s
[21:33:00.320] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:84:0)
[21:33:00.359] Build status: STARTED
[21:33:00.359] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:85:0) after 3m14s
[21:36:14.431] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:85:0)
[21:36:14.449] Build status: STARTED
[21:36:14.449] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:86:0) after 7m6s
[21:43:20.466] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:86:0)
[21:43:20.504] Build status: STARTED
[21:43:20.504] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:87:0) after 6m29s
[21:49:49.630] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:87:0)
[21:49:49.649] Build status: STARTED
[21:49:49.649] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:88:0) after 9m55s
[21:59:44.686] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:88:0)
[21:59:44.717] Build status: STARTED
[21:59:44.717] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:89:0) after 6m40s
[22:06:24.745] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:89:0)
[22:06:24.763] Build status: STARTED
[22:06:24.763] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:90:0) after 2m0s
[22:08:24.777] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:90:0)
[22:08:24.838] Build status: STARTED
[22:08:24.838] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:91:0) after 6m35s
[22:14:59.878] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:91:0)
[22:14:59.895] Build status: STARTED
[22:14:59.895] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:92:0) after 4m23s
[22:19:22.919] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:92:0)
[22:19:22.941] Build status: STARTED
[22:19:22.941] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:93:0) after 9m49s
[22:29:12.032] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:93:0)
[22:29:12.046] Build status: STARTED
[22:29:12.046] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:94:0) after 1m27s
[22:30:39.122] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:94:0)
[22:30:39.138] Build status: STARTED
[22:30:39.138] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:95:0) after 1m4s
[22:31:43.165] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:95:0)
[22:31:43.181] Build status: STARTED
[22:31:43.181] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:96:0) after 2m36s
[22:34:19.205] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:96:0)
[22:34:19.222] Build status: STARTED
[22:34:19.222] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:97:0) after 1m35s
[22:35:54.244] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:97:0)
[22:35:54.350] Build status: STARTED
[22:35:54.350] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:98:0) after 3m13s
[22:39:07.538] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:98:0)
[22:39:07.639] Build status: STARTED
[22:39:07.639] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:99:0) after 7m23s
[22:46:30.662] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:99:0)
[22:46:30.700] Build status: STARTED
[22:46:30.700] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:100:0) after 3m55s
[22:50:25.868] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:100:0)
[22:50:25.883] Build status: STARTED
[22:50:25.883] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:101:0) after 1m51s
[22:52:16.907] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:101:0)
[22:52:16.966] Build status: STARTED
[22:52:16.966] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:102:0) after 4m24s
[22:56:41.698] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:102:0)
[22:56:41.737] Build status: STARTED
[22:56:41.737] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:103:0) after 4m5s
[23:00:46.913] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:103:0)
[23:00:46.926] Build status: STARTED
[23:00:46.926] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:104:0) after 2m50s
[23:03:37.032] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:104:0)
[23:03:37.051] Build status: STARTED
[23:03:37.051] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:105:0) after 7m0s
[23:10:37.262] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:105:0)
[23:10:37.281] Build status: STARTED
[23:10:37.281] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:106:0) after 7m31s
[23:18:08.308] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:106:0)
[23:18:08.379] Build status: STARTED
[23:18:08.379] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:107:0) after 9m10s
[23:27:18.400] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:107:0)
[23:27:18.427] Build status: STARTED
[23:27:18.427] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:108:0) after 2m51s
[23:30:09.455] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:108:0)
[23:30:09.477] Build status: STARTED
[23:30:09.477] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:109:0) after 5m10s
[23:35:19.505] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:109:0)
[23:35:19.521] Build status: STARTED
[23:35:19.521] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:110:0) after 1m5s
[23:36:24.551] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:110:0)
[23:36:24.585] Build status: STARTED
[23:36:24.585] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:111:0) after 3m35s
[23:39:59.892] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:111:0)
[23:39:59.976] Build status: STARTED
[23:39:59.976] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:112:0) after 9m1s
[23:49:01.082] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:112:0)
[23:49:01.108] Build status: STARTED
[23:49:01.109] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:113:0) after 3m23s
[23:52:24.225] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:113:0)
[23:52:24.241] Build status: STARTED
[23:52:24.241] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:114:0) after 4m3s
[23:56:27.774] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:114:0)
[23:56:27.807] Build status: STARTED
[23:56:27.807] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:115:0) after 7m52s
[00:04:21.056] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:115:0)
[00:04:21.079] Build status: STARTED
[00:04:21.080] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:116:0) after 5m42s
[00:10:03.409] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:116:0)
[00:10:03.440] Build status: STARTED
[00:10:03.440] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:117:0) after 2m46s
[00:12:49.466] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:117:0)
[00:12:49.493] Build status: STARTED
[00:12:49.493] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:118:0) after 4m3s
[00:16:52.528] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:118:0)
[00:16:52.567] Build status: STARTED
[00:16:52.567] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:119:0) after 6m32s
[00:23:24.862] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:119:0)
[00:23:24.882] Build status: STARTED
[00:23:24.882] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:120:0) after 3m11s
[00:26:35.906] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:120:0)
[00:26:35.985] Build status: STARTED
[00:26:35.985] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:121:0) after 5m19s
[00:31:55.105] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:121:0)
[00:31:55.165] Build status: STARTED
[00:31:55.165] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:122:0) after 3m2s
[00:34:57.394] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:122:0)
[00:34:57.449] Build status: STARTED
[00:34:57.449] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:123:0) after 6m40s
[00:41:37.577] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:123:0)
[00:41:37.596] Build status: STARTED
[00:41:37.596] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:124:0) after 2m27s
[00:44:04.780] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:124:0)
[00:44:04.801] Build status: STARTED
[00:44:04.801] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:125:0) after 6m33s
[00:50:37.868] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:125:0)
[00:50:37.914] Build status: STARTED
[00:50:37.914] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:126:0) after 3m7s
[00:53:44.954] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:126:0)
[00:53:45.223] Build status: STARTED
[00:53:45.223] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:127:0) after 8m33s
[01:02:18.312] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:127:0)
[01:02:18.333] Build status: STARTED
[01:02:18.333] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:128:0) after 7m19s
[01:09:37.361] Handling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:128:0)
[01:09:37.494] Build status: STARTED
[01:09:37.494] Scheduling timer "check-buildbucket-build-status" (chromium/Win x64 Cross Builder (reclient compare):8893002555555938640:129:0) after 6m1s
[01:10:48.159] Received PubSub notification, asking Buildbucket for the build status
[01:10:48.176] Build:
{
"id": "8727572493591628881",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Win x64 Cross Builder (reclient compare)"
},
"number": 2936,
"createdBy": "project:chromium",
"createTime": "2024-12-25T14:01:30.496809739Z",
"startTime": "2024-12-25T14:06:42.098517782Z",
"endTime": "2024-12-26T01:10:47.783482999Z",
"updateTime": "2024-12-26T01:10:47.783482999Z",
"status": "INFRA_FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "786fc458f2835e20dbe9af8d2710e9239dd822cb",
"ref": "refs/heads/main"
}
}
}
[01:10:48.176] Invocation finished in 11h9m18.937840818s with status FAILED