[21:42:02.936] New invocation is queued and will start shortly
[21:42:06.433] Starting the invocation (attempt 1)
[21:42:06.465] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/6a240ca9f529a4e6accb7bb302f684c03b00315c"
[21:42:06.465] Popped gitiles commit info from properties and tags
[21:42:06.465] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:42:06.466] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:42:06.466] Buildbucket request:
{
"requestId": "8938543853629315904",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6a240ca9f529a4e6accb7bb302f684c03b00315c",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8938543853629315904"
},
{
"key": "scheduler_job_id",
"value": "chromium/Windows Cross deterministic"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:42:07.087] Scheduled build:
{
"id": "8773113789179862257",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"number": 1602,
"createdBy": "project:chromium",
"createTime": "2023-08-10T21:42:06.557175313Z",
"updateTime": "2023-08-10T21:42:06.557175313Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6a240ca9f529a4e6accb7bb302f684c03b00315c",
"ref": "refs/heads/main"
}
}
}
[21:42:07.087] Task URL: https://cr-buildbucket.appspot.com/build/8773113789179862257
[21:42:07.087] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:2:0) after 8m59s
[21:42:08.016] Received PubSub notification, asking Buildbucket for the build status
[21:42:08.045] Build status: SCHEDULED
[21:42:34.697] Received PubSub notification, asking Buildbucket for the build status
[21:42:34.734] Build status: STARTED
[21:51:06.225] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:2:0)
[21:51:06.253] Build status: STARTED
[21:51:06.254] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:5:0) after 5m26s
[21:56:32.278] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:5:0)
[21:56:32.301] Build status: STARTED
[21:56:32.301] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:6:0) after 2m39s
[21:59:11.499] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:6:0)
[21:59:11.523] Build status: STARTED
[21:59:11.523] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:7:0) after 1m37s
[22:00:48.568] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:7:0)
[22:00:48.591] Build status: STARTED
[22:00:48.591] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:8:0) after 4m36s
[22:05:25.857] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:8:0)
[22:05:25.887] Build status: STARTED
[22:05:25.887] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:9:0) after 8m3s
[22:13:28.927] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:9:0)
[22:13:28.961] Build status: STARTED
[22:13:28.961] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:10:0) after 3m45s
[22:17:14.014] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:10:0)
[22:17:14.071] Build status: STARTED
[22:17:14.071] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:11:0) after 6m45s
[22:23:59.101] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:11:0)
[22:23:59.122] Build status: STARTED
[22:23:59.122] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:12:0) after 5m51s
[22:29:50.807] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:12:0)
[22:29:50.837] Build status: STARTED
[22:29:50.837] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:13:0) after 2m50s
[22:32:41.059] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:13:0)
[22:32:41.077] Build status: STARTED
[22:32:41.078] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:14:0) after 1m38s
[22:34:19.358] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:14:0)
[22:34:19.377] Build status: STARTED
[22:34:19.377] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:15:0) after 6m19s
[22:40:38.413] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:15:0)
[22:40:38.435] Build status: STARTED
[22:40:38.435] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:16:0) after 7m1s
[22:47:39.448] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:16:0)
[22:47:39.488] Build status: STARTED
[22:47:39.488] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:17:0) after 5m5s
[22:52:44.817] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:17:0)
[22:52:44.844] Build status: STARTED
[22:52:44.844] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:18:0) after 3m1s
[22:55:46.440] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:18:0)
[22:55:46.459] Build status: STARTED
[22:55:46.459] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:19:0) after 3m7s
[22:58:53.551] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:19:0)
[22:58:53.572] Build status: STARTED
[22:58:53.572] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:20:0) after 7m58s
[23:06:51.808] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:20:0)
[23:06:51.825] Build status: STARTED
[23:06:51.825] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:21:0) after 2m24s
[23:09:15.928] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:21:0)
[23:09:15.998] Build status: STARTED
[23:09:15.998] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:22:0) after 2m37s
[23:11:53.331] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:22:0)
[23:11:53.359] Build status: STARTED
[23:11:53.359] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:23:0) after 8m17s
[23:20:10.603] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:23:0)
[23:20:10.628] Build status: STARTED
[23:20:10.628] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:24:0) after 5m20s
[23:25:30.661] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:24:0)
[23:25:30.693] Build status: STARTED
[23:25:30.693] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:25:0) after 3m32s
[23:29:02.878] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:25:0)
[23:29:02.898] Build status: STARTED
[23:29:02.898] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:26:0) after 1m29s
[23:30:31.987] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:26:0)
[23:30:32.039] Build status: STARTED
[23:30:32.039] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:27:0) after 7m52s
[23:38:24.455] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:27:0)
[23:38:24.480] Build status: STARTED
[23:38:24.480] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:28:0) after 4m27s
[23:42:51.470] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:28:0)
[23:42:51.488] Build status: STARTED
[23:42:51.488] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:29:0) after 4m58s
[23:47:49.681] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:29:0)
[23:47:49.700] Build status: STARTED
[23:47:49.700] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:30:0) after 5m50s
[23:53:39.726] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:30:0)
[23:53:39.750] Build status: STARTED
[23:53:39.750] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:31:0) after 7m50s
[00:01:29.954] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:31:0)
[00:01:30.399] Build status: STARTED
[00:01:30.399] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:32:0) after 1m27s
[00:02:57.463] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:32:0)
[00:02:57.523] Build status: STARTED
[00:02:57.523] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:33:0) after 7m34s
[00:10:31.671] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:33:0)
[00:10:31.690] Build status: STARTED
[00:10:31.690] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:34:0) after 1m38s
[00:12:09.730] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:34:0)
[00:12:09.771] Build status: STARTED
[00:12:09.771] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:35:0) after 2m59s
[00:15:08.788] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:35:0)
[00:15:08.824] Build status: STARTED
[00:15:08.824] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:36:0) after 4m5s
[00:19:13.882] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:36:0)
[00:19:13.900] Build status: STARTED
[00:19:13.900] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:37:0) after 2m38s
[00:21:51.982] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:37:0)
[00:21:52.021] Build status: STARTED
[00:21:52.021] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:38:0) after 3m8s
[00:25:00.048] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:38:0)
[00:25:00.107] Build status: STARTED
[00:25:00.107] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:39:0) after 1m22s
[00:26:22.132] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:39:0)
[00:26:22.167] Build status: STARTED
[00:26:22.167] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:40:0) after 9m0s
[00:35:22.195] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:40:0)
[00:35:22.213] Build status: STARTED
[00:35:22.214] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:41:0) after 9m20s
[00:44:42.354] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:41:0)
[00:44:42.376] Build status: STARTED
[00:44:42.376] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:42:0) after 5m11s
[00:49:53.618] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:42:0)
[00:49:53.637] Build status: STARTED
[00:49:53.637] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:43:0) after 3m45s
[00:53:38.853] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:43:0)
[00:53:38.872] Build status: STARTED
[00:53:38.872] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:44:0) after 7m12s
[01:00:50.912] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:44:0)
[01:00:50.956] Build status: STARTED
[01:00:50.956] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:45:0) after 3m37s
[01:04:28.052] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:45:0)
[01:04:28.096] Build status: STARTED
[01:04:28.096] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:46:0) after 4m48s
[01:09:16.124] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:46:0)
[01:09:16.146] Build status: STARTED
[01:09:16.146] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:47:0) after 9m57s
[01:19:13.278] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:47:0)
[01:19:13.295] Build status: STARTED
[01:19:13.295] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:48:0) after 2m27s
[01:21:40.323] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:48:0)
[01:21:40.344] Build status: STARTED
[01:21:40.344] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:49:0) after 6m39s
[01:28:19.429] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:49:0)
[01:28:19.456] Build status: STARTED
[01:28:19.456] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:50:0) after 3m26s
[01:31:45.691] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:50:0)
[01:31:45.711] Build status: STARTED
[01:31:45.711] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:51:0) after 9m46s
[01:41:32.019] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:51:0)
[01:41:32.039] Build status: STARTED
[01:41:32.039] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:52:0) after 2m58s
[01:44:30.245] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:52:0)
[01:44:30.266] Build status: STARTED
[01:44:30.267] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:53:0) after 2m42s
[01:47:12.292] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:53:0)
[01:47:12.329] Build status: STARTED
[01:47:12.329] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:54:0) after 6m57s
[01:54:09.367] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:54:0)
[01:54:09.385] Build status: STARTED
[01:54:09.385] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:55:0) after 5m57s
[02:00:06.392] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:55:0)
[02:00:06.418] Build status: STARTED
[02:00:06.418] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:56:0) after 3m8s
[02:03:14.689] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:56:0)
[02:03:14.714] Build status: STARTED
[02:03:14.714] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:57:0) after 5m40s
[02:08:54.862] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:57:0)
[02:08:54.893] Build status: STARTED
[02:08:54.893] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:58:0) after 1m54s
[02:10:49.395] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:58:0)
[02:10:49.413] Build status: STARTED
[02:10:49.413] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:59:0) after 8m26s
[02:19:15.765] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:59:0)
[02:19:15.787] Build status: STARTED
[02:19:15.787] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:60:0) after 9m5s
[02:28:21.113] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:60:0)
[02:28:21.133] Build status: STARTED
[02:28:21.133] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:61:0) after 6m22s
[02:34:43.583] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:61:0)
[02:34:43.608] Build status: STARTED
[02:34:43.608] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:62:0) after 9m44s
[02:44:27.640] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:62:0)
[02:44:27.666] Build status: STARTED
[02:44:27.666] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:63:0) after 8m53s
[02:53:21.006] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:63:0)
[02:53:21.026] Build status: STARTED
[02:53:21.026] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:64:0) after 5m9s
[02:58:30.057] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:64:0)
[02:58:30.091] Build status: STARTED
[02:58:30.091] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:65:0) after 7m23s
[03:05:53.236] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:65:0)
[03:05:53.255] Build status: STARTED
[03:05:53.255] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:66:0) after 1m27s
[03:07:20.327] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:66:0)
[03:07:20.346] Build status: STARTED
[03:07:20.346] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:67:0) after 9m41s
[03:17:01.365] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:67:0)
[03:17:01.386] Build status: STARTED
[03:17:01.386] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:68:0) after 7m24s
[03:24:25.699] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:68:0)
[03:24:25.718] Build status: STARTED
[03:24:25.718] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:69:0) after 3m59s
[03:28:24.759] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:69:0)
[03:28:24.867] Build status: STARTED
[03:28:24.867] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:70:0) after 8m43s
[03:37:07.906] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:70:0)
[03:37:07.931] Build status: STARTED
[03:37:07.931] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:71:0) after 1m39s
[03:38:46.961] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:71:0)
[03:38:46.978] Build status: STARTED
[03:38:46.978] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:72:0) after 5m10s
[03:43:57.254] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:72:0)
[03:43:57.270] Build status: STARTED
[03:43:57.270] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:73:0) after 1m14s
[03:45:11.431] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:73:0)
[03:45:11.457] Build status: STARTED
[03:45:11.457] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:74:0) after 9m56s
[03:55:07.489] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:74:0)
[03:55:07.521] Build status: STARTED
[03:55:07.521] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:75:0) after 2m29s
[03:57:36.558] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:75:0)
[03:57:36.638] Build status: STARTED
[03:57:36.638] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:76:0) after 8m47s
[04:06:23.703] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:76:0)
[04:06:23.758] Build status: STARTED
[04:06:23.758] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:77:0) after 6m59s
[04:13:22.842] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:77:0)
[04:13:22.892] Build status: STARTED
[04:13:22.892] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:78:0) after 5m41s
[04:19:03.914] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:78:0)
[04:19:03.931] Build status: STARTED
[04:19:03.931] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:79:0) after 6m35s
[04:25:39.101] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:79:0)
[04:25:39.132] Build status: STARTED
[04:25:39.132] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:80:0) after 5m1s
[04:30:40.166] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:80:0)
[04:30:40.199] Build status: STARTED
[04:30:40.199] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:81:0) after 7m12s
[04:37:52.334] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:81:0)
[04:37:52.357] Build status: STARTED
[04:37:52.357] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:82:0) after 1m43s
[04:39:35.408] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:82:0)
[04:39:35.427] Build status: STARTED
[04:39:35.427] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:83:0) after 9m42s
[04:49:17.489] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:83:0)
[04:49:17.509] Build status: STARTED
[04:49:17.509] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:84:0) after 4m8s
[04:53:25.537] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:84:0)
[04:53:25.567] Build status: STARTED
[04:53:25.567] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:85:0) after 1m16s
[04:54:41.603] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:85:0)
[04:54:41.619] Build status: STARTED
[04:54:41.619] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:86:0) after 4m6s
[04:58:47.721] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:86:0)
[04:58:47.738] Build status: STARTED
[04:58:47.738] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:87:0) after 9m51s
[05:08:38.872] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:87:0)
[05:08:38.902] Build status: STARTED
[05:08:38.902] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:88:0) after 8m30s
[05:17:08.965] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:88:0)
[05:17:09.001] Build status: STARTED
[05:17:09.001] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:89:0) after 7m47s
[05:24:56.261] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:89:0)
[05:24:56.284] Build status: STARTED
[05:24:56.284] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:90:0) after 8m34s
[05:33:30.862] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:90:0)
[05:33:30.883] Build status: STARTED
[05:33:30.883] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:91:0) after 4m0s
[05:37:30.916] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:91:0)
[05:37:30.937] Build status: STARTED
[05:37:30.937] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:92:0) after 4m20s
[05:41:51.064] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:92:0)
[05:41:51.086] Build status: STARTED
[05:41:51.086] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:93:0) after 3m43s
[05:45:34.114] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:93:0)
[05:45:34.133] Build status: STARTED
[05:45:34.133] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:94:0) after 3m8s
[05:48:42.180] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:94:0)
[05:48:42.199] Build status: STARTED
[05:48:42.200] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:95:0) after 5m27s
[05:54:09.504] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:95:0)
[05:54:09.533] Build status: STARTED
[05:54:09.533] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:96:0) after 8m14s
[06:02:23.566] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:96:0)
[06:02:23.584] Build status: STARTED
[06:02:23.584] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:97:0) after 4m54s
[06:07:17.773] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:97:0)
[06:07:17.793] Build status: STARTED
[06:07:17.793] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:98:0) after 2m51s
[06:10:09.528] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:98:0)
[06:10:09.568] Build status: STARTED
[06:10:09.568] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:99:0) after 8m9s
[06:18:18.630] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:99:0)
[06:18:18.647] Build status: STARTED
[06:18:18.647] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:100:0) after 9m2s
[06:27:20.833] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:100:0)
[06:27:20.867] Build status: STARTED
[06:27:20.867] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:101:0) after 3m25s
[06:30:45.913] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:101:0)
[06:30:45.938] Build status: STARTED
[06:30:45.938] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:102:0) after 7m57s
[06:38:42.979] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:102:0)
[06:38:43.048] Build status: STARTED
[06:38:43.048] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:103:0) after 6m47s
[06:45:30.073] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:103:0)
[06:45:30.101] Build status: STARTED
[06:45:30.101] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:104:0) after 5m9s
[06:50:39.136] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:104:0)
[06:50:39.155] Build status: STARTED
[06:50:39.155] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:105:0) after 3m48s
[06:54:27.326] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:105:0)
[06:54:27.346] Build status: STARTED
[06:54:27.346] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:106:0) after 1m19s
[06:55:46.460] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:106:0)
[06:55:46.551] Build status: STARTED
[06:55:46.551] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:107:0) after 1m49s
[06:57:35.953] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:107:0)
[06:57:35.972] Build status: STARTED
[06:57:35.972] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:108:0) after 4m4s
[07:01:40.092] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:108:0)
[07:01:40.188] Build status: STARTED
[07:01:40.188] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:109:0) after 7m18s
[07:08:58.378] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:109:0)
[07:08:58.403] Build status: STARTED
[07:08:58.404] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:110:0) after 2m54s
[07:11:52.568] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:110:0)
[07:11:52.643] Build status: STARTED
[07:11:52.643] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:111:0) after 6m12s
[07:18:04.902] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:111:0)
[07:18:04.925] Build status: STARTED
[07:18:04.925] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:112:0) after 2m53s
[07:20:58.189] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:112:0)
[07:20:58.216] Build status: STARTED
[07:20:58.216] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:113:0) after 8m49s
[07:29:47.262] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:113:0)
[07:29:47.282] Build status: STARTED
[07:29:47.282] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:114:0) after 3m0s
[07:32:47.484] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:114:0)
[07:32:47.503] Build status: STARTED
[07:32:47.503] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:115:0) after 1m14s
[07:34:01.742] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:115:0)
[07:34:01.815] Build status: STARTED
[07:34:01.816] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:116:0) after 8m16s
[07:42:17.986] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:116:0)
[07:42:18.007] Build status: STARTED
[07:42:18.007] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:117:0) after 9m34s
[07:51:52.123] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:117:0)
[07:51:52.163] Build status: STARTED
[07:51:52.163] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:118:0) after 2m17s
[07:54:09.201] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:118:0)
[07:54:09.221] Build status: STARTED
[07:54:09.221] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:119:0) after 3m59s
[07:58:08.247] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:119:0)
[07:58:08.267] Build status: STARTED
[07:58:08.267] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:120:0) after 9m29s
[08:07:37.501] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:120:0)
[08:07:37.521] Build status: STARTED
[08:07:37.521] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:121:0) after 3m57s
[08:11:34.823] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:121:0)
[08:11:34.848] Build status: STARTED
[08:11:34.848] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:122:0) after 4m17s
[08:15:51.906] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:122:0)
[08:15:52.001] Build status: STARTED
[08:15:52.001] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:123:0) after 3m51s
[08:19:43.039] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:123:0)
[08:19:43.060] Build status: STARTED
[08:19:43.060] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:124:0) after 7m10s
[08:26:53.182] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:124:0)
[08:26:53.232] Build status: STARTED
[08:26:53.232] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:125:0) after 6m15s
[08:33:08.268] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:125:0)
[08:33:08.297] Build status: STARTED
[08:33:08.297] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:126:0) after 2m31s
[08:35:39.383] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:126:0)
[08:35:39.403] Build status: STARTED
[08:35:39.403] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:127:0) after 8m19s
[08:43:58.636] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:127:0)
[08:43:58.655] Build status: STARTED
[08:43:58.655] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:128:0) after 5m56s
[08:49:54.771] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:128:0)
[08:49:54.796] Build status: STARTED
[08:49:54.797] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:129:0) after 1m33s
[08:51:27.851] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:129:0)
[08:51:27.924] Build status: STARTED
[08:51:27.924] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:130:0) after 4m22s
[08:55:50.073] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:130:0)
[08:55:50.127] Build status: STARTED
[08:55:50.127] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:131:0) after 3m5s
[08:58:55.200] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:131:0)
[08:58:55.219] Build status: STARTED
[08:58:55.220] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:132:0) after 7m1s
[09:05:56.246] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:132:0)
[09:05:56.264] Build status: STARTED
[09:05:56.264] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:133:0) after 2m3s
[09:07:59.290] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:133:0)
[09:07:59.312] Build status: STARTED
[09:07:59.312] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:134:0) after 7m29s
[09:15:28.627] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:134:0)
[09:15:28.645] Build status: STARTED
[09:15:28.645] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:135:0) after 8m3s
[09:23:31.678] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:135:0)
[09:23:31.698] Build status: STARTED
[09:23:31.698] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:136:0) after 8m30s
[09:32:01.955] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:136:0)
[09:32:01.974] Build status: STARTED
[09:32:01.975] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:137:0) after 5m15s
[09:37:17.260] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:137:0)
[09:37:17.302] Build status: STARTED
[09:37:17.302] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:138:0) after 9m1s
[09:46:18.512] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:138:0)
[09:46:18.532] Build status: STARTED
[09:46:18.532] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:139:0) after 3m0s
[09:49:18.985] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:139:0)
[09:49:19.022] Build status: STARTED
[09:49:19.022] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:140:0) after 9m41s
[09:59:00.050] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:140:0)
[09:59:00.073] Build status: STARTED
[09:59:00.073] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:141:0) after 5m24s
[10:04:24.143] Handling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:141:0)
[10:04:24.163] Build status: STARTED
[10:04:24.163] Scheduling timer "check-buildbucket-build-status" (chromium/Windows Cross deterministic:8938543853629315904:142:0) after 7m10s
[10:05:20.499] Received PubSub notification, asking Buildbucket for the build status
[10:05:20.571] Build:
{
"id": "8773113789179862257",
"builder": {
"project": "chromium",
"bucket": "reclient",
"builder": "Windows Cross deterministic"
},
"number": 1602,
"createdBy": "project:chromium",
"createTime": "2023-08-10T21:42:06.557175313Z",
"startTime": "2023-08-10T21:42:33.612507Z",
"endTime": "2023-08-11T10:05:19.763739Z",
"updateTime": "2023-08-11T10:05:19.763739Z",
"status": "INFRA_FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6a240ca9f529a4e6accb7bb302f684c03b00315c",
"ref": "refs/heads/main"
}
}
}
[10:05:20.571] Invocation finished in 12h23m17.640243106s with status FAILED