[02:50:37.191] New invocation is queued and will start shortly
[02:50:38.256] Starting the invocation (attempt 1)
[02:50:38.279] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/c0390bcd64ba1fd6594fbc9f6246a1649662d683"
[02:50:38.279] Popped gitiles commit info from properties and tags
[02:50:38.279] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[02:50:38.279] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[02:50:38.279] Buildbucket request:
{
"requestId": "8850645382616617648",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0390bcd64ba1fd6594fbc9f6246a1649662d683",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850645382616617648"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[02:50:38.839] Scheduled build:
{
"id": "8685215320756852641",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2405,
"createdBy": "project:chromium",
"createTime": "2026-04-07T02:50:38.344593568Z",
"updateTime": "2026-04-07T02:50:38.344593568Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0390bcd64ba1fd6594fbc9f6246a1649662d683",
"ref": "refs/heads/main"
}
},
"canary": true
}
[02:50:38.839] Task URL: https://cr-buildbucket.appspot.com/build/8685215320756852641
[02:50:38.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:2:0) after 9m40s
[02:50:39.730] Received PubSub notification, asking Buildbucket for the build status
[02:50:39.748] Build status: SCHEDULED
[02:51:23.193] Received PubSub notification, asking Buildbucket for the build status
[02:51:23.212] Build status: STARTED
[03:00:19.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:2:0)
[03:00:19.034] Build status: STARTED
[03:00:19.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:5:0) after 5m6s
[03:05:25.094] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:5:0)
[03:05:25.114] Build status: STARTED
[03:05:25.114] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:6:0) after 2m10s
[03:07:35.130] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:6:0)
[03:07:35.147] Build status: STARTED
[03:07:35.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:7:0) after 5m12s
[03:12:47.139] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:7:0)
[03:12:47.153] Build status: STARTED
[03:12:47.153] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:8:0) after 4m47s
[03:17:34.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:8:0)
[03:17:34.241] Build status: STARTED
[03:17:34.241] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:9:0) after 4m24s
[03:21:58.250] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:9:0)
[03:21:58.264] Build status: STARTED
[03:21:58.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:10:0) after 9m49s
[03:31:47.285] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:10:0)
[03:31:47.430] Build status: STARTED
[03:31:47.431] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:11:0) after 5m24s
[03:37:11.451] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:11:0)
[03:37:11.473] Build status: STARTED
[03:37:11.473] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:12:0) after 8m57s
[03:46:08.554] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:12:0)
[03:46:08.574] Build status: STARTED
[03:46:08.574] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:13:0) after 9m59s
[03:56:07.643] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:13:0)
[03:56:07.703] Build status: STARTED
[03:56:07.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:14:0) after 5m52s
[04:01:59.723] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:14:0)
[04:01:59.746] Build status: STARTED
[04:01:59.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:15:0) after 6m17s
[04:08:16.809] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:15:0)
[04:08:16.829] Build status: STARTED
[04:08:16.830] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:16:0) after 3m6s
[04:11:22.844] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:16:0)
[04:11:22.863] Build status: STARTED
[04:11:22.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:17:0) after 5m43s
[04:17:05.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:17:0)
[04:17:05.921] Build status: STARTED
[04:17:05.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:18:0) after 4m59s
[04:22:04.938] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:18:0)
[04:22:04.957] Build status: STARTED
[04:22:04.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:19:0) after 3m7s
[04:25:12.023] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:19:0)
[04:25:12.040] Build status: STARTED
[04:25:12.040] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:20:0) after 3m15s
[04:28:27.203] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:20:0)
[04:28:27.226] Build status: STARTED
[04:28:27.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:21:0) after 2m23s
[04:30:50.282] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:21:0)
[04:30:50.302] Build status: STARTED
[04:30:50.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:22:0) after 5m31s
[04:36:21.290] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:22:0)
[04:36:21.311] Build status: STARTED
[04:36:21.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:23:0) after 2m13s
[04:38:34.331] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:23:0)
[04:38:34.349] Build status: STARTED
[04:38:34.349] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:24:0) after 9m26s
[04:48:00.345] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:24:0)
[04:48:00.360] Build status: STARTED
[04:48:00.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:25:0) after 4m10s
[04:52:10.497] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:25:0)
[04:52:10.531] Build status: STARTED
[04:52:10.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:26:0) after 5m58s
[04:58:08.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:26:0)
[04:58:08.539] Build status: STARTED
[04:58:08.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:27:0) after 8m4s
[05:06:12.558] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:27:0)
[05:06:12.617] Build status: STARTED
[05:06:12.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:28:0) after 4m2s
[05:10:14.675] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:28:0)
[05:10:14.695] Build status: STARTED
[05:10:14.695] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:29:0) after 9m32s
[05:19:46.713] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:29:0)
[05:19:46.731] Build status: STARTED
[05:19:46.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:30:0) after 3m56s
[05:23:42.709] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:30:0)
[05:23:42.760] Build status: STARTED
[05:23:42.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:31:0) after 3m47s
[05:27:29.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:31:0)
[05:27:29.809] Build status: STARTED
[05:27:29.809] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:32:0) after 3m40s
[05:31:09.803] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:32:0)
[05:31:09.833] Build status: STARTED
[05:31:09.833] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:33:0) after 6m48s
[05:37:57.848] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:33:0)
[05:37:57.867] Build status: STARTED
[05:37:57.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:34:0) after 5m53s
[05:43:50.883] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:34:0)
[05:43:50.901] Build status: STARTED
[05:43:50.902] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:35:0) after 4m54s
[05:48:45.194] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:35:0)
[05:48:45.256] Build status: STARTED
[05:48:45.256] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:36:0) after 5m2s
[05:53:47.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:36:0)
[05:53:47.337] Build status: STARTED
[05:53:47.337] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:37:0) after 3m12s
[05:56:59.354] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:37:0)
[05:56:59.382] Build status: STARTED
[05:56:59.382] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:38:0) after 5m58s
[06:02:57.432] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:38:0)
[06:02:57.453] Build status: STARTED
[06:02:57.454] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:39:0) after 7m34s
[06:10:31.546] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:39:0)
[06:10:31.587] Build status: STARTED
[06:10:31.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:40:0) after 2m56s
[06:13:27.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:40:0)
[06:13:27.604] Build status: STARTED
[06:13:27.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:41:0) after 3m2s
[06:16:29.626] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:41:0)
[06:16:29.641] Build status: STARTED
[06:16:29.641] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:42:0) after 7m22s
[06:23:51.698] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:42:0)
[06:23:51.775] Build status: STARTED
[06:23:51.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:43:0) after 6m6s
[06:29:57.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:43:0)
[06:29:57.808] Build status: STARTED
[06:29:57.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:44:0) after 5m56s
[06:35:53.825] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:44:0)
[06:35:53.870] Build status: STARTED
[06:35:53.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:45:0) after 5m33s
[06:41:26.887] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:45:0)
[06:41:26.912] Build status: STARTED
[06:41:26.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:46:0) after 5m42s
[06:47:09.247] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:46:0)
[06:47:09.267] Build status: STARTED
[06:47:09.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:47:0) after 2m42s
[06:49:51.872] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:47:0)
[06:49:51.893] Build status: STARTED
[06:49:51.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:48:0) after 8m53s
[06:58:45.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:48:0)
[06:58:45.176] Build status: STARTED
[06:58:45.176] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:49:0) after 5m37s
[07:04:22.190] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:49:0)
[07:04:22.226] Build status: STARTED
[07:04:22.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:50:0) after 3m1s
[07:07:23.244] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:50:0)
[07:07:23.330] Build status: STARTED
[07:07:23.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:51:0) after 2m1s
[07:11:51.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:51:0)
[07:11:51.086] Build status: STARTED
[07:11:51.086] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:52:0) after 2m26s
[07:14:17.103] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:52:0)
[07:14:17.129] Build status: STARTED
[07:14:17.129] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:53:0) after 7m10s
[07:21:27.390] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:53:0)
[07:21:27.579] Build status: STARTED
[07:21:27.579] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:54:0) after 2m12s
[07:23:39.597] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:54:0)
[07:23:39.614] Build status: STARTED
[07:23:39.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:55:0) after 3m52s
[07:27:31.702] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:55:0)
[07:27:31.720] Build status: STARTED
[07:27:31.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:56:0) after 3m43s
[07:31:14.787] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:56:0)
[07:31:14.805] Build status: STARTED
[07:31:14.805] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850645382616617648:57:0) after 4m7s
[07:31:22.862] Received PubSub notification, asking Buildbucket for the build status
[07:31:22.888] Build:
{
"id": "8685215320756852641",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2405,
"createdBy": "project:chromium",
"createTime": "2026-04-07T02:50:38.344593568Z",
"startTime": "2026-04-07T02:51:23.032580311Z",
"endTime": "2026-04-07T07:31:22.612107790Z",
"updateTime": "2026-04-07T07:31:22.612107790Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c0390bcd64ba1fd6594fbc9f6246a1649662d683",
"ref": "refs/heads/main"
}
},
"canary": true
}
[07:31:22.888] Invocation finished in 4h40m45.702116696s with status FAILED