[15:56:18.528] New invocation is queued and will start shortly
[15:56:19.588] Starting the invocation (attempt 1)
[15:56:19.610] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/604671125a8dc5cab351574b96650812f8677bde"
[15:56:19.610] Popped gitiles commit info from properties and tags
[15:56:19.610] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[15:56:19.610] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[15:56:19.610] Buildbucket request:
{
"requestId": "8852407890674618656",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "604671125a8dc5cab351574b96650812f8677bde",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8852407890674618656"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[15:56:20.132] Scheduled build:
{
"id": "8686977828832138353",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2132,
"createdBy": "project:chromium",
"createTime": "2026-03-18T15:56:19.660710800Z",
"updateTime": "2026-03-18T15:56:19.660710800Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "604671125a8dc5cab351574b96650812f8677bde",
"ref": "refs/heads/main"
}
},
"canary": true
}
[15:56:20.132] Task URL: https://cr-buildbucket.appspot.com/build/8686977828832138353
[15:56:20.132] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:2:0) after 1m23s
[15:56:20.480] Received PubSub notification, asking Buildbucket for the build status
[15:56:20.504] Build status: SCHEDULED
[15:57:11.931] Received PubSub notification, asking Buildbucket for the build status
[15:57:11.958] Build status: STARTED
[15:57:43.153] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:2:0)
[15:57:43.238] Build status: STARTED
[15:57:43.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:5:0) after 8m26s
[16:06:09.310] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:5:0)
[16:06:09.345] Build status: STARTED
[16:06:09.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:6:0) after 8m43s
[16:14:52.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:6:0)
[16:14:52.527] Build status: STARTED
[16:14:52.527] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:7:0) after 7m24s
[16:22:16.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:7:0)
[16:22:16.756] Build status: STARTED
[16:22:16.756] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:8:0) after 4m50s
[16:27:06.811] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:8:0)
[16:27:06.832] Build status: STARTED
[16:27:06.832] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:9:0) after 8m13s
[16:35:19.859] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:9:0)
[16:35:19.875] Build status: STARTED
[16:35:19.875] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:10:0) after 2m4s
[16:37:23.895] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:10:0)
[16:37:23.940] Build status: STARTED
[16:37:23.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:11:0) after 8m51s
[16:46:14.992] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:11:0)
[16:46:15.019] Build status: STARTED
[16:46:15.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:12:0) after 8m18s
[16:54:33.018] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:12:0)
[16:54:33.035] Build status: STARTED
[16:54:33.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:13:0) after 9m27s
[17:04:00.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:13:0)
[17:04:00.078] Build status: STARTED
[17:04:00.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:14:0) after 9m4s
[17:13:04.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:14:0)
[17:13:04.366] Build status: STARTED
[17:13:04.366] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:15:0) after 2m58s
[17:16:02.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:15:0)
[17:16:02.407] Build status: STARTED
[17:16:02.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:16:0) after 1m7s
[17:17:09.387] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:16:0)
[17:17:09.407] Build status: STARTED
[17:17:09.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:17:0) after 4m48s
[17:21:57.398] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:17:0)
[17:21:57.421] Build status: STARTED
[17:21:57.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:18:0) after 4m3s
[17:26:00.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:18:0)
[17:26:00.467] Build status: STARTED
[17:26:00.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:19:0) after 1m30s
[17:27:30.484] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:19:0)
[17:27:30.501] Build status: STARTED
[17:27:30.501] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:20:0) after 4m57s
[17:32:27.521] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:20:0)
[17:32:27.586] Build status: STARTED
[17:32:27.586] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:21:0) after 4m22s
[17:36:49.557] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:21:0)
[17:36:49.586] Build status: STARTED
[17:36:49.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:22:0) after 1m21s
[17:38:10.655] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:22:0)
[17:38:10.671] Build status: STARTED
[17:38:10.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:23:0) after 2m43s
[17:40:53.687] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:23:0)
[17:40:53.708] Build status: STARTED
[17:40:53.708] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:24:0) after 2m26s
[17:43:19.723] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:24:0)
[17:43:19.739] Build status: STARTED
[17:43:19.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:25:0) after 7m36s
[17:50:55.793] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:25:0)
[17:50:55.812] Build status: STARTED
[17:50:55.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:26:0) after 7m35s
[17:58:30.832] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:26:0)
[17:58:30.847] Build status: STARTED
[17:58:30.847] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:27:0) after 7m35s
[18:06:05.872] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:27:0)
[18:06:05.890] Build status: STARTED
[18:06:05.890] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:28:0) after 4m29s
[18:10:35.106] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:28:0)
[18:10:35.186] Build status: STARTED
[18:10:35.186] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:29:0) after 2m8s
[18:12:43.270] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:29:0)
[18:12:43.295] Build status: STARTED
[18:12:43.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:30:0) after 2m45s
[18:15:28.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:30:0)
[18:15:28.370] Build status: STARTED
[18:15:28.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:31:0) after 1m6s
[18:16:34.415] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:31:0)
[18:16:34.432] Build status: STARTED
[18:16:34.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:32:0) after 2m32s
[18:19:06.535] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:32:0)
[18:19:06.551] Build status: STARTED
[18:19:06.551] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:33:0) after 3m35s
[18:22:41.667] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:33:0)
[18:22:41.683] Build status: STARTED
[18:22:41.683] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:34:0) after 4m12s
[18:26:53.701] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:34:0)
[18:26:53.720] Build status: STARTED
[18:26:53.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:35:0) after 3m31s
[18:30:24.712] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:35:0)
[18:30:24.733] Build status: STARTED
[18:30:24.733] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:36:0) after 3m52s
[18:34:16.724] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:36:0)
[18:34:16.745] Build status: STARTED
[18:34:16.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:37:0) after 9m6s
[18:43:23.023] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:37:0)
[18:43:23.055] Build status: STARTED
[18:43:23.055] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:38:0) after 1m16s
[18:44:39.108] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:38:0)
[18:44:39.142] Build status: STARTED
[18:44:39.142] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:39:0) after 1m8s
[18:45:47.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:39:0)
[18:45:47.233] Build status: STARTED
[18:45:47.233] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:40:0) after 4m14s
[18:50:01.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:40:0)
[18:50:01.276] Build status: STARTED
[18:50:01.276] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:41:0) after 9m39s
[18:59:40.294] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:41:0)
[18:59:40.311] Build status: STARTED
[18:59:40.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:42:0) after 8m49s
[19:08:29.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:42:0)
[19:08:29.946] Build status: STARTED
[19:08:29.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:43:0) after 1m40s
[19:10:10.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:43:0)
[19:10:10.148] Build status: STARTED
[19:10:10.148] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:44:0) after 8m10s
[19:18:20.280] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:44:0)
[19:18:20.334] Build status: STARTED
[19:18:20.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:45:0) after 8m39s
[19:26:59.395] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:45:0)
[19:26:59.417] Build status: STARTED
[19:26:59.417] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:46:0) after 3m55s
[19:30:54.442] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:46:0)
[19:30:54.533] Build status: STARTED
[19:30:54.533] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:47:0) after 1m8s
[19:32:02.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:47:0)
[19:32:02.576] Build status: STARTED
[19:32:02.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:48:0) after 3m28s
[19:35:30.604] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:48:0)
[19:35:30.670] Build status: STARTED
[19:35:30.670] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:49:0) after 8m43s
[19:44:13.680] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:49:0)
[19:44:13.749] Build status: STARTED
[19:44:13.749] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:50:0) after 2m21s
[19:46:34.766] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:50:0)
[19:46:34.860] Build status: STARTED
[19:46:34.860] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:51:0) after 7m45s
[19:54:19.884] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:51:0)
[19:54:19.902] Build status: STARTED
[19:54:19.902] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:52:0) after 8m33s
[20:02:52.948] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:52:0)
[20:02:52.969] Build status: STARTED
[20:02:52.969] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:53:0) after 4m41s
[20:07:33.992] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:53:0)
[20:07:34.007] Build status: STARTED
[20:07:34.007] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:54:0) after 2m27s
[20:10:01.076] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:54:0)
[20:10:01.094] Build status: STARTED
[20:10:01.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:55:0) after 6m8s
[20:16:09.192] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:55:0)
[20:16:09.213] Build status: STARTED
[20:16:09.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:56:0) after 6m26s
[20:22:35.408] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:56:0)
[20:22:35.432] Build status: STARTED
[20:22:35.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:57:0) after 6m10s
[20:28:45.458] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:57:0)
[20:28:45.479] Build status: STARTED
[20:28:45.479] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:58:0) after 8m43s
[20:37:28.589] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:58:0)
[20:37:28.903] Build status: STARTED
[20:37:28.903] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8852407890674618656:59:0) after 5m14s
[20:41:13.306] Received PubSub notification, asking Buildbucket for the build status
[20:41:13.332] Build:
{
"id": "8686977828832138353",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2132,
"createdBy": "project:chromium",
"createTime": "2026-03-18T15:56:19.660710800Z",
"startTime": "2026-03-18T15:57:11.754880787Z",
"endTime": "2026-03-18T20:41:13.030644234Z",
"updateTime": "2026-03-18T20:41:13.030644234Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "604671125a8dc5cab351574b96650812f8677bde",
"ref": "refs/heads/main"
}
},
"canary": true
}
[20:41:13.332] Invocation finished in 4h44m54.812874204s with status FAILED