[12:24:07.812] New invocation is queued and will start shortly
[12:24:09.653] Starting the invocation (attempt 1)
[12:24:09.684] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d10011d53a45c0788cb36dacac09af48a9c31715"
[12:24:09.684] Popped gitiles commit info from properties and tags
[12:24:09.684] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[12:24:09.685] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[12:24:09.685] Buildbucket request:
{
"requestId": "8857585266885876464",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d10011d53a45c0788cb36dacac09af48a9c31715",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857585266885876464"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[12:24:10.254] Scheduled build:
{
"id": "8692155204149479281",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1214,
"createdBy": "project:chromium",
"createTime": "2026-01-20T12:24:09.793647816Z",
"updateTime": "2026-01-20T12:24:09.793647816Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d10011d53a45c0788cb36dacac09af48a9c31715",
"ref": "refs/heads/main"
}
},
"canary": true
}
[12:24:10.254] Task URL: https://cr-buildbucket.appspot.com/build/8692155204149479281
[12:24:10.254] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:2:0) after 5m15s
[12:24:11.204] Received PubSub notification, asking Buildbucket for the build status
[12:24:11.228] Build status: SCHEDULED
[12:25:19.110] Received PubSub notification, asking Buildbucket for the build status
[12:25:19.149] Build status: STARTED
[12:29:25.622] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:2:0)
[12:29:25.646] Build status: STARTED
[12:29:25.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:5:0) after 1m59s
[12:31:24.691] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:5:0)
[12:31:24.719] Build status: STARTED
[12:31:24.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:6:0) after 9m2s
[12:40:26.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:6:0)
[12:40:26.947] Build status: STARTED
[12:40:26.947] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:7:0) after 4m7s
[12:44:34.157] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:7:0)
[12:44:34.175] Build status: STARTED
[12:44:34.175] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:8:0) after 5m12s
[12:49:46.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:8:0)
[12:49:46.442] Build status: STARTED
[12:49:46.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:9:0) after 1m32s
[12:51:18.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:9:0)
[12:51:18.511] Build status: STARTED
[12:51:18.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:10:0) after 8m11s
[12:59:29.602] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:10:0)
[12:59:29.618] Build status: STARTED
[12:59:29.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:11:0) after 8m42s
[13:08:11.782] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:11:0)
[13:08:11.795] Build status: STARTED
[13:08:11.795] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:12:0) after 8m6s
[13:16:17.915] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:12:0)
[13:16:17.945] Build status: STARTED
[13:16:17.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:13:0) after 7m26s
[13:23:43.962] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:13:0)
[13:23:43.976] Build status: STARTED
[13:23:43.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:14:0) after 4m55s
[13:28:39.050] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:14:0)
[13:28:39.263] Build status: STARTED
[13:28:39.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:15:0) after 5m10s
[13:33:49.280] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:15:0)
[13:33:49.293] Build status: STARTED
[13:33:49.293] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:16:0) after 6m4s
[13:39:53.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:16:0)
[13:39:53.419] Build status: STARTED
[13:39:53.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:17:0) after 7m39s
[13:47:32.539] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:17:0)
[13:47:32.554] Build status: STARTED
[13:47:32.554] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:18:0) after 4m5s
[13:51:37.601] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:18:0)
[13:51:37.616] Build status: STARTED
[13:51:37.616] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:19:0) after 6m58s
[13:58:35.654] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:19:0)
[13:58:35.668] Build status: STARTED
[13:58:35.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:20:0) after 3m36s
[14:02:11.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:20:0)
[14:02:11.767] Build status: STARTED
[14:02:11.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:21:0) after 4m9s
[14:06:20.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:21:0)
[14:06:20.803] Build status: STARTED
[14:06:20.803] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:22:0) after 3m21s
[14:09:42.028] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:22:0)
[14:09:42.046] Build status: STARTED
[14:09:42.047] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:23:0) after 9m10s
[14:18:52.056] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:23:0)
[14:18:52.093] Build status: STARTED
[14:18:52.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:24:0) after 4m23s
[14:23:15.393] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:24:0)
[14:23:15.408] Build status: STARTED
[14:23:15.408] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:25:0) after 3m40s
[14:26:55.572] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:25:0)
[14:26:55.586] Build status: STARTED
[14:26:55.586] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:26:0) after 4m32s
[14:31:27.607] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:26:0)
[14:31:27.627] Build status: STARTED
[14:31:27.627] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:27:0) after 2m56s
[14:34:23.661] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:27:0)
[14:34:23.679] Build status: STARTED
[14:34:23.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:28:0) after 1m29s
[14:35:52.728] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:28:0)
[14:35:52.755] Build status: STARTED
[14:35:52.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:29:0) after 3m53s
[14:39:45.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:29:0)
[14:39:45.794] Build status: STARTED
[14:39:45.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:30:0) after 3m49s
[14:43:34.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:30:0)
[14:43:34.839] Build status: STARTED
[14:43:34.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:31:0) after 3m31s
[14:47:05.860] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:31:0)
[14:47:05.889] Build status: STARTED
[14:47:05.889] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:32:0) after 6m23s
[14:53:29.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:32:0)
[14:53:29.023] Build status: STARTED
[14:53:29.023] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:33:0) after 3m31s
[14:57:00.046] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:33:0)
[14:57:00.069] Build status: STARTED
[14:57:00.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:34:0) after 7m42s
[15:04:42.090] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:34:0)
[15:04:42.108] Build status: STARTED
[15:04:42.109] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:35:0) after 7m31s
[15:12:13.190] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:35:0)
[15:12:13.211] Build status: STARTED
[15:12:13.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:36:0) after 8m54s
[15:21:07.261] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:36:0)
[15:21:07.347] Build status: STARTED
[15:21:07.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:37:0) after 2m51s
[15:23:58.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:37:0)
[15:23:58.490] Build status: STARTED
[15:23:58.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:38:0) after 8m48s
[15:32:46.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:38:0)
[15:32:46.488] Build status: STARTED
[15:32:46.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:39:0) after 9m33s
[15:42:19.557] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:39:0)
[15:42:19.575] Build status: STARTED
[15:42:19.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:40:0) after 8m12s
[15:50:31.597] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:40:0)
[15:50:31.612] Build status: STARTED
[15:50:31.612] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:41:0) after 8m56s
[15:59:27.604] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:41:0)
[15:59:27.623] Build status: STARTED
[15:59:27.623] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:42:0) after 4m29s
[16:03:56.640] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:42:0)
[16:03:56.685] Build status: STARTED
[16:03:56.685] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:43:0) after 2m20s
[16:06:16.745] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:43:0)
[16:06:16.763] Build status: STARTED
[16:06:16.764] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:44:0) after 9m0s
[16:15:16.795] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:44:0)
[16:15:16.812] Build status: STARTED
[16:15:16.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:45:0) after 3m6s
[16:18:22.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:45:0)
[16:18:22.814] Build status: STARTED
[16:18:22.814] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:46:0) after 6m35s
[16:24:57.831] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:46:0)
[16:24:57.872] Build status: STARTED
[16:24:57.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:47:0) after 6m31s
[16:31:28.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:47:0)
[16:31:28.996] Build status: STARTED
[16:31:28.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:48:0) after 7m42s
[16:39:11.016] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:48:0)
[16:39:11.032] Build status: STARTED
[16:39:11.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:49:0) after 3m44s
[16:42:55.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:49:0)
[16:42:55.069] Build status: STARTED
[16:42:55.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:50:0) after 8m34s
[16:51:29.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:50:0)
[16:51:29.108] Build status: STARTED
[16:51:29.108] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:51:0) after 7m53s
[16:59:22.127] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:51:0)
[16:59:22.157] Build status: STARTED
[16:59:22.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:52:0) after 1m7s
[17:00:29.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:52:0)
[17:00:29.203] Build status: STARTED
[17:00:29.203] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:53:0) after 8m21s
[17:08:50.227] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:53:0)
[17:08:50.315] Build status: STARTED
[17:08:50.315] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:54:0) after 9m56s
[17:18:46.341] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:54:0)
[17:18:46.356] Build status: STARTED
[17:18:46.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:55:0) after 4m1s
[17:22:47.375] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:55:0)
[17:22:47.440] Build status: STARTED
[17:22:47.440] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:56:0) after 9m40s
[17:32:27.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:56:0)
[17:32:27.483] Build status: STARTED
[17:32:27.483] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857585266885876464:57:0) after 5m36s
[17:35:43.307] Received PubSub notification, asking Buildbucket for the build status
[17:35:43.320] Build:
{
"id": "8692155204149479281",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1214,
"createdBy": "project:chromium",
"createTime": "2026-01-20T12:24:09.793647816Z",
"startTime": "2026-01-20T12:25:18.956754241Z",
"endTime": "2026-01-20T17:35:42.997286221Z",
"updateTime": "2026-01-20T17:35:42.997286221Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d10011d53a45c0788cb36dacac09af48a9c31715",
"ref": "refs/heads/main"
}
},
"canary": true
}
[17:35:43.320] Invocation finished in 5h11m35.519766386s with status FAILED