[16:33:31.922] New invocation is queued and will start shortly
[16:33:32.994] Starting the invocation (attempt 1)
[16:33:33.019] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/9e5d2a8555e3bc795a1bea5eedb88babdfc27a3a"
[16:33:33.019] Popped gitiles commit info from properties and tags
[16:33:33.019] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[16:33:33.019] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[16:33:33.019] Buildbucket request:
{
"requestId": "8854398682051354448",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9e5d2a8555e3bc795a1bea5eedb88babdfc27a3a",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8854398682051354448"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[16:33:33.485] Scheduled build:
{
"id": "8688968620187464369",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1416,
"createdBy": "project:chromium",
"createTime": "2026-02-24T16:33:33.075822675Z",
"updateTime": "2026-02-24T16:33:33.075822675Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9e5d2a8555e3bc795a1bea5eedb88babdfc27a3a",
"ref": "refs/heads/main"
}
}
}
[16:33:33.485] Task URL: https://cr-buildbucket.appspot.com/build/8688968620187464369
[16:33:33.485] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:2:0) after 5m10s
[16:33:34.446] Received PubSub notification, asking Buildbucket for the build status
[16:33:34.466] Build status: SCHEDULED
[16:33:36.847] Received PubSub notification, asking Buildbucket for the build status
[16:33:36.862] Build status: STARTED
[16:38:43.508] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:2:0)
[16:38:43.525] Build status: STARTED
[16:38:43.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:5:0) after 7m25s
[16:46:08.592] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:5:0)
[16:46:08.687] Build status: STARTED
[16:46:08.687] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:6:0) after 1m6s
[16:47:14.758] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:6:0)
[16:47:14.785] Build status: STARTED
[16:47:14.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:7:0) after 9m24s
[16:56:38.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:7:0)
[16:56:38.809] Build status: STARTED
[16:56:38.809] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:8:0) after 1m9s
[16:57:47.889] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:8:0)
[16:57:47.933] Build status: STARTED
[16:57:47.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:9:0) after 9m11s
[17:06:58.923] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:9:0)
[17:06:58.941] Build status: STARTED
[17:06:58.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:10:0) after 4m25s
[17:11:24.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:10:0)
[17:11:24.027] Build status: STARTED
[17:11:24.027] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:11:0) after 6m34s
[17:17:58.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:11:0)
[17:17:58.079] Build status: STARTED
[17:17:58.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:12:0) after 7m36s
[17:25:34.129] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:12:0)
[17:25:34.214] Build status: STARTED
[17:25:34.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:13:0) after 5m21s
[17:30:55.194] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:13:0)
[17:30:55.214] Build status: STARTED
[17:30:55.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:14:0) after 8m35s
[17:39:30.230] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:14:0)
[17:39:30.246] Build status: STARTED
[17:39:30.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:15:0) after 4m5s
[17:43:35.263] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:15:0)
[17:43:35.281] Build status: STARTED
[17:43:35.281] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:16:0) after 8m35s
[17:52:10.279] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:16:0)
[17:52:10.295] Build status: STARTED
[17:52:10.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:17:0) after 3m29s
[17:55:39.512] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:17:0)
[17:55:39.618] Build status: STARTED
[17:55:39.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:18:0) after 3m56s
[17:59:35.635] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:18:0)
[17:59:35.761] Build status: STARTED
[17:59:35.761] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:19:0) after 2m59s
[18:02:34.831] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:19:0)
[18:02:34.852] Build status: STARTED
[18:02:34.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:20:0) after 6m9s
[18:08:43.873] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:20:0)
[18:08:43.890] Build status: STARTED
[18:08:43.890] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:21:0) after 2m7s
[18:10:50.899] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:21:0)
[18:10:50.911] Build status: STARTED
[18:10:50.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:22:0) after 1m44s
[18:12:34.972] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:22:0)
[18:12:34.987] Build status: STARTED
[18:12:34.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:23:0) after 7m32s
[18:20:07.003] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:23:0)
[18:20:07.090] Build status: STARTED
[18:20:07.090] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:24:0) after 7m24s
[18:27:31.106] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:24:0)
[18:27:31.119] Build status: STARTED
[18:27:31.120] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:25:0) after 9m19s
[18:36:50.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:25:0)
[18:36:50.203] Build status: STARTED
[18:36:50.203] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:26:0) after 3m55s
[18:40:45.283] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:26:0)
[18:40:45.300] Build status: STARTED
[18:40:45.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:27:0) after 7m48s
[18:48:33.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:27:0)
[18:48:33.385] Build status: STARTED
[18:48:33.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:28:0) after 4m8s
[18:52:41.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:28:0)
[18:52:41.556] Build status: STARTED
[18:52:41.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:29:0) after 9m40s
[19:02:21.643] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:29:0)
[19:02:21.669] Build status: STARTED
[19:02:21.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:30:0) after 7m56s
[19:10:17.732] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:30:0)
[19:10:17.743] Build status: STARTED
[19:10:17.743] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:31:0) after 8m6s
[19:18:23.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:31:0)
[19:18:23.786] Build status: STARTED
[19:18:23.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:32:0) after 7m13s
[19:25:36.850] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:32:0)
[19:25:36.918] Build status: STARTED
[19:25:36.918] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:33:0) after 9m24s
[19:35:00.986] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:33:0)
[19:35:01.052] Build status: STARTED
[19:35:01.052] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:34:0) after 4m26s
[19:39:27.075] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:34:0)
[19:39:27.089] Build status: STARTED
[19:39:27.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:35:0) after 6m21s
[19:45:48.166] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:35:0)
[19:45:48.192] Build status: STARTED
[19:45:48.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:36:0) after 5m42s
[19:51:30.230] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:36:0)
[19:51:30.250] Build status: STARTED
[19:51:30.250] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:37:0) after 8m57s
[20:00:27.270] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:37:0)
[20:00:27.305] Build status: STARTED
[20:00:27.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:38:0) after 5m10s
[20:05:37.284] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:38:0)
[20:05:37.300] Build status: STARTED
[20:05:37.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:39:0) after 5m44s
[20:11:21.317] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:39:0)
[20:11:21.341] Build status: STARTED
[20:11:21.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:40:0) after 1m58s
[20:13:19.356] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:40:0)
[20:13:19.383] Build status: STARTED
[20:13:19.383] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:41:0) after 9m45s
[20:23:04.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:41:0)
[20:23:04.438] Build status: STARTED
[20:23:04.438] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:42:0) after 4m42s
[20:27:46.540] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:42:0)
[20:27:46.556] Build status: STARTED
[20:27:46.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854398682051354448:43:0) after 5m20s
[20:27:55.579] Received PubSub notification, asking Buildbucket for the build status
[20:27:55.704] Build:
{
"id": "8688968620187464369",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1416,
"createdBy": "project:chromium",
"createTime": "2026-02-24T16:33:33.075822675Z",
"startTime": "2026-02-24T16:33:36.674001970Z",
"endTime": "2026-02-24T20:27:55.269794440Z",
"updateTime": "2026-02-24T20:27:55.269794440Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9e5d2a8555e3bc795a1bea5eedb88babdfc27a3a",
"ref": "refs/heads/main"
}
}
}
[20:27:55.704] Invocation finished in 3h54m23.789613674s with status FAILED