[21:53:50.121] New invocation is queued and will start shortly
[21:53:51.205] Starting the invocation (attempt 1)
[21:53:51.228] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/3fc8eeea79a767de233ec31a7dc67f3213d1c786"
[21:53:51.228] Popped gitiles commit info from properties and tags
[21:53:51.228] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:53:51.228] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:53:51.228] Buildbucket request:
{
"requestId": "8848761518390690624",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3fc8eeea79a767de233ec31a7dc67f3213d1c786",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848761518390690624"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:53:51.713] Scheduled build:
{
"id": "8683331456508902017",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2487,
"createdBy": "project:chromium",
"createTime": "2026-04-27T21:53:51.292788464Z",
"updateTime": "2026-04-27T21:53:51.292788464Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3fc8eeea79a767de233ec31a7dc67f3213d1c786",
"ref": "refs/heads/main"
}
}
}
[21:53:51.713] Task URL: https://cr-buildbucket.appspot.com/build/8683331456508902017
[21:53:51.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:2:0) after 8m19s
[21:53:52.657] Received PubSub notification, asking Buildbucket for the build status
[21:53:52.681] Build status: SCHEDULED
[21:53:56.003] Received PubSub notification, asking Buildbucket for the build status
[21:53:56.040] Build status: STARTED
[22:02:10.734] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:2:0)
[22:02:10.760] Build status: STARTED
[22:02:10.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:5:0) after 7m23s
[22:09:33.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:5:0)
[22:09:33.816] Build status: STARTED
[22:09:33.816] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:6:0) after 7m50s
[22:17:23.867] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:6:0)
[22:17:23.893] Build status: STARTED
[22:17:23.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:7:0) after 3m29s
[22:20:53.126] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:7:0)
[22:20:53.152] Build status: STARTED
[22:20:53.152] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:8:0) after 2m15s
[22:23:08.164] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:8:0)
[22:23:08.182] Build status: STARTED
[22:23:08.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:9:0) after 8m35s
[22:31:43.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:9:0)
[22:31:43.256] Build status: STARTED
[22:31:43.256] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:10:0) after 3m32s
[22:35:15.335] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:10:0)
[22:35:15.353] Build status: STARTED
[22:35:15.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:11:0) after 3m12s
[22:38:27.373] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:11:0)
[22:38:27.396] Build status: STARTED
[22:38:27.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:12:0) after 5m37s
[22:44:04.409] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:12:0)
[22:44:04.427] Build status: STARTED
[22:44:04.427] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:13:0) after 7m57s
[22:52:01.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:13:0)
[22:52:01.457] Build status: STARTED
[22:52:01.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:14:0) after 1m59s
[22:54:00.510] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:14:0)
[22:54:00.532] Build status: STARTED
[22:54:00.532] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:15:0) after 1m37s
[22:55:37.664] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:15:0)
[22:55:37.679] Build status: STARTED
[22:55:37.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:16:0) after 2m31s
[22:58:08.749] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:16:0)
[22:58:08.770] Build status: STARTED
[22:58:08.770] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:17:0) after 5m17s
[23:03:25.785] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:17:0)
[23:03:25.810] Build status: STARTED
[23:03:25.810] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:18:0) after 6m54s
[23:10:19.825] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:18:0)
[23:10:19.841] Build status: STARTED
[23:10:19.841] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:19:0) after 1m8s
[23:11:27.862] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:19:0)
[23:11:27.882] Build status: STARTED
[23:11:27.883] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:20:0) after 9m24s
[23:20:51.899] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:20:0)
[23:20:51.917] Build status: STARTED
[23:20:51.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:21:0) after 3m48s
[23:24:39.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:21:0)
[23:24:40.037] Build status: STARTED
[23:24:40.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:22:0) after 2m29s
[23:27:09.143] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:22:0)
[23:27:09.213] Build status: STARTED
[23:27:09.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:23:0) after 4m23s
[23:31:32.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:23:0)
[23:31:32.245] Build status: STARTED
[23:31:32.245] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:24:0) after 4m1s
[23:35:33.261] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:24:0)
[23:35:33.282] Build status: STARTED
[23:35:33.282] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:25:0) after 1m47s
[23:37:20.324] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:25:0)
[23:37:20.344] Build status: STARTED
[23:37:20.344] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:26:0) after 6m8s
[23:43:28.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:26:0)
[23:43:28.567] Build status: STARTED
[23:43:28.567] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:27:0) after 5m41s
[23:49:09.634] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:27:0)
[23:49:09.672] Build status: STARTED
[23:49:09.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:28:0) after 5m59s
[23:55:08.687] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:28:0)
[23:55:08.744] Build status: STARTED
[23:55:08.744] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:29:0) after 2m45s
[23:57:53.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:29:0)
[23:57:53.793] Build status: STARTED
[23:57:53.793] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:30:0) after 8m56s
[00:06:49.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:30:0)
[00:06:49.830] Build status: STARTED
[00:06:49.830] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:31:0) after 4m1s
[00:10:50.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:31:0)
[00:10:50.880] Build status: STARTED
[00:10:50.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:32:0) after 7m33s
[00:18:23.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:32:0)
[00:18:23.960] Build status: STARTED
[00:18:23.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:33:0) after 4m6s
[00:22:29.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:33:0)
[00:22:29.950] Build status: STARTED
[00:22:29.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:34:0) after 6m33s
[00:29:03.090] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:34:0)
[00:29:03.183] Build status: STARTED
[00:29:03.183] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:35:0) after 7m26s
[00:36:29.198] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:35:0)
[00:36:29.218] Build status: STARTED
[00:36:29.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:36:0) after 6m58s
[00:43:27.262] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:36:0)
[00:43:27.280] Build status: STARTED
[00:43:27.280] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:37:0) after 3m47s
[00:47:14.254] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:37:0)
[00:47:14.277] Build status: STARTED
[00:47:14.277] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:38:0) after 9m6s
[00:56:20.293] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:38:0)
[00:56:20.343] Build status: STARTED
[00:56:20.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:39:0) after 9m48s
[01:06:08.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:39:0)
[01:06:08.452] Build status: STARTED
[01:06:08.452] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:40:0) after 5m45s
[01:11:53.508] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:40:0)
[01:11:53.524] Build status: STARTED
[01:11:53.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:41:0) after 4m14s
[01:16:07.563] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:41:0)
[01:16:07.621] Build status: STARTED
[01:16:07.621] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:42:0) after 1m15s
[01:17:22.712] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:42:0)
[01:17:22.731] Build status: STARTED
[01:17:22.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:43:0) after 5m53s
[01:23:15.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:43:0)
[01:23:15.777] Build status: STARTED
[01:23:15.777] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:44:0) after 3m39s
[01:26:55.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:44:0)
[01:26:55.069] Build status: STARTED
[01:26:55.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:45:0) after 7m48s
[01:34:43.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:45:0)
[01:34:43.263] Build status: STARTED
[01:34:43.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:46:0) after 9m38s
[01:44:21.328] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:46:0)
[01:44:21.346] Build status: STARTED
[01:44:21.346] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:47:0) after 4m58s
[01:49:19.400] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:47:0)
[01:49:19.450] Build status: STARTED
[01:49:19.450] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:48:0) after 1m8s
[01:50:27.464] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:48:0)
[01:50:27.483] Build status: STARTED
[01:50:27.483] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:49:0) after 5m22s
[01:55:49.625] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:49:0)
[01:55:49.722] Build status: STARTED
[01:55:49.722] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:50:0) after 2m7s
[01:57:56.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:50:0)
[01:57:56.753] Build status: STARTED
[01:57:56.753] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:51:0) after 7m56s
[02:05:52.774] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:51:0)
[02:05:52.811] Build status: STARTED
[02:05:52.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:52:0) after 2m42s
[02:08:34.824] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:52:0)
[02:08:34.856] Build status: STARTED
[02:08:34.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:53:0) after 6m21s
[02:14:55.871] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:53:0)
[02:14:55.889] Build status: STARTED
[02:14:55.889] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:54:0) after 3m59s
[02:18:54.954] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:54:0)
[02:18:54.971] Build status: STARTED
[02:18:54.971] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:55:0) after 6m4s
[02:24:58.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:55:0)
[02:24:59.010] Build status: STARTED
[02:24:59.010] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:56:0) after 6m20s
[02:31:19.101] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:56:0)
[02:31:19.136] Build status: STARTED
[02:31:19.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:57:0) after 8m50s
[02:40:09.306] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:57:0)
[02:40:09.331] Build status: STARTED
[02:40:09.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:58:0) after 3m21s
[02:43:30.350] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:58:0)
[02:43:30.369] Build status: STARTED
[02:43:30.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:59:0) after 1m34s
[02:45:04.385] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:59:0)
[02:45:04.457] Build status: STARTED
[02:45:04.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:60:0) after 3m12s
[02:48:16.532] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:60:0)
[02:48:16.549] Build status: STARTED
[02:48:16.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:61:0) after 2m42s
[02:50:58.583] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:61:0)
[02:50:58.603] Build status: STARTED
[02:50:58.603] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:62:0) after 3m39s
[02:54:37.701] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:62:0)
[02:54:37.730] Build status: STARTED
[02:54:37.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848761518390690624:63:0) after 5m39s
[02:57:23.225] Received PubSub notification, asking Buildbucket for the build status
[02:57:23.295] Build:
{
"id": "8683331456508902017",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2487,
"createdBy": "project:chromium",
"createTime": "2026-04-27T21:53:51.292788464Z",
"startTime": "2026-04-27T21:53:55.848962273Z",
"endTime": "2026-04-28T02:57:22.748087952Z",
"updateTime": "2026-04-28T02:57:22.748087952Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3fc8eeea79a767de233ec31a7dc67f3213d1c786",
"ref": "refs/heads/main"
}
}
}
[02:57:23.295] Invocation finished in 5h3m33.180497129s with status FAILED