[12:49:10.058] New invocation is queued and will start shortly
[12:49:11.138] Starting the invocation (attempt 1)
[12:49:11.167] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/a675fa71fcd14c573853385fb62eb7c677b39875"
[12:49:11.167] Popped gitiles commit info from properties and tags
[12:49:11.167] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[12:49:11.167] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[12:49:11.167] Buildbucket request:
{
"requestId": "8850879516151986224",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a675fa71fcd14c573853385fb62eb7c677b39875",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850879516151986224"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[12:49:12.192] Scheduled build:
{
"id": "8685449454252775761",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2085,
"createdBy": "project:chromium",
"createTime": "2026-04-04T12:49:11.242597655Z",
"updateTime": "2026-04-04T12:49:11.242597655Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a675fa71fcd14c573853385fb62eb7c677b39875",
"ref": "refs/heads/main"
}
}
}
[12:49:12.192] Task URL: https://cr-buildbucket.appspot.com/build/8685449454252775761
[12:49:12.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:2:0) after 6m8s
[12:49:12.955] Received PubSub notification, asking Buildbucket for the build status
[12:49:12.975] Build status: SCHEDULED
[12:49:59.859] Received PubSub notification, asking Buildbucket for the build status
[12:49:59.881] Build status: STARTED
[12:55:20.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:2:0)
[12:55:20.394] Build status: STARTED
[12:55:20.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:5:0) after 9m6s
[13:04:26.367] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:5:0)
[13:04:26.386] Build status: STARTED
[13:04:26.386] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:6:0) after 9m48s
[13:14:14.402] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:6:0)
[13:14:14.498] Build status: STARTED
[13:14:14.498] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:7:0) after 6m15s
[13:20:29.513] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:7:0)
[13:20:29.535] Build status: STARTED
[13:20:29.535] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:8:0) after 1m11s
[13:21:40.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:8:0)
[13:21:40.566] Build status: STARTED
[13:21:40.567] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:9:0) after 7m24s
[13:29:04.845] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:9:0)
[13:29:04.915] Build status: STARTED
[13:29:04.915] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:10:0) after 4m48s
[13:33:52.933] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:10:0)
[13:33:52.957] Build status: STARTED
[13:33:52.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:11:0) after 2m56s
[13:36:49.103] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:11:0)
[13:36:49.126] Build status: STARTED
[13:36:49.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:12:0) after 1m17s
[13:38:06.140] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:12:0)
[13:38:06.164] Build status: STARTED
[13:38:06.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:13:0) after 2m45s
[13:40:51.177] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:13:0)
[13:40:51.229] Build status: STARTED
[13:40:51.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:14:0) after 7m0s
[13:47:51.271] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:14:0)
[13:47:51.289] Build status: STARTED
[13:47:51.289] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:15:0) after 5m39s
[13:53:30.305] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:15:0)
[13:53:30.323] Build status: STARTED
[13:53:30.323] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:16:0) after 2m52s
[13:56:22.375] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:16:0)
[13:56:22.434] Build status: STARTED
[13:56:22.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:17:0) after 2m37s
[13:58:59.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:17:0)
[13:58:59.509] Build status: STARTED
[13:58:59.509] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:18:0) after 7m14s
[14:06:13.614] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:18:0)
[14:06:13.642] Build status: STARTED
[14:06:13.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:19:0) after 3m43s
[14:09:56.656] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:19:0)
[14:09:56.674] Build status: STARTED
[14:09:56.674] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:20:0) after 5m22s
[14:15:18.907] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:20:0)
[14:15:18.924] Build status: STARTED
[14:15:18.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:21:0) after 5m30s
[14:20:49.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:21:0)
[14:20:49.419] Build status: STARTED
[14:20:49.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:22:0) after 9m46s
[14:30:35.600] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:22:0)
[14:30:35.614] Build status: STARTED
[14:30:35.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:23:0) after 8m16s
[14:38:51.630] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:23:0)
[14:38:51.647] Build status: STARTED
[14:38:51.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:24:0) after 3m16s
[14:42:07.671] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:24:0)
[14:42:07.719] Build status: STARTED
[14:42:07.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:25:0) after 4m46s
[14:46:53.772] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:25:0)
[14:46:53.796] Build status: STARTED
[14:46:53.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:26:0) after 5m51s
[14:52:44.952] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:26:0)
[14:52:44.965] Build status: STARTED
[14:52:44.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:27:0) after 6m10s
[14:58:55.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:27:0)
[14:58:55.262] Build status: STARTED
[14:58:55.262] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:28:0) after 8m40s
[15:07:35.281] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:28:0)
[15:07:35.298] Build status: STARTED
[15:07:35.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:29:0) after 5m34s
[15:13:09.314] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:29:0)
[15:13:09.331] Build status: STARTED
[15:13:09.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:30:0) after 6m40s
[15:19:49.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:30:0)
[15:19:49.439] Build status: STARTED
[15:19:49.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:31:0) after 3m35s
[15:23:24.465] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:31:0)
[15:23:24.522] Build status: STARTED
[15:23:24.523] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:32:0) after 8m57s
[15:32:21.673] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:32:0)
[15:32:21.690] Build status: STARTED
[15:32:21.690] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:33:0) after 1m54s
[15:34:15.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:33:0)
[15:34:15.804] Build status: STARTED
[15:34:15.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:34:0) after 4m27s
[15:38:42.820] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:34:0)
[15:38:42.878] Build status: STARTED
[15:38:42.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:35:0) after 8m7s
[15:46:50.197] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:35:0)
[15:46:50.228] Build status: STARTED
[15:46:50.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:36:0) after 7m41s
[15:54:31.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:36:0)
[15:54:31.339] Build status: STARTED
[15:54:31.340] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:37:0) after 3m3s
[15:57:34.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:37:0)
[15:57:34.507] Build status: STARTED
[15:57:34.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:38:0) after 2m54s
[16:00:28.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:38:0)
[16:00:28.796] Build status: STARTED
[16:00:28.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:39:0) after 8m57s
[16:09:25.814] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:39:0)
[16:09:25.946] Build status: STARTED
[16:09:25.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:40:0) after 5m39s
[16:15:04.980] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:40:0)
[16:15:04.998] Build status: STARTED
[16:15:04.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:41:0) after 5m41s
[16:20:46.103] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:41:0)
[16:20:46.147] Build status: STARTED
[16:20:46.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:42:0) after 9m25s
[16:30:11.260] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:42:0)
[16:30:11.282] Build status: STARTED
[16:30:11.282] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:43:0) after 9m44s
[16:39:55.295] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:43:0)
[16:39:55.357] Build status: STARTED
[16:39:55.357] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:44:0) after 9m2s
[16:48:57.371] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:44:0)
[16:48:57.385] Build status: STARTED
[16:48:57.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:45:0) after 1m2s
[16:49:59.426] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:45:0)
[16:49:59.502] Build status: STARTED
[16:49:59.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:46:0) after 7m29s
[16:57:28.578] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:46:0)
[16:57:28.642] Build status: STARTED
[16:57:28.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:47:0) after 1m55s
[16:59:23.759] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:47:0)
[16:59:23.778] Build status: STARTED
[16:59:23.778] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:48:0) after 6m22s
[17:05:45.936] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:48:0)
[17:05:46.001] Build status: STARTED
[17:05:46.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:49:0) after 1m57s
[17:07:43.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:49:0)
[17:07:43.047] Build status: STARTED
[17:07:43.047] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:50:0) after 2m39s
[17:10:22.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:50:0)
[17:10:22.172] Build status: STARTED
[17:10:22.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:51:0) after 4m55s
[17:15:17.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:51:0)
[17:15:17.230] Build status: STARTED
[17:15:17.230] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:52:0) after 7m31s
[17:22:48.305] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:52:0)
[17:22:48.322] Build status: STARTED
[17:22:48.322] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:53:0) after 3m52s
[17:26:40.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:53:0)
[17:26:40.487] Build status: STARTED
[17:26:40.487] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:54:0) after 8m39s
[17:35:19.463] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:54:0)
[17:35:19.485] Build status: STARTED
[17:35:19.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:55:0) after 9m6s
[17:44:25.502] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:55:0)
[17:44:25.523] Build status: STARTED
[17:44:25.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:56:0) after 6m35s
[17:51:00.803] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:56:0)
[17:51:00.820] Build status: STARTED
[17:51:00.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:57:0) after 5m39s
[17:56:39.835] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:57:0)
[17:56:39.851] Build status: STARTED
[17:56:39.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:58:0) after 4m37s
[18:01:16.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:58:0)
[18:01:16.946] Build status: STARTED
[18:01:16.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:59:0) after 6m0s
[18:07:16.999] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:59:0)
[18:07:17.027] Build status: STARTED
[18:07:17.027] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:60:0) after 5m31s
[18:12:48.089] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:60:0)
[18:12:48.106] Build status: STARTED
[18:12:48.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:61:0) after 4m50s
[18:17:38.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:61:0)
[18:17:38.306] Build status: STARTED
[18:17:38.306] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:62:0) after 2m51s
[18:20:29.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:62:0)
[18:20:29.372] Build status: STARTED
[18:20:29.372] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850879516151986224:63:0) after 6m21s
[18:24:07.663] Received PubSub notification, asking Buildbucket for the build status
[18:24:07.683] Build:
{
"id": "8685449454252775761",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2085,
"createdBy": "project:chromium",
"createTime": "2026-04-04T12:49:11.242597655Z",
"startTime": "2026-04-04T12:49:59.682151741Z",
"endTime": "2026-04-04T18:24:07.354528381Z",
"updateTime": "2026-04-04T18:24:07.354528381Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a675fa71fcd14c573853385fb62eb7c677b39875",
"ref": "refs/heads/main"
}
}
}
[18:24:07.683] Invocation finished in 5h34m57.63641711s with status FAILED