[21:48:07.043] New invocation is queued and will start shortly
[21:48:08.379] Starting the invocation (attempt 1)
[21:48:08.419] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d9271f850d9d73971d23d8e7085d814eae320346"
[21:48:08.419] Popped gitiles commit info from properties and tags
[21:48:08.419] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:48:08.419] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:48:08.419] Buildbucket request:
{
"requestId": "8841151732961574944",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d9271f850d9d73971d23d8e7085d814eae320346",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841151732961574944"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:48:09.031] Scheduled build:
{
"id": "8675721670772595649",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3815,
"createdBy": "project:chromium",
"createTime": "2026-07-20T21:48:08.503673100Z",
"updateTime": "2026-07-20T21:48:08.503673100Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d9271f850d9d73971d23d8e7085d814eae320346",
"ref": "refs/heads/main"
}
}
}
[21:48:09.031] Task URL: https://cr-buildbucket.appspot.com/build/8675721670772595649
[21:48:09.031] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:2:0) after 3m20s
[21:48:09.973] Received PubSub notification, asking Buildbucket for the build status
[21:48:09.992] Build status: SCHEDULED
[21:48:55.212] Received PubSub notification, asking Buildbucket for the build status
[21:48:55.229] Build status: STARTED
[21:51:29.006] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:2:0)
[21:51:29.026] Build status: STARTED
[21:51:29.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:5:0) after 5m31s
[21:57:00.049] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:5:0)
[21:57:00.063] Build status: STARTED
[21:57:00.063] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:6:0) after 6m30s
[22:03:30.082] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:6:0)
[22:03:30.104] Build status: STARTED
[22:03:30.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:7:0) after 3m3s
[22:06:33.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:7:0)
[22:06:33.285] Build status: STARTED
[22:06:33.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:8:0) after 1m30s
[22:08:03.437] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:8:0)
[22:08:03.469] Build status: STARTED
[22:08:03.469] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:9:0) after 6m57s
[22:15:00.497] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:9:0)
[22:15:00.526] Build status: STARTED
[22:15:00.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:10:0) after 5m28s
[22:20:28.548] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:10:0)
[22:20:28.572] Build status: STARTED
[22:20:28.572] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:11:0) after 9m9s
[22:29:37.594] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:11:0)
[22:29:37.624] Build status: STARTED
[22:29:37.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:12:0) after 7m3s
[22:36:40.642] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:12:0)
[22:36:40.659] Build status: STARTED
[22:36:40.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:13:0) after 6m13s
[22:42:53.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:13:0)
[22:42:53.792] Build status: STARTED
[22:42:53.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:14:0) after 1m39s
[22:44:32.814] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:14:0)
[22:44:32.847] Build status: STARTED
[22:44:32.847] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:15:0) after 4m54s
[22:49:26.869] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:15:0)
[22:49:26.942] Build status: STARTED
[22:49:26.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:16:0) after 7m21s
[22:56:47.923] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:16:0)
[22:56:47.943] Build status: STARTED
[22:56:47.943] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:17:0) after 4m50s
[23:01:38.039] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:17:0)
[23:01:38.089] Build status: STARTED
[23:01:38.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:18:0) after 9m16s
[23:10:54.113] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:18:0)
[23:10:54.150] Build status: STARTED
[23:10:54.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:19:0) after 4m9s
[23:15:03.198] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:19:0)
[23:15:03.214] Build status: STARTED
[23:15:03.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:20:0) after 7m4s
[23:22:07.285] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:20:0)
[23:22:07.337] Build status: STARTED
[23:22:07.337] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:21:0) after 3m50s
[23:25:57.365] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:21:0)
[23:25:57.401] Build status: STARTED
[23:25:57.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:22:0) after 7m10s
[23:33:07.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:22:0)
[23:33:07.439] Build status: STARTED
[23:33:07.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:23:0) after 4m59s
[23:38:06.640] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:23:0)
[23:38:06.659] Build status: STARTED
[23:38:06.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:24:0) after 8m48s
[23:46:54.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:24:0)
[23:46:54.796] Build status: STARTED
[23:46:54.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:25:0) after 8m18s
[23:55:12.917] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:25:0)
[23:55:13.009] Build status: STARTED
[23:55:13.009] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:26:0) after 4m0s
[23:59:13.061] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:26:0)
[23:59:13.084] Build status: STARTED
[23:59:13.084] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:27:0) after 4m23s
[00:03:36.090] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:27:0)
[00:03:36.108] Build status: STARTED
[00:03:36.108] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:28:0) after 7m30s
[00:11:06.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:28:0)
[00:11:06.294] Build status: STARTED
[00:11:06.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:29:0) after 1m20s
[00:12:26.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:29:0)
[00:12:26.370] Build status: STARTED
[00:12:26.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:30:0) after 7m51s
[00:20:17.442] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:30:0)
[00:20:17.466] Build status: STARTED
[00:20:17.466] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:31:0) after 3m43s
[00:24:00.479] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:31:0)
[00:24:00.495] Build status: STARTED
[00:24:00.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:32:0) after 7m47s
[00:31:47.590] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:32:0)
[00:31:47.608] Build status: STARTED
[00:31:47.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:33:0) after 9m12s
[00:40:59.682] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:33:0)
[00:40:59.786] Build status: STARTED
[00:40:59.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:34:0) after 2m24s
[00:43:23.841] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:34:0)
[00:43:23.869] Build status: STARTED
[00:43:23.869] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:35:0) after 7m12s
[00:50:35.937] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:35:0)
[00:50:35.957] Build status: STARTED
[00:50:35.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:36:0) after 5m57s
[00:56:33.060] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:36:0)
[00:56:33.080] Build status: STARTED
[00:56:33.080] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:37:0) after 3m52s
[01:00:25.150] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:37:0)
[01:00:25.166] Build status: STARTED
[01:00:25.166] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:38:0) after 8m14s
[01:08:39.394] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:38:0)
[01:08:39.441] Build status: STARTED
[01:08:39.441] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:39:0) after 9m35s
[01:18:14.602] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:39:0)
[01:18:14.619] Build status: STARTED
[01:18:14.619] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:40:0) after 3m58s
[01:22:12.642] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:40:0)
[01:22:12.685] Build status: STARTED
[01:22:12.685] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:41:0) after 9m53s
[01:32:05.739] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:41:0)
[01:32:05.765] Build status: STARTED
[01:32:05.765] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:42:0) after 9m55s
[01:42:00.784] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:42:0)
[01:42:00.804] Build status: STARTED
[01:42:00.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:43:0) after 7m49s
[01:49:49.816] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:43:0)
[01:49:49.839] Build status: STARTED
[01:49:49.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:44:0) after 3m40s
[01:53:29.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:44:0)
[01:53:29.964] Build status: STARTED
[01:53:29.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:45:0) after 5m12s
[01:58:42.048] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:45:0)
[01:58:42.119] Build status: STARTED
[01:58:42.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:46:0) after 2m49s
[02:01:31.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:46:0)
[02:01:31.187] Build status: STARTED
[02:01:31.187] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:47:0) after 5m33s
[02:07:04.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:47:0)
[02:07:04.234] Build status: STARTED
[02:07:04.234] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:48:0) after 3m45s
[02:10:49.253] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:48:0)
[02:10:49.270] Build status: STARTED
[02:10:49.270] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:49:0) after 3m17s
[02:14:06.263] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:49:0)
[02:14:06.278] Build status: STARTED
[02:14:06.278] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:50:0) after 3m2s
[02:17:08.297] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:50:0)
[02:17:08.346] Build status: STARTED
[02:17:08.346] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:51:0) after 1m30s
[02:18:38.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:51:0)
[02:18:38.400] Build status: STARTED
[02:18:38.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:52:0) after 5m29s
[02:24:07.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:52:0)
[02:24:07.451] Build status: STARTED
[02:24:07.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:53:0) after 2m53s
[02:27:00.473] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:53:0)
[02:27:00.500] Build status: STARTED
[02:27:00.500] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:54:0) after 4m36s
[02:31:36.522] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:54:0)
[02:31:36.585] Build status: STARTED
[02:31:36.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:55:0) after 4m24s
[02:36:00.633] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:55:0)
[02:36:00.651] Build status: STARTED
[02:36:00.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:56:0) after 4m56s
[02:40:56.674] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:56:0)
[02:40:56.692] Build status: STARTED
[02:40:56.692] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:57:0) after 3m53s
[02:44:49.734] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:57:0)
[02:44:49.750] Build status: STARTED
[02:44:49.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:58:0) after 1m55s
[02:46:44.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:58:0)
[02:46:44.840] Build status: STARTED
[02:46:44.840] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841151732961574944:59:0) after 4m54s
[02:51:04.957] Received PubSub notification, asking Buildbucket for the build status
[02:51:04.980] Build:
{
"id": "8675721670772595649",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3815,
"createdBy": "project:chromium",
"createTime": "2026-07-20T21:48:08.503673100Z",
"startTime": "2026-07-20T21:48:54.986317273Z",
"endTime": "2026-07-21T02:51:04.561354637Z",
"updateTime": "2026-07-21T02:51:04.561354637Z",
"status": "INFRA_FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d9271f850d9d73971d23d8e7085d814eae320346",
"ref": "refs/heads/main"
}
}
}
[02:51:04.980] Invocation finished in 5h2m57.949341667s with status FAILED