[00:53:03.176] New invocation is queued and will start shortly
[00:53:04.384] Starting the invocation (attempt 1)
[00:53:04.409] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/bb456ac464ce520ee72ce12fa6f8c73c94e17d38"
[00:53:04.409] Popped gitiles commit info from properties and tags
[00:53:04.409] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[00:53:04.409] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[00:53:04.409] Buildbucket request:
{
"requestId": "8848569049060567696",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bb456ac464ce520ee72ce12fa6f8c73c94e17d38",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848569049060567696"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[00:53:04.891] Scheduled build:
{
"id": "8683138987048397441",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2406,
"createdBy": "project:chromium",
"createTime": "2026-04-30T00:53:04.474320651Z",
"updateTime": "2026-04-30T00:53:04.474320651Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bb456ac464ce520ee72ce12fa6f8c73c94e17d38",
"ref": "refs/heads/main"
}
}
}
[00:53:04.891] Task URL: https://cr-buildbucket.appspot.com/build/8683138987048397441
[00:53:04.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:2:0) after 4m37s
[00:53:05.853] Received PubSub notification, asking Buildbucket for the build status
[00:53:05.874] Build status: SCHEDULED
[00:53:08.649] Received PubSub notification, asking Buildbucket for the build status
[00:53:08.688] Build status: STARTED
[00:57:41.953] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:2:0)
[00:57:41.976] Build status: STARTED
[00:57:41.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:5:0) after 6m38s
[01:04:20.123] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:5:0)
[01:04:20.141] Build status: STARTED
[01:04:20.142] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:6:0) after 7m1s
[01:11:21.166] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:6:0)
[01:11:21.193] Build status: STARTED
[01:11:21.193] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:7:0) after 3m49s
[01:15:10.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:7:0)
[01:15:10.799] Build status: STARTED
[01:15:10.799] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:8:0) after 4m35s
[01:19:45.817] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:8:0)
[01:19:45.877] Build status: STARTED
[01:19:45.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:9:0) after 9m19s
[01:29:04.948] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:9:0)
[01:29:04.973] Build status: STARTED
[01:29:04.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:10:0) after 6m37s
[01:35:42.179] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:10:0)
[01:35:42.204] Build status: STARTED
[01:35:42.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:11:0) after 5m52s
[01:41:34.273] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:11:0)
[01:41:34.291] Build status: STARTED
[01:41:34.291] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:12:0) after 8m35s
[01:50:09.361] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:12:0)
[01:50:09.403] Build status: STARTED
[01:50:09.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:13:0) after 9m20s
[01:59:29.467] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:13:0)
[01:59:29.533] Build status: STARTED
[01:59:29.533] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:14:0) after 1m15s
[02:00:44.767] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:14:0)
[02:00:44.797] Build status: STARTED
[02:00:44.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:15:0) after 4m28s
[02:05:12.895] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:15:0)
[02:05:12.916] Build status: STARTED
[02:05:12.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:16:0) after 7m39s
[02:12:52.077] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:16:0)
[02:12:52.105] Build status: STARTED
[02:12:52.105] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:17:0) after 1m23s
[02:14:15.491] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:17:0)
[02:14:15.508] Build status: STARTED
[02:14:15.508] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:18:0) after 8m49s
[02:23:04.695] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:18:0)
[02:23:04.713] Build status: STARTED
[02:23:04.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:19:0) after 5m37s
[02:28:41.741] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:19:0)
[02:28:41.779] Build status: STARTED
[02:28:41.779] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:20:0) after 3m42s
[02:32:23.768] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:20:0)
[02:32:23.783] Build status: STARTED
[02:32:23.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:21:0) after 5m44s
[02:38:07.807] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:21:0)
[02:38:07.828] Build status: STARTED
[02:38:07.828] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:22:0) after 9m10s
[02:47:17.837] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:22:0)
[02:47:17.858] Build status: STARTED
[02:47:17.858] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:23:0) after 9m20s
[02:56:38.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:23:0)
[02:56:38.045] Build status: STARTED
[02:56:38.045] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:24:0) after 7m19s
[03:03:57.111] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:24:0)
[03:03:57.133] Build status: STARTED
[03:03:57.133] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:25:0) after 4m48s
[03:08:45.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:25:0)
[03:08:45.210] Build status: STARTED
[03:08:45.210] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:26:0) after 2m35s
[03:11:20.402] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:26:0)
[03:11:20.421] Build status: STARTED
[03:11:20.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:27:0) after 2m58s
[03:14:18.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:27:0)
[03:14:18.460] Build status: STARTED
[03:14:18.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:28:0) after 3m9s
[03:17:27.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:28:0)
[03:17:27.503] Build status: STARTED
[03:17:27.503] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:29:0) after 8m11s
[03:25:38.635] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:29:0)
[03:25:38.664] Build status: STARTED
[03:25:38.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:30:0) after 8m30s
[03:34:08.823] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:30:0)
[03:34:08.839] Build status: STARTED
[03:34:08.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:31:0) after 6m7s
[03:40:15.939] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:31:0)
[03:40:15.959] Build status: STARTED
[03:40:15.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:32:0) after 5m43s
[03:45:58.956] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:32:0)
[03:45:58.992] Build status: STARTED
[03:45:58.992] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:33:0) after 1m25s
[03:47:24.172] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:33:0)
[03:47:24.194] Build status: STARTED
[03:47:24.194] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:34:0) after 7m50s
[03:55:14.307] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:34:0)
[03:55:14.325] Build status: STARTED
[03:55:14.325] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:35:0) after 7m45s
[04:02:59.376] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:35:0)
[04:02:59.401] Build status: STARTED
[04:02:59.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:36:0) after 9m40s
[04:12:39.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:36:0)
[04:12:39.443] Build status: STARTED
[04:12:39.443] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:37:0) after 8m41s
[04:21:20.512] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:37:0)
[04:21:20.530] Build status: STARTED
[04:21:20.530] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:38:0) after 2m50s
[04:24:10.600] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:38:0)
[04:24:10.619] Build status: STARTED
[04:24:10.619] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:39:0) after 8m7s
[04:32:17.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:39:0)
[04:32:17.675] Build status: STARTED
[04:32:17.675] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:40:0) after 2m50s
[04:35:08.009] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:40:0)
[04:35:08.030] Build status: STARTED
[04:35:08.030] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:41:0) after 6m50s
[04:41:58.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:41:0)
[04:41:58.107] Build status: STARTED
[04:41:58.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:42:0) after 8m25s
[04:50:23.201] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:42:0)
[04:50:23.224] Build status: STARTED
[04:50:23.224] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:43:0) after 6m57s
[04:57:20.294] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:43:0)
[04:57:20.336] Build status: STARTED
[04:57:20.336] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:44:0) after 3m33s
[05:00:53.354] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:44:0)
[05:00:53.379] Build status: STARTED
[05:00:53.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:45:0) after 5m14s
[05:06:07.447] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:45:0)
[05:06:07.476] Build status: STARTED
[05:06:07.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:46:0) after 8m38s
[05:14:45.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:46:0)
[05:14:45.610] Build status: STARTED
[05:14:45.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:47:0) after 6m45s
[05:21:30.718] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:47:0)
[05:21:30.774] Build status: STARTED
[05:21:30.774] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:48:0) after 7m14s
[05:28:45.053] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:48:0)
[05:28:45.080] Build status: STARTED
[05:28:45.080] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:49:0) after 9m26s
[05:38:11.150] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:49:0)
[05:38:11.211] Build status: STARTED
[05:38:11.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:50:0) after 4m41s
[05:42:52.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:50:0)
[05:42:52.296] Build status: STARTED
[05:42:52.296] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:51:0) after 4m12s
[05:47:04.369] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:51:0)
[05:47:04.397] Build status: STARTED
[05:47:04.397] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:52:0) after 7m51s
[05:54:55.417] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:52:0)
[05:54:55.466] Build status: STARTED
[05:54:55.466] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:53:0) after 9m41s
[06:04:36.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:53:0)
[06:04:36.710] Build status: STARTED
[06:04:36.710] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:54:0) after 8m4s
[06:12:40.730] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:54:0)
[06:12:40.747] Build status: STARTED
[06:12:40.747] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:55:0) after 1m42s
[06:14:22.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:55:0)
[06:14:22.973] Build status: STARTED
[06:14:22.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:56:0) after 8m26s
[06:22:48.994] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:56:0)
[06:22:49.014] Build status: STARTED
[06:22:49.014] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:57:0) after 9m48s
[06:32:37.047] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:57:0)
[06:32:37.118] Build status: STARTED
[06:32:37.118] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:58:0) after 8m37s
[06:41:14.173] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:58:0)
[06:41:14.192] Build status: STARTED
[06:41:14.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:59:0) after 5m40s
[06:46:54.182] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:59:0)
[06:46:54.201] Build status: STARTED
[06:46:54.201] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:60:0) after 5m25s
[06:52:19.186] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:60:0)
[06:52:19.209] Build status: STARTED
[06:52:19.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:61:0) after 8m29s
[07:00:48.475] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:61:0)
[07:00:48.510] Build status: STARTED
[07:00:48.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:62:0) after 3m54s
[07:04:42.585] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:62:0)
[07:04:42.615] Build status: STARTED
[07:04:42.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:63:0) after 2m25s
[07:07:07.686] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:63:0)
[07:07:07.703] Build status: STARTED
[07:07:07.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:64:0) after 7m11s
[07:14:18.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:64:0)
[07:14:18.798] Build status: STARTED
[07:14:18.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848569049060567696:65:0) after 7m44s
[07:20:15.350] Received PubSub notification, asking Buildbucket for the build status
[07:20:15.369] Build:
{
"id": "8683138987048397441",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2406,
"createdBy": "project:chromium",
"createTime": "2026-04-30T00:53:04.474320651Z",
"startTime": "2026-04-30T00:53:08.306183379Z",
"endTime": "2026-04-30T07:20:15.050386742Z",
"updateTime": "2026-04-30T07:20:15.050386742Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bb456ac464ce520ee72ce12fa6f8c73c94e17d38",
"ref": "refs/heads/main"
}
}
}
[07:20:15.369] Invocation finished in 6h27m12.197884092s with status FAILED