[07:17:32.803] New invocation is queued and will start shortly
[07:17:34.056] Starting the invocation (attempt 1)
[07:17:34.081] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/0240fe13de43422984168f02c5912f731d8d5f19"
[07:17:34.081] Popped gitiles commit info from properties and tags
[07:17:34.081] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:17:34.082] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:17:34.083] Buildbucket request:
{
"requestId": "8843652622620135360",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0240fe13de43422984168f02c5912f731d8d5f19",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8843652622620135360"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:17:34.642] Scheduled build:
{
"id": "8678222560492191777",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3524,
"createdBy": "project:chromium",
"createTime": "2026-06-23T07:17:34.209590013Z",
"updateTime": "2026-06-23T07:17:34.209590013Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0240fe13de43422984168f02c5912f731d8d5f19",
"ref": "refs/heads/main"
}
}
}
[07:17:34.642] Task URL: https://cr-buildbucket.appspot.com/build/8678222560492191777
[07:17:34.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:2:0) after 6m20s
[07:17:35.650] Received PubSub notification, asking Buildbucket for the build status
[07:17:35.669] Build status: SCHEDULED
[07:17:39.099] Received PubSub notification, asking Buildbucket for the build status
[07:17:39.122] Build status: STARTED
[07:23:54.660] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:2:0)
[07:23:54.679] Build status: STARTED
[07:23:54.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:5:0) after 2m46s
[07:26:40.711] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:5:0)
[07:26:40.731] Build status: STARTED
[07:26:40.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:6:0) after 3m12s
[07:29:52.757] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:6:0)
[07:29:52.804] Build status: STARTED
[07:29:52.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:7:0) after 7m53s
[07:37:45.819] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:7:0)
[07:37:45.839] Build status: STARTED
[07:37:45.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:8:0) after 3m43s
[07:41:28.856] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:8:0)
[07:41:28.875] Build status: STARTED
[07:41:28.875] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:9:0) after 6m32s
[07:48:00.980] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:9:0)
[07:48:00.996] Build status: STARTED
[07:48:00.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:10:0) after 8m1s
[07:56:02.018] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:10:0)
[07:56:02.048] Build status: STARTED
[07:56:02.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:11:0) after 2m21s
[07:58:23.199] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:11:0)
[07:58:23.217] Build status: STARTED
[07:58:23.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:12:0) after 5m14s
[08:03:37.335] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:12:0)
[08:03:37.355] Build status: STARTED
[08:03:37.355] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:13:0) after 9m2s
[08:12:39.471] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:13:0)
[08:12:39.488] Build status: STARTED
[08:12:39.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:14:0) after 1m15s
[08:13:54.500] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:14:0)
[08:13:54.588] Build status: STARTED
[08:13:54.588] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:15:0) after 8m39s
[08:22:33.711] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:15:0)
[08:22:33.730] Build status: STARTED
[08:22:33.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:16:0) after 2m14s
[08:24:47.746] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:16:0)
[08:24:47.762] Build status: STARTED
[08:24:47.762] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:17:0) after 6m22s
[08:31:09.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:17:0)
[08:31:09.798] Build status: STARTED
[08:31:09.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:18:0) after 1m11s
[08:32:20.861] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:18:0)
[08:32:20.878] Build status: STARTED
[08:32:20.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:19:0) after 1m26s
[08:33:46.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:19:0)
[08:33:46.921] Build status: STARTED
[08:33:46.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:20:0) after 6m32s
[08:40:18.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:20:0)
[08:40:18.995] Build status: STARTED
[08:40:18.995] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:21:0) after 7m43s
[08:48:02.026] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:21:0)
[08:48:02.048] Build status: STARTED
[08:48:02.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:22:0) after 6m29s
[08:54:31.115] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:22:0)
[08:54:31.134] Build status: STARTED
[08:54:31.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:23:0) after 9m6s
[09:03:37.198] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:23:0)
[09:03:37.219] Build status: STARTED
[09:03:37.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:24:0) after 4m55s
[09:08:32.334] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:24:0)
[09:08:32.367] Build status: STARTED
[09:08:32.367] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:25:0) after 3m41s
[09:12:13.382] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:25:0)
[09:12:13.400] Build status: STARTED
[09:12:13.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:26:0) after 1m18s
[09:13:31.415] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:26:0)
[09:13:31.442] Build status: STARTED
[09:13:31.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:27:0) after 2m3s
[09:15:34.676] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:27:0)
[09:15:34.697] Build status: STARTED
[09:15:34.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:28:0) after 9m1s
[09:24:35.762] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:28:0)
[09:24:35.822] Build status: STARTED
[09:24:35.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:29:0) after 9m12s
[09:33:47.889] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:29:0)
[09:33:47.909] Build status: STARTED
[09:33:47.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:30:0) after 5m4s
[09:38:51.963] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:30:0)
[09:38:51.980] Build status: STARTED
[09:38:51.980] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:31:0) after 5m5s
[09:43:56.995] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:31:0)
[09:43:57.029] Build status: STARTED
[09:43:57.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:32:0) after 4m43s
[09:48:40.046] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:32:0)
[09:48:40.064] Build status: STARTED
[09:48:40.064] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:33:0) after 1m8s
[09:49:48.163] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:33:0)
[09:49:48.266] Build status: STARTED
[09:49:48.266] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:34:0) after 8m52s
[09:58:40.281] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:34:0)
[09:58:40.338] Build status: STARTED
[09:58:40.338] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:35:0) after 8m16s
[10:06:56.354] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:35:0)
[10:06:56.421] Build status: STARTED
[10:06:56.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:36:0) after 4m35s
[10:11:31.493] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:36:0)
[10:11:31.512] Build status: STARTED
[10:11:31.513] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:37:0) after 5m25s
[10:16:56.530] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:37:0)
[10:16:56.558] Build status: STARTED
[10:16:56.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:38:0) after 2m56s
[10:19:52.644] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:38:0)
[10:19:52.666] Build status: STARTED
[10:19:52.666] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:39:0) after 2m31s
[10:22:23.711] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:39:0)
[10:22:23.734] Build status: STARTED
[10:22:23.734] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:40:0) after 1m21s
[10:23:44.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:40:0)
[10:23:44.883] Build status: STARTED
[10:23:44.883] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:41:0) after 7m16s
[10:31:00.953] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:41:0)
[10:31:00.975] Build status: STARTED
[10:31:00.975] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:42:0) after 6m37s
[10:37:37.976] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:42:0)
[10:37:37.997] Build status: STARTED
[10:37:37.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:43:0) after 4m24s
[10:42:02.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:43:0)
[10:42:02.192] Build status: STARTED
[10:42:02.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:44:0) after 9m9s
[10:51:11.208] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:44:0)
[10:51:11.222] Build status: STARTED
[10:51:11.222] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:45:0) after 8m44s
[10:59:55.331] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:45:0)
[10:59:55.348] Build status: STARTED
[10:59:55.349] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:46:0) after 8m7s
[11:08:02.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:46:0)
[11:08:02.423] Build status: STARTED
[11:08:02.423] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:47:0) after 4m45s
[11:12:47.438] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:47:0)
[11:12:47.456] Build status: STARTED
[11:12:47.456] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:48:0) after 1m23s
[11:14:10.469] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:48:0)
[11:14:10.495] Build status: STARTED
[11:14:10.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:49:0) after 6m12s
[11:20:22.529] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:49:0)
[11:20:22.569] Build status: STARTED
[11:20:22.569] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:50:0) after 2m24s
[11:22:46.605] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:50:0)
[11:22:46.647] Build status: STARTED
[11:22:46.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:51:0) after 8m29s
[11:31:15.711] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:51:0)
[11:31:15.729] Build status: STARTED
[11:31:15.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:52:0) after 6m10s
[11:37:25.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:52:0)
[11:37:25.749] Build status: STARTED
[11:37:25.750] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:53:0) after 3m26s
[11:40:51.765] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:53:0)
[11:40:51.813] Build status: STARTED
[11:40:51.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:54:0) after 8m6s
[11:48:57.909] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:54:0)
[11:48:57.996] Build status: STARTED
[11:48:57.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:55:0) after 4m29s
[11:53:27.044] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:55:0)
[11:53:27.079] Build status: STARTED
[11:53:27.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:56:0) after 7m49s
[12:01:16.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:56:0)
[12:01:16.134] Build status: STARTED
[12:01:16.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:57:0) after 7m31s
[12:08:47.138] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:57:0)
[12:08:47.168] Build status: STARTED
[12:08:47.168] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8843652622620135360:58:0) after 8m3s
[12:14:15.890] Received PubSub notification, asking Buildbucket for the build status
[12:14:15.910] Build:
{
"id": "8678222560492191777",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3524,
"createdBy": "project:chromium",
"createTime": "2026-06-23T07:17:34.209590013Z",
"startTime": "2026-06-23T07:17:38.459850509Z",
"endTime": "2026-06-23T12:14:15.610551665Z",
"updateTime": "2026-06-23T12:14:15.610551665Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0240fe13de43422984168f02c5912f731d8d5f19",
"ref": "refs/heads/main"
}
}
}
[12:14:15.910] Invocation finished in 4h56m43.113704005s with status FAILED