[11:23:21.063] New invocation is queued and will start shortly
[11:23:22.146] Starting the invocation (attempt 1)
[11:23:22.173] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/b6ce5d73ebf67c4ad646db07800aebe7edc2a217"
[11:23:22.173] Popped gitiles commit info from properties and tags
[11:23:22.173] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:23:22.173] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:23:22.173] Buildbucket request:
{
"requestId": "8848710588062253264",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b6ce5d73ebf67c4ad646db07800aebe7edc2a217",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848710588062253264"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:23:22.760] Scheduled build:
{
"id": "8683280526175454929",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2400,
"createdBy": "project:chromium",
"createTime": "2026-04-28T11:23:22.243185647Z",
"updateTime": "2026-04-28T11:23:22.243185647Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b6ce5d73ebf67c4ad646db07800aebe7edc2a217",
"ref": "refs/heads/main"
}
}
}
[11:23:22.760] Task URL: https://cr-buildbucket.appspot.com/build/8683280526175454929
[11:23:22.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:2:0) after 6m12s
[11:23:23.737] Received PubSub notification, asking Buildbucket for the build status
[11:23:23.759] Build status: SCHEDULED
[11:23:28.020] Received PubSub notification, asking Buildbucket for the build status
[11:23:28.041] Build status: STARTED
[11:29:34.859] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:2:0)
[11:29:34.931] Build status: STARTED
[11:29:34.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:5:0) after 3m25s
[11:32:59.949] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:5:0)
[11:32:59.974] Build status: STARTED
[11:32:59.974] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:6:0) after 6m25s
[11:39:24.992] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:6:0)
[11:39:25.022] Build status: STARTED
[11:39:25.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:7:0) after 4m58s
[11:44:23.135] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:7:0)
[11:44:23.150] Build status: STARTED
[11:44:23.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:8:0) after 7m13s
[11:51:36.283] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:8:0)
[11:51:36.305] Build status: STARTED
[11:51:36.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:9:0) after 5m43s
[11:57:19.368] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:9:0)
[11:57:19.385] Build status: STARTED
[11:57:19.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:10:0) after 5m43s
[12:03:02.353] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:10:0)
[12:03:02.409] Build status: STARTED
[12:03:02.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:11:0) after 5m17s
[12:08:19.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:11:0)
[12:08:19.558] Build status: STARTED
[12:08:19.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:12:0) after 6m36s
[12:14:55.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:12:0)
[12:14:55.721] Build status: STARTED
[12:14:55.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:13:0) after 7m15s
[12:22:10.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:13:0)
[12:22:10.800] Build status: STARTED
[12:22:10.800] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:14:0) after 4m57s
[12:27:07.992] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:14:0)
[12:27:08.011] Build status: STARTED
[12:27:08.011] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:15:0) after 1m37s
[12:28:45.081] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:15:0)
[12:28:45.101] Build status: STARTED
[12:28:45.101] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:16:0) after 1m54s
[12:30:39.120] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:16:0)
[12:30:39.140] Build status: STARTED
[12:30:39.140] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:17:0) after 2m36s
[12:33:15.259] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:17:0)
[12:33:15.294] Build status: STARTED
[12:33:15.294] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:18:0) after 5m48s
[12:39:03.359] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:18:0)
[12:39:03.371] Build status: STARTED
[12:39:03.371] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:19:0) after 1m26s
[12:40:29.397] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:19:0)
[12:40:29.444] Build status: STARTED
[12:40:29.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:20:0) after 2m37s
[12:43:06.464] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:20:0)
[12:43:06.483] Build status: STARTED
[12:43:06.483] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:21:0) after 7m14s
[12:50:20.499] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:21:0)
[12:50:20.517] Build status: STARTED
[12:50:20.517] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:22:0) after 9m24s
[12:59:44.504] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:22:0)
[12:59:44.521] Build status: STARTED
[12:59:44.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:23:0) after 7m25s
[13:07:09.491] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:23:0)
[13:07:09.515] Build status: STARTED
[13:07:09.515] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:24:0) after 1m41s
[13:08:50.536] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:24:0)
[13:08:50.554] Build status: STARTED
[13:08:50.554] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:25:0) after 4m6s
[13:12:56.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:25:0)
[13:12:56.658] Build status: STARTED
[13:12:56.658] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:26:0) after 2m41s
[13:15:37.708] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:26:0)
[13:15:37.731] Build status: STARTED
[13:15:37.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:27:0) after 2m10s
[13:17:47.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:27:0)
[13:17:47.769] Build status: STARTED
[13:17:47.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:28:0) after 6m32s
[13:24:19.802] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:28:0)
[13:24:19.911] Build status: STARTED
[13:24:19.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:29:0) after 9m21s
[13:33:40.927] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:29:0)
[13:33:40.948] Build status: STARTED
[13:33:40.948] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:30:0) after 8m50s
[13:42:30.968] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:30:0)
[13:42:30.987] Build status: STARTED
[13:42:30.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:31:0) after 9m26s
[13:51:57.046] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:31:0)
[13:51:57.068] Build status: STARTED
[13:51:57.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:32:0) after 5m27s
[13:57:24.114] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:32:0)
[13:57:24.135] Build status: STARTED
[13:57:24.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:33:0) after 8m15s
[14:05:39.199] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:33:0)
[14:05:39.223] Build status: STARTED
[14:05:39.223] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:34:0) after 7m7s
[14:12:46.263] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:34:0)
[14:12:46.280] Build status: STARTED
[14:12:46.280] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:35:0) after 8m13s
[14:20:59.302] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:35:0)
[14:20:59.358] Build status: STARTED
[14:20:59.358] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:36:0) after 9m2s
[14:30:01.559] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:36:0)
[14:30:01.649] Build status: STARTED
[14:30:01.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:37:0) after 3m43s
[14:33:44.702] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:37:0)
[14:33:44.729] Build status: STARTED
[14:33:44.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:38:0) after 2m3s
[14:35:47.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:38:0)
[14:35:47.769] Build status: STARTED
[14:35:47.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:39:0) after 3m52s
[14:39:39.796] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:39:0)
[14:39:39.813] Build status: STARTED
[14:39:39.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:40:0) after 1m12s
[14:40:51.830] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:40:0)
[14:40:51.851] Build status: STARTED
[14:40:51.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:41:0) after 9m42s
[14:50:33.867] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:41:0)
[14:50:33.887] Build status: STARTED
[14:50:33.887] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:42:0) after 5m27s
[14:56:00.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:42:0)
[14:56:00.962] Build status: STARTED
[14:56:00.963] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:43:0) after 8m15s
[15:04:15.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:43:0)
[15:04:15.996] Build status: STARTED
[15:04:15.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:44:0) after 9m2s
[15:13:18.065] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:44:0)
[15:13:18.084] Build status: STARTED
[15:13:18.084] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:45:0) after 7m58s
[15:21:16.105] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:45:0)
[15:21:16.119] Build status: STARTED
[15:21:16.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:46:0) after 9m22s
[15:30:38.189] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:46:0)
[15:30:38.208] Build status: STARTED
[15:30:38.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:47:0) after 7m7s
[15:37:45.284] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:47:0)
[15:37:45.305] Build status: STARTED
[15:37:45.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:48:0) after 7m52s
[15:45:37.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:48:0)
[15:45:37.467] Build status: STARTED
[15:45:37.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:49:0) after 1m7s
[15:46:44.472] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:49:0)
[15:46:44.549] Build status: STARTED
[15:46:44.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:50:0) after 2m58s
[15:49:42.717] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:50:0)
[15:49:42.739] Build status: STARTED
[15:49:42.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:51:0) after 9m55s
[15:59:38.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:51:0)
[15:59:38.026] Build status: STARTED
[15:59:38.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:52:0) after 8m53s
[16:08:31.145] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:52:0)
[16:08:31.176] Build status: STARTED
[16:08:31.176] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:53:0) after 7m45s
[16:16:16.211] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:53:0)
[16:16:16.232] Build status: STARTED
[16:16:16.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:54:0) after 6m39s
[16:22:55.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:54:0)
[16:22:55.399] Build status: STARTED
[16:22:55.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:55:0) after 8m24s
[16:31:19.515] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:55:0)
[16:31:19.531] Build status: STARTED
[16:31:19.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:56:0) after 4m19s
[16:35:38.503] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:56:0)
[16:35:38.521] Build status: STARTED
[16:35:38.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:57:0) after 5m59s
[16:41:37.539] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:57:0)
[16:41:37.554] Build status: STARTED
[16:41:37.554] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:58:0) after 4m26s
[16:46:03.559] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:58:0)
[16:46:03.592] Build status: STARTED
[16:46:03.592] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:59:0) after 5m29s
[16:51:32.705] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:59:0)
[16:51:32.759] Build status: STARTED
[16:51:32.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:60:0) after 2m11s
[16:53:43.787] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:60:0)
[16:53:43.808] Build status: STARTED
[16:53:43.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:61:0) after 8m53s
[17:02:36.874] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:61:0)
[17:02:36.898] Build status: STARTED
[17:02:36.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:62:0) after 6m13s
[17:08:49.918] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:62:0)
[17:08:49.934] Build status: STARTED
[17:08:49.934] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:63:0) after 3m30s
[17:12:20.061] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:63:0)
[17:12:20.080] Build status: STARTED
[17:12:20.080] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:64:0) after 9m12s
[17:21:32.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:64:0)
[17:21:32.125] Build status: STARTED
[17:21:32.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:65:0) after 5m45s
[17:27:17.293] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:65:0)
[17:27:17.320] Build status: STARTED
[17:27:17.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:66:0) after 8m11s
[17:35:28.508] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:66:0)
[17:35:28.528] Build status: STARTED
[17:35:28.528] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:67:0) after 8m54s
[17:44:22.595] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:67:0)
[17:44:22.613] Build status: STARTED
[17:44:22.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848710588062253264:68:0) after 5m51s
[17:49:54.674] Received PubSub notification, asking Buildbucket for the build status
[17:49:54.695] Build:
{
"id": "8683280526175454929",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2400,
"createdBy": "project:chromium",
"createTime": "2026-04-28T11:23:22.243185647Z",
"startTime": "2026-04-28T11:23:27.750132293Z",
"endTime": "2026-04-28T17:49:54.344671132Z",
"updateTime": "2026-04-28T17:49:54.344671132Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b6ce5d73ebf67c4ad646db07800aebe7edc2a217",
"ref": "refs/heads/main"
}
}
}
[17:49:54.695] Invocation finished in 6h26m33.635390996s with status FAILED