[16:16:16.463] New invocation is queued and will start shortly
[16:16:17.645] Starting the invocation (attempt 1)
[16:16:17.667] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/0a4735880df6637c27ee5a2581d31288b365141c"
[16:16:17.667] Popped gitiles commit info from properties and tags
[16:16:17.667] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[16:16:17.667] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[16:16:17.667] Buildbucket request:
{
"requestId": "8848963949819232944",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848963949819232944"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[16:16:18.274] Scheduled build:
{
"id": "8683533887812623665",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2384,
"createdBy": "project:chromium",
"createTime": "2026-04-25T16:16:17.757735270Z",
"updateTime": "2026-04-25T16:16:17.757735270Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
}
}
}
[16:16:18.274] Task URL: https://cr-buildbucket.appspot.com/build/8683533887812623665
[16:16:18.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:2:0) after 3m46s
[16:16:19.755] Received PubSub notification, asking Buildbucket for the build status
[16:16:19.771] Build status: SCHEDULED
[16:16:26.664] Received PubSub notification, asking Buildbucket for the build status
[16:16:26.682] Build status: STARTED
[16:20:04.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:2:0)
[16:20:04.600] Build status: STARTED
[16:20:04.600] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:5:0) after 8m25s
[16:28:29.616] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:5:0)
[16:28:29.651] Build status: STARTED
[16:28:29.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:6:0) after 1m5s
[16:29:34.856] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:6:0)
[16:29:34.900] Build status: STARTED
[16:29:34.900] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:7:0) after 1m32s
[16:31:06.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:7:0)
[16:31:06.934] Build status: STARTED
[16:31:06.934] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:8:0) after 6m35s
[16:37:42.204] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:8:0)
[16:37:42.226] Build status: STARTED
[16:37:42.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:9:0) after 3m7s
[16:40:49.337] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:9:0)
[16:40:49.361] Build status: STARTED
[16:40:49.361] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:10:0) after 4m23s
[16:45:12.397] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:10:0)
[16:45:12.501] Build status: STARTED
[16:45:12.501] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:11:0) after 1m34s
[16:46:46.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:11:0)
[16:46:46.576] Build status: STARTED
[16:46:46.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:12:0) after 8m27s
[16:55:13.589] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:12:0)
[16:55:13.606] Build status: STARTED
[16:55:13.606] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:13:0) after 1m43s
[16:56:56.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:13:0)
[16:56:56.945] Build status: STARTED
[16:56:56.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:14:0) after 7m0s
[17:03:57.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:14:0)
[17:03:57.406] Build status: STARTED
[17:03:57.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:15:0) after 5m26s
[17:09:23.703] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:15:0)
[17:09:23.721] Build status: STARTED
[17:09:23.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:16:0) after 1m24s
[17:10:47.767] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:16:0)
[17:10:47.783] Build status: STARTED
[17:10:47.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:17:0) after 5m49s
[17:16:36.838] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:17:0)
[17:16:36.859] Build status: STARTED
[17:16:36.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:18:0) after 6m4s
[17:22:40.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:18:0)
[17:22:40.917] Build status: STARTED
[17:22:40.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:19:0) after 5m29s
[17:28:10.030] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:19:0)
[17:28:10.046] Build status: STARTED
[17:28:10.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:20:0) after 9m48s
[17:37:58.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:20:0)
[17:37:58.077] Build status: STARTED
[17:37:58.077] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:21:0) after 5m31s
[17:43:29.300] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:21:0)
[17:43:29.379] Build status: STARTED
[17:43:29.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:22:0) after 8m0s
[17:51:29.423] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:22:0)
[17:51:29.439] Build status: STARTED
[17:51:29.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:23:0) after 3m3s
[17:54:32.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:23:0)
[17:54:32.759] Build status: STARTED
[17:54:32.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:24:0) after 6m22s
[18:00:54.878] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:24:0)
[18:00:54.896] Build status: STARTED
[18:00:54.896] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:25:0) after 9m26s
[18:10:20.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:25:0)
[18:10:20.976] Build status: STARTED
[18:10:20.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:26:0) after 7m59s
[18:18:20.219] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:26:0)
[18:18:20.236] Build status: STARTED
[18:18:20.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:27:0) after 6m54s
[18:25:14.355] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:27:0)
[18:25:14.398] Build status: STARTED
[18:25:14.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:28:0) after 2m47s
[18:28:01.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:28:0)
[18:28:01.431] Build status: STARTED
[18:28:01.431] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:29:0) after 7m50s
[18:35:51.448] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:29:0)
[18:35:51.463] Build status: STARTED
[18:35:51.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:30:0) after 2m18s
[18:38:09.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:30:0)
[18:38:09.541] Build status: STARTED
[18:38:09.541] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:31:0) after 8m47s
[18:46:56.588] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:31:0)
[18:46:56.604] Build status: STARTED
[18:46:56.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:32:0) after 6m59s
[18:53:55.621] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:32:0)
[18:53:55.661] Build status: STARTED
[18:53:55.661] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:33:0) after 6m43s
[19:00:38.774] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:33:0)
[19:00:38.793] Build status: STARTED
[19:00:38.793] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:34:0) after 5m27s
[19:06:05.807] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:34:0)
[19:06:05.824] Build status: STARTED
[19:06:05.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:35:0) after 5m58s
[19:12:03.839] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:35:0)
[19:12:03.857] Build status: STARTED
[19:12:03.857] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:36:0) after 6m18s
[19:18:22.338] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:36:0)
[19:18:22.369] Build status: STARTED
[19:18:22.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:37:0) after 7m14s
[19:25:36.410] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:37:0)
[19:25:36.432] Build status: STARTED
[19:25:36.433] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:38:0) after 4m52s
[19:30:28.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:38:0)
[19:30:28.560] Build status: STARTED
[19:30:28.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:39:0) after 5m59s
[19:36:27.555] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:39:0)
[19:36:27.582] Build status: STARTED
[19:36:27.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:40:0) after 6m13s
[19:42:40.597] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:40:0)
[19:42:40.613] Build status: STARTED
[19:42:40.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:41:0) after 7m27s
[19:50:07.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:41:0)
[19:50:07.663] Build status: STARTED
[19:50:07.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:42:0) after 8m38s
[19:58:45.662] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:42:0)
[19:58:45.735] Build status: STARTED
[19:58:45.735] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:43:0) after 3m59s
[20:02:44.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:43:0)
[20:02:44.844] Build status: STARTED
[20:02:44.844] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:44:0) after 7m2s
[20:09:46.881] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:44:0)
[20:09:46.904] Build status: STARTED
[20:09:46.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:45:0) after 4m57s
[20:14:44.020] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:45:0)
[20:14:44.068] Build status: STARTED
[20:14:44.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:46:0) after 6m36s
[20:21:20.331] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:46:0)
[20:21:20.413] Build status: STARTED
[20:21:20.413] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:47:0) after 6m13s
[20:27:33.464] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:47:0)
[20:27:33.481] Build status: STARTED
[20:27:33.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:48:0) after 6m13s
[20:33:46.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:48:0)
[20:33:46.517] Build status: STARTED
[20:33:46.517] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:49:0) after 4m47s
[20:38:33.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:49:0)
[20:38:33.604] Build status: STARTED
[20:38:33.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:50:0) after 9m19s
[20:47:52.829] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:50:0)
[20:47:52.849] Build status: STARTED
[20:47:52.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:51:0) after 5m13s
[20:53:05.869] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:51:0)
[20:53:05.893] Build status: STARTED
[20:53:05.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:52:0) after 8m34s
[21:01:40.392] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:52:0)
[21:01:40.418] Build status: STARTED
[21:01:40.418] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:53:0) after 4m5s
[21:05:45.522] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:53:0)
[21:05:45.565] Build status: STARTED
[21:05:45.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:54:0) after 5m43s
[21:11:28.790] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:54:0)
[21:11:28.807] Build status: STARTED
[21:11:28.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:55:0) after 2m52s
[21:14:21.045] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:55:0)
[21:14:21.060] Build status: STARTED
[21:14:21.061] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:56:0) after 3m44s
[21:18:05.127] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:56:0)
[21:18:05.143] Build status: STARTED
[21:18:05.143] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:57:0) after 3m57s
[21:22:02.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:57:0)
[21:22:02.570] Build status: STARTED
[21:22:02.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:58:0) after 2m53s
[21:24:55.585] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:58:0)
[21:24:55.627] Build status: STARTED
[21:24:55.627] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848963949819232944:59:0) after 6m9s
[21:26:33.786] Received PubSub notification, asking Buildbucket for the build status
[21:26:33.803] Build:
{
"id": "8683533887812623665",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2384,
"createdBy": "project:chromium",
"createTime": "2026-04-25T16:16:17.757735270Z",
"startTime": "2026-04-25T16:16:26.425504995Z",
"endTime": "2026-04-25T21:26:33.545336970Z",
"updateTime": "2026-04-25T21:26:33.545336970Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
}
}
}
[21:26:33.803] Invocation finished in 5h10m17.343121959s with status FAILED