[13:18:56.267] New invocation is queued and will start shortly
[13:18:57.310] Starting the invocation (attempt 1)
[13:18:57.332] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/4d564e1ca5359231126dccaca95689080af2da7f"
[13:18:57.332] Popped gitiles commit info from properties and tags
[13:18:57.332] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[13:18:57.332] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[13:18:57.332] Buildbucket request:
{
"requestId": "8848793912942238320",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4d564e1ca5359231126dccaca95689080af2da7f",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848793912942238320"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[13:18:57.877] Scheduled build:
{
"id": "8683363851111706081",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2396,
"createdBy": "project:chromium",
"createTime": "2026-04-27T13:18:57.392781108Z",
"updateTime": "2026-04-27T13:18:57.392781108Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4d564e1ca5359231126dccaca95689080af2da7f",
"ref": "refs/heads/main"
}
}
}
[13:18:57.877] Task URL: https://cr-buildbucket.appspot.com/build/8683363851111706081
[13:18:57.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:2:0) after 3m46s
[13:18:58.834] Received PubSub notification, asking Buildbucket for the build status
[13:18:58.860] Build status: SCHEDULED
[13:19:01.916] Received PubSub notification, asking Buildbucket for the build status
[13:19:01.932] Build status: STARTED
[13:22:43.884] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:2:0)
[13:22:43.977] Build status: STARTED
[13:22:43.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:5:0) after 9m11s
[13:31:55.069] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:5:0)
[13:31:55.102] Build status: STARTED
[13:31:55.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:6:0) after 8m41s
[13:40:36.233] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:6:0)
[13:40:36.254] Build status: STARTED
[13:40:36.254] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:7:0) after 1m4s
[13:41:40.275] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:7:0)
[13:41:40.320] Build status: STARTED
[13:41:40.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:8:0) after 3m22s
[13:45:02.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:8:0)
[13:45:02.376] Build status: STARTED
[13:45:02.376] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:9:0) after 6m8s
[13:51:10.668] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:9:0)
[13:51:10.691] Build status: STARTED
[13:51:10.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:10:0) after 3m37s
[13:54:47.708] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:10:0)
[13:54:47.727] Build status: STARTED
[13:54:47.727] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:11:0) after 9m46s
[14:04:33.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:11:0)
[14:04:33.816] Build status: STARTED
[14:04:33.816] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:12:0) after 6m37s
[14:11:10.835] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:12:0)
[14:11:10.854] Build status: STARTED
[14:11:10.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:13:0) after 7m28s
[14:18:38.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:13:0)
[14:18:38.986] Build status: STARTED
[14:18:38.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:14:0) after 9m21s
[14:28:00.005] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:14:0)
[14:28:00.044] Build status: STARTED
[14:28:00.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:15:0) after 8m15s
[14:36:15.064] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:15:0)
[14:36:15.093] Build status: STARTED
[14:36:15.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:16:0) after 8m13s
[14:44:28.199] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:16:0)
[14:44:28.225] Build status: STARTED
[14:44:28.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:17:0) after 6m9s
[14:50:37.250] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:17:0)
[14:50:37.290] Build status: STARTED
[14:50:37.291] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:18:0) after 2m13s
[14:52:50.339] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:18:0)
[14:52:50.364] Build status: STARTED
[14:52:50.364] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:19:0) after 2m11s
[14:55:01.400] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:19:0)
[14:55:01.421] Build status: STARTED
[14:55:01.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:20:0) after 1m21s
[14:56:22.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:20:0)
[14:56:22.702] Build status: STARTED
[14:56:22.702] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:21:0) after 1m7s
[14:57:29.722] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:21:0)
[14:57:29.743] Build status: STARTED
[14:57:29.743] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:22:0) after 6m35s
[15:04:04.910] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:22:0)
[15:04:04.926] Build status: STARTED
[15:04:04.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:23:0) after 1m54s
[15:05:58.972] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:23:0)
[15:05:58.997] Build status: STARTED
[15:05:58.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:24:0) after 8m31s
[15:14:30.015] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:24:0)
[15:14:30.038] Build status: STARTED
[15:14:30.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:25:0) after 2m34s
[15:17:04.058] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:25:0)
[15:17:04.076] Build status: STARTED
[15:17:04.076] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:26:0) after 1m53s
[15:18:57.160] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:26:0)
[15:18:57.176] Build status: STARTED
[15:18:57.176] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:27:0) after 8m11s
[15:27:08.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:27:0)
[15:27:08.259] Build status: STARTED
[15:27:08.260] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:28:0) after 4m30s
[15:31:38.341] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:28:0)
[15:31:38.421] Build status: STARTED
[15:31:38.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:29:0) after 3m41s
[15:35:19.423] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:29:0)
[15:35:19.489] Build status: STARTED
[15:35:19.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:30:0) after 4m42s
[15:40:01.547] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:30:0)
[15:40:01.579] Build status: STARTED
[15:40:01.579] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:31:0) after 8m14s
[15:48:15.563] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:31:0)
[15:48:15.582] Build status: STARTED
[15:48:15.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:32:0) after 2m57s
[15:51:12.610] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:32:0)
[15:51:12.647] Build status: STARTED
[15:51:12.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:33:0) after 5m2s
[15:56:14.670] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:33:0)
[15:56:14.693] Build status: STARTED
[15:56:14.693] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:34:0) after 6m40s
[16:02:54.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:34:0)
[16:02:54.794] Build status: STARTED
[16:02:54.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:35:0) after 8m32s
[16:11:26.809] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:35:0)
[16:11:26.834] Build status: STARTED
[16:11:26.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:36:0) after 3m2s
[16:14:28.852] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:36:0)
[16:14:28.873] Build status: STARTED
[16:14:28.873] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:37:0) after 4m8s
[16:18:36.899] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:37:0)
[16:18:36.916] Build status: STARTED
[16:18:36.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:38:0) after 3m56s
[16:22:32.969] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:38:0)
[16:22:32.987] Build status: STARTED
[16:22:32.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:39:0) after 3m2s
[16:25:35.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:39:0)
[16:25:35.072] Build status: STARTED
[16:25:35.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:40:0) after 3m56s
[16:29:31.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:40:0)
[16:29:31.164] Build status: STARTED
[16:29:31.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:41:0) after 6m18s
[16:35:49.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:41:0)
[16:35:49.339] Build status: STARTED
[16:35:49.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:42:0) after 3m48s
[16:39:37.343] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:42:0)
[16:39:37.418] Build status: STARTED
[16:39:37.418] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:43:0) after 6m32s
[16:46:09.543] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:43:0)
[16:46:09.564] Build status: STARTED
[16:46:09.564] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:44:0) after 5m12s
[16:51:21.579] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:44:0)
[16:51:21.611] Build status: STARTED
[16:51:21.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:45:0) after 5m6s
[16:56:27.678] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:45:0)
[16:56:27.697] Build status: STARTED
[16:56:27.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:46:0) after 3m49s
[17:00:16.730] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:46:0)
[17:00:16.748] Build status: STARTED
[17:00:16.748] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:47:0) after 9m0s
[17:09:16.797] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:47:0)
[17:09:16.820] Build status: STARTED
[17:09:16.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:48:0) after 6m21s
[17:15:37.891] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:48:0)
[17:15:37.910] Build status: STARTED
[17:15:37.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:49:0) after 8m22s
[17:23:59.947] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:49:0)
[17:23:59.976] Build status: STARTED
[17:23:59.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:50:0) after 5m27s
[17:29:27.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:50:0)
[17:29:27.028] Build status: STARTED
[17:29:27.028] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:51:0) after 1m52s
[17:31:19.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:51:0)
[17:31:19.169] Build status: STARTED
[17:31:19.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:52:0) after 2m25s
[17:33:44.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:52:0)
[17:33:44.240] Build status: STARTED
[17:33:44.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:53:0) after 2m37s
[17:36:21.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:53:0)
[17:36:21.241] Build status: STARTED
[17:36:21.241] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:54:0) after 3m55s
[17:40:16.275] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:54:0)
[17:40:16.294] Build status: STARTED
[17:40:16.294] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:55:0) after 7m20s
[17:47:36.363] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:55:0)
[17:47:36.385] Build status: STARTED
[17:47:36.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:56:0) after 1m38s
[17:49:14.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:56:0)
[17:49:14.428] Build status: STARTED
[17:49:14.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:57:0) after 3m5s
[17:52:19.449] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:57:0)
[17:52:19.471] Build status: STARTED
[17:52:19.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:58:0) after 8m10s
[18:00:29.527] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:58:0)
[18:00:29.543] Build status: STARTED
[18:00:29.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:59:0) after 1m40s
[18:02:09.754] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:59:0)
[18:02:09.769] Build status: STARTED
[18:02:09.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:60:0) after 5m19s
[18:07:28.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:60:0)
[18:07:28.811] Build status: STARTED
[18:07:28.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:61:0) after 4m58s
[18:12:26.784] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:61:0)
[18:12:26.800] Build status: STARTED
[18:12:26.800] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:62:0) after 7m2s
[18:19:28.837] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:62:0)
[18:19:28.856] Build status: STARTED
[18:19:28.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848793912942238320:63:0) after 8m19s
[18:27:16.522] Received PubSub notification, asking Buildbucket for the build status
[18:27:16.559] Build:
{
"id": "8683363851111706081",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2396,
"createdBy": "project:chromium",
"createTime": "2026-04-27T13:18:57.392781108Z",
"startTime": "2026-04-27T13:19:01.660928218Z",
"endTime": "2026-04-27T18:27:15.985921933Z",
"updateTime": "2026-04-27T18:27:15.985921933Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4d564e1ca5359231126dccaca95689080af2da7f",
"ref": "refs/heads/main"
}
}
}
[18:27:16.559] Invocation finished in 5h8m20.295773353s with status FAILED