[20:53:21.177] New invocation is queued and will start shortly
[20:53:22.445] Starting the invocation (attempt 1)
[20:53:22.469] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/178e326593880b6bbab543d2946d07357d15baaf"
[20:53:22.470] Popped gitiles commit info from properties and tags
[20:53:22.470] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[20:53:22.470] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[20:53:22.470] Buildbucket request:
{
"requestId": "8850577262943694448",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "178e326593880b6bbab543d2946d07357d15baaf",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850577262943694448"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[20:53:23.155] Scheduled build:
{
"id": "8685147200867259633",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2061,
"createdBy": "project:chromium",
"createTime": "2026-04-07T20:53:22.533832645Z",
"updateTime": "2026-04-07T20:53:22.533832645Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "178e326593880b6bbab543d2946d07357d15baaf",
"ref": "refs/heads/main"
}
}
}
[20:53:23.155] Task URL: https://cr-buildbucket.appspot.com/build/8685147200867259633
[20:53:23.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:2:0) after 8m22s
[20:53:24.026] Received PubSub notification, asking Buildbucket for the build status
[20:53:24.058] Build status: SCHEDULED
[20:53:26.544] Received PubSub notification, asking Buildbucket for the build status
[20:53:26.564] Build status: STARTED
[21:01:45.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:2:0)
[21:01:45.215] Build status: STARTED
[21:01:45.215] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:5:0) after 1m56s
[21:03:41.279] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:5:0)
[21:03:41.299] Build status: STARTED
[21:03:41.299] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:6:0) after 8m34s
[21:12:15.366] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:6:0)
[21:12:15.393] Build status: STARTED
[21:12:15.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:7:0) after 5m51s
[21:18:06.565] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:7:0)
[21:18:06.582] Build status: STARTED
[21:18:06.583] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:8:0) after 8m13s
[21:26:19.696] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:8:0)
[21:26:19.715] Build status: STARTED
[21:26:19.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:9:0) after 5m19s
[21:31:39.014] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:9:0)
[21:31:39.050] Build status: STARTED
[21:31:39.050] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:10:0) after 5m37s
[21:37:16.072] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:10:0)
[21:37:16.103] Build status: STARTED
[21:37:16.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:11:0) after 4m45s
[21:42:01.207] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:11:0)
[21:42:01.234] Build status: STARTED
[21:42:01.234] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:12:0) after 1m46s
[21:43:47.401] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:12:0)
[21:43:47.420] Build status: STARTED
[21:43:47.420] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:13:0) after 4m41s
[21:48:28.551] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:13:0)
[21:48:28.576] Build status: STARTED
[21:48:28.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:14:0) after 5m9s
[21:53:37.795] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:14:0)
[21:53:37.812] Build status: STARTED
[21:53:37.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:15:0) after 4m7s
[21:57:44.865] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:15:0)
[21:57:44.885] Build status: STARTED
[21:57:44.885] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:16:0) after 8m29s
[22:06:13.952] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:16:0)
[22:06:13.970] Build status: STARTED
[22:06:13.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:17:0) after 7m47s
[22:14:01.082] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:17:0)
[22:14:01.101] Build status: STARTED
[22:14:01.101] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:18:0) after 5m54s
[22:19:55.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:18:0)
[22:19:55.195] Build status: STARTED
[22:19:55.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:19:0) after 7m8s
[22:27:03.510] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:19:0)
[22:27:03.528] Build status: STARTED
[22:27:03.528] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:20:0) after 6m12s
[22:33:15.605] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:20:0)
[22:33:15.626] Build status: STARTED
[22:33:15.626] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:21:0) after 7m12s
[22:40:27.653] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:21:0)
[22:40:27.671] Build status: STARTED
[22:40:27.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:22:0) after 3m15s
[22:43:42.813] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:22:0)
[22:43:42.836] Build status: STARTED
[22:43:42.836] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:23:0) after 9m31s
[22:53:14.228] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:23:0)
[22:53:14.245] Build status: STARTED
[22:53:14.245] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:24:0) after 7m35s
[23:00:49.433] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:24:0)
[23:00:49.489] Build status: STARTED
[23:00:49.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:25:0) after 8m13s
[23:09:02.541] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:25:0)
[23:09:02.563] Build status: STARTED
[23:09:02.563] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:26:0) after 3m18s
[23:12:20.584] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:26:0)
[23:12:20.605] Build status: STARTED
[23:12:20.605] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:27:0) after 2m42s
[23:15:02.678] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:27:0)
[23:15:02.732] Build status: STARTED
[23:15:02.732] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:28:0) after 9m7s
[23:24:09.830] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:28:0)
[23:24:09.847] Build status: STARTED
[23:24:09.847] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:29:0) after 6m54s
[23:31:03.924] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:29:0)
[23:31:03.953] Build status: STARTED
[23:31:03.953] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:30:0) after 5m58s
[23:37:01.977] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:30:0)
[23:37:02.059] Build status: STARTED
[23:37:02.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:31:0) after 3m33s
[23:40:35.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:31:0)
[23:40:35.120] Build status: STARTED
[23:40:35.120] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:32:0) after 6m1s
[23:46:36.241] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:32:0)
[23:46:36.267] Build status: STARTED
[23:46:36.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:33:0) after 2m19s
[23:48:55.254] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:33:0)
[23:48:55.275] Build status: STARTED
[23:48:55.275] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:34:0) after 2m19s
[23:51:14.445] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:34:0)
[23:51:14.512] Build status: STARTED
[23:51:14.512] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:35:0) after 9m59s
[00:01:13.535] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:35:0)
[00:01:13.556] Build status: STARTED
[00:01:13.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:36:0) after 1m8s
[00:02:21.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:36:0)
[00:02:21.607] Build status: STARTED
[00:02:21.607] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:37:0) after 7m50s
[00:10:11.634] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:37:0)
[00:10:11.661] Build status: STARTED
[00:10:11.661] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:38:0) after 5m40s
[00:15:51.687] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:38:0)
[00:15:51.709] Build status: STARTED
[00:15:51.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:39:0) after 7m3s
[00:22:54.749] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:39:0)
[00:22:54.782] Build status: STARTED
[00:22:54.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:40:0) after 3m33s
[00:26:27.973] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:40:0)
[00:26:28.010] Build status: STARTED
[00:26:28.010] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:41:0) after 2m32s
[00:28:59.982] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:41:0)
[00:29:00.001] Build status: STARTED
[00:29:00.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:42:0) after 1m6s
[00:30:06.242] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:42:0)
[00:30:06.269] Build status: STARTED
[00:30:06.269] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:43:0) after 9m16s
[00:39:22.424] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:43:0)
[00:39:22.448] Build status: STARTED
[00:39:22.449] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:44:0) after 4m49s
[00:44:11.667] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:44:0)
[00:44:11.684] Build status: STARTED
[00:44:11.684] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:45:0) after 6m50s
[00:51:01.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:45:0)
[00:51:01.859] Build status: STARTED
[00:51:01.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:46:0) after 7m27s
[00:58:28.953] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:46:0)
[00:58:28.970] Build status: STARTED
[00:58:28.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:47:0) after 2m57s
[01:01:26.220] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:47:0)
[01:01:26.292] Build status: STARTED
[01:01:26.293] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:48:0) after 9m19s
[01:10:45.428] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:48:0)
[01:10:45.514] Build status: STARTED
[01:10:45.514] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:49:0) after 7m31s
[01:18:16.540] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:49:0)
[01:18:16.556] Build status: STARTED
[01:18:16.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:50:0) after 3m16s
[01:21:32.581] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:50:0)
[01:21:32.605] Build status: STARTED
[01:21:32.605] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:51:0) after 5m22s
[01:26:54.615] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:51:0)
[01:26:54.633] Build status: STARTED
[01:26:54.633] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:52:0) after 5m4s
[01:31:58.661] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:52:0)
[01:31:58.677] Build status: STARTED
[01:31:58.677] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:53:0) after 1m55s
[01:33:53.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:53:0)
[01:33:53.785] Build status: STARTED
[01:33:53.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:54:0) after 4m57s
[01:38:50.877] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:54:0)
[01:38:50.899] Build status: STARTED
[01:38:50.899] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:55:0) after 1m32s
[01:40:23.068] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:55:0)
[01:40:23.089] Build status: STARTED
[01:40:23.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:56:0) after 2m39s
[01:43:02.233] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:56:0)
[01:43:02.257] Build status: STARTED
[01:43:02.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:57:0) after 5m17s
[01:48:19.293] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:57:0)
[01:48:19.312] Build status: STARTED
[01:48:19.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:58:0) after 5m19s
[01:53:38.392] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:58:0)
[01:53:38.409] Build status: STARTED
[01:53:38.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:59:0) after 7m5s
[02:00:43.520] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:59:0)
[02:00:43.536] Build status: STARTED
[02:00:43.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:60:0) after 6m28s
[02:07:11.645] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:60:0)
[02:07:11.667] Build status: STARTED
[02:07:11.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:61:0) after 5m38s
[02:12:49.832] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:61:0)
[02:12:49.854] Build status: STARTED
[02:12:49.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:62:0) after 1m5s
[02:13:54.948] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:62:0)
[02:13:54.969] Build status: STARTED
[02:13:54.969] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:63:0) after 7m38s
[02:21:33.141] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:63:0)
[02:21:33.158] Build status: STARTED
[02:21:33.158] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8850577262943694448:64:0) after 3m22s
[02:22:40.906] Received PubSub notification, asking Buildbucket for the build status
[02:22:40.975] Build:
{
"id": "8685147200867259633",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2061,
"createdBy": "project:chromium",
"createTime": "2026-04-07T20:53:22.533832645Z",
"startTime": "2026-04-07T20:53:26.323666635Z",
"endTime": "2026-04-08T02:22:40.596953008Z",
"updateTime": "2026-04-08T02:22:40.596953008Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "178e326593880b6bbab543d2946d07357d15baaf",
"ref": "refs/heads/main"
}
}
}
[02:22:40.975] Invocation finished in 5h29m19.806246784s with status FAILED