[05:36:33.214] New invocation is queued and will start shortly
[05:36:34.361] Starting the invocation (attempt 1)
[05:36:34.379] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/5c2d9996316971d5b4fdd556f77200d44e78a77d"
[05:36:34.379] Popped gitiles commit info from properties and tags
[05:36:34.379] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[05:36:34.379] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[05:36:34.379] Buildbucket request:
{
"requestId": "8850816136910686208",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5c2d9996316971d5b4fdd556f77200d44e78a77d",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850816136910686208"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[05:36:35.106] Scheduled build:
{
"id": "8685386074917057761",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2088,
"createdBy": "project:chromium",
"createTime": "2026-04-05T05:36:34.487898656Z",
"updateTime": "2026-04-05T05:36:34.487898656Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5c2d9996316971d5b4fdd556f77200d44e78a77d",
"ref": "refs/heads/main"
}
}
}
[05:36:35.106] Task URL: https://cr-buildbucket.appspot.com/build/8685386074917057761
[05:36:35.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:2:0) after 2m34s
[05:36:35.948] Received PubSub notification, asking Buildbucket for the build status
[05:36:35.972] Build status: SCHEDULED
[05:37:16.451] Received PubSub notification, asking Buildbucket for the build status
[05:37:16.472] Build status: STARTED
[05:39:09.120] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:2:0)
[05:39:09.145] Build status: STARTED
[05:39:09.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:5:0) after 6m3s
[05:45:12.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:5:0)
[05:45:12.240] Build status: STARTED
[05:45:12.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:6:0) after 5m32s
[05:50:44.252] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:6:0)
[05:50:44.273] Build status: STARTED
[05:50:44.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:7:0) after 7m6s
[05:57:50.286] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:7:0)
[05:57:50.298] Build status: STARTED
[05:57:50.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:8:0) after 2m8s
[05:59:58.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:8:0)
[05:59:58.330] Build status: STARTED
[05:59:58.330] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:9:0) after 6m10s
[06:06:08.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:9:0)
[06:06:08.467] Build status: STARTED
[06:06:08.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:10:0) after 3m37s
[06:09:45.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:10:0)
[06:09:45.679] Build status: STARTED
[06:09:45.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:11:0) after 9m36s
[06:19:21.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:11:0)
[06:19:21.759] Build status: STARTED
[06:19:21.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:12:0) after 1m6s
[06:20:27.772] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:12:0)
[06:20:27.789] Build status: STARTED
[06:20:27.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:13:0) after 3m59s
[06:24:26.807] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:13:0)
[06:24:26.822] Build status: STARTED
[06:24:26.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:14:0) after 3m12s
[06:27:39.025] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:14:0)
[06:27:39.043] Build status: STARTED
[06:27:39.043] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:15:0) after 9m36s
[06:37:15.265] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:15:0)
[06:37:15.421] Build status: STARTED
[06:37:15.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:16:0) after 8m59s
[06:46:14.650] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:16:0)
[06:46:14.669] Build status: STARTED
[06:46:14.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:17:0) after 9m32s
[06:55:46.680] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:17:0)
[06:55:46.724] Build status: STARTED
[06:55:46.724] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:18:0) after 9m27s
[07:05:13.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:18:0)
[07:05:13.771] Build status: STARTED
[07:05:13.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:19:0) after 7m9s
[07:12:23.630] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:19:0)
[07:12:23.645] Build status: STARTED
[07:12:23.645] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:20:0) after 9m19s
[07:21:42.657] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:20:0)
[07:21:42.672] Build status: STARTED
[07:21:42.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:21:0) after 2m43s
[07:24:25.814] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:21:0)
[07:24:25.831] Build status: STARTED
[07:24:25.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:22:0) after 3m24s
[07:27:49.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:22:0)
[07:27:49.991] Build status: STARTED
[07:27:49.991] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:23:0) after 7m26s
[07:35:16.005] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:23:0)
[07:35:16.029] Build status: STARTED
[07:35:16.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:24:0) after 4m20s
[07:39:36.045] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:24:0)
[07:39:36.069] Build status: STARTED
[07:39:36.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:25:0) after 7m41s
[07:47:17.082] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:25:0)
[07:47:17.195] Build status: STARTED
[07:47:17.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:26:0) after 8m17s
[07:55:34.219] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:26:0)
[07:55:34.263] Build status: STARTED
[07:55:34.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:27:0) after 3m34s
[07:59:08.557] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:27:0)
[07:59:08.580] Build status: STARTED
[07:59:08.580] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:28:0) after 4m38s
[08:03:46.614] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:28:0)
[08:03:46.630] Build status: STARTED
[08:03:46.630] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:29:0) after 8m45s
[08:12:31.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:29:0)
[08:12:31.831] Build status: STARTED
[08:12:31.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:30:0) after 3m0s
[08:15:31.825] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:30:0)
[08:15:31.859] Build status: STARTED
[08:15:31.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:31:0) after 4m42s
[08:20:13.914] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:31:0)
[08:20:13.943] Build status: STARTED
[08:20:13.943] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:32:0) after 4m8s
[08:24:22.098] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:32:0)
[08:24:22.120] Build status: STARTED
[08:24:22.120] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:33:0) after 9m16s
[08:33:38.134] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:33:0)
[08:33:38.179] Build status: STARTED
[08:33:38.179] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:34:0) after 5m36s
[08:39:14.192] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:34:0)
[08:39:14.209] Build status: STARTED
[08:39:14.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:35:0) after 4m10s
[08:43:24.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:35:0)
[08:43:24.266] Build status: STARTED
[08:43:24.266] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:36:0) after 5m19s
[08:48:43.287] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:36:0)
[08:48:43.305] Build status: STARTED
[08:48:43.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:37:0) after 3m4s
[08:51:47.455] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:37:0)
[08:51:47.485] Build status: STARTED
[08:51:47.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:38:0) after 7m10s
[08:58:57.905] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:38:0)
[08:58:57.926] Build status: STARTED
[08:58:57.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:39:0) after 9m40s
[09:08:37.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:39:0)
[09:08:37.962] Build status: STARTED
[09:08:37.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:40:0) after 5m54s
[09:14:31.998] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:40:0)
[09:14:32.021] Build status: STARTED
[09:14:32.021] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:41:0) after 8m37s
[09:23:09.112] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:41:0)
[09:23:09.124] Build status: STARTED
[09:23:09.124] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:42:0) after 4m21s
[09:27:30.137] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:42:0)
[09:27:30.155] Build status: STARTED
[09:27:30.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:43:0) after 1m34s
[09:29:04.170] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:43:0)
[09:29:04.191] Build status: STARTED
[09:29:04.191] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:44:0) after 4m48s
[09:33:52.323] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:44:0)
[09:33:52.340] Build status: STARTED
[09:33:52.340] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:45:0) after 1m17s
[09:35:09.445] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:45:0)
[09:35:09.467] Build status: STARTED
[09:35:09.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:46:0) after 5m45s
[09:40:54.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:46:0)
[09:40:54.565] Build status: STARTED
[09:40:54.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:47:0) after 4m26s
[09:45:20.660] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:47:0)
[09:45:20.730] Build status: STARTED
[09:45:20.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:48:0) after 3m17s
[09:48:37.867] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:48:0)
[09:48:37.961] Build status: STARTED
[09:48:37.961] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:49:0) after 2m59s
[09:51:36.976] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:49:0)
[09:51:36.993] Build status: STARTED
[09:51:36.993] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:50:0) after 6m29s
[09:58:06.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:50:0)
[09:58:06.072] Build status: STARTED
[09:58:06.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:51:0) after 6m3s
[10:04:09.089] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:51:0)
[10:04:09.107] Build status: STARTED
[10:04:09.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:52:0) after 5m36s
[10:09:45.122] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:52:0)
[10:09:45.143] Build status: STARTED
[10:09:45.143] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:53:0) after 7m22s
[10:17:07.161] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:53:0)
[10:17:07.181] Build status: STARTED
[10:17:07.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:54:0) after 6m34s
[10:23:41.201] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:54:0)
[10:23:41.220] Build status: STARTED
[10:23:41.220] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:55:0) after 3m48s
[10:27:29.334] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:55:0)
[10:27:29.406] Build status: STARTED
[10:27:29.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:56:0) after 7m15s
[10:34:44.621] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:56:0)
[10:34:44.705] Build status: STARTED
[10:34:44.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:57:0) after 9m53s
[10:44:37.802] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:57:0)
[10:44:37.823] Build status: STARTED
[10:44:37.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:58:0) after 7m23s
[10:52:00.838] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:58:0)
[10:52:00.852] Build status: STARTED
[10:52:00.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:59:0) after 3m24s
[10:55:24.871] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:59:0)
[10:55:24.897] Build status: STARTED
[10:55:24.897] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:60:0) after 1m50s
[10:57:14.870] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:60:0)
[10:57:14.886] Build status: STARTED
[10:57:14.886] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:61:0) after 6m21s
[11:03:36.199] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:61:0)
[11:03:36.220] Build status: STARTED
[11:03:36.220] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:62:0) after 1m15s
[11:04:51.234] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:62:0)
[11:04:51.251] Build status: STARTED
[11:04:51.251] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:63:0) after 4m18s
[11:09:09.300] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:63:0)
[11:09:09.484] Build status: STARTED
[11:09:09.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:64:0) after 4m32s
[11:13:41.618] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:64:0)
[11:13:41.653] Build status: STARTED
[11:13:41.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850816136910686208:65:0) after 4m29s
[11:13:53.862] Received PubSub notification, asking Buildbucket for the build status
[11:13:53.883] Build:
{
"id": "8685386074917057761",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2088,
"createdBy": "project:chromium",
"createTime": "2026-04-05T05:36:34.487898656Z",
"startTime": "2026-04-05T05:37:14.896757158Z",
"endTime": "2026-04-05T11:13:53.460310157Z",
"updateTime": "2026-04-05T11:13:53.460310157Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5c2d9996316971d5b4fdd556f77200d44e78a77d",
"ref": "refs/heads/main"
}
}
}
[11:13:53.883] Invocation finished in 5h37m20.68149284s with status FAILED