[07:21:33.668] New invocation is queued and will start shortly
[07:21:34.720] Starting the invocation (attempt 1)
[07:21:34.744] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/8294ce49192bbf79bb258a021755fc060c9043b1"
[07:21:34.744] Popped gitiles commit info from properties and tags
[07:21:34.744] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:21:34.745] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:21:34.746] Buildbucket request:
{
"requestId": "8858510272549254272",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8294ce49192bbf79bb258a021755fc060c9043b1",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8858510272549254272"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:21:35.361] Scheduled build:
{
"id": "8693080210704097073",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1154,
"createdBy": "project:chromium",
"createTime": "2026-01-10T07:21:34.800061739Z",
"updateTime": "2026-01-10T07:21:34.800061739Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8294ce49192bbf79bb258a021755fc060c9043b1",
"ref": "refs/heads/main"
}
}
}
[07:21:35.361] Task URL: https://cr-buildbucket.appspot.com/build/8693080210704097073
[07:21:35.361] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:2:0) after 9m12s
[07:21:37.006] Received PubSub notification, asking Buildbucket for the build status
[07:21:37.120] Build status: SCHEDULED
[07:21:41.355] Received PubSub notification, asking Buildbucket for the build status
[07:21:41.368] Build status: STARTED
[07:30:47.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:2:0)
[07:30:47.562] Build status: STARTED
[07:30:47.562] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:5:0) after 2m51s
[07:33:38.759] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:5:0)
[07:33:38.817] Build status: STARTED
[07:33:38.817] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:6:0) after 5m5s
[07:38:43.833] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:6:0)
[07:38:43.856] Build status: STARTED
[07:38:43.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:7:0) after 6m21s
[07:45:04.972] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:7:0)
[07:45:04.998] Build status: STARTED
[07:45:04.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:8:0) after 1m51s
[07:46:56.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:8:0)
[07:46:56.232] Build status: STARTED
[07:46:56.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:9:0) after 7m28s
[07:54:24.206] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:9:0)
[07:54:24.239] Build status: STARTED
[07:54:24.239] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:10:0) after 1m43s
[07:56:07.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:10:0)
[07:56:07.238] Build status: STARTED
[07:56:07.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:11:0) after 9m43s
[08:05:50.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:11:0)
[08:05:50.236] Build status: STARTED
[08:05:50.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:12:0) after 5m32s
[08:11:22.280] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:12:0)
[08:11:22.308] Build status: STARTED
[08:11:22.308] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:13:0) after 2m7s
[08:13:29.321] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:13:0)
[08:13:29.337] Build status: STARTED
[08:13:29.338] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:14:0) after 1m25s
[08:14:54.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:14:0)
[08:14:54.574] Build status: STARTED
[08:14:54.574] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:15:0) after 6m25s
[08:21:19.875] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:15:0)
[08:21:19.890] Build status: STARTED
[08:21:19.890] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:16:0) after 5m54s
[08:27:13.909] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:16:0)
[08:27:13.924] Build status: STARTED
[08:27:13.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:17:0) after 8m46s
[08:36:00.286] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:17:0)
[08:36:00.311] Build status: STARTED
[08:36:00.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:18:0) after 1m42s
[08:37:42.327] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:18:0)
[08:37:42.341] Build status: STARTED
[08:37:42.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:19:0) after 3m17s
[08:40:59.467] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:19:0)
[08:40:59.490] Build status: STARTED
[08:40:59.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:20:0) after 1m44s
[08:42:43.531] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:20:0)
[08:42:43.556] Build status: STARTED
[08:42:43.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:21:0) after 5m1s
[08:47:44.945] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:21:0)
[08:47:44.959] Build status: STARTED
[08:47:44.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:22:0) after 1m7s
[08:48:52.004] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:22:0)
[08:48:52.017] Build status: STARTED
[08:48:52.017] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:23:0) after 3m28s
[08:52:20.120] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:23:0)
[08:52:20.141] Build status: STARTED
[08:52:20.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:24:0) after 6m35s
[08:58:55.236] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:24:0)
[08:58:55.248] Build status: STARTED
[08:58:55.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:25:0) after 1m26s
[09:00:21.240] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:25:0)
[09:00:21.267] Build status: STARTED
[09:00:21.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:26:0) after 1m16s
[09:01:37.285] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:26:0)
[09:01:37.300] Build status: STARTED
[09:01:37.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:27:0) after 6m9s
[09:07:46.309] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:27:0)
[09:07:46.324] Build status: STARTED
[09:07:46.324] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:28:0) after 8m59s
[09:16:45.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:28:0)
[09:16:45.334] Build status: STARTED
[09:16:45.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:29:0) after 8m10s
[09:24:55.429] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:29:0)
[09:24:55.453] Build status: STARTED
[09:24:55.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:30:0) after 7m4s
[09:31:59.809] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:30:0)
[09:31:59.827] Build status: STARTED
[09:31:59.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:31:0) after 9m19s
[09:41:19.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:31:0)
[09:41:19.569] Build status: STARTED
[09:41:19.569] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:32:0) after 4m3s
[09:45:22.583] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:32:0)
[09:45:22.600] Build status: STARTED
[09:45:22.600] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:33:0) after 3m46s
[09:49:08.575] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:33:0)
[09:49:08.589] Build status: STARTED
[09:49:08.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:34:0) after 4m34s
[09:53:42.590] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:34:0)
[09:53:42.604] Build status: STARTED
[09:53:42.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:35:0) after 5m35s
[09:59:17.944] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:35:0)
[09:59:17.981] Build status: STARTED
[09:59:17.981] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:36:0) after 2m44s
[10:02:02.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:36:0)
[10:02:02.107] Build status: STARTED
[10:02:02.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:37:0) after 9m36s
[10:11:38.406] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:37:0)
[10:11:38.424] Build status: STARTED
[10:11:38.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:38:0) after 4m0s
[10:15:38.543] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:38:0)
[10:15:38.565] Build status: STARTED
[10:15:38.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:39:0) after 4m26s
[10:20:04.863] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:39:0)
[10:20:04.894] Build status: STARTED
[10:20:04.894] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:40:0) after 6m15s
[10:26:19.913] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:40:0)
[10:26:19.939] Build status: STARTED
[10:26:19.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:41:0) after 9m35s
[10:35:55.004] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:41:0)
[10:35:55.037] Build status: STARTED
[10:35:55.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:42:0) after 7m51s
[10:43:46.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:42:0)
[10:43:46.065] Build status: STARTED
[10:43:46.065] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:43:0) after 3m49s
[10:47:35.230] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:43:0)
[10:47:35.249] Build status: STARTED
[10:47:35.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:44:0) after 9m33s
[10:57:08.263] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:44:0)
[10:57:08.275] Build status: STARTED
[10:57:08.275] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:45:0) after 6m49s
[11:03:57.341] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:45:0)
[11:03:57.354] Build status: STARTED
[11:03:57.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:46:0) after 8m47s
[11:12:44.362] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:46:0)
[11:12:44.384] Build status: STARTED
[11:12:44.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:47:0) after 2m26s
[11:15:10.655] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:47:0)
[11:15:10.672] Build status: STARTED
[11:15:10.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:48:0) after 8m55s
[11:24:05.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:48:0)
[11:24:05.709] Build status: STARTED
[11:24:05.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:49:0) after 1m16s
[11:25:22.144] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:49:0)
[11:25:22.161] Build status: STARTED
[11:25:22.161] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:50:0) after 5m51s
[11:31:13.174] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:50:0)
[11:31:13.194] Build status: STARTED
[11:31:13.194] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:51:0) after 1m31s
[11:32:44.209] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:51:0)
[11:32:44.225] Build status: STARTED
[11:32:44.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:52:0) after 9m1s
[11:41:45.240] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:52:0)
[11:41:45.253] Build status: STARTED
[11:41:45.253] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:53:0) after 9m56s
[11:51:41.270] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:53:0)
[11:51:41.284] Build status: STARTED
[11:51:41.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:54:0) after 1m41s
[11:53:22.297] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:54:0)
[11:53:22.313] Build status: STARTED
[11:53:22.313] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:55:0) after 1m51s
[11:55:13.407] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:55:0)
[11:55:13.422] Build status: STARTED
[11:55:13.422] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:56:0) after 8m34s
[12:03:47.438] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:56:0)
[12:03:47.451] Build status: STARTED
[12:03:47.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:57:0) after 3m35s
[12:07:22.663] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:57:0)
[12:07:22.676] Build status: STARTED
[12:07:22.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:58:0) after 1m45s
[12:09:07.697] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:58:0)
[12:09:07.713] Build status: STARTED
[12:09:07.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:59:0) after 2m52s
[12:11:59.828] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:59:0)
[12:11:59.842] Build status: STARTED
[12:11:59.842] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:60:0) after 3m4s
[12:15:03.956] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:60:0)
[12:15:03.969] Build status: STARTED
[12:15:03.969] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:61:0) after 8m13s
[12:23:17.001] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:61:0)
[12:23:17.017] Build status: STARTED
[12:23:17.017] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858510272549254272:62:0) after 7m56s
[12:28:58.300] Received PubSub notification, asking Buildbucket for the build status
[12:28:58.314] Build:
{
"id": "8693080210704097073",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1154,
"createdBy": "project:chromium",
"createTime": "2026-01-10T07:21:34.800061739Z",
"startTime": "2026-01-10T07:21:39.874509345Z",
"endTime": "2026-01-10T12:28:58.001174987Z",
"updateTime": "2026-01-10T12:28:58.001174987Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8294ce49192bbf79bb258a021755fc060c9043b1",
"ref": "refs/heads/main"
}
}
}
[12:28:58.314] Invocation finished in 5h7m24.658103866s with status FAILED