[07:09:58.373] New invocation is queued and will start shortly
[07:09:59.507] Starting the invocation (attempt 1)
[07:09:59.526] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/f55e430f13c2fbbc2b153a1957b961fa2909044f"
[07:09:59.526] Popped gitiles commit info from properties and tags
[07:09:59.526] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:09:59.526] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:09:59.526] Buildbucket request:
{
"requestId": "8850900856444011840",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f55e430f13c2fbbc2b153a1957b961fa2909044f",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850900856444011840"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:10:00.487] Scheduled build:
{
"id": "8685470794463997153",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2084,
"createdBy": "project:chromium",
"createTime": "2026-04-04T07:09:59.631822004Z",
"updateTime": "2026-04-04T07:09:59.631822004Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f55e430f13c2fbbc2b153a1957b961fa2909044f",
"ref": "refs/heads/main"
}
}
}
[07:10:00.487] Task URL: https://cr-buildbucket.appspot.com/build/8685470794463997153
[07:10:00.487] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:2:0) after 1m6s
[07:10:01.576] Received PubSub notification, asking Buildbucket for the build status
[07:10:01.613] Build status: SCHEDULED
[07:10:40.558] Received PubSub notification, asking Buildbucket for the build status
[07:10:40.585] Build status: STARTED
[07:11:06.863] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:2:0)
[07:11:06.880] Build status: STARTED
[07:11:06.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:5:0) after 7m34s
[07:18:40.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:5:0)
[07:18:41.008] Build status: STARTED
[07:18:41.008] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:6:0) after 1m8s
[07:19:49.256] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:6:0)
[07:19:49.285] Build status: STARTED
[07:19:49.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:7:0) after 3m31s
[07:23:20.351] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:7:0)
[07:23:20.372] Build status: STARTED
[07:23:20.372] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:8:0) after 4m30s
[07:27:50.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:8:0)
[07:27:50.745] Build status: STARTED
[07:27:50.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:9:0) after 7m35s
[07:35:25.795] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:9:0)
[07:35:25.820] Build status: STARTED
[07:35:25.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:10:0) after 3m13s
[07:38:38.839] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:10:0)
[07:38:38.868] Build status: STARTED
[07:38:38.868] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:11:0) after 5m17s
[07:43:55.886] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:11:0)
[07:43:55.908] Build status: STARTED
[07:43:55.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:12:0) after 4m36s
[07:48:31.947] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:12:0)
[07:48:31.964] Build status: STARTED
[07:48:31.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:13:0) after 3m10s
[07:51:42.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:13:0)
[07:51:42.029] Build status: STARTED
[07:51:42.030] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:14:0) after 5m14s
[07:56:56.210] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:14:0)
[07:56:56.240] Build status: STARTED
[07:56:56.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:15:0) after 5m46s
[08:02:42.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:15:0)
[08:02:42.386] Build status: STARTED
[08:02:42.386] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:16:0) after 7m48s
[08:10:30.400] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:16:0)
[08:10:30.420] Build status: STARTED
[08:10:30.420] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:17:0) after 1m33s
[08:12:03.651] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:17:0)
[08:12:03.711] Build status: STARTED
[08:12:03.711] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:18:0) after 7m37s
[08:19:40.725] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:18:0)
[08:19:40.767] Build status: STARTED
[08:19:40.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:19:0) after 1m42s
[08:21:22.818] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:19:0)
[08:21:22.838] Build status: STARTED
[08:21:22.838] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:20:0) after 1m44s
[08:23:06.851] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:20:0)
[08:23:06.870] Build status: STARTED
[08:23:06.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:21:0) after 1m54s
[08:25:00.887] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:21:0)
[08:25:00.907] Build status: STARTED
[08:25:00.907] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:22:0) after 3m53s
[08:28:53.973] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:22:0)
[08:28:54.107] Build status: STARTED
[08:28:54.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:23:0) after 7m17s
[08:36:11.294] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:23:0)
[08:36:11.309] Build status: STARTED
[08:36:11.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:24:0) after 2m3s
[08:38:14.473] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:24:0)
[08:38:14.493] Build status: STARTED
[08:38:14.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:25:0) after 2m39s
[08:40:53.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:25:0)
[08:40:53.540] Build status: STARTED
[08:40:53.540] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:26:0) after 5m29s
[08:46:22.682] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:26:0)
[08:46:22.719] Build status: STARTED
[08:46:22.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:27:0) after 8m13s
[08:54:35.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:27:0)
[08:54:35.765] Build status: STARTED
[08:54:35.766] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:28:0) after 9m11s
[09:03:46.911] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:28:0)
[09:03:46.942] Build status: STARTED
[09:03:46.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:29:0) after 3m44s
[09:07:31.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:29:0)
[09:07:31.141] Build status: STARTED
[09:07:31.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:30:0) after 6m4s
[09:13:35.844] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:30:0)
[09:13:35.866] Build status: STARTED
[09:13:35.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:31:0) after 5m43s
[09:19:19.223] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:31:0)
[09:19:19.393] Build status: STARTED
[09:19:19.393] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:32:0) after 4m15s
[09:23:34.364] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:32:0)
[09:23:34.390] Build status: STARTED
[09:23:34.390] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:33:0) after 2m21s
[09:25:55.405] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:33:0)
[09:25:55.424] Build status: STARTED
[09:25:55.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:34:0) after 9m20s
[09:35:15.437] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:34:0)
[09:35:15.460] Build status: STARTED
[09:35:15.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:35:0) after 4m58s
[09:40:13.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:35:0)
[09:40:13.471] Build status: STARTED
[09:40:13.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:36:0) after 8m34s
[09:48:47.533] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:36:0)
[09:48:47.549] Build status: STARTED
[09:48:47.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:37:0) after 7m37s
[09:56:24.640] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:37:0)
[09:56:24.656] Build status: STARTED
[09:56:24.656] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:38:0) after 3m44s
[10:00:08.706] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:38:0)
[10:00:08.738] Build status: STARTED
[10:00:08.738] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:39:0) after 8m32s
[10:08:41.055] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:39:0)
[10:08:41.162] Build status: STARTED
[10:08:41.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:40:0) after 7m30s
[10:16:11.284] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:40:0)
[10:16:11.320] Build status: STARTED
[10:16:11.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:41:0) after 5m6s
[10:21:17.327] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:41:0)
[10:21:17.356] Build status: STARTED
[10:21:17.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:42:0) after 6m34s
[10:27:51.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:42:0)
[10:27:51.396] Build status: STARTED
[10:27:51.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:43:0) after 9m2s
[10:36:53.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:43:0)
[10:36:53.429] Build status: STARTED
[10:36:53.429] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:44:0) after 5m0s
[10:41:53.490] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:44:0)
[10:41:53.510] Build status: STARTED
[10:41:53.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:45:0) after 7m10s
[10:49:03.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:45:0)
[10:49:03.543] Build status: STARTED
[10:49:03.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:46:0) after 8m20s
[10:57:23.575] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:46:0)
[10:57:23.609] Build status: STARTED
[10:57:23.609] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:47:0) after 7m49s
[11:05:12.634] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:47:0)
[11:05:12.646] Build status: STARTED
[11:05:12.646] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:48:0) after 6m57s
[11:12:09.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:48:0)
[11:12:09.929] Build status: STARTED
[11:12:09.929] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:49:0) after 9m41s
[11:21:50.944] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:49:0)
[11:21:50.959] Build status: STARTED
[11:21:50.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:50:0) after 6m19s
[11:28:09.952] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:50:0)
[11:28:09.972] Build status: STARTED
[11:28:09.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:51:0) after 7m2s
[11:35:12.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:51:0)
[11:35:12.259] Build status: STARTED
[11:35:12.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:52:0) after 3m51s
[11:39:03.299] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:52:0)
[11:39:03.319] Build status: STARTED
[11:39:03.319] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:53:0) after 7m38s
[11:46:41.656] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:53:0)
[11:46:41.676] Build status: STARTED
[11:46:41.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:54:0) after 8m31s
[11:55:12.862] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:54:0)
[11:55:12.940] Build status: STARTED
[11:55:12.940] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:55:0) after 8m25s
[12:03:37.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:55:0)
[12:03:37.962] Build status: STARTED
[12:03:37.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:56:0) after 2m9s
[12:05:46.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:56:0)
[12:05:47.001] Build status: STARTED
[12:05:47.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:57:0) after 9m38s
[12:15:25.576] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:57:0)
[12:15:25.614] Build status: STARTED
[12:15:25.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:58:0) after 4m52s
[12:20:17.823] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:58:0)
[12:20:17.845] Build status: STARTED
[12:20:17.845] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:59:0) after 8m42s
[12:28:59.964] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:59:0)
[12:28:59.987] Build status: STARTED
[12:28:59.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:60:0) after 9m28s
[12:38:28.006] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:60:0)
[12:38:28.032] Build status: STARTED
[12:38:28.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:61:0) after 4m48s
[12:43:16.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:61:0)
[12:43:16.197] Build status: STARTED
[12:43:16.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:62:0) after 5m30s
[12:48:46.201] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:62:0)
[12:48:46.216] Build status: STARTED
[12:48:46.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850900856444011840:63:0) after 4m7s
[12:49:08.836] Received PubSub notification, asking Buildbucket for the build status
[12:49:08.857] Build:
{
"id": "8685470794463997153",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2084,
"createdBy": "project:chromium",
"createTime": "2026-04-04T07:09:59.631822004Z",
"startTime": "2026-04-04T07:10:40.358629690Z",
"endTime": "2026-04-04T12:49:08.571021425Z",
"updateTime": "2026-04-04T12:49:08.571021425Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f55e430f13c2fbbc2b153a1957b961fa2909044f",
"ref": "refs/heads/main"
}
}
}
[12:49:08.857] Invocation finished in 5h39m10.497622568s with status FAILED