[07:09:30.806] New invocation is queued and will start shortly
[07:09:31.925] Starting the invocation (attempt 1)
[07:09:31.946] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/f83a5847ab6844142529d1ab43f7e92d6674a4c3"
[07:09:31.946] Popped gitiles commit info from properties and tags
[07:09:31.946] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:09:31.947] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:09:31.947] Buildbucket request:
{
"requestId": "8851806855009008528",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f83a5847ab6844142529d1ab43f7e92d6674a4c3",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851806855009008528"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:09:32.734] Scheduled build:
{
"id": "8686376793090799553",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2002,
"createdBy": "project:chromium",
"createTime": "2026-03-25T07:09:32.010542263Z",
"updateTime": "2026-03-25T07:09:32.010542263Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f83a5847ab6844142529d1ab43f7e92d6674a4c3",
"ref": "refs/heads/main"
}
}
}
[07:09:32.735] Task URL: https://cr-buildbucket.appspot.com/build/8686376793090799553
[07:09:32.735] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:2:0) after 2m0s
[07:09:33.615] Received PubSub notification, asking Buildbucket for the build status
[07:09:33.640] Build status: SCHEDULED
[07:10:18.643] Received PubSub notification, asking Buildbucket for the build status
[07:10:18.664] Build status: STARTED
[07:11:32.758] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:2:0)
[07:11:32.781] Build status: STARTED
[07:11:32.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:5:0) after 1m45s
[07:13:17.822] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:5:0)
[07:13:17.842] Build status: STARTED
[07:13:17.842] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:6:0) after 3m37s
[07:16:54.860] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:6:0)
[07:16:54.876] Build status: STARTED
[07:16:54.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:7:0) after 3m11s
[07:20:05.892] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:7:0)
[07:20:05.914] Build status: STARTED
[07:20:05.914] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:8:0) after 6m49s
[07:26:54.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:8:0)
[07:26:54.965] Build status: STARTED
[07:26:54.966] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:9:0) after 4m27s
[07:31:22.197] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:9:0)
[07:31:22.214] Build status: STARTED
[07:31:22.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:10:0) after 8m21s
[07:39:43.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:10:0)
[07:39:43.512] Build status: STARTED
[07:39:43.512] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:11:0) after 3m7s
[07:42:50.536] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:11:0)
[07:42:50.604] Build status: STARTED
[07:42:50.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:12:0) after 7m24s
[07:50:14.672] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:12:0)
[07:50:14.696] Build status: STARTED
[07:50:14.696] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:13:0) after 3m25s
[07:53:39.742] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:13:0)
[07:53:39.761] Build status: STARTED
[07:53:39.761] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:14:0) after 2m42s
[07:56:21.790] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:14:0)
[07:56:21.808] Build status: STARTED
[07:56:21.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:15:0) after 6m39s
[08:03:00.830] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:15:0)
[08:03:00.881] Build status: STARTED
[08:03:00.881] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:16:0) after 5m19s
[08:08:19.870] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:16:0)
[08:08:19.889] Build status: STARTED
[08:08:19.889] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:17:0) after 8m6s
[08:16:25.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:17:0)
[08:16:25.987] Build status: STARTED
[08:16:25.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:18:0) after 3m47s
[08:20:13.012] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:18:0)
[08:20:13.077] Build status: STARTED
[08:20:13.077] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:19:0) after 8m35s
[08:28:48.101] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:19:0)
[08:28:48.122] Build status: STARTED
[08:28:48.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:20:0) after 1m3s
[08:29:51.154] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:20:0)
[08:29:51.171] Build status: STARTED
[08:29:51.171] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:21:0) after 5m9s
[08:35:00.240] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:21:0)
[08:35:00.259] Build status: STARTED
[08:35:00.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:22:0) after 8m56s
[08:43:56.281] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:22:0)
[08:43:56.300] Build status: STARTED
[08:43:56.301] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:23:0) after 1m0s
[08:44:56.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:23:0)
[08:44:56.339] Build status: STARTED
[08:44:56.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:24:0) after 5m54s
[08:50:50.378] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:24:0)
[08:50:50.474] Build status: STARTED
[08:50:50.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:25:0) after 1m39s
[08:52:29.685] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:25:0)
[08:52:29.703] Build status: STARTED
[08:52:29.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:26:0) after 3m12s
[08:55:41.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:26:0)
[08:55:41.752] Build status: STARTED
[08:55:41.752] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:27:0) after 5m47s
[09:01:28.783] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:27:0)
[09:01:28.803] Build status: STARTED
[09:01:28.803] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:28:0) after 6m21s
[09:07:49.820] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:28:0)
[09:07:49.835] Build status: STARTED
[09:07:49.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:29:0) after 8m55s
[09:16:44.857] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:29:0)
[09:16:44.886] Build status: STARTED
[09:16:44.886] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:30:0) after 9m28s
[09:26:12.966] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:30:0)
[09:26:12.996] Build status: STARTED
[09:26:12.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:31:0) after 5m22s
[09:31:34.994] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:31:0)
[09:31:35.014] Build status: STARTED
[09:31:35.014] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:32:0) after 2m31s
[09:34:06.032] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:32:0)
[09:34:06.049] Build status: STARTED
[09:34:06.049] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:33:0) after 8m35s
[09:42:41.065] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:33:0)
[09:42:41.082] Build status: STARTED
[09:42:41.082] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:34:0) after 8m29s
[09:51:10.147] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:34:0)
[09:51:10.183] Build status: STARTED
[09:51:10.183] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:35:0) after 7m37s
[09:58:47.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:35:0)
[09:58:47.229] Build status: STARTED
[09:58:47.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:36:0) after 6m44s
[10:05:31.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:36:0)
[10:05:31.263] Build status: STARTED
[10:05:31.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:37:0) after 2m42s
[10:08:13.297] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:37:0)
[10:08:13.338] Build status: STARTED
[10:08:13.338] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:38:0) after 8m59s
[10:17:12.383] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:38:0)
[10:17:12.457] Build status: STARTED
[10:17:12.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:39:0) after 5m37s
[10:22:49.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:39:0)
[10:22:49.526] Build status: STARTED
[10:22:49.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:40:0) after 9m44s
[10:32:33.835] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:40:0)
[10:32:33.863] Build status: STARTED
[10:32:33.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:41:0) after 3m32s
[10:36:06.035] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:41:0)
[10:36:06.052] Build status: STARTED
[10:36:06.053] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:42:0) after 9m34s
[10:45:40.658] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:42:0)
[10:45:40.675] Build status: STARTED
[10:45:40.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:43:0) after 4m30s
[10:50:10.731] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:43:0)
[10:50:10.767] Build status: STARTED
[10:50:10.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:44:0) after 4m40s
[10:54:50.822] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:44:0)
[10:54:50.848] Build status: STARTED
[10:54:50.848] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:45:0) after 4m15s
[10:59:05.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:45:0)
[10:59:05.903] Build status: STARTED
[10:59:05.903] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:46:0) after 3m52s
[11:02:57.922] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:46:0)
[11:02:57.999] Build status: STARTED
[11:02:57.999] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:47:0) after 2m24s
[11:05:22.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:47:0)
[11:05:22.037] Build status: STARTED
[11:05:22.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:48:0) after 4m6s
[11:09:28.253] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:48:0)
[11:09:28.276] Build status: STARTED
[11:09:28.277] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:49:0) after 1m34s
[11:11:02.294] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:49:0)
[11:11:02.318] Build status: STARTED
[11:11:02.318] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:50:0) after 6m15s
[11:17:17.391] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:50:0)
[11:17:17.405] Build status: STARTED
[11:17:17.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:51:0) after 1m46s
[11:19:03.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:51:0)
[11:19:03.450] Build status: STARTED
[11:19:03.450] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:52:0) after 6m53s
[11:25:56.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:52:0)
[11:25:56.754] Build status: STARTED
[11:25:56.754] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:53:0) after 5m58s
[11:31:55.042] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:53:0)
[11:31:55.058] Build status: STARTED
[11:31:55.058] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:54:0) after 3m36s
[11:35:31.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:54:0)
[11:35:31.317] Build status: STARTED
[11:35:31.317] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:55:0) after 2m38s
[11:38:09.525] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:55:0)
[11:38:09.609] Build status: STARTED
[11:38:09.609] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:56:0) after 3m32s
[11:41:41.675] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:56:0)
[11:41:41.691] Build status: STARTED
[11:41:41.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:57:0) after 4m24s
[11:46:05.698] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:57:0)
[11:46:05.716] Build status: STARTED
[11:46:05.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:58:0) after 6m45s
[11:52:50.740] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:58:0)
[11:52:50.767] Build status: STARTED
[11:52:50.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:59:0) after 7m22s
[12:00:12.842] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:59:0)
[12:00:12.859] Build status: STARTED
[12:00:12.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:60:0) after 7m24s
[12:07:37.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:60:0)
[12:07:37.228] Build status: STARTED
[12:07:37.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:61:0) after 5m18s
[12:12:55.504] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:61:0)
[12:12:55.521] Build status: STARTED
[12:12:55.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:62:0) after 4m32s
[12:17:27.522] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:62:0)
[12:17:27.548] Build status: STARTED
[12:17:27.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:63:0) after 7m47s
[12:25:14.587] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:63:0)
[12:25:14.606] Build status: STARTED
[12:25:14.606] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:64:0) after 5m36s
[12:30:50.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:64:0)
[12:30:50.777] Build status: STARTED
[12:30:50.777] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:65:0) after 9m46s
[12:40:36.931] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:65:0)
[12:40:36.968] Build status: STARTED
[12:40:36.968] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851806855009008528:66:0) after 8m5s
[12:47:14.218] Received PubSub notification, asking Buildbucket for the build status
[12:47:14.244] Build:
{
"id": "8686376793090799553",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2002,
"createdBy": "project:chromium",
"createTime": "2026-03-25T07:09:32.010542263Z",
"startTime": "2026-03-25T07:10:18.450604179Z",
"endTime": "2026-03-25T12:46:57.778584557Z",
"updateTime": "2026-03-25T12:46:57.778584557Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f83a5847ab6844142529d1ab43f7e92d6674a4c3",
"ref": "refs/heads/main"
}
}
}
[12:47:14.244] Invocation finished in 5h37m43.447398569s with status FAILED