[18:39:43.095] New invocation is queued and will start shortly
[18:39:44.172] Starting the invocation (attempt 1)
[18:39:44.197] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/ba944c6b8c640b3e259eb3c2bb2929d9626fc64f"
[18:39:44.197] Popped gitiles commit info from properties and tags
[18:39:44.197] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[18:39:44.197] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[18:39:44.198] Buildbucket request:
{
"requestId": "8851491640172338448",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ba944c6b8c640b3e259eb3c2bb2929d9626fc64f",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851491640172338448"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[18:39:44.801] Scheduled build:
{
"id": "8686061578305137137",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2058,
"createdBy": "project:chromium",
"createTime": "2026-03-28T18:39:44.255551272Z",
"updateTime": "2026-03-28T18:39:44.255551272Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ba944c6b8c640b3e259eb3c2bb2929d9626fc64f",
"ref": "refs/heads/main"
}
}
}
[18:39:44.801] Task URL: https://cr-buildbucket.appspot.com/build/8686061578305137137
[18:39:44.801] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:2:0) after 5m10s
[18:39:45.736] Received PubSub notification, asking Buildbucket for the build status
[18:39:45.787] Build status: SCHEDULED
[18:40:26.570] Received PubSub notification, asking Buildbucket for the build status
[18:40:26.620] Build status: STARTED
[18:44:55.103] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:2:0)
[18:44:55.131] Build status: STARTED
[18:44:55.131] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:5:0) after 7m13s
[18:52:08.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:5:0)
[18:52:08.347] Build status: STARTED
[18:52:08.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:6:0) after 6m8s
[18:58:16.363] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:6:0)
[18:58:16.382] Build status: STARTED
[18:58:16.382] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:7:0) after 9m7s
[19:07:23.571] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:7:0)
[19:07:23.588] Build status: STARTED
[19:07:23.588] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:8:0) after 8m46s
[19:16:09.798] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:8:0)
[19:16:10.044] Build status: STARTED
[19:16:10.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:9:0) after 3m19s
[19:19:29.060] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:9:0)
[19:19:29.078] Build status: STARTED
[19:19:29.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:10:0) after 7m55s
[19:27:24.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:10:0)
[19:27:24.116] Build status: STARTED
[19:27:24.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:11:0) after 2m35s
[19:29:59.245] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:11:0)
[19:29:59.262] Build status: STARTED
[19:29:59.262] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:12:0) after 8m0s
[19:37:59.819] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:12:0)
[19:37:59.876] Build status: STARTED
[19:37:59.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:13:0) after 7m45s
[19:45:45.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:13:0)
[19:45:45.316] Build status: STARTED
[19:45:45.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:14:0) after 4m1s
[19:49:46.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:14:0)
[19:49:46.539] Build status: STARTED
[19:49:46.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:15:0) after 3m54s
[19:53:40.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:15:0)
[19:53:40.573] Build status: STARTED
[19:53:40.573] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:16:0) after 9m59s
[20:03:39.592] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:16:0)
[20:03:39.614] Build status: STARTED
[20:03:39.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:17:0) after 5m52s
[20:09:31.697] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:17:0)
[20:09:31.719] Build status: STARTED
[20:09:31.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:18:0) after 4m27s
[20:13:58.734] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:18:0)
[20:13:58.757] Build status: STARTED
[20:13:58.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:19:0) after 1m58s
[20:15:56.766] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:19:0)
[20:15:56.785] Build status: STARTED
[20:15:56.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:20:0) after 8m1s
[20:23:57.805] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:20:0)
[20:23:57.825] Build status: STARTED
[20:23:57.825] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:21:0) after 7m50s
[20:31:47.850] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:21:0)
[20:31:47.872] Build status: STARTED
[20:31:47.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:22:0) after 9m39s
[20:41:26.932] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:22:0)
[20:41:26.948] Build status: STARTED
[20:41:26.948] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:23:0) after 1m42s
[20:43:08.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:23:0)
[20:43:09.012] Build status: STARTED
[20:43:09.012] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:24:0) after 5m32s
[20:48:41.321] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:24:0)
[20:48:41.354] Build status: STARTED
[20:48:41.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:25:0) after 8m58s
[20:57:39.328] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:25:0)
[20:57:39.355] Build status: STARTED
[20:57:39.355] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:26:0) after 3m24s
[21:01:03.373] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:26:0)
[21:01:03.427] Build status: STARTED
[21:01:03.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:27:0) after 2m10s
[21:03:13.443] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:27:0)
[21:03:13.462] Build status: STARTED
[21:03:13.462] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:28:0) after 1m2s
[21:04:15.475] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:28:0)
[21:04:15.492] Build status: STARTED
[21:04:15.492] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:29:0) after 2m3s
[21:06:18.505] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:29:0)
[21:06:18.540] Build status: STARTED
[21:06:18.540] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:30:0) after 6m48s
[21:13:06.557] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:30:0)
[21:13:06.590] Build status: STARTED
[21:13:06.590] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:31:0) after 4m11s
[21:17:17.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:31:0)
[21:17:17.826] Build status: STARTED
[21:17:17.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:32:0) after 6m31s
[21:23:49.070] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:32:0)
[21:23:49.088] Build status: STARTED
[21:23:49.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:33:0) after 6m15s
[21:30:04.343] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:33:0)
[21:30:04.361] Build status: STARTED
[21:30:04.361] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:34:0) after 5m43s
[21:35:47.427] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:34:0)
[21:35:47.447] Build status: STARTED
[21:35:47.447] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:35:0) after 5m43s
[21:41:31.132] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:35:0)
[21:41:31.155] Build status: STARTED
[21:41:31.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:36:0) after 7m12s
[21:48:43.154] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:36:0)
[21:48:43.168] Build status: STARTED
[21:48:43.168] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:37:0) after 3m31s
[21:52:14.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:37:0)
[21:52:14.389] Build status: STARTED
[21:52:14.389] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:38:0) after 2m9s
[21:54:23.452] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:38:0)
[21:54:23.480] Build status: STARTED
[21:54:23.480] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:39:0) after 2m50s
[21:57:13.497] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:39:0)
[21:57:13.511] Build status: STARTED
[21:57:13.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:40:0) after 7m44s
[22:04:57.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:40:0)
[22:04:57.542] Build status: STARTED
[22:04:57.542] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:41:0) after 4m41s
[22:09:38.710] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:41:0)
[22:09:38.781] Build status: STARTED
[22:09:38.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:42:0) after 3m14s
[22:12:52.794] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:42:0)
[22:12:52.812] Build status: STARTED
[22:12:52.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:43:0) after 6m59s
[22:19:51.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:43:0)
[22:19:51.822] Build status: STARTED
[22:19:51.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:44:0) after 7m38s
[22:27:29.917] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:44:0)
[22:27:29.935] Build status: STARTED
[22:27:29.935] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:45:0) after 2m19s
[22:29:49.065] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:45:0)
[22:29:49.106] Build status: STARTED
[22:29:49.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:46:0) after 9m48s
[22:39:37.242] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:46:0)
[22:39:37.264] Build status: STARTED
[22:39:37.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:47:0) after 6m42s
[22:46:19.387] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:47:0)
[22:46:19.401] Build status: STARTED
[22:46:19.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:48:0) after 9m49s
[22:56:08.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:48:0)
[22:56:08.438] Build status: STARTED
[22:56:08.438] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:49:0) after 5m45s
[23:01:53.499] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:49:0)
[23:01:53.555] Build status: STARTED
[23:01:53.555] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:50:0) after 4m40s
[23:06:33.572] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:50:0)
[23:06:33.589] Build status: STARTED
[23:06:33.590] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:51:0) after 6m13s
[23:12:46.764] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:51:0)
[23:12:46.787] Build status: STARTED
[23:12:46.787] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:52:0) after 2m44s
[23:15:31.169] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:52:0)
[23:15:31.184] Build status: STARTED
[23:15:31.184] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:53:0) after 8m8s
[23:23:39.299] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:53:0)
[23:23:39.334] Build status: STARTED
[23:23:39.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:54:0) after 5m51s
[23:29:30.433] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:54:0)
[23:29:30.455] Build status: STARTED
[23:29:30.455] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:55:0) after 5m8s
[23:34:38.901] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:55:0)
[23:34:38.920] Build status: STARTED
[23:34:38.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:56:0) after 4m15s
[23:38:53.938] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:56:0)
[23:38:54.004] Build status: STARTED
[23:38:54.004] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:57:0) after 1m50s
[23:40:44.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:57:0)
[23:40:44.243] Build status: STARTED
[23:40:44.243] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:58:0) after 7m16s
[23:48:00.318] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:58:0)
[23:48:00.334] Build status: STARTED
[23:48:00.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:59:0) after 7m12s
[23:55:12.349] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:59:0)
[23:55:12.370] Build status: STARTED
[23:55:12.371] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:60:0) after 5m55s
[00:01:07.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:60:0)
[00:01:07.401] Build status: STARTED
[00:01:07.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:61:0) after 3m55s
[00:05:02.373] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:61:0)
[00:05:02.391] Build status: STARTED
[00:05:02.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:62:0) after 4m44s
[00:09:46.409] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:62:0)
[00:09:46.446] Build status: STARTED
[00:09:46.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851491640172338448:63:0) after 7m48s
[00:14:42.415] Received PubSub notification, asking Buildbucket for the build status
[00:14:42.632] Build:
{
"id": "8686061578305137137",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2058,
"createdBy": "project:chromium",
"createTime": "2026-03-28T18:39:44.255551272Z",
"startTime": "2026-03-28T18:40:26.336272763Z",
"endTime": "2026-03-29T00:14:42.045480460Z",
"updateTime": "2026-03-29T00:14:42.045480460Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ba944c6b8c640b3e259eb3c2bb2929d9626fc64f",
"ref": "refs/heads/main"
}
}
}
[00:14:42.632] Invocation finished in 5h34m59.541152375s with status FAILED