[03:17:38.905] New invocation is queued and will start shortly
[03:17:40.019] Starting the invocation (attempt 1)
[03:17:40.059] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/96fd268df03cfce5ba9914a72a9e0b2a94d49115"
[03:17:40.060] Popped gitiles commit info from properties and tags
[03:17:40.060] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[03:17:40.060] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[03:17:40.060] Buildbucket request:
{
"requestId": "8831980707056041232",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "96fd268df03cfce5ba9914a72a9e0b2a94d49115",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8831980707056041232"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[03:17:40.831] Scheduled build:
{
"id": "8666550645074268961",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 6041,
"createdBy": "project:chromium",
"createTime": "2026-10-30T03:17:40.166061834Z",
"updateTime": "2026-10-30T03:17:40.166061834Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "96fd268df03cfce5ba9914a72a9e0b2a94d49115",
"ref": "refs/heads/main"
}
},
"canary": true
}
[03:17:40.831] Task URL: https://cr-buildbucket.appspot.com/build/8666550645074268961
[03:17:40.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:2:0) after 6m47s
[03:17:41.841] Received PubSub notification, asking Buildbucket for the build status
[03:17:41.868] Build status: SCHEDULED
[03:17:45.626] Received PubSub notification, asking Buildbucket for the build status
[03:17:45.644] Build status: STARTED
[03:24:27.903] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:2:0)
[03:24:27.951] Build status: STARTED
[03:24:27.956] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:5:0) after 3m49s
[03:28:16.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:5:0)
[03:28:17.016] Build status: STARTED
[03:28:17.016] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:6:0) after 2m17s
[03:30:34.038] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:6:0)
[03:30:34.078] Build status: STARTED
[03:30:34.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:7:0) after 9m13s
[03:39:47.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:7:0)
[03:39:47.171] Build status: STARTED
[03:39:47.171] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:8:0) after 7m46s
[03:47:33.192] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:8:0)
[03:47:33.246] Build status: STARTED
[03:47:33.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:9:0) after 1m48s
[03:49:21.355] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:9:0)
[03:49:21.379] Build status: STARTED
[03:49:21.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:10:0) after 1m21s
[03:50:42.392] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:10:0)
[03:50:42.478] Build status: STARTED
[03:50:42.478] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:11:0) after 7m44s
[03:58:26.548] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:11:0)
[03:58:26.585] Build status: STARTED
[03:58:26.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:12:0) after 5m30s
[04:03:56.706] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:12:0)
[04:03:56.727] Build status: STARTED
[04:03:56.727] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:13:0) after 3m50s
[04:07:46.888] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:13:0)
[04:07:46.926] Build status: STARTED
[04:07:46.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:14:0) after 9m24s
[04:17:10.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:14:0)
[04:17:10.931] Build status: STARTED
[04:17:10.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:15:0) after 8m42s
[04:25:53.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:15:0)
[04:25:53.070] Build status: STARTED
[04:25:53.070] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:16:0) after 9m51s
[04:35:44.074] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:16:0)
[04:35:44.173] Build status: STARTED
[04:35:44.173] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:17:0) after 9m55s
[04:45:39.301] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:17:0)
[04:45:39.321] Build status: STARTED
[04:45:39.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:18:0) after 6m58s
[04:52:37.346] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:18:0)
[04:52:37.441] Build status: STARTED
[04:52:37.441] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:19:0) after 5m12s
[04:57:49.466] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:19:0)
[04:57:49.501] Build status: STARTED
[04:57:49.501] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:20:0) after 1m54s
[04:59:43.523] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:20:0)
[04:59:43.543] Build status: STARTED
[04:59:43.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:21:0) after 8m4s
[05:07:47.596] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:21:0)
[05:07:47.614] Build status: STARTED
[05:07:47.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:22:0) after 1m29s
[05:09:16.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:22:0)
[05:09:16.666] Build status: STARTED
[05:09:16.666] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:23:0) after 5m0s
[05:14:16.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:23:0)
[05:14:16.830] Build status: STARTED
[05:14:16.830] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:24:0) after 1m55s
[05:16:11.908] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:24:0)
[05:16:12.017] Build status: STARTED
[05:16:12.017] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:25:0) after 1m47s
[05:17:59.143] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:25:0)
[05:17:59.197] Build status: STARTED
[05:17:59.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:26:0) after 3m19s
[05:21:18.223] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:26:0)
[05:21:18.292] Build status: STARTED
[05:21:18.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:27:0) after 1m14s
[05:22:32.387] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:27:0)
[05:22:32.407] Build status: STARTED
[05:22:32.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:28:0) after 5m53s
[05:28:25.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:28:0)
[05:28:25.453] Build status: STARTED
[05:28:25.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:29:0) after 3m0s
[05:31:25.486] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:29:0)
[05:31:25.503] Build status: STARTED
[05:31:25.503] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:30:0) after 5m4s
[05:36:29.527] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:30:0)
[05:36:29.552] Build status: STARTED
[05:36:29.552] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:31:0) after 1m37s
[05:38:06.580] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:31:0)
[05:38:06.614] Build status: STARTED
[05:38:06.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:32:0) after 3m6s
[05:41:12.673] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:32:0)
[05:41:12.709] Build status: STARTED
[05:41:12.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:33:0) after 4m57s
[05:46:09.747] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:33:0)
[05:46:09.775] Build status: STARTED
[05:46:09.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:34:0) after 4m11s
[05:50:20.782] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:34:0)
[05:50:20.848] Build status: STARTED
[05:50:20.848] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:35:0) after 5m51s
[05:56:11.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:35:0)
[05:56:11.932] Build status: STARTED
[05:56:11.932] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:36:0) after 6m46s
[06:02:57.951] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:36:0)
[06:02:57.970] Build status: STARTED
[06:02:57.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:37:0) after 7m47s
[06:10:45.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:37:0)
[06:10:45.079] Build status: STARTED
[06:10:45.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:38:0) after 4m59s
[06:15:44.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:38:0)
[06:15:44.184] Build status: STARTED
[06:15:44.184] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:39:0) after 1m45s
[06:17:29.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:39:0)
[06:17:29.274] Build status: STARTED
[06:17:29.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:40:0) after 7m5s
[06:24:34.448] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:40:0)
[06:24:34.511] Build status: STARTED
[06:24:34.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:41:0) after 8m7s
[06:32:41.590] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:41:0)
[06:32:41.611] Build status: STARTED
[06:32:41.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8831980707056041232:42:0) after 7m21s
[06:34:06.121] Received PubSub notification, asking Buildbucket for the build status
[06:34:06.148] Build:
{
"id": "8666550645074268961",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 6041,
"createdBy": "project:chromium",
"createTime": "2026-10-30T03:17:40.166061834Z",
"startTime": "2026-10-30T03:17:45.432166006Z",
"endTime": "2026-10-30T06:34:05.874928868Z",
"updateTime": "2026-10-30T06:34:05.874928868Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "96fd268df03cfce5ba9914a72a9e0b2a94d49115",
"ref": "refs/heads/main"
}
},
"canary": true
}
[06:34:06.148] Invocation finished in 3h16m27.255475822s with status FAILED