[01:15:46.910] New invocation is queued and will start shortly
[01:15:47.990] Starting the invocation (attempt 1)
[01:15:48.017] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/cbc0d6a414e2ab8f57afbea4a93fab98a6f783d3"
[01:15:48.017] Popped gitiles commit info from properties and tags
[01:15:48.017] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:15:48.017] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:15:48.017] Buildbucket request:
{
"requestId": "8842860010200152336",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "cbc0d6a414e2ab8f57afbea4a93fab98a6f783d3",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842860010200152336"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:15:49.668] Scheduled build:
{
"id": "8677429948314568529",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3597,
"createdBy": "project:chromium",
"createTime": "2026-07-02T01:15:48.085594126Z",
"updateTime": "2026-07-02T01:15:48.085594126Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "cbc0d6a414e2ab8f57afbea4a93fab98a6f783d3",
"ref": "refs/heads/main"
}
}
}
[01:15:49.668] Task URL: https://cr-buildbucket.appspot.com/build/8677429948314568529
[01:15:49.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:2:0) after 9m12s
[01:15:50.167] Received PubSub notification, asking Buildbucket for the build status
[01:15:50.181] Build status: SCHEDULED
[01:15:56.422] Received PubSub notification, asking Buildbucket for the build status
[01:15:56.452] Build status: STARTED
[01:25:01.732] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:2:0)
[01:25:01.751] Build status: STARTED
[01:25:01.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:5:0) after 5m16s
[01:30:17.801] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:5:0)
[01:30:17.857] Build status: STARTED
[01:30:17.857] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:6:0) after 7m31s
[01:37:48.872] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:6:0)
[01:37:48.929] Build status: STARTED
[01:37:48.929] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:7:0) after 4m25s
[01:42:13.907] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:7:0)
[01:42:13.926] Build status: STARTED
[01:42:13.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:8:0) after 5m45s
[01:47:58.989] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:8:0)
[01:47:59.015] Build status: STARTED
[01:47:59.015] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:9:0) after 3m24s
[01:51:23.085] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:9:0)
[01:51:23.103] Build status: STARTED
[01:51:23.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:10:0) after 5m10s
[01:56:33.121] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:10:0)
[01:56:33.137] Build status: STARTED
[01:56:33.137] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:11:0) after 6m59s
[02:03:32.259] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:11:0)
[02:03:32.280] Build status: STARTED
[02:03:32.280] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:12:0) after 2m38s
[02:06:10.287] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:12:0)
[02:06:10.311] Build status: STARTED
[02:06:10.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:13:0) after 9m18s
[02:15:28.420] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:13:0)
[02:15:28.447] Build status: STARTED
[02:15:28.447] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:14:0) after 9m42s
[02:25:10.490] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:14:0)
[02:25:10.507] Build status: STARTED
[02:25:10.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:15:0) after 7m45s
[02:32:55.593] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:15:0)
[02:32:55.613] Build status: STARTED
[02:32:55.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:16:0) after 3m48s
[02:36:43.629] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:16:0)
[02:36:43.648] Build status: STARTED
[02:36:43.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:17:0) after 6m18s
[02:43:01.746] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:17:0)
[02:43:01.764] Build status: STARTED
[02:43:01.764] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:18:0) after 2m56s
[02:45:57.841] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:18:0)
[02:45:57.865] Build status: STARTED
[02:45:57.865] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:19:0) after 3m44s
[02:49:42.154] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:19:0)
[02:49:42.175] Build status: STARTED
[02:49:42.176] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:20:0) after 2m2s
[02:51:44.171] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:20:0)
[02:51:44.192] Build status: STARTED
[02:51:44.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:21:0) after 5m12s
[02:56:56.222] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:21:0)
[02:56:56.257] Build status: STARTED
[02:56:56.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:22:0) after 5m17s
[03:02:13.289] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:22:0)
[03:02:13.342] Build status: STARTED
[03:02:13.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:23:0) after 4m34s
[03:06:47.397] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:23:0)
[03:06:47.437] Build status: STARTED
[03:06:47.438] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:24:0) after 3m6s
[03:09:53.475] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:24:0)
[03:09:53.504] Build status: STARTED
[03:09:53.504] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:25:0) after 5m23s
[03:15:16.558] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:25:0)
[03:15:16.575] Build status: STARTED
[03:15:16.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:26:0) after 5m3s
[03:20:19.593] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:26:0)
[03:20:19.620] Build status: STARTED
[03:20:19.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:27:0) after 9m48s
[03:30:07.636] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:27:0)
[03:30:07.655] Build status: STARTED
[03:30:07.655] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:28:0) after 7m7s
[03:37:14.671] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:28:0)
[03:37:14.703] Build status: STARTED
[03:37:14.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:29:0) after 9m31s
[03:46:45.730] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:29:0)
[03:46:45.745] Build status: STARTED
[03:46:45.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:30:0) after 7m52s
[03:54:37.796] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:30:0)
[03:54:37.861] Build status: STARTED
[03:54:37.862] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:31:0) after 6m39s
[04:01:16.904] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:31:0)
[04:01:16.921] Build status: STARTED
[04:01:16.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:32:0) after 3m11s
[04:04:27.962] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:32:0)
[04:04:27.990] Build status: STARTED
[04:04:27.990] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:33:0) after 6m17s
[04:10:45.034] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:33:0)
[04:10:45.139] Build status: STARTED
[04:10:45.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:34:0) after 9m18s
[04:20:03.207] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:34:0)
[04:20:03.226] Build status: STARTED
[04:20:03.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:35:0) after 2m25s
[04:22:28.292] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:35:0)
[04:22:28.312] Build status: STARTED
[04:22:28.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:36:0) after 6m29s
[04:28:57.330] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:36:0)
[04:28:57.350] Build status: STARTED
[04:28:57.350] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:37:0) after 5m39s
[04:34:36.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:37:0)
[04:34:36.446] Build status: STARTED
[04:34:36.447] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:38:0) after 8m27s
[04:43:03.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:38:0)
[04:43:03.539] Build status: STARTED
[04:43:03.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:39:0) after 4m30s
[04:47:33.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:39:0)
[04:47:33.617] Build status: STARTED
[04:47:33.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:40:0) after 4m15s
[04:51:48.639] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:40:0)
[04:51:48.676] Build status: STARTED
[04:51:48.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:41:0) after 5m12s
[04:57:00.695] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:41:0)
[04:57:00.735] Build status: STARTED
[04:57:00.736] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:42:0) after 7m44s
[05:04:44.801] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:42:0)
[05:04:44.815] Build status: STARTED
[05:04:44.815] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:43:0) after 4m56s
[05:09:40.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:43:0)
[05:09:40.946] Build status: STARTED
[05:09:40.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:44:0) after 1m13s
[05:10:54.006] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:44:0)
[05:10:54.056] Build status: STARTED
[05:10:54.056] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:45:0) after 3m12s
[05:14:06.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:45:0)
[05:14:06.126] Build status: STARTED
[05:14:06.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:46:0) after 3m47s
[05:17:53.202] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:46:0)
[05:17:53.225] Build status: STARTED
[05:17:53.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:47:0) after 2m50s
[05:20:43.244] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:47:0)
[05:20:43.259] Build status: STARTED
[05:20:43.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:48:0) after 8m13s
[05:28:56.275] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:48:0)
[05:28:56.293] Build status: STARTED
[05:28:56.293] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:49:0) after 1m41s
[05:30:37.467] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:49:0)
[05:30:37.560] Build status: STARTED
[05:30:37.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:50:0) after 2m48s
[05:33:25.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:50:0)
[05:33:25.788] Build status: STARTED
[05:33:25.788] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:51:0) after 6m54s
[05:40:19.848] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:51:0)
[05:40:19.867] Build status: STARTED
[05:40:19.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:52:0) after 8m48s
[05:49:07.885] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:52:0)
[05:49:07.987] Build status: STARTED
[05:49:07.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:53:0) after 3m10s
[05:52:18.005] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:53:0)
[05:52:18.052] Build status: STARTED
[05:52:18.052] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:54:0) after 4m52s
[05:57:10.070] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:54:0)
[05:57:10.103] Build status: STARTED
[05:57:10.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:55:0) after 9m29s
[06:06:39.132] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:55:0)
[06:06:39.149] Build status: STARTED
[06:06:39.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:56:0) after 1m58s
[06:08:37.211] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:56:0)
[06:08:37.234] Build status: STARTED
[06:08:37.234] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842860010200152336:57:0) after 5m5s
[06:09:46.133] Received PubSub notification, asking Buildbucket for the build status
[06:09:46.151] Build:
{
"id": "8677429948314568529",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3597,
"createdBy": "project:chromium",
"createTime": "2026-07-02T01:15:48.085594126Z",
"startTime": "2026-07-02T01:15:56.246828495Z",
"endTime": "2026-07-02T06:09:45.896015167Z",
"updateTime": "2026-07-02T06:09:45.896015167Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "cbc0d6a414e2ab8f57afbea4a93fab98a6f783d3",
"ref": "refs/heads/main"
}
}
}
[06:09:46.151] Invocation finished in 4h53m59.247379742s with status FAILED