[01:55:41.317] New invocation is queued and will start shortly
[01:55:42.362] Starting the invocation (attempt 1)
[01:55:42.385] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/40917eb28d3941d7ac9c5341f6e0399adb1f7e55"
[01:55:42.385] Popped gitiles commit info from properties and tags
[01:55:42.385] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:55:42.385] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:55:42.386] Buildbucket request:
{
"requestId": "8843672872179492592",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "40917eb28d3941d7ac9c5341f6e0399adb1f7e55",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8843672872179492592"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:55:42.992] Scheduled build:
{
"id": "8678242810331362545",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3538,
"createdBy": "project:chromium",
"createTime": "2026-06-23T01:55:42.457138830Z",
"updateTime": "2026-06-23T01:55:42.457138830Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "40917eb28d3941d7ac9c5341f6e0399adb1f7e55",
"ref": "refs/heads/main"
}
}
}
[01:55:42.993] Task URL: https://cr-buildbucket.appspot.com/build/8678242810331362545
[01:55:42.993] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:2:0) after 8m21s
[01:55:44.021] Received PubSub notification, asking Buildbucket for the build status
[01:55:44.073] Build status: SCHEDULED
[01:55:46.926] Received PubSub notification, asking Buildbucket for the build status
[01:55:46.942] Build status: STARTED
[02:04:04.038] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:2:0)
[02:04:04.065] Build status: STARTED
[02:04:04.065] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:5:0) after 7m55s
[02:11:59.156] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:5:0)
[02:11:59.178] Build status: STARTED
[02:11:59.178] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:6:0) after 9m39s
[02:21:38.197] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:6:0)
[02:21:38.218] Build status: STARTED
[02:21:38.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:7:0) after 7m32s
[02:29:10.242] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:7:0)
[02:29:10.263] Build status: STARTED
[02:29:10.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:8:0) after 9m30s
[02:38:40.377] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:8:0)
[02:38:40.435] Build status: STARTED
[02:38:40.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:9:0) after 2m54s
[02:41:34.438] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:9:0)
[02:41:34.516] Build status: STARTED
[02:41:34.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:10:0) after 9m50s
[02:51:24.624] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:10:0)
[02:51:24.647] Build status: STARTED
[02:51:24.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:11:0) after 2m4s
[02:53:28.669] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:11:0)
[02:53:28.686] Build status: STARTED
[02:53:28.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:12:0) after 7m58s
[03:01:26.753] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:12:0)
[03:01:26.771] Build status: STARTED
[03:01:26.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:13:0) after 5m38s
[03:07:04.857] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:13:0)
[03:07:04.882] Build status: STARTED
[03:07:04.882] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:14:0) after 5m37s
[03:12:41.935] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:14:0)
[03:12:41.976] Build status: STARTED
[03:12:41.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:15:0) after 1m1s
[03:13:42.950] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:15:0)
[03:13:42.969] Build status: STARTED
[03:13:42.969] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:16:0) after 3m14s
[03:16:56.989] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:16:0)
[03:16:57.009] Build status: STARTED
[03:16:57.009] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:17:0) after 4m35s
[03:21:32.039] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:17:0)
[03:21:32.084] Build status: STARTED
[03:21:32.084] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:18:0) after 1m44s
[03:23:16.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:18:0)
[03:23:16.182] Build status: STARTED
[03:23:16.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:19:0) after 3m18s
[03:26:34.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:19:0)
[03:26:34.330] Build status: STARTED
[03:26:34.330] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:20:0) after 3m56s
[03:30:30.350] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:20:0)
[03:30:30.467] Build status: STARTED
[03:30:30.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:21:0) after 9m53s
[03:40:23.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:21:0)
[03:40:23.563] Build status: STARTED
[03:40:23.563] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:22:0) after 1m43s
[03:42:06.631] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:22:0)
[03:42:06.649] Build status: STARTED
[03:42:06.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:23:0) after 6m15s
[03:48:21.670] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:23:0)
[03:48:21.689] Build status: STARTED
[03:48:21.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:24:0) after 3m8s
[03:51:29.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:24:0)
[03:51:29.755] Build status: STARTED
[03:51:29.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:25:0) after 6m14s
[03:57:43.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:25:0)
[03:57:43.917] Build status: STARTED
[03:57:43.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:26:0) after 1m37s
[03:59:20.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:26:0)
[03:59:20.954] Build status: STARTED
[03:59:20.954] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:27:0) after 6m0s
[04:05:21.069] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:27:0)
[04:05:21.089] Build status: STARTED
[04:05:21.090] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:28:0) after 4m43s
[04:10:04.066] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:28:0)
[04:10:04.086] Build status: STARTED
[04:10:04.086] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:29:0) after 6m2s
[04:16:06.201] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:29:0)
[04:16:06.221] Build status: STARTED
[04:16:06.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:30:0) after 1m43s
[04:17:49.264] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:30:0)
[04:17:49.287] Build status: STARTED
[04:17:49.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:31:0) after 5m50s
[04:23:39.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:31:0)
[04:23:39.367] Build status: STARTED
[04:23:39.367] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:32:0) after 7m30s
[04:31:09.436] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:32:0)
[04:31:09.460] Build status: STARTED
[04:31:09.461] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:33:0) after 6m48s
[04:37:57.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:33:0)
[04:37:57.571] Build status: STARTED
[04:37:57.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:34:0) after 3m9s
[04:41:06.589] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:34:0)
[04:41:06.611] Build status: STARTED
[04:41:06.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:35:0) after 5m50s
[04:46:56.632] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:35:0)
[04:46:56.657] Build status: STARTED
[04:46:56.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:36:0) after 3m42s
[04:50:38.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:36:0)
[04:50:38.746] Build status: STARTED
[04:50:38.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:37:0) after 1m33s
[04:52:11.816] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:37:0)
[04:52:11.832] Build status: STARTED
[04:52:11.833] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:38:0) after 6m42s
[04:58:54.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:38:0)
[04:58:54.080] Build status: STARTED
[04:58:54.080] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:39:0) after 8m40s
[05:07:34.073] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:39:0)
[05:07:34.158] Build status: STARTED
[05:07:34.158] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:40:0) after 1m8s
[05:08:42.190] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:40:0)
[05:08:42.221] Build status: STARTED
[05:08:42.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:41:0) after 1m33s
[05:10:15.288] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:41:0)
[05:10:15.303] Build status: STARTED
[05:10:15.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:42:0) after 2m4s
[05:12:19.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:42:0)
[05:12:19.348] Build status: STARTED
[05:12:19.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:43:0) after 1m8s
[05:13:27.429] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:43:0)
[05:13:27.451] Build status: STARTED
[05:13:27.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:44:0) after 2m20s
[05:15:47.494] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:44:0)
[05:15:47.548] Build status: STARTED
[05:15:47.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:45:0) after 1m47s
[05:17:34.605] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:45:0)
[05:17:34.636] Build status: STARTED
[05:17:34.636] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:46:0) after 1m6s
[05:18:40.658] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:46:0)
[05:18:40.710] Build status: STARTED
[05:18:40.710] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:47:0) after 5m4s
[05:23:44.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:47:0)
[05:23:44.802] Build status: STARTED
[05:23:44.802] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:48:0) after 3m4s
[05:26:48.832] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:48:0)
[05:26:48.850] Build status: STARTED
[05:26:48.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:49:0) after 3m1s
[05:29:49.822] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:49:0)
[05:29:49.909] Build status: STARTED
[05:29:49.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:50:0) after 1m25s
[05:31:14.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:50:0)
[05:31:14.945] Build status: STARTED
[05:31:14.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:51:0) after 6m15s
[05:37:30.005] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:51:0)
[05:37:30.025] Build status: STARTED
[05:37:30.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:52:0) after 7m13s
[05:44:43.045] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:52:0)
[05:44:43.066] Build status: STARTED
[05:44:43.066] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:53:0) after 6m8s
[05:50:51.083] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:53:0)
[05:50:51.103] Build status: STARTED
[05:50:51.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:54:0) after 3m12s
[05:54:03.169] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:54:0)
[05:54:03.192] Build status: STARTED
[05:54:03.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:55:0) after 5m32s
[05:59:35.211] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:55:0)
[05:59:35.230] Build status: STARTED
[05:59:35.230] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:56:0) after 4m49s
[06:04:24.250] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:56:0)
[06:04:24.274] Build status: STARTED
[06:04:24.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:57:0) after 4m17s
[06:08:41.342] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:57:0)
[06:08:41.363] Build status: STARTED
[06:08:41.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:58:0) after 8m52s
[06:17:33.527] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:58:0)
[06:17:33.550] Build status: STARTED
[06:17:33.550] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:59:0) after 5m43s
[06:23:16.571] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:59:0)
[06:23:16.611] Build status: STARTED
[06:23:16.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:60:0) after 6m38s
[06:29:54.633] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:60:0)
[06:29:54.650] Build status: STARTED
[06:29:54.650] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:61:0) after 9m10s
[06:39:04.686] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:61:0)
[06:39:04.722] Build status: STARTED
[06:39:04.723] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:62:0) after 2m26s
[06:41:30.741] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:62:0)
[06:41:30.763] Build status: STARTED
[06:41:30.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:63:0) after 2m19s
[06:43:49.741] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:63:0)
[06:43:49.769] Build status: STARTED
[06:43:49.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8843672872179492592:64:0) after 8m41s
[06:47:26.204] Received PubSub notification, asking Buildbucket for the build status
[06:47:26.226] Build:
{
"id": "8678242810331362545",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3538,
"createdBy": "project:chromium",
"createTime": "2026-06-23T01:55:42.457138830Z",
"startTime": "2026-06-23T01:55:46.726506777Z",
"endTime": "2026-06-23T06:47:20.451715113Z",
"updateTime": "2026-06-23T06:47:20.451715113Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "40917eb28d3941d7ac9c5341f6e0399adb1f7e55",
"ref": "refs/heads/main"
}
}
}
[06:47:26.226] Invocation finished in 4h51m44.914466998s with status FAILED