[22:55:04.439] New invocation is queued and will start shortly
[22:55:05.604] Starting the invocation (attempt 1)
[22:55:05.629] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/4ebb65aa4e2c23524138556ee24b5665c5afaccd"
[22:55:05.629] Popped gitiles commit info from properties and tags
[22:55:05.629] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[22:55:05.630] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[22:55:05.630] Buildbucket request:
{
"requestId": "8841238117375158752",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4ebb65aa4e2c23524138556ee24b5665c5afaccd",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841238117375158752"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[22:55:06.092] Scheduled build:
{
"id": "8675808055408395905",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3888,
"createdBy": "project:chromium",
"createTime": "2026-07-19T22:55:05.694208771Z",
"updateTime": "2026-07-19T22:55:05.694208771Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4ebb65aa4e2c23524138556ee24b5665c5afaccd",
"ref": "refs/heads/main"
}
}
}
[22:55:06.093] Task URL: https://cr-buildbucket.appspot.com/build/8675808055408395905
[22:55:06.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:2:0) after 3m11s
[22:55:07.034] Received PubSub notification, asking Buildbucket for the build status
[22:55:07.051] Build status: SCHEDULED
[22:55:10.078] Received PubSub notification, asking Buildbucket for the build status
[22:55:10.098] Build status: STARTED
[22:58:17.232] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:2:0)
[22:58:17.268] Build status: STARTED
[22:58:17.268] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:5:0) after 2m55s
[23:01:12.531] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:5:0)
[23:01:12.557] Build status: STARTED
[23:01:12.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:6:0) after 6m0s
[23:07:13.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:6:0)
[23:07:13.068] Build status: STARTED
[23:07:13.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:7:0) after 1m12s
[23:08:25.144] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:7:0)
[23:08:25.165] Build status: STARTED
[23:08:25.165] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:8:0) after 8m52s
[23:17:17.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:8:0)
[23:17:17.235] Build status: STARTED
[23:17:17.235] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:9:0) after 6m4s
[23:23:21.334] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:9:0)
[23:23:21.352] Build status: STARTED
[23:23:21.352] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:10:0) after 1m22s
[23:24:43.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:10:0)
[23:24:43.390] Build status: STARTED
[23:24:43.390] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:11:0) after 4m14s
[23:28:57.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:11:0)
[23:28:57.472] Build status: STARTED
[23:28:57.472] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:12:0) after 6m10s
[23:35:07.488] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:12:0)
[23:35:07.510] Build status: STARTED
[23:35:07.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:13:0) after 5m35s
[23:40:42.604] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:13:0)
[23:40:42.625] Build status: STARTED
[23:40:42.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:14:0) after 4m29s
[23:45:11.618] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:14:0)
[23:45:11.653] Build status: STARTED
[23:45:11.653] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:15:0) after 5m30s
[23:50:41.666] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:15:0)
[23:50:41.683] Build status: STARTED
[23:50:41.683] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:16:0) after 8m0s
[23:58:41.968] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:16:0)
[23:58:41.982] Build status: STARTED
[23:58:41.982] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:17:0) after 5m30s
[00:04:12.001] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:17:0)
[00:04:12.022] Build status: STARTED
[00:04:12.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:18:0) after 6m4s
[00:10:16.057] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:18:0)
[00:10:16.077] Build status: STARTED
[00:10:16.077] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:19:0) after 5m22s
[00:15:38.535] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:19:0)
[00:15:38.616] Build status: STARTED
[00:15:38.616] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:20:0) after 8m55s
[00:24:33.887] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:20:0)
[00:24:33.912] Build status: STARTED
[00:24:33.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:21:0) after 7m35s
[00:32:08.975] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:21:0)
[00:32:09.004] Build status: STARTED
[00:32:09.004] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:22:0) after 2m0s
[00:34:09.145] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:22:0)
[00:34:09.167] Build status: STARTED
[00:34:09.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:23:0) after 8m6s
[00:42:15.277] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:23:0)
[00:42:15.293] Build status: STARTED
[00:42:15.293] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:24:0) after 5m28s
[00:47:43.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:24:0)
[00:47:43.327] Build status: STARTED
[00:47:43.327] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:25:0) after 8m46s
[00:56:29.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:25:0)
[00:56:29.642] Build status: STARTED
[00:56:29.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:26:0) after 2m46s
[00:59:15.658] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:26:0)
[00:59:15.675] Build status: STARTED
[00:59:15.675] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:27:0) after 4m54s
[01:04:09.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:27:0)
[01:04:09.715] Build status: STARTED
[01:04:09.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:28:0) after 4m44s
[01:08:53.913] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:28:0)
[01:08:53.928] Build status: STARTED
[01:08:53.928] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:29:0) after 3m9s
[01:12:03.213] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:29:0)
[01:12:03.236] Build status: STARTED
[01:12:03.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:30:0) after 7m58s
[01:20:01.329] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:30:0)
[01:20:01.348] Build status: STARTED
[01:20:01.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:31:0) after 5m25s
[01:25:26.427] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:31:0)
[01:25:26.441] Build status: STARTED
[01:25:26.441] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:32:0) after 4m50s
[01:30:16.494] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:32:0)
[01:30:16.531] Build status: STARTED
[01:30:16.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:33:0) after 1m3s
[01:31:19.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:33:0)
[01:31:19.686] Build status: STARTED
[01:31:19.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:34:0) after 5m57s
[01:37:16.769] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:34:0)
[01:37:16.794] Build status: STARTED
[01:37:16.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:35:0) after 5m43s
[01:42:59.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:35:0)
[01:42:59.831] Build status: STARTED
[01:42:59.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:36:0) after 2m17s
[01:45:16.846] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:36:0)
[01:45:16.888] Build status: STARTED
[01:45:16.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:37:0) after 3m22s
[01:48:39.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:37:0)
[01:48:39.076] Build status: STARTED
[01:48:39.076] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:38:0) after 2m23s
[01:51:02.145] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:38:0)
[01:51:02.162] Build status: STARTED
[01:51:02.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:39:0) after 3m44s
[01:54:46.223] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:39:0)
[01:54:46.240] Build status: STARTED
[01:54:46.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:40:0) after 2m28s
[01:57:14.356] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:40:0)
[01:57:14.374] Build status: STARTED
[01:57:14.374] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:41:0) after 6m6s
[02:03:20.391] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:41:0)
[02:03:20.409] Build status: STARTED
[02:03:20.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:42:0) after 8m10s
[02:11:30.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:42:0)
[02:11:30.488] Build status: STARTED
[02:11:30.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:43:0) after 1m44s
[02:13:14.505] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:43:0)
[02:13:14.523] Build status: STARTED
[02:13:14.523] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:44:0) after 4m6s
[02:17:20.542] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:44:0)
[02:17:20.559] Build status: STARTED
[02:17:20.559] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:45:0) after 1m1s
[02:18:21.681] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:45:0)
[02:18:21.705] Build status: STARTED
[02:18:21.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:46:0) after 7m15s
[02:25:36.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:46:0)
[02:25:36.764] Build status: STARTED
[02:25:36.764] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:47:0) after 2m2s
[02:27:38.788] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:47:0)
[02:27:38.935] Build status: STARTED
[02:27:38.935] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:48:0) after 3m22s
[02:31:01.000] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:48:0)
[02:31:01.025] Build status: STARTED
[02:31:01.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:49:0) after 9m5s
[02:40:06.122] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:49:0)
[02:40:06.144] Build status: STARTED
[02:40:06.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:50:0) after 3m56s
[02:44:02.160] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:50:0)
[02:44:02.258] Build status: STARTED
[02:44:02.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:51:0) after 1m30s
[02:45:32.290] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:51:0)
[02:45:32.346] Build status: STARTED
[02:45:32.346] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:52:0) after 9m5s
[02:54:37.412] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:52:0)
[02:54:37.435] Build status: STARTED
[02:54:37.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:53:0) after 1m43s
[02:56:20.490] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:53:0)
[02:56:20.510] Build status: STARTED
[02:56:20.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:54:0) after 4m55s
[03:01:15.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:54:0)
[03:01:15.544] Build status: STARTED
[03:01:15.544] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:55:0) after 8m40s
[03:09:55.570] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:55:0)
[03:09:55.589] Build status: STARTED
[03:09:55.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:56:0) after 7m15s
[03:17:10.630] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:56:0)
[03:17:10.647] Build status: STARTED
[03:17:10.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:57:0) after 6m12s
[03:23:22.703] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:57:0)
[03:23:22.720] Build status: STARTED
[03:23:22.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:58:0) after 9m16s
[03:32:38.784] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:58:0)
[03:32:38.837] Build status: STARTED
[03:32:38.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:59:0) after 5m13s
[03:37:51.853] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:59:0)
[03:37:51.887] Build status: STARTED
[03:37:51.887] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:60:0) after 2m9s
[03:40:00.918] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:60:0)
[03:40:00.939] Build status: STARTED
[03:40:00.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:61:0) after 2m3s
[03:42:04.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:61:0)
[03:42:04.030] Build status: STARTED
[03:42:04.030] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:62:0) after 4m59s
[03:47:03.013] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:62:0)
[03:47:03.028] Build status: STARTED
[03:47:03.028] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841238117375158752:63:0) after 7m44s
[03:53:59.791] Received PubSub notification, asking Buildbucket for the build status
[03:53:59.840] Build:
{
"id": "8675808055408395905",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3888,
"createdBy": "project:chromium",
"createTime": "2026-07-19T22:55:05.694208771Z",
"startTime": "2026-07-19T22:55:09.901616633Z",
"endTime": "2026-07-20T03:53:59.539828214Z",
"updateTime": "2026-07-20T03:53:59.539828214Z",
"status": "INFRA_FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4ebb65aa4e2c23524138556ee24b5665c5afaccd",
"ref": "refs/heads/main"
}
}
}
[03:53:59.840] Invocation finished in 4h58m55.405532242s with status FAILED