[21:42:41.307] New invocation is queued and will start shortly
[21:42:42.356] Starting the invocation (attempt 1)
[21:42:42.380] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/522dfde6f6ff8dc7536883ee7da83309a50f26f4"
[21:42:42.380] Popped gitiles commit info from properties and tags
[21:42:42.380] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:42:42.381] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:42:42.381] Buildbucket request:
{
"requestId": "8858456095100938176",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "522dfde6f6ff8dc7536883ee7da83309a50f26f4",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8858456095100938176"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:42:43.068] Scheduled build:
{
"id": "8693026033220891841",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1157,
"createdBy": "project:chromium",
"createTime": "2026-01-10T21:42:42.474670211Z",
"updateTime": "2026-01-10T21:42:42.474670211Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "522dfde6f6ff8dc7536883ee7da83309a50f26f4",
"ref": "refs/heads/main"
}
}
}
[21:42:43.068] Task URL: https://cr-buildbucket.appspot.com/build/8693026033220891841
[21:42:43.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:2:0) after 5m20s
[21:42:43.982] Received PubSub notification, asking Buildbucket for the build status
[21:42:44.005] Build status: SCHEDULED
[21:42:48.202] Received PubSub notification, asking Buildbucket for the build status
[21:42:48.226] Build status: STARTED
[21:48:03.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:2:0)
[21:48:03.138] Build status: STARTED
[21:48:03.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:5:0) after 5m57s
[21:54:00.157] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:5:0)
[21:54:00.236] Build status: STARTED
[21:54:00.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:6:0) after 2m54s
[21:56:54.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:6:0)
[21:56:54.225] Build status: STARTED
[21:56:54.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:7:0) after 4m18s
[22:01:12.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:7:0)
[22:01:12.258] Build status: STARTED
[22:01:12.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:8:0) after 5m27s
[22:06:39.285] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:8:0)
[22:06:39.301] Build status: STARTED
[22:06:39.301] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:9:0) after 7m3s
[22:13:42.615] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:9:0)
[22:13:42.638] Build status: STARTED
[22:13:42.638] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:10:0) after 6m9s
[22:19:51.765] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:10:0)
[22:19:51.782] Build status: STARTED
[22:19:51.782] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:11:0) after 7m57s
[22:27:48.798] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:11:0)
[22:27:48.872] Build status: STARTED
[22:27:48.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:12:0) after 1m37s
[22:29:25.910] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:12:0)
[22:29:25.925] Build status: STARTED
[22:29:25.925] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:13:0) after 7m55s
[22:37:20.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:13:0)
[22:37:20.968] Build status: STARTED
[22:37:20.968] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:14:0) after 7m19s
[22:44:40.015] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:14:0)
[22:44:40.039] Build status: STARTED
[22:44:40.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:15:0) after 5m46s
[22:50:26.472] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:15:0)
[22:50:26.499] Build status: STARTED
[22:50:26.499] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:16:0) after 9m32s
[22:59:58.513] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:16:0)
[22:59:58.539] Build status: STARTED
[22:59:58.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:17:0) after 5m34s
[23:05:32.756] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:17:0)
[23:05:32.849] Build status: STARTED
[23:05:32.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:18:0) after 8m30s
[23:14:02.861] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:18:0)
[23:14:02.912] Build status: STARTED
[23:14:02.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:19:0) after 7m50s
[23:21:53.184] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:19:0)
[23:21:53.200] Build status: STARTED
[23:21:53.200] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:20:0) after 6m1s
[23:27:54.435] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:20:0)
[23:27:54.453] Build status: STARTED
[23:27:54.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:21:0) after 9m41s
[23:37:35.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:21:0)
[23:37:35.460] Build status: STARTED
[23:37:35.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:22:0) after 5m3s
[23:42:38.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:22:0)
[23:42:38.511] Build status: STARTED
[23:42:38.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:23:0) after 8m34s
[23:51:12.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:23:0)
[23:51:12.547] Build status: STARTED
[23:51:12.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:24:0) after 1m30s
[23:52:42.636] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:24:0)
[23:52:42.651] Build status: STARTED
[23:52:42.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:25:0) after 3m24s
[23:56:06.618] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:25:0)
[23:56:06.633] Build status: STARTED
[23:56:06.633] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:26:0) after 1m15s
[23:57:21.746] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:26:0)
[23:57:21.759] Build status: STARTED
[23:57:21.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:27:0) after 7m40s
[00:05:01.779] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:27:0)
[00:05:01.826] Build status: STARTED
[00:05:01.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:28:0) after 1m51s
[00:06:52.839] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:28:0)
[00:06:52.852] Build status: STARTED
[00:06:52.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:29:0) after 7m53s
[00:14:46.167] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:29:0)
[00:14:46.190] Build status: STARTED
[00:14:46.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:30:0) after 3m1s
[00:17:47.394] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:30:0)
[00:17:47.424] Build status: STARTED
[00:17:47.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:31:0) after 5m55s
[00:23:42.710] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:31:0)
[00:23:42.724] Build status: STARTED
[00:23:42.724] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:32:0) after 9m12s
[00:32:54.742] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:32:0)
[00:32:54.757] Build status: STARTED
[00:32:54.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:33:0) after 7m19s
[00:40:13.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:33:0)
[00:40:13.798] Build status: STARTED
[00:40:13.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:34:0) after 8m44s
[00:48:58.103] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:34:0)
[00:48:58.361] Build status: STARTED
[00:48:58.362] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:35:0) after 5m21s
[00:54:19.377] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:35:0)
[00:54:19.396] Build status: STARTED
[00:54:19.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:36:0) after 5m41s
[01:00:00.492] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:36:0)
[01:00:00.516] Build status: STARTED
[01:00:00.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:37:0) after 8m16s
[01:08:16.533] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:37:0)
[01:08:16.549] Build status: STARTED
[01:08:16.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:38:0) after 6m58s
[01:15:14.829] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:38:0)
[01:15:14.848] Build status: STARTED
[01:15:14.848] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:39:0) after 7m49s
[01:23:04.101] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:39:0)
[01:23:04.135] Build status: STARTED
[01:23:04.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:40:0) after 3m31s
[01:26:35.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:40:0)
[01:26:35.158] Build status: STARTED
[01:26:35.158] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:41:0) after 5m42s
[01:32:17.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:41:0)
[01:32:17.209] Build status: STARTED
[01:32:17.210] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:42:0) after 2m17s
[01:34:34.389] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:42:0)
[01:34:34.403] Build status: STARTED
[01:34:34.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:43:0) after 3m38s
[01:38:12.416] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:43:0)
[01:38:12.428] Build status: STARTED
[01:38:12.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:44:0) after 8m53s
[01:47:05.520] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:44:0)
[01:47:05.542] Build status: STARTED
[01:47:05.542] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:45:0) after 8m41s
[01:55:46.596] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:45:0)
[01:55:46.613] Build status: STARTED
[01:55:46.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:46:0) after 4m55s
[02:00:41.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:46:0)
[02:00:41.771] Build status: STARTED
[02:00:41.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:47:0) after 1m12s
[02:01:53.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:47:0)
[02:01:53.813] Build status: STARTED
[02:01:53.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:48:0) after 7m35s
[02:09:28.830] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:48:0)
[02:09:28.846] Build status: STARTED
[02:09:28.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:49:0) after 9m42s
[02:19:10.884] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:49:0)
[02:19:10.947] Build status: STARTED
[02:19:10.947] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:50:0) after 5m33s
[02:24:43.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:50:0)
[02:24:43.976] Build status: STARTED
[02:24:43.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:51:0) after 5m59s
[02:30:43.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:51:0)
[02:30:43.103] Build status: STARTED
[02:30:43.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:52:0) after 3m33s
[02:34:16.200] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:52:0)
[02:34:16.227] Build status: STARTED
[02:34:16.227] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:53:0) after 3m45s
[02:38:01.388] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:53:0)
[02:38:01.482] Build status: STARTED
[02:38:01.482] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:54:0) after 4m49s
[02:42:50.681] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:54:0)
[02:42:50.699] Build status: STARTED
[02:42:50.699] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:55:0) after 1m20s
[02:44:11.178] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:55:0)
[02:44:11.197] Build status: STARTED
[02:44:11.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:56:0) after 3m54s
[02:48:05.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:56:0)
[02:48:05.228] Build status: STARTED
[02:48:05.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858456095100938176:57:0) after 9m17s
[02:49:06.445] Received PubSub notification, asking Buildbucket for the build status
[02:49:06.465] Build:
{
"id": "8693026033220891841",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1157,
"createdBy": "project:chromium",
"createTime": "2026-01-10T21:42:42.474670211Z",
"startTime": "2026-01-10T21:42:47.353228541Z",
"endTime": "2026-01-11T02:49:06.217970525Z",
"updateTime": "2026-01-11T02:49:06.217970525Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "522dfde6f6ff8dc7536883ee7da83309a50f26f4",
"ref": "refs/heads/main"
}
}
}
[02:49:06.465] Invocation finished in 5h6m25.168188486s with status FAILED