[12:34:23.665] New invocation is queued and will start shortly
[12:34:24.755] Starting the invocation (attempt 1)
[12:34:24.780] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/38bc659a726437e1c8eeaddb449facd9838ac91b"
[12:34:24.781] Popped gitiles commit info from properties and tags
[12:34:24.781] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[12:34:24.781] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[12:34:24.781] Buildbucket request:
{
"requestId": "8858309396850760784",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "38bc659a726437e1c8eeaddb449facd9838ac91b",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8858309396850760784"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[12:34:25.246] Scheduled build:
{
"id": "8692879334967872449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1165,
"createdBy": "project:chromium",
"createTime": "2026-01-12T12:34:24.830164003Z",
"updateTime": "2026-01-12T12:34:24.830164003Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "38bc659a726437e1c8eeaddb449facd9838ac91b",
"ref": "refs/heads/main"
}
}
}
[12:34:25.246] Task URL: https://cr-buildbucket.appspot.com/build/8692879334967872449
[12:34:25.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:2:0) after 5m50s
[12:34:26.284] Received PubSub notification, asking Buildbucket for the build status
[12:34:26.324] Build status: SCHEDULED
[12:34:28.436] Received PubSub notification, asking Buildbucket for the build status
[12:34:28.454] Build status: STARTED
[12:40:15.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:2:0)
[12:40:15.358] Build status: STARTED
[12:40:15.358] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:5:0) after 7m25s
[12:47:40.476] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:5:0)
[12:47:40.500] Build status: STARTED
[12:47:40.500] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:6:0) after 1m30s
[12:49:10.531] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:6:0)
[12:49:10.556] Build status: STARTED
[12:49:10.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:7:0) after 9m12s
[12:58:22.574] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:7:0)
[12:58:22.627] Build status: STARTED
[12:58:22.627] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:8:0) after 9m22s
[13:07:44.642] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:8:0)
[13:07:44.668] Build status: STARTED
[13:07:44.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:9:0) after 8m33s
[13:16:17.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:9:0)
[13:16:17.762] Build status: STARTED
[13:16:17.762] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:10:0) after 9m27s
[13:25:44.826] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:10:0)
[13:25:44.843] Build status: STARTED
[13:25:44.843] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:11:0) after 1m43s
[13:27:27.857] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:11:0)
[13:27:27.873] Build status: STARTED
[13:27:27.873] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:12:0) after 9m20s
[13:36:47.903] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:12:0)
[13:36:47.939] Build status: STARTED
[13:36:47.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:13:0) after 9m59s
[13:46:47.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:13:0)
[13:46:47.150] Build status: STARTED
[13:46:47.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:14:0) after 8m5s
[13:54:52.163] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:14:0)
[13:54:52.180] Build status: STARTED
[13:54:52.180] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:15:0) after 4m13s
[13:59:05.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:15:0)
[13:59:05.372] Build status: STARTED
[13:59:05.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:16:0) after 2m24s
[14:01:29.478] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:16:0)
[14:01:29.493] Build status: STARTED
[14:01:29.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:17:0) after 5m50s
[14:07:19.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:17:0)
[14:07:19.529] Build status: STARTED
[14:07:19.529] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:18:0) after 9m40s
[14:16:59.555] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:18:0)
[14:17:00.232] Build status: STARTED
[14:17:00.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:19:0) after 4m56s
[14:21:56.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:19:0)
[14:21:56.266] Build status: STARTED
[14:21:56.266] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:20:0) after 9m27s
[14:31:23.285] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:20:0)
[14:31:23.299] Build status: STARTED
[14:31:23.299] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:21:0) after 4m51s
[14:36:14.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:21:0)
[14:36:14.432] Build status: STARTED
[14:36:14.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:22:0) after 3m1s
[14:39:15.450] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:22:0)
[14:39:15.465] Build status: STARTED
[14:39:15.465] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:23:0) after 3m22s
[14:42:37.617] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:23:0)
[14:42:37.633] Build status: STARTED
[14:42:37.634] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:24:0) after 3m50s
[14:46:27.673] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:24:0)
[14:46:27.688] Build status: STARTED
[14:46:27.688] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:25:0) after 3m34s
[14:50:01.754] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:25:0)
[14:50:01.768] Build status: STARTED
[14:50:01.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:26:0) after 9m11s
[14:59:12.828] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:26:0)
[14:59:12.862] Build status: STARTED
[14:59:12.862] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:27:0) after 1m13s
[15:00:25.932] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:27:0)
[15:00:25.950] Build status: STARTED
[15:00:25.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:28:0) after 8m31s
[15:08:57.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:28:0)
[15:08:57.068] Build status: STARTED
[15:08:57.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:29:0) after 1m23s
[15:10:20.037] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:29:0)
[15:10:20.152] Build status: STARTED
[15:10:20.152] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:30:0) after 2m14s
[15:12:34.235] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:30:0)
[15:12:34.250] Build status: STARTED
[15:12:34.250] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:31:0) after 9m17s
[15:21:51.271] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:31:0)
[15:21:51.299] Build status: STARTED
[15:21:51.299] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:32:0) after 2m0s
[15:23:51.430] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:32:0)
[15:23:51.523] Build status: STARTED
[15:23:51.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:33:0) after 5m14s
[15:29:05.674] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:33:0)
[15:29:05.691] Build status: STARTED
[15:29:05.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:34:0) after 3m16s
[15:32:21.708] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:34:0)
[15:32:21.725] Build status: STARTED
[15:32:21.725] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:35:0) after 1m43s
[15:34:04.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:35:0)
[15:34:04.854] Build status: STARTED
[15:34:04.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:36:0) after 9m9s
[15:43:13.877] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:36:0)
[15:43:13.904] Build status: STARTED
[15:43:13.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:37:0) after 6m21s
[15:49:34.991] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:37:0)
[15:49:35.005] Build status: STARTED
[15:49:35.005] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:38:0) after 9m0s
[15:58:35.044] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:38:0)
[15:58:35.079] Build status: STARTED
[15:58:35.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:39:0) after 7m57s
[16:06:32.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:39:0)
[16:06:32.125] Build status: STARTED
[16:06:32.125] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:40:0) after 7m10s
[16:13:42.153] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:40:0)
[16:13:42.235] Build status: STARTED
[16:13:42.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:41:0) after 2m15s
[16:15:57.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:41:0)
[16:15:57.386] Build status: STARTED
[16:15:57.386] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:42:0) after 3m1s
[16:18:58.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:42:0)
[16:18:58.449] Build status: STARTED
[16:18:58.449] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:43:0) after 9m28s
[16:28:26.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:43:0)
[16:28:26.494] Build status: STARTED
[16:28:26.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:44:0) after 5m48s
[16:34:14.558] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:44:0)
[16:34:14.600] Build status: STARTED
[16:34:14.600] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:45:0) after 1m45s
[16:35:59.616] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:45:0)
[16:35:59.664] Build status: STARTED
[16:35:59.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:46:0) after 5m26s
[16:41:25.681] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:46:0)
[16:41:25.696] Build status: STARTED
[16:41:25.696] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:47:0) after 5m8s
[16:46:33.677] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:47:0)
[16:46:33.701] Build status: STARTED
[16:46:33.701] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:48:0) after 8m26s
[16:54:59.722] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:48:0)
[16:54:59.804] Build status: STARTED
[16:54:59.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:49:0) after 5m18s
[17:00:17.825] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:49:0)
[17:00:17.843] Build status: STARTED
[17:00:17.843] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:50:0) after 5m31s
[17:05:48.905] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:50:0)
[17:05:48.923] Build status: STARTED
[17:05:48.923] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:51:0) after 5m35s
[17:11:23.950] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:51:0)
[17:11:23.966] Build status: STARTED
[17:11:23.966] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:52:0) after 2m15s
[17:13:38.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:52:0)
[17:13:39.025] Build status: STARTED
[17:13:39.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:53:0) after 8m11s
[17:21:50.068] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:53:0)
[17:21:50.127] Build status: STARTED
[17:21:50.128] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:54:0) after 8m12s
[17:30:02.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:54:0)
[17:30:02.231] Build status: STARTED
[17:30:02.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:55:0) after 4m0s
[17:34:02.271] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:55:0)
[17:34:02.303] Build status: STARTED
[17:34:02.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:56:0) after 5m46s
[17:39:48.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:56:0)
[17:39:48.403] Build status: STARTED
[17:39:48.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:57:0) after 3m59s
[17:43:47.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:57:0)
[17:43:47.420] Build status: STARTED
[17:43:47.420] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8858309396850760784:58:0) after 8m50s
[17:44:55.559] Received PubSub notification, asking Buildbucket for the build status
[17:44:55.573] Build:
{
"id": "8692879334967872449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1165,
"createdBy": "project:chromium",
"createTime": "2026-01-12T12:34:24.830164003Z",
"startTime": "2026-01-12T12:34:28.293875895Z",
"endTime": "2026-01-12T17:44:40.662148994Z",
"updateTime": "2026-01-12T17:44:40.662148994Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "38bc659a726437e1c8eeaddb449facd9838ac91b",
"ref": "refs/heads/main"
}
}
}
[17:44:55.573] Invocation finished in 5h10m31.921898792s with status FAILED