[08:55:09.194] New invocation is queued and will start shortly
[08:55:10.295] Starting the invocation (attempt 1)
[08:55:10.316] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e88a023e6b41d55eeb9ab9f7418b758e6d54bcde"
[08:55:10.316] Popped gitiles commit info from properties and tags
[08:55:10.316] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[08:55:10.316] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[08:55:10.316] Buildbucket request:
{
"requestId": "8857417220638366544",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e88a023e6b41d55eeb9ab9f7418b758e6d54bcde",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857417220638366544"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[08:55:10.816] Scheduled build:
{
"id": "8691987158744817425",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1223,
"createdBy": "project:chromium",
"createTime": "2026-01-22T08:55:10.380687177Z",
"updateTime": "2026-01-22T08:55:10.380687177Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e88a023e6b41d55eeb9ab9f7418b758e6d54bcde",
"ref": "refs/heads/main"
}
}
}
[08:55:10.816] Task URL: https://cr-buildbucket.appspot.com/build/8691987158744817425
[08:55:10.816] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:2:0) after 7m14s
[08:55:11.747] Received PubSub notification, asking Buildbucket for the build status
[08:55:11.766] Build status: SCHEDULED
[08:55:13.649] Received PubSub notification, asking Buildbucket for the build status
[08:55:13.670] Build status: STARTED
[09:02:25.010] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:2:0)
[09:02:25.025] Build status: STARTED
[09:02:25.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:5:0) after 2m22s
[09:04:47.126] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:5:0)
[09:04:47.144] Build status: STARTED
[09:04:47.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:6:0) after 2m28s
[09:07:15.579] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:6:0)
[09:07:15.596] Build status: STARTED
[09:07:15.596] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:7:0) after 4m19s
[09:11:34.611] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:7:0)
[09:11:34.660] Build status: STARTED
[09:11:34.660] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:8:0) after 2m42s
[09:14:16.756] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:8:0)
[09:14:16.779] Build status: STARTED
[09:14:16.779] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:9:0) after 5m19s
[09:19:35.857] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:9:0)
[09:19:35.916] Build status: STARTED
[09:19:35.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:10:0) after 7m11s
[09:26:46.967] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:10:0)
[09:26:47.003] Build status: STARTED
[09:26:47.003] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:11:0) after 3m39s
[09:30:25.989] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:11:0)
[09:30:26.051] Build status: STARTED
[09:30:26.051] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:12:0) after 3m9s
[09:33:35.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:12:0)
[09:33:35.173] Build status: STARTED
[09:33:35.173] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:13:0) after 3m49s
[09:37:24.194] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:13:0)
[09:37:24.214] Build status: STARTED
[09:37:24.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:14:0) after 3m29s
[09:40:53.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:14:0)
[09:40:53.300] Build status: STARTED
[09:40:53.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:15:0) after 4m11s
[09:45:04.317] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:15:0)
[09:45:04.369] Build status: STARTED
[09:45:04.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:16:0) after 8m53s
[09:53:57.414] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:16:0)
[09:53:57.434] Build status: STARTED
[09:53:57.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:17:0) after 3m13s
[09:57:10.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:17:0)
[09:57:10.465] Build status: STARTED
[09:57:10.465] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:18:0) after 4m8s
[10:01:18.437] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:18:0)
[10:01:18.452] Build status: STARTED
[10:01:18.452] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:19:0) after 8m41s
[10:09:59.473] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:19:0)
[10:09:59.489] Build status: STARTED
[10:09:59.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:20:0) after 5m28s
[10:15:27.511] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:20:0)
[10:15:27.525] Build status: STARTED
[10:15:27.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:21:0) after 2m4s
[10:17:31.524] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:21:0)
[10:17:31.545] Build status: STARTED
[10:17:31.545] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:22:0) after 1m35s
[10:19:06.559] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:22:0)
[10:19:06.646] Build status: STARTED
[10:19:06.646] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:23:0) after 7m47s
[10:26:53.760] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:23:0)
[10:26:53.796] Build status: STARTED
[10:26:53.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:24:0) after 1m23s
[10:28:16.892] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:24:0)
[10:28:16.906] Build status: STARTED
[10:28:16.906] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:25:0) after 3m29s
[10:31:45.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:25:0)
[10:31:45.958] Build status: STARTED
[10:31:45.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:26:0) after 5m57s
[10:37:43.055] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:26:0)
[10:37:43.074] Build status: STARTED
[10:37:43.074] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:27:0) after 5m8s
[10:42:51.232] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:27:0)
[10:42:51.247] Build status: STARTED
[10:42:51.247] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:28:0) after 5m39s
[10:48:30.261] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:28:0)
[10:48:30.281] Build status: STARTED
[10:48:30.281] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:29:0) after 8m32s
[10:57:02.363] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:29:0)
[10:57:02.379] Build status: STARTED
[10:57:02.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:30:0) after 1m42s
[10:58:44.482] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:30:0)
[10:58:44.495] Build status: STARTED
[10:58:44.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:31:0) after 5m24s
[11:04:08.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:31:0)
[11:04:08.548] Build status: STARTED
[11:04:08.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:32:0) after 5m15s
[11:09:23.601] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:32:0)
[11:09:23.617] Build status: STARTED
[11:09:23.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:33:0) after 5m7s
[11:14:30.691] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:33:0)
[11:14:30.705] Build status: STARTED
[11:14:30.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:34:0) after 7m34s
[11:22:04.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:34:0)
[11:22:04.768] Build status: STARTED
[11:22:04.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:35:0) after 5m26s
[11:27:30.835] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:35:0)
[11:27:30.855] Build status: STARTED
[11:27:30.855] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:36:0) after 2m5s
[11:29:35.869] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:36:0)
[11:29:35.883] Build status: STARTED
[11:29:35.883] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:37:0) after 3m57s
[11:33:32.902] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:37:0)
[11:33:32.982] Build status: STARTED
[11:33:32.982] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:38:0) after 1m6s
[11:34:38.958] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:38:0)
[11:34:38.972] Build status: STARTED
[11:34:38.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:39:0) after 2m29s
[11:37:07.984] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:39:0)
[11:37:08.000] Build status: STARTED
[11:37:08.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:40:0) after 3m20s
[11:40:28.016] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:40:0)
[11:40:28.029] Build status: STARTED
[11:40:28.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:41:0) after 3m21s
[11:43:49.034] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:41:0)
[11:43:49.048] Build status: STARTED
[11:43:49.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:42:0) after 9m7s
[11:52:56.067] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:42:0)
[11:52:56.111] Build status: STARTED
[11:52:56.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:43:0) after 9m35s
[12:02:31.131] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:43:0)
[12:02:31.147] Build status: STARTED
[12:02:31.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:44:0) after 4m40s
[12:07:11.266] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:44:0)
[12:07:11.285] Build status: STARTED
[12:07:11.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:45:0) after 2m37s
[12:09:48.362] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:45:0)
[12:09:48.378] Build status: STARTED
[12:09:48.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:46:0) after 1m11s
[12:10:59.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:46:0)
[12:10:59.472] Build status: STARTED
[12:10:59.472] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:47:0) after 8m5s
[12:19:04.496] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:47:0)
[12:19:04.522] Build status: STARTED
[12:19:04.522] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:48:0) after 1m43s
[12:20:47.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:48:0)
[12:20:47.598] Build status: STARTED
[12:20:47.598] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:49:0) after 3m25s
[12:24:12.616] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:49:0)
[12:24:12.631] Build status: STARTED
[12:24:12.631] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:50:0) after 9m27s
[12:33:39.639] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:50:0)
[12:33:39.655] Build status: STARTED
[12:33:39.655] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:51:0) after 1m33s
[12:35:12.675] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:51:0)
[12:35:12.693] Build status: STARTED
[12:35:12.694] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:52:0) after 2m45s
[12:37:57.716] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:52:0)
[12:37:57.733] Build status: STARTED
[12:37:57.733] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:53:0) after 7m31s
[12:45:28.778] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:53:0)
[12:45:28.796] Build status: STARTED
[12:45:28.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:54:0) after 4m14s
[12:49:42.846] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:54:0)
[12:49:42.898] Build status: STARTED
[12:49:42.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:55:0) after 5m49s
[12:55:31.915] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:55:0)
[12:55:31.931] Build status: STARTED
[12:55:31.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:56:0) after 8m5s
[13:03:36.998] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:56:0)
[13:03:37.013] Build status: STARTED
[13:03:37.013] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:57:0) after 7m49s
[13:11:26.031] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:57:0)
[13:11:26.069] Build status: STARTED
[13:11:26.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:58:0) after 4m57s
[13:16:23.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:58:0)
[13:16:23.141] Build status: STARTED
[13:16:23.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857417220638366544:59:0) after 2m56s
[13:17:52.685] Received PubSub notification, asking Buildbucket for the build status
[13:17:52.850] Build:
{
"id": "8691987158744817425",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1223,
"createdBy": "project:chromium",
"createTime": "2026-01-22T08:55:10.380687177Z",
"startTime": "2026-01-22T08:55:13.496916417Z",
"endTime": "2026-01-22T13:17:52.458496820Z",
"updateTime": "2026-01-22T13:17:52.458496820Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e88a023e6b41d55eeb9ab9f7418b758e6d54bcde",
"ref": "refs/heads/main"
}
}
}
[13:17:52.850] Invocation finished in 4h22m43.660593033s with status FAILED