[17:04:16.448] New invocation is queued and will start shortly
[17:04:17.622] Starting the invocation (attempt 1)
[17:04:17.649] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/236f4f586d0dea9e137167c5da705c7161506b57"
[17:04:17.649] Popped gitiles commit info from properties and tags
[17:04:17.649] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[17:04:17.650] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[17:04:17.650] Buildbucket request:
{
"requestId": "8854306150957128784",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "236f4f586d0dea9e137167c5da705c7161506b57",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8854306150957128784"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[17:04:18.475] Scheduled build:
{
"id": "8688876088990526401",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1422,
"createdBy": "project:chromium",
"createTime": "2026-02-25T17:04:17.701336666Z",
"updateTime": "2026-02-25T17:04:17.701336666Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "236f4f586d0dea9e137167c5da705c7161506b57",
"ref": "refs/heads/main"
}
}
}
[17:04:18.475] Task URL: https://cr-buildbucket.appspot.com/build/8688876088990526401
[17:04:18.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:2:0) after 2m57s
[17:04:19.341] Received PubSub notification, asking Buildbucket for the build status
[17:04:19.374] Build status: SCHEDULED
[17:05:32.480] Received PubSub notification, asking Buildbucket for the build status
[17:05:32.509] Build status: STARTED
[17:07:15.486] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:2:0)
[17:07:15.505] Build status: STARTED
[17:07:15.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:5:0) after 2m6s
[17:09:21.558] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:5:0)
[17:09:21.624] Build status: STARTED
[17:09:21.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:6:0) after 7m10s
[17:16:31.641] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:6:0)
[17:16:31.660] Build status: STARTED
[17:16:31.660] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:7:0) after 3m16s
[17:19:47.674] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:7:0)
[17:19:47.745] Build status: STARTED
[17:19:47.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:8:0) after 7m9s
[17:26:56.797] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:8:0)
[17:26:56.813] Build status: STARTED
[17:26:56.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:9:0) after 1m53s
[17:28:49.879] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:9:0)
[17:28:49.930] Build status: STARTED
[17:28:49.930] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:10:0) after 3m35s
[17:32:24.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:10:0)
[17:32:25.001] Build status: STARTED
[17:32:25.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:11:0) after 1m11s
[17:33:36.073] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:11:0)
[17:33:36.088] Build status: STARTED
[17:33:36.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:12:0) after 9m10s
[17:42:46.178] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:12:0)
[17:42:46.213] Build status: STARTED
[17:42:46.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:13:0) after 1m11s
[17:43:57.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:13:0)
[17:43:57.248] Build status: STARTED
[17:43:57.248] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:14:0) after 4m15s
[17:48:12.326] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:14:0)
[17:48:12.345] Build status: STARTED
[17:48:12.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:15:0) after 2m36s
[17:50:48.428] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:15:0)
[17:50:48.443] Build status: STARTED
[17:50:48.443] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:16:0) after 8m46s
[17:59:34.539] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:16:0)
[17:59:34.585] Build status: STARTED
[17:59:34.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:17:0) after 1m40s
[18:01:14.659] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:17:0)
[18:01:14.688] Build status: STARTED
[18:01:14.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:18:0) after 2m28s
[18:03:42.709] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:18:0)
[18:03:42.727] Build status: STARTED
[18:03:42.727] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:19:0) after 2m56s
[18:06:38.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:19:0)
[18:06:38.769] Build status: STARTED
[18:06:38.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:20:0) after 1m6s
[18:07:44.843] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:20:0)
[18:07:44.865] Build status: STARTED
[18:07:44.865] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:21:0) after 9m33s
[18:17:17.865] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:21:0)
[18:17:17.878] Build status: STARTED
[18:17:17.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:22:0) after 9m7s
[18:26:24.883] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:22:0)
[18:26:24.910] Build status: STARTED
[18:26:24.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:23:0) after 4m47s
[18:31:12.023] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:23:0)
[18:31:12.058] Build status: STARTED
[18:31:12.058] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:24:0) after 6m33s
[18:37:45.079] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:24:0)
[18:37:45.096] Build status: STARTED
[18:37:45.096] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:25:0) after 9m10s
[18:46:55.127] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:25:0)
[18:46:55.149] Build status: STARTED
[18:46:55.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:26:0) after 5m54s
[18:52:49.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:26:0)
[18:52:49.212] Build status: STARTED
[18:52:49.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:27:0) after 8m10s
[19:00:59.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:27:0)
[19:00:59.261] Build status: STARTED
[19:00:59.261] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:28:0) after 9m44s
[19:10:43.286] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:28:0)
[19:10:43.306] Build status: STARTED
[19:10:43.306] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:29:0) after 1m26s
[19:12:09.378] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:29:0)
[19:12:09.403] Build status: STARTED
[19:12:09.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:30:0) after 9m54s
[19:22:03.425] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:30:0)
[19:22:03.444] Build status: STARTED
[19:22:03.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:31:0) after 6m23s
[19:28:26.497] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:31:0)
[19:28:26.524] Build status: STARTED
[19:28:26.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:32:0) after 3m59s
[19:32:25.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:32:0)
[19:32:25.604] Build status: STARTED
[19:32:25.605] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:33:0) after 6m43s
[19:39:08.672] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:33:0)
[19:39:08.691] Build status: STARTED
[19:39:08.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:34:0) after 8m33s
[19:47:41.759] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:34:0)
[19:47:41.773] Build status: STARTED
[19:47:41.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:35:0) after 7m0s
[19:54:42.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:35:0)
[19:54:42.123] Build status: STARTED
[19:54:42.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:36:0) after 7m10s
[20:01:52.143] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:36:0)
[20:01:52.160] Build status: STARTED
[20:01:52.160] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:37:0) after 8m53s
[20:10:45.202] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:37:0)
[20:10:45.216] Build status: STARTED
[20:10:45.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:38:0) after 5m13s
[20:15:58.246] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:38:0)
[20:15:58.261] Build status: STARTED
[20:15:58.261] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:39:0) after 8m6s
[20:24:04.464] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:39:0)
[20:24:04.487] Build status: STARTED
[20:24:04.487] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:40:0) after 3m22s
[20:27:26.595] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:40:0)
[20:27:26.612] Build status: STARTED
[20:27:26.612] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:41:0) after 3m32s
[20:30:58.635] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:41:0)
[20:30:58.739] Build status: STARTED
[20:30:58.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:42:0) after 6m8s
[20:37:07.007] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:42:0)
[20:37:07.022] Build status: STARTED
[20:37:07.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:43:0) after 7m19s
[20:44:26.218] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:43:0)
[20:44:26.240] Build status: STARTED
[20:44:26.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:44:0) after 9m10s
[20:53:36.632] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:44:0)
[20:53:36.660] Build status: STARTED
[20:53:36.660] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:45:0) after 8m42s
[21:02:18.728] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:45:0)
[21:02:18.757] Build status: STARTED
[21:02:18.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:46:0) after 1m16s
[21:03:34.815] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:46:0)
[21:03:34.835] Build status: STARTED
[21:03:34.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:47:0) after 7m15s
[21:10:49.982] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:47:0)
[21:10:49.999] Build status: STARTED
[21:10:49.999] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:48:0) after 4m25s
[21:15:15.211] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:48:0)
[21:15:15.228] Build status: STARTED
[21:15:15.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:49:0) after 6m40s
[21:21:55.276] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:49:0)
[21:21:55.292] Build status: STARTED
[21:21:55.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:50:0) after 4m19s
[21:26:14.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:50:0)
[21:26:14.536] Build status: STARTED
[21:26:14.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:51:0) after 9m39s
[21:35:53.637] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:51:0)
[21:35:53.656] Build status: STARTED
[21:35:53.656] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:52:0) after 7m22s
[21:43:15.783] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:52:0)
[21:43:15.809] Build status: STARTED
[21:43:15.809] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:53:0) after 5m40s
[21:48:56.043] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:53:0)
[21:48:56.058] Build status: STARTED
[21:48:56.058] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:54:0) after 2m47s
[21:51:43.091] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:54:0)
[21:51:43.110] Build status: STARTED
[21:51:43.110] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:55:0) after 2m7s
[21:53:50.122] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:55:0)
[21:53:50.138] Build status: STARTED
[21:53:50.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:56:0) after 7m17s
[22:01:07.213] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:56:0)
[22:01:07.228] Build status: STARTED
[22:01:07.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8854306150957128784:57:0) after 9m33s
[22:09:27.707] Received PubSub notification, asking Buildbucket for the build status
[22:09:27.720] Build:
{
"id": "8688876088990526401",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1422,
"createdBy": "project:chromium",
"createTime": "2026-02-25T17:04:17.701336666Z",
"startTime": "2026-02-25T17:05:32.015942596Z",
"endTime": "2026-02-25T22:09:27.438156299Z",
"updateTime": "2026-02-25T22:09:27.438156299Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "236f4f586d0dea9e137167c5da705c7161506b57",
"ref": "refs/heads/main"
}
}
}
[22:09:27.720] Invocation finished in 5h5m11.277324433s with status FAILED