[07:31:03.004] New invocation is queued and will start shortly
[07:31:04.129] Starting the invocation (attempt 1)
[07:31:04.155] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/6e57487fb582e5e24854e03264479803c7261853"
[07:31:04.155] Popped gitiles commit info from properties and tags
[07:31:04.155] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:31:04.155] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:31:04.155] Buildbucket request:
{
"requestId": "8848544009247058496",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6e57487fb582e5e24854e03264479803c7261853",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848544009247058496"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:31:04.675] Scheduled build:
{
"id": "8683113947325830289",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2408,
"createdBy": "project:chromium",
"createTime": "2026-04-30T07:31:04.214987124Z",
"updateTime": "2026-04-30T07:31:04.214987124Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6e57487fb582e5e24854e03264479803c7261853",
"ref": "refs/heads/main"
}
}
}
[07:31:04.675] Task URL: https://cr-buildbucket.appspot.com/build/8683113947325830289
[07:31:04.675] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:2:0) after 2m43s
[07:31:05.305] Received PubSub notification, asking Buildbucket for the build status
[07:31:05.353] Build status: SCHEDULED
[07:31:13.437] Received PubSub notification, asking Buildbucket for the build status
[07:31:13.464] Build status: STARTED
[07:33:47.687] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:2:0)
[07:33:47.704] Build status: STARTED
[07:33:47.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:5:0) after 1m3s
[07:34:50.669] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:5:0)
[07:34:50.683] Build status: STARTED
[07:34:50.683] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:6:0) after 9m58s
[07:44:48.682] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:6:0)
[07:44:48.700] Build status: STARTED
[07:44:48.700] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:7:0) after 8m8s
[07:52:56.715] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:7:0)
[07:52:56.739] Build status: STARTED
[07:52:56.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:8:0) after 3m40s
[07:56:36.777] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:8:0)
[07:56:36.843] Build status: STARTED
[07:56:36.843] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:9:0) after 5m36s
[08:02:12.860] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:9:0)
[08:02:12.877] Build status: STARTED
[08:02:12.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:10:0) after 4m43s
[08:06:55.873] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:10:0)
[08:06:55.894] Build status: STARTED
[08:06:55.894] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:11:0) after 2m19s
[08:09:14.917] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:11:0)
[08:09:14.932] Build status: STARTED
[08:09:14.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:12:0) after 6m22s
[08:15:36.997] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:12:0)
[08:15:37.016] Build status: STARTED
[08:15:37.016] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:13:0) after 8m17s
[08:23:54.130] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:13:0)
[08:23:54.149] Build status: STARTED
[08:23:54.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:14:0) after 6m47s
[08:30:41.163] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:14:0)
[08:30:41.184] Build status: STARTED
[08:30:41.184] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:15:0) after 5m31s
[08:36:12.253] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:15:0)
[08:36:12.276] Build status: STARTED
[08:36:12.276] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:16:0) after 1m6s
[08:37:18.270] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:16:0)
[08:37:18.341] Build status: STARTED
[08:37:18.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:17:0) after 3m1s
[08:40:19.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:17:0)
[08:40:19.451] Build status: STARTED
[08:40:19.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:18:0) after 8m48s
[08:49:07.479] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:18:0)
[08:49:07.500] Build status: STARTED
[08:49:07.500] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:19:0) after 8m19s
[08:57:26.512] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:19:0)
[08:57:26.530] Build status: STARTED
[08:57:26.530] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:20:0) after 4m3s
[09:01:29.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:20:0)
[09:01:29.567] Build status: STARTED
[09:01:29.567] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:21:0) after 5m40s
[09:07:09.579] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:21:0)
[09:07:09.637] Build status: STARTED
[09:07:09.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:22:0) after 5m27s
[09:12:36.711] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:22:0)
[09:12:36.733] Build status: STARTED
[09:12:36.734] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:23:0) after 7m36s
[09:20:12.858] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:23:0)
[09:20:12.916] Build status: STARTED
[09:20:12.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:24:0) after 2m31s
[09:22:43.976] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:24:0)
[09:22:44.073] Build status: STARTED
[09:22:44.074] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:25:0) after 5m9s
[09:27:53.184] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:25:0)
[09:27:53.201] Build status: STARTED
[09:27:53.201] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:26:0) after 3m21s
[09:31:14.219] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:26:0)
[09:31:14.235] Build status: STARTED
[09:31:14.235] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:27:0) after 5m57s
[09:37:11.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:27:0)
[09:37:11.453] Build status: STARTED
[09:37:11.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:28:0) after 8m47s
[09:45:58.563] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:28:0)
[09:45:58.581] Build status: STARTED
[09:45:58.581] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:29:0) after 8m49s
[09:54:47.644] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:29:0)
[09:54:47.658] Build status: STARTED
[09:54:47.658] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:30:0) after 7m42s
[10:02:29.729] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:30:0)
[10:02:29.748] Build status: STARTED
[10:02:29.748] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:31:0) after 6m37s
[10:09:06.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:31:0)
[10:09:06.949] Build status: STARTED
[10:09:06.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:32:0) after 2m2s
[10:11:08.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:32:0)
[10:11:08.944] Build status: STARTED
[10:11:08.944] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:33:0) after 6m23s
[10:17:31.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:33:0)
[10:17:31.988] Build status: STARTED
[10:17:31.988] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:34:0) after 7m57s
[10:25:29.004] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:34:0)
[10:25:29.077] Build status: STARTED
[10:25:29.077] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:35:0) after 1m53s
[10:27:22.088] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:35:0)
[10:27:22.107] Build status: STARTED
[10:27:22.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:36:0) after 7m5s
[10:34:27.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:36:0)
[10:34:27.169] Build status: STARTED
[10:34:27.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:37:0) after 8m27s
[10:42:54.232] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:37:0)
[10:42:54.258] Build status: STARTED
[10:42:54.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:38:0) after 2m9s
[10:45:03.276] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:38:0)
[10:45:03.341] Build status: STARTED
[10:45:03.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:39:0) after 8m56s
[10:53:59.501] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:39:0)
[10:53:59.555] Build status: STARTED
[10:53:59.555] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:40:0) after 7m30s
[11:01:29.660] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:40:0)
[11:01:29.685] Build status: STARTED
[11:01:29.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:41:0) after 5m10s
[11:06:39.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:41:0)
[11:06:39.707] Build status: STARTED
[11:06:39.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:42:0) after 6m58s
[11:13:37.757] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:42:0)
[11:13:37.775] Build status: STARTED
[11:13:37.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:43:0) after 8m38s
[11:22:15.870] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:43:0)
[11:22:15.888] Build status: STARTED
[11:22:15.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:44:0) after 5m4s
[11:27:20.000] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:44:0)
[11:27:20.050] Build status: STARTED
[11:27:20.050] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:45:0) after 7m32s
[11:34:52.063] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:45:0)
[11:34:52.081] Build status: STARTED
[11:34:52.081] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:46:0) after 2m22s
[11:37:14.246] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:46:0)
[11:37:14.325] Build status: STARTED
[11:37:14.325] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:47:0) after 6m19s
[11:43:33.342] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:47:0)
[11:43:33.377] Build status: STARTED
[11:43:33.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:48:0) after 4m2s
[11:47:35.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:48:0)
[11:47:35.402] Build status: STARTED
[11:47:35.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:49:0) after 7m35s
[11:55:10.525] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:49:0)
[11:55:10.683] Build status: STARTED
[11:55:10.683] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:50:0) after 9m19s
[12:04:29.768] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:50:0)
[12:04:29.796] Build status: STARTED
[12:04:29.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:51:0) after 5m58s
[12:10:27.823] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:51:0)
[12:10:27.866] Build status: STARTED
[12:10:27.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:52:0) after 5m8s
[12:15:36.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:52:0)
[12:15:36.039] Build status: STARTED
[12:15:36.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:53:0) after 6m45s
[12:22:21.135] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:53:0)
[12:22:21.153] Build status: STARTED
[12:22:21.153] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:54:0) after 9m2s
[12:31:23.172] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:54:0)
[12:31:23.210] Build status: STARTED
[12:31:23.210] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:55:0) after 5m6s
[12:36:29.194] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:55:0)
[12:36:29.229] Build status: STARTED
[12:36:29.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:56:0) after 4m58s
[12:41:27.240] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:56:0)
[12:41:27.275] Build status: STARTED
[12:41:27.275] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8848544009247058496:57:0) after 6m54s
[12:43:31.845] Received PubSub notification, asking Buildbucket for the build status
[12:43:32.044] Build:
{
"id": "8683113947325830289",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2408,
"createdBy": "project:chromium",
"createTime": "2026-04-30T07:31:04.214987124Z",
"startTime": "2026-04-30T07:31:08.879767327Z",
"endTime": "2026-04-30T12:43:28.834501380Z",
"updateTime": "2026-04-30T12:43:28.834501380Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6e57487fb582e5e24854e03264479803c7261853",
"ref": "refs/heads/main"
}
}
}
[12:43:32.044] Invocation finished in 5h12m29.046428271s with status FAILED