[16:21:41.831] New invocation is queued and will start shortly
[16:21:42.890] Starting the invocation (attempt 1)
[16:21:42.917] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/0a4735880df6637c27ee5a2581d31288b365141c"
[16:21:42.918] Popped gitiles commit info from properties and tags
[16:21:42.918] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[16:21:42.918] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[16:21:42.918] Buildbucket request:
{
"requestId": "8848963608649052464",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848963608649052464"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[16:21:43.465] Scheduled build:
{
"id": "8683533546785978065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20445,
"createdBy": "project:chromium",
"createTime": "2026-04-25T16:21:42.986998870Z",
"updateTime": "2026-04-25T16:21:42.986998870Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
}
}
}
[16:21:43.465] Task URL: https://cr-buildbucket.appspot.com/build/8683533546785978065
[16:21:43.465] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:2:0) after 5m35s
[16:21:43.829] Received PubSub notification, asking Buildbucket for the build status
[16:21:43.849] Build status: SCHEDULED
[16:22:25.901] Received PubSub notification, asking Buildbucket for the build status
[16:22:25.928] Build status: STARTED
[16:27:18.863] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:2:0)
[16:27:18.971] Build status: STARTED
[16:27:18.971] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:5:0) after 3m13s
[16:30:32.077] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:5:0)
[16:30:32.092] Build status: STARTED
[16:30:32.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:6:0) after 3m52s
[16:34:24.126] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:6:0)
[16:34:24.185] Build status: STARTED
[16:34:24.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:7:0) after 1m4s
[16:35:28.226] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:7:0)
[16:35:28.249] Build status: STARTED
[16:35:28.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:8:0) after 8m55s
[16:44:23.264] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:8:0)
[16:44:23.287] Build status: STARTED
[16:44:23.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:9:0) after 8m7s
[16:52:30.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:9:0)
[16:52:30.383] Build status: STARTED
[16:52:30.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:10:0) after 5m8s
[16:57:38.470] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:10:0)
[16:57:38.495] Build status: STARTED
[16:57:38.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:11:0) after 8m39s
[17:06:17.516] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:11:0)
[17:06:17.537] Build status: STARTED
[17:06:17.537] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:12:0) after 6m19s
[17:12:36.655] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:12:0)
[17:12:36.670] Build status: STARTED
[17:12:36.670] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:13:0) after 2m24s
[17:15:00.789] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:13:0)
[17:15:00.805] Build status: STARTED
[17:15:00.805] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:14:0) after 8m2s
[17:23:02.821] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:14:0)
[17:23:02.858] Build status: STARTED
[17:23:02.858] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:15:0) after 4m41s
[17:27:43.941] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:15:0)
[17:27:43.958] Build status: STARTED
[17:27:43.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:16:0) after 3m1s
[17:30:44.978] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:16:0)
[17:30:45.001] Build status: STARTED
[17:30:45.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:17:0) after 5m56s
[17:36:41.074] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:17:0)
[17:36:41.112] Build status: STARTED
[17:36:41.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:18:0) after 2m18s
[17:38:59.124] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:18:0)
[17:38:59.142] Build status: STARTED
[17:38:59.142] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:19:0) after 9m55s
[17:48:54.156] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:19:0)
[17:48:54.174] Build status: STARTED
[17:48:54.174] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:20:0) after 3m56s
[17:52:50.723] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:20:0)
[17:52:50.749] Build status: STARTED
[17:52:50.749] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:21:0) after 8m14s
[18:01:04.856] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:21:0)
[18:01:04.876] Build status: STARTED
[18:01:04.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:22:0) after 5m39s
[18:06:43.891] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:22:0)
[18:06:43.906] Build status: STARTED
[18:06:43.906] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:23:0) after 5m0s
[18:11:43.954] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:23:0)
[18:11:43.972] Build status: STARTED
[18:11:43.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:24:0) after 6m36s
[18:18:20.322] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:24:0)
[18:18:20.348] Build status: STARTED
[18:18:20.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:25:0) after 6m51s
[18:25:11.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:25:0)
[18:25:11.377] Build status: STARTED
[18:25:11.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:26:0) after 2m44s
[18:27:55.501] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:26:0)
[18:27:55.516] Build status: STARTED
[18:27:55.517] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:27:0) after 8m23s
[18:36:18.612] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:27:0)
[18:36:18.630] Build status: STARTED
[18:36:18.630] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:28:0) after 4m10s
[18:40:28.643] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:28:0)
[18:40:28.657] Build status: STARTED
[18:40:28.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:29:0) after 7m10s
[18:47:38.805] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:29:0)
[18:47:38.822] Build status: STARTED
[18:47:38.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:30:0) after 7m57s
[18:55:36.387] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:30:0)
[18:55:36.468] Build status: STARTED
[18:55:36.468] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:31:0) after 1m48s
[18:57:24.493] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:31:0)
[18:57:24.560] Build status: STARTED
[18:57:24.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:32:0) after 3m27s
[19:00:51.569] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:32:0)
[19:00:51.584] Build status: STARTED
[19:00:51.584] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:33:0) after 2m26s
[19:03:17.597] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:33:0)
[19:03:17.614] Build status: STARTED
[19:03:17.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:34:0) after 8m57s
[19:12:14.628] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:34:0)
[19:12:14.649] Build status: STARTED
[19:12:14.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:35:0) after 3m58s
[19:16:12.757] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:35:0)
[19:16:12.781] Build status: STARTED
[19:16:12.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:36:0) after 2m50s
[19:19:02.895] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:36:0)
[19:19:02.913] Build status: STARTED
[19:19:02.913] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:37:0) after 3m48s
[19:22:51.042] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:37:0)
[19:22:51.070] Build status: STARTED
[19:22:51.070] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:38:0) after 4m6s
[19:26:57.085] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:38:0)
[19:26:57.102] Build status: STARTED
[19:26:57.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:39:0) after 9m19s
[19:36:16.441] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:39:0)
[19:36:16.460] Build status: STARTED
[19:36:16.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:40:0) after 1m6s
[19:37:22.479] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:40:0)
[19:37:22.558] Build status: STARTED
[19:37:22.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:41:0) after 4m10s
[19:41:33.003] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:41:0)
[19:41:33.019] Build status: STARTED
[19:41:33.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:42:0) after 2m36s
[19:44:09.034] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:42:0)
[19:44:09.099] Build status: STARTED
[19:44:09.099] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:43:0) after 6m29s
[19:50:38.216] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:43:0)
[19:50:38.232] Build status: STARTED
[19:50:38.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:44:0) after 4m0s
[19:54:38.245] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:44:0)
[19:54:38.264] Build status: STARTED
[19:54:38.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:45:0) after 9m54s
[20:04:32.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:45:0)
[20:04:32.384] Build status: STARTED
[20:04:32.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:46:0) after 2m14s
[20:06:46.399] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:46:0)
[20:06:46.415] Build status: STARTED
[20:06:46.415] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:47:0) after 6m20s
[20:13:06.760] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:47:0)
[20:13:06.783] Build status: STARTED
[20:13:06.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:48:0) after 4m25s
[20:17:31.766] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:48:0)
[20:17:31.781] Build status: STARTED
[20:17:31.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:49:0) after 9m39s
[20:27:10.822] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:49:0)
[20:27:10.850] Build status: STARTED
[20:27:10.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:50:0) after 8m34s
[20:35:44.867] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:50:0)
[20:35:44.884] Build status: STARTED
[20:35:44.884] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:51:0) after 9m31s
[20:45:16.118] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:51:0)
[20:45:16.139] Build status: STARTED
[20:45:16.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:52:0) after 7m23s
[20:52:39.223] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:52:0)
[20:52:39.240] Build status: STARTED
[20:52:39.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:53:0) after 9m14s
[21:01:53.425] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:53:0)
[21:01:53.476] Build status: STARTED
[21:01:53.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:54:0) after 2m35s
[21:04:28.594] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:54:0)
[21:04:28.608] Build status: STARTED
[21:04:28.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:55:0) after 3m8s
[21:07:36.625] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:55:0)
[21:07:36.642] Build status: STARTED
[21:07:36.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:56:0) after 6m24s
[21:14:00.655] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:56:0)
[21:14:00.686] Build status: STARTED
[21:14:00.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:57:0) after 6m56s
[21:20:56.970] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:57:0)
[21:20:56.985] Build status: STARTED
[21:20:56.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:58:0) after 8m8s
[21:29:05.003] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:58:0)
[21:29:05.024] Build status: STARTED
[21:29:05.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:59:0) after 7m16s
[21:36:21.069] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:59:0)
[21:36:21.104] Build status: STARTED
[21:36:21.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:60:0) after 1m20s
[21:37:41.268] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:60:0)
[21:37:41.287] Build status: STARTED
[21:37:41.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848963608649052464:61:0) after 9m30s
[21:40:31.896] Received PubSub notification, asking Buildbucket for the build status
[21:40:31.924] Build:
{
"id": "8683533546785978065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20445,
"createdBy": "project:chromium",
"createTime": "2026-04-25T16:21:42.986998870Z",
"startTime": "2026-04-25T16:22:25.700113519Z",
"endTime": "2026-04-25T21:40:31.655772595Z",
"updateTime": "2026-04-25T21:40:31.655772595Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "0a4735880df6637c27ee5a2581d31288b365141c",
"ref": "refs/heads/main"
}
}
}
[21:40:31.924] Invocation finished in 5h18m50.098271799s with status FAILED