[23:27:46.977] New invocation is queued and will start shortly
[23:27:48.050] Starting the invocation (attempt 1)
[23:27:48.070] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e878af67a395216c86b4e59813e343b361a5c511"
[23:27:48.070] Popped gitiles commit info from properties and tags
[23:27:48.070] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[23:27:48.070] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[23:27:48.071] Buildbucket request:
{
"requestId": "8848846204683530336",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e878af67a395216c86b4e59813e343b361a5c511",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848846204683530336"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[23:27:48.888] Scheduled build:
{
"id": "8683416142820488561",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20451,
"createdBy": "project:chromium",
"createTime": "2026-04-26T23:27:48.132413955Z",
"updateTime": "2026-04-26T23:27:48.132413955Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e878af67a395216c86b4e59813e343b361a5c511",
"ref": "refs/heads/main"
}
}
}
[23:27:48.888] Task URL: https://cr-buildbucket.appspot.com/build/8683416142820488561
[23:27:48.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:2:0) after 8m18s
[23:27:49.529] Received PubSub notification, asking Buildbucket for the build status
[23:27:49.544] Build status: SCHEDULED
[23:28:47.931] Received PubSub notification, asking Buildbucket for the build status
[23:28:48.099] Build status: STARTED
[23:36:06.976] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:2:0)
[23:36:06.995] Build status: STARTED
[23:36:06.995] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:5:0) after 2m1s
[23:38:08.295] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:5:0)
[23:38:08.310] Build status: STARTED
[23:38:08.310] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:6:0) after 7m30s
[23:45:38.489] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:6:0)
[23:45:38.512] Build status: STARTED
[23:45:38.512] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:7:0) after 4m46s
[23:50:24.526] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:7:0)
[23:50:24.549] Build status: STARTED
[23:50:24.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:8:0) after 2m7s
[23:52:31.757] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:8:0)
[23:52:31.898] Build status: STARTED
[23:52:31.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:9:0) after 5m54s
[23:58:25.935] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:9:0)
[23:58:26.035] Build status: STARTED
[23:58:26.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:10:0) after 8m47s
[00:07:13.051] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:10:0)
[00:07:13.076] Build status: STARTED
[00:07:13.076] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:11:0) after 2m42s
[00:09:55.109] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:11:0)
[00:09:55.128] Build status: STARTED
[00:09:55.128] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:12:0) after 4m14s
[00:14:09.246] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:12:0)
[00:14:09.305] Build status: STARTED
[00:14:09.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:13:0) after 9m21s
[00:23:30.485] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:13:0)
[00:23:30.535] Build status: STARTED
[00:23:30.535] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:14:0) after 4m19s
[00:27:49.648] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:14:0)
[00:27:49.669] Build status: STARTED
[00:27:49.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:15:0) after 2m38s
[00:30:27.729] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:15:0)
[00:30:27.760] Build status: STARTED
[00:30:27.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:16:0) after 3m29s
[00:33:56.776] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:16:0)
[00:33:56.792] Build status: STARTED
[00:33:56.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:17:0) after 7m5s
[00:41:01.809] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:17:0)
[00:41:01.825] Build status: STARTED
[00:41:01.825] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:18:0) after 8m13s
[00:49:14.963] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:18:0)
[00:49:15.059] Build status: STARTED
[00:49:15.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:19:0) after 1m41s
[00:50:56.080] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:19:0)
[00:50:56.099] Build status: STARTED
[00:50:56.100] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:20:0) after 7m3s
[00:57:59.152] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:20:0)
[00:57:59.181] Build status: STARTED
[00:57:59.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:21:0) after 2m27s
[01:00:26.195] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:21:0)
[01:00:26.218] Build status: STARTED
[01:00:26.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:22:0) after 1m37s
[01:02:03.317] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:22:0)
[01:02:03.335] Build status: STARTED
[01:02:03.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:23:0) after 5m59s
[01:08:02.449] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:23:0)
[01:08:02.466] Build status: STARTED
[01:08:02.466] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:24:0) after 5m11s
[01:13:13.617] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:24:0)
[01:13:13.638] Build status: STARTED
[01:13:13.639] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:25:0) after 6m31s
[01:19:44.653] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:25:0)
[01:19:44.670] Build status: STARTED
[01:19:44.670] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:26:0) after 9m11s
[01:28:55.747] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:26:0)
[01:28:55.762] Build status: STARTED
[01:28:55.762] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:27:0) after 9m43s
[01:38:38.756] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:27:0)
[01:38:38.776] Build status: STARTED
[01:38:38.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:28:0) after 4m55s
[01:43:33.857] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:28:0)
[01:43:33.876] Build status: STARTED
[01:43:33.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:29:0) after 7m57s
[01:51:30.843] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:29:0)
[01:51:30.859] Build status: STARTED
[01:51:30.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:30:0) after 3m1s
[01:54:32.131] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:30:0)
[01:54:32.149] Build status: STARTED
[01:54:32.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:31:0) after 7m1s
[02:01:33.289] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:31:0)
[02:01:33.316] Build status: STARTED
[02:01:33.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:32:0) after 3m11s
[02:04:44.336] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:32:0)
[02:04:44.355] Build status: STARTED
[02:04:44.355] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:33:0) after 3m2s
[02:07:46.471] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:33:0)
[02:07:46.511] Build status: STARTED
[02:07:46.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:34:0) after 8m23s
[02:16:10.036] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:34:0)
[02:16:10.056] Build status: STARTED
[02:16:10.056] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:35:0) after 4m38s
[02:20:48.078] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:35:0)
[02:20:48.095] Build status: STARTED
[02:20:48.095] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:36:0) after 6m20s
[02:27:08.308] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:36:0)
[02:27:08.323] Build status: STARTED
[02:27:08.323] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:37:0) after 7m10s
[02:34:18.342] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:37:0)
[02:34:18.363] Build status: STARTED
[02:34:18.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:38:0) after 8m21s
[02:42:39.425] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:38:0)
[02:42:39.484] Build status: STARTED
[02:42:39.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:39:0) after 5m22s
[02:48:01.504] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:39:0)
[02:48:01.519] Build status: STARTED
[02:48:01.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:40:0) after 3m24s
[02:51:25.536] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:40:0)
[02:51:25.671] Build status: STARTED
[02:51:25.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:41:0) after 9m47s
[03:01:12.744] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:41:0)
[03:01:12.763] Build status: STARTED
[03:01:12.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:42:0) after 7m53s
[03:09:05.821] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:42:0)
[03:09:05.850] Build status: STARTED
[03:09:05.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:43:0) after 2m13s
[03:11:18.919] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:43:0)
[03:11:18.941] Build status: STARTED
[03:11:18.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:44:0) after 8m14s
[03:19:32.954] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:44:0)
[03:19:32.987] Build status: STARTED
[03:19:32.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:45:0) after 3m54s
[03:23:27.043] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:45:0)
[03:23:27.058] Build status: STARTED
[03:23:27.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:46:0) after 9m44s
[03:33:11.930] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:46:0)
[03:33:11.945] Build status: STARTED
[03:33:11.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:47:0) after 2m49s
[03:36:01.017] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:47:0)
[03:36:01.038] Build status: STARTED
[03:36:01.038] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:48:0) after 4m14s
[03:40:15.106] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:48:0)
[03:40:15.131] Build status: STARTED
[03:40:15.131] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:49:0) after 6m42s
[03:46:57.192] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:49:0)
[03:46:57.219] Build status: STARTED
[03:46:57.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:50:0) after 4m4s
[03:51:01.240] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:50:0)
[03:51:01.267] Build status: STARTED
[03:51:01.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:51:0) after 7m3s
[03:58:04.284] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:51:0)
[03:58:04.309] Build status: STARTED
[03:58:04.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:52:0) after 2m12s
[04:00:16.324] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:52:0)
[04:00:16.341] Build status: STARTED
[04:00:16.341] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:53:0) after 6m11s
[04:06:27.359] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:53:0)
[04:06:27.379] Build status: STARTED
[04:06:27.379] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:54:0) after 8m25s
[04:14:52.392] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:54:0)
[04:14:52.408] Build status: STARTED
[04:14:52.408] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848846204683530336:55:0) after 6m54s
[04:21:37.481] Received PubSub notification, asking Buildbucket for the build status
[04:21:37.513] Build:
{
"id": "8683416142820488561",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20451,
"createdBy": "project:chromium",
"createTime": "2026-04-26T23:27:48.132413955Z",
"startTime": "2026-04-26T23:28:47.798676882Z",
"endTime": "2026-04-27T04:21:34.486670015Z",
"updateTime": "2026-04-27T04:21:34.486670015Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e878af67a395216c86b4e59813e343b361a5c511",
"ref": "refs/heads/main"
}
}
}
[04:21:37.513] Invocation finished in 4h53m50.541089102s with status FAILED