[09:36:10.333] New invocation is queued and will start shortly
[09:36:11.979] Starting the invocation (attempt 1)
[09:36:12.002] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/3005728580e2f4cfe5dee4a344173eadd4634e44"
[09:36:12.002] Popped gitiles commit info from properties and tags
[09:36:12.002] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[09:36:12.002] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[09:36:12.002] Buildbucket request:
{
"requestId": "8848807928139742288",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3005728580e2f4cfe5dee4a344173eadd4634e44",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848807928139742288"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[09:36:12.739] Scheduled build:
{
"id": "8683377865621746529",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20454,
"createdBy": "project:chromium",
"createTime": "2026-04-27T09:36:12.114418688Z",
"updateTime": "2026-04-27T09:36:12.114418688Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3005728580e2f4cfe5dee4a344173eadd4634e44",
"ref": "refs/heads/main"
}
},
"canary": true
}
[09:36:12.739] Task URL: https://cr-buildbucket.appspot.com/build/8683377865621746529
[09:36:12.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:2:0) after 1m9s
[09:36:28.987] Received PubSub notification, asking Buildbucket for the build status
[09:36:29.011] Build status: STARTED
[09:36:33.486] Received PubSub notification, asking Buildbucket for the build status
[09:36:33.643] Build status: STARTED
[09:37:21.822] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:2:0)
[09:37:21.841] Build status: STARTED
[09:37:21.841] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:5:0) after 3m43s
[09:41:04.933] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:5:0)
[09:41:04.987] Build status: STARTED
[09:41:04.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:6:0) after 9m57s
[09:51:01.987] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:6:0)
[09:51:02.006] Build status: STARTED
[09:51:02.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:7:0) after 4m34s
[09:55:36.076] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:7:0)
[09:55:36.102] Build status: STARTED
[09:55:36.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:8:0) after 1m31s
[09:57:07.155] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:8:0)
[09:57:07.211] Build status: STARTED
[09:57:07.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:9:0) after 4m10s
[10:01:17.229] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:9:0)
[10:01:17.249] Build status: STARTED
[10:01:17.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:10:0) after 6m57s
[10:08:14.286] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:10:0)
[10:08:14.311] Build status: STARTED
[10:08:14.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:11:0) after 9m39s
[10:17:53.319] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:11:0)
[10:17:53.353] Build status: STARTED
[10:17:53.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:12:0) after 1m58s
[10:19:52.118] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:12:0)
[10:19:52.135] Build status: STARTED
[10:19:52.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:13:0) after 9m35s
[10:29:27.444] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:13:0)
[10:29:27.479] Build status: STARTED
[10:29:27.479] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:14:0) after 3m0s
[10:32:27.504] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:14:0)
[10:32:27.536] Build status: STARTED
[10:32:27.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:15:0) after 2m14s
[10:34:41.558] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:15:0)
[10:34:41.574] Build status: STARTED
[10:34:41.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:16:0) after 8m54s
[10:43:35.587] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:16:0)
[10:43:35.608] Build status: STARTED
[10:43:35.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:17:0) after 6m12s
[10:49:48.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:17:0)
[10:49:48.820] Build status: STARTED
[10:49:48.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:18:0) after 8m47s
[10:58:35.862] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:18:0)
[10:58:35.879] Build status: STARTED
[10:58:35.879] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:19:0) after 7m13s
[11:05:49.024] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:19:0)
[11:05:49.105] Build status: STARTED
[11:05:49.105] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:20:0) after 5m14s
[11:11:03.096] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:20:0)
[11:11:03.115] Build status: STARTED
[11:11:03.115] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:21:0) after 9m10s
[11:20:13.135] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:21:0)
[11:20:13.155] Build status: STARTED
[11:20:13.156] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:22:0) after 6m11s
[11:26:24.170] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:22:0)
[11:26:24.189] Build status: STARTED
[11:26:24.189] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:23:0) after 7m39s
[11:34:03.243] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:23:0)
[11:34:03.263] Build status: STARTED
[11:34:03.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:24:0) after 6m36s
[11:40:39.329] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:24:0)
[11:40:39.343] Build status: STARTED
[11:40:39.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:25:0) after 8m28s
[11:49:07.488] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:25:0)
[11:49:07.506] Build status: STARTED
[11:49:07.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:26:0) after 7m35s
[11:56:42.829] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:26:0)
[11:56:42.858] Build status: STARTED
[11:56:42.858] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:27:0) after 7m8s
[12:03:50.873] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:27:0)
[12:03:50.906] Build status: STARTED
[12:03:50.906] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:28:0) after 2m27s
[12:06:17.927] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:28:0)
[12:06:17.946] Build status: STARTED
[12:06:17.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:29:0) after 7m27s
[12:13:45.000] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:29:0)
[12:13:45.022] Build status: STARTED
[12:13:45.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:30:0) after 2m41s
[12:16:26.043] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:30:0)
[12:16:26.059] Build status: STARTED
[12:16:26.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:31:0) after 6m25s
[12:22:51.056] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:31:0)
[12:22:51.076] Build status: STARTED
[12:22:51.076] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:32:0) after 1m36s
[12:24:27.133] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:32:0)
[12:24:27.216] Build status: STARTED
[12:24:27.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:33:0) after 5m29s
[12:29:56.356] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:33:0)
[12:29:56.377] Build status: STARTED
[12:29:56.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:34:0) after 3m43s
[12:33:39.361] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:34:0)
[12:33:39.395] Build status: STARTED
[12:33:39.395] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:35:0) after 6m1s
[12:39:40.407] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:35:0)
[12:39:40.436] Build status: STARTED
[12:39:40.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:36:0) after 8m24s
[12:48:04.498] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:36:0)
[12:48:04.516] Build status: STARTED
[12:48:04.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:37:0) after 6m21s
[12:54:25.580] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:37:0)
[12:54:25.596] Build status: STARTED
[12:54:25.596] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:38:0) after 5m48s
[13:00:13.635] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:38:0)
[13:00:13.871] Build status: STARTED
[13:00:13.871] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:39:0) after 6m42s
[13:06:56.027] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:39:0)
[13:06:56.050] Build status: STARTED
[13:06:56.050] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:40:0) after 6m9s
[13:13:05.065] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:40:0)
[13:13:05.105] Build status: STARTED
[13:13:05.105] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:41:0) after 7m18s
[13:20:23.171] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:41:0)
[13:20:23.195] Build status: STARTED
[13:20:23.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:42:0) after 8m17s
[13:28:40.213] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:42:0)
[13:28:40.234] Build status: STARTED
[13:28:40.234] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:43:0) after 5m49s
[13:34:29.306] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:43:0)
[13:34:29.358] Build status: STARTED
[13:34:29.358] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:44:0) after 9m56s
[13:44:26.013] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:44:0)
[13:44:26.036] Build status: STARTED
[13:44:26.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:45:0) after 7m26s
[13:51:52.105] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:45:0)
[13:51:52.128] Build status: STARTED
[13:51:52.128] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:46:0) after 1m52s
[13:53:44.094] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:46:0)
[13:53:44.119] Build status: STARTED
[13:53:44.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:47:0) after 1m35s
[13:55:19.203] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:47:0)
[13:55:19.224] Build status: STARTED
[13:55:19.224] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:48:0) after 6m17s
[14:01:36.247] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:48:0)
[14:01:36.301] Build status: STARTED
[14:01:36.301] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:49:0) after 3m6s
[14:04:42.322] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:49:0)
[14:04:42.468] Build status: STARTED
[14:04:42.468] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:50:0) after 5m44s
[14:10:26.484] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:50:0)
[14:10:26.507] Build status: STARTED
[14:10:26.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:51:0) after 5m11s
[14:15:37.536] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:51:0)
[14:15:37.549] Build status: STARTED
[14:15:37.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:52:0) after 4m44s
[14:20:21.582] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:52:0)
[14:20:21.656] Build status: STARTED
[14:20:21.656] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:53:0) after 5m54s
[14:26:15.671] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:53:0)
[14:26:15.755] Build status: STARTED
[14:26:15.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:54:0) after 8m8s
[14:34:23.773] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:54:0)
[14:34:23.791] Build status: STARTED
[14:34:23.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:55:0) after 1m8s
[14:35:31.906] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:55:0)
[14:35:32.670] Build status: STARTED
[14:35:32.670] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:56:0) after 6m20s
[14:41:52.688] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:56:0)
[14:41:52.715] Build status: STARTED
[14:41:52.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:57:0) after 6m20s
[14:48:12.785] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:57:0)
[14:48:12.810] Build status: STARTED
[14:48:12.810] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:58:0) after 1m27s
[14:49:39.881] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:58:0)
[14:49:39.904] Build status: STARTED
[14:49:39.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8848807928139742288:59:0) after 4m8s
[14:52:41.483] Received PubSub notification, asking Buildbucket for the build status
[14:52:41.508] Build:
{
"id": "8683377865621746529",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20454,
"createdBy": "project:chromium",
"createTime": "2026-04-27T09:36:12.114418688Z",
"startTime": "2026-04-27T09:36:18.299140199Z",
"endTime": "2026-04-27T14:52:39.797366121Z",
"updateTime": "2026-04-27T14:52:39.797366121Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3005728580e2f4cfe5dee4a344173eadd4634e44",
"ref": "refs/heads/main"
}
},
"canary": true
}
[14:52:41.508] Invocation finished in 5h16m31.178882649s with status FAILED