[23:45:37.178] New invocation is queued and will start shortly
[23:45:38.225] Starting the invocation (attempt 1)
[23:45:38.259] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/ea686ad2b10c878c4374ac8a7cedbcbecf349e25"
[23:45:38.259] Popped gitiles commit info from properties and tags
[23:45:38.259] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[23:45:38.259] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[23:45:38.259] Buildbucket request:
{
"requestId": "8839785385856269232",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ea686ad2b10c878c4374ac8a7cedbcbecf349e25",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8839785385856269232"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[23:45:38.895] Scheduled build:
{
"id": "8674355323903614449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 22168,
"createdBy": "project:chromium",
"createTime": "2026-08-04T23:45:38.419914078Z",
"updateTime": "2026-08-04T23:45:38.419914078Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ea686ad2b10c878c4374ac8a7cedbcbecf349e25",
"ref": "refs/heads/main"
}
}
}
[23:45:38.895] Task URL: https://cr-buildbucket.appspot.com/build/8674355323903614449
[23:45:38.895] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:2:0) after 7m33s
[23:45:42.373] Received PubSub notification, asking Buildbucket for the build status
[23:45:42.403] Build status: SCHEDULED
[23:46:51.471] Received PubSub notification, asking Buildbucket for the build status
[23:46:51.497] Build status: STARTED
[23:53:11.915] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:2:0)
[23:53:11.941] Build status: STARTED
[23:53:11.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:5:0) after 4m51s
[23:58:03.095] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:5:0)
[23:58:03.117] Build status: STARTED
[23:58:03.117] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:6:0) after 9m8s
[00:07:11.141] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:6:0)
[00:07:11.162] Build status: STARTED
[00:07:11.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:7:0) after 7m30s
[00:14:41.196] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:7:0)
[00:14:41.347] Build status: STARTED
[00:14:41.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:8:0) after 5m36s
[00:20:17.496] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:8:0)
[00:20:17.516] Build status: STARTED
[00:20:17.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:9:0) after 5m38s
[00:25:55.544] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:9:0)
[00:25:55.565] Build status: STARTED
[00:25:55.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:10:0) after 8m56s
[00:34:51.749] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:10:0)
[00:34:51.773] Build status: STARTED
[00:34:51.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:11:0) after 1m6s
[00:35:57.870] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:11:0)
[00:35:57.965] Build status: STARTED
[00:35:57.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:12:0) after 9m34s
[00:45:32.138] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:12:0)
[00:45:32.163] Build status: STARTED
[00:45:32.163] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:13:0) after 3m47s
[00:49:19.189] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:13:0)
[00:49:19.232] Build status: STARTED
[00:49:19.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:14:0) after 9m19s
[00:58:38.258] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:14:0)
[00:58:38.285] Build status: STARTED
[00:58:38.286] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:15:0) after 7m44s
[01:06:22.286] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:15:0)
[01:06:22.381] Build status: STARTED
[01:06:22.381] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:16:0) after 8m49s
[01:15:11.442] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:16:0)
[01:15:11.518] Build status: STARTED
[01:15:11.518] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:17:0) after 9m35s
[01:24:46.657] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:17:0)
[01:24:46.676] Build status: STARTED
[01:24:46.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:18:0) after 7m18s
[01:32:04.658] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:18:0)
[01:32:04.708] Build status: STARTED
[01:32:04.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:19:0) after 4m48s
[01:36:52.732] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:19:0)
[01:36:52.746] Build status: STARTED
[01:36:52.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:20:0) after 7m2s
[01:43:54.793] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:20:0)
[01:43:54.813] Build status: STARTED
[01:43:54.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:21:0) after 1m56s
[01:45:50.790] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:21:0)
[01:45:50.821] Build status: STARTED
[01:45:50.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:22:0) after 1m36s
[01:47:26.848] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:22:0)
[01:47:26.864] Build status: STARTED
[01:47:26.864] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:23:0) after 2m3s
[01:49:29.971] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:23:0)
[01:49:29.988] Build status: STARTED
[01:49:29.988] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:24:0) after 2m8s
[01:51:38.066] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:24:0)
[01:51:38.087] Build status: STARTED
[01:51:38.087] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:25:0) after 8m30s
[02:00:08.196] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:25:0)
[02:00:08.217] Build status: STARTED
[02:00:08.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:26:0) after 1m13s
[02:01:21.273] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:26:0)
[02:01:21.383] Build status: STARTED
[02:01:21.383] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:27:0) after 5m20s
[02:06:41.421] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:27:0)
[02:06:41.438] Build status: STARTED
[02:06:41.438] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:28:0) after 7m9s
[02:13:50.545] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:28:0)
[02:13:50.565] Build status: STARTED
[02:13:50.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:29:0) after 5m14s
[02:19:04.772] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:29:0)
[02:19:04.793] Build status: STARTED
[02:19:04.793] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:30:0) after 1m4s
[02:20:08.876] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:30:0)
[02:20:08.970] Build status: STARTED
[02:20:08.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:31:0) after 3m55s
[02:24:04.088] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:31:0)
[02:24:04.146] Build status: STARTED
[02:24:04.146] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:32:0) after 4m20s
[02:28:24.193] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:32:0)
[02:28:24.252] Build status: STARTED
[02:28:24.252] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:33:0) after 2m7s
[02:30:31.339] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:33:0)
[02:30:31.363] Build status: STARTED
[02:30:31.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:34:0) after 8m59s
[02:39:30.439] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:34:0)
[02:39:30.459] Build status: STARTED
[02:39:30.459] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:35:0) after 5m22s
[02:44:52.559] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:35:0)
[02:44:52.583] Build status: STARTED
[02:44:52.583] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:36:0) after 3m7s
[02:47:59.624] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:36:0)
[02:47:59.648] Build status: STARTED
[02:47:59.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:37:0) after 6m27s
[02:54:26.676] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:37:0)
[02:54:26.715] Build status: STARTED
[02:54:26.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:38:0) after 1m51s
[02:56:17.724] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:38:0)
[02:56:17.742] Build status: STARTED
[02:56:17.742] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:39:0) after 5m53s
[03:02:10.898] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:39:0)
[03:02:10.933] Build status: STARTED
[03:02:10.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:40:0) after 5m19s
[03:07:30.005] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:40:0)
[03:07:30.025] Build status: STARTED
[03:07:30.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:41:0) after 3m2s
[03:10:32.055] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:41:0)
[03:10:32.081] Build status: STARTED
[03:10:32.081] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:42:0) after 1m27s
[03:11:59.142] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:42:0)
[03:11:59.162] Build status: STARTED
[03:11:59.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:43:0) after 7m23s
[03:19:22.299] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:43:0)
[03:19:22.317] Build status: STARTED
[03:19:22.317] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:44:0) after 7m42s
[03:27:04.382] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:44:0)
[03:27:04.400] Build status: STARTED
[03:27:04.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:45:0) after 5m13s
[03:32:17.567] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:45:0)
[03:32:17.595] Build status: STARTED
[03:32:17.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:46:0) after 2m14s
[03:34:31.664] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:46:0)
[03:34:31.685] Build status: STARTED
[03:34:31.685] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:47:0) after 5m47s
[03:40:18.740] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:47:0)
[03:40:18.759] Build status: STARTED
[03:40:18.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:48:0) after 5m1s
[03:45:19.750] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:48:0)
[03:45:19.774] Build status: STARTED
[03:45:19.774] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:49:0) after 4m20s
[03:49:39.924] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:49:0)
[03:49:39.957] Build status: STARTED
[03:49:39.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:50:0) after 5m8s
[03:54:48.122] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:50:0)
[03:54:48.146] Build status: STARTED
[03:54:48.146] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:51:0) after 5m13s
[04:00:01.270] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:51:0)
[04:00:01.317] Build status: STARTED
[04:00:01.317] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:52:0) after 7m32s
[04:07:33.413] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:52:0)
[04:07:33.428] Build status: STARTED
[04:07:33.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:53:0) after 9m32s
[04:17:05.641] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:53:0)
[04:17:05.659] Build status: STARTED
[04:17:05.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:54:0) after 8m24s
[04:25:29.698] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:54:0)
[04:25:29.717] Build status: STARTED
[04:25:29.718] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:55:0) after 8m44s
[04:34:13.777] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:55:0)
[04:34:13.812] Build status: STARTED
[04:34:13.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:56:0) after 6m34s
[04:40:47.925] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:56:0)
[04:40:47.943] Build status: STARTED
[04:40:47.943] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:57:0) after 1m1s
[04:41:49.007] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:57:0)
[04:41:49.035] Build status: STARTED
[04:41:49.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:58:0) after 8m21s
[04:50:10.119] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:58:0)
[04:50:10.141] Build status: STARTED
[04:50:10.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:59:0) after 9m30s
[04:59:40.215] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:59:0)
[04:59:40.288] Build status: STARTED
[04:59:40.288] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:60:0) after 3m17s
[05:02:57.393] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:60:0)
[05:02:57.425] Build status: STARTED
[05:02:57.425] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:61:0) after 9m19s
[05:12:16.454] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:61:0)
[05:12:16.472] Build status: STARTED
[05:12:16.472] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:62:0) after 7m40s
[05:19:56.515] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:62:0)
[05:19:56.537] Build status: STARTED
[05:19:56.537] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:63:0) after 1m0s
[05:20:56.532] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:63:0)
[05:20:56.555] Build status: STARTED
[05:20:56.555] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8839785385856269232:64:0) after 9m2s
[05:27:40.390] Received PubSub notification, asking Buildbucket for the build status
[05:27:40.410] Build:
{
"id": "8674355323903614449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 22168,
"createdBy": "project:chromium",
"createTime": "2026-08-04T23:45:38.419914078Z",
"startTime": "2026-08-04T23:46:51.296153692Z",
"endTime": "2026-08-05T05:27:39.982356646Z",
"updateTime": "2026-08-05T05:27:39.982356646Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ea686ad2b10c878c4374ac8a7cedbcbecf349e25",
"ref": "refs/heads/main"
}
}
}
[05:27:40.410] Invocation finished in 5h42m3.236384408s with status FAILED