[15:25:04.429] New invocation is queued and will start shortly
[15:25:05.672] Starting the invocation (attempt 1)
[15:25:05.695] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/8e01df3d3559a3e9ad0f06f7dc4264da64b8fa92"
[15:25:05.695] Popped gitiles commit info from properties and tags
[15:25:05.696] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[15:25:05.696] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[15:25:05.696] Buildbucket request:
{
"requestId": "8842081801648172304",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8e01df3d3559a3e9ad0f06f7dc4264da64b8fa92",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842081801648172304"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[15:25:06.284] Scheduled build:
{
"id": "8676651739581444881",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21705,
"createdBy": "project:chromium",
"createTime": "2026-07-10T15:25:05.767187255Z",
"updateTime": "2026-07-10T15:25:05.767187255Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8e01df3d3559a3e9ad0f06f7dc4264da64b8fa92",
"ref": "refs/heads/main"
}
}
}
[15:25:06.284] Task URL: https://cr-buildbucket.appspot.com/build/8676651739581444881
[15:25:06.284] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:2:0) after 7m51s
[15:25:07.174] Received PubSub notification, asking Buildbucket for the build status
[15:25:07.207] Build status: SCHEDULED
[15:26:11.615] Received PubSub notification, asking Buildbucket for the build status
[15:26:11.674] Build status: STARTED
[15:32:57.437] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:2:0)
[15:32:57.453] Build status: STARTED
[15:32:57.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:5:0) after 1m16s
[15:34:13.616] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:5:0)
[15:34:13.653] Build status: STARTED
[15:34:13.653] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:6:0) after 7m39s
[15:41:52.784] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:6:0)
[15:41:52.809] Build status: STARTED
[15:41:52.809] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:7:0) after 3m47s
[15:45:39.826] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:7:0)
[15:45:39.860] Build status: STARTED
[15:45:39.860] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:8:0) after 8m56s
[15:54:35.911] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:8:0)
[15:54:35.932] Build status: STARTED
[15:54:35.932] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:9:0) after 7m8s
[16:01:44.035] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:9:0)
[16:01:44.055] Build status: STARTED
[16:01:44.055] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:10:0) after 2m1s
[16:03:45.076] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:10:0)
[16:03:45.163] Build status: STARTED
[16:03:45.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:11:0) after 6m20s
[16:10:05.177] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:11:0)
[16:10:05.199] Build status: STARTED
[16:10:05.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:12:0) after 6m19s
[16:16:24.242] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:12:0)
[16:16:24.274] Build status: STARTED
[16:16:24.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:13:0) after 7m49s
[16:24:13.312] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:13:0)
[16:24:13.338] Build status: STARTED
[16:24:13.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:14:0) after 8m43s
[16:32:56.492] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:14:0)
[16:32:56.520] Build status: STARTED
[16:32:56.520] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:15:0) after 6m23s
[16:39:19.533] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:15:0)
[16:39:19.548] Build status: STARTED
[16:39:19.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:16:0) after 8m50s
[16:48:09.526] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:16:0)
[16:48:09.572] Build status: STARTED
[16:48:09.572] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:17:0) after 9m47s
[16:57:56.688] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:17:0)
[16:57:56.728] Build status: STARTED
[16:57:56.728] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:18:0) after 1m41s
[16:59:37.800] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:18:0)
[16:59:37.836] Build status: STARTED
[16:59:37.836] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:19:0) after 4m56s
[17:04:33.905] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:19:0)
[17:04:33.934] Build status: STARTED
[17:04:33.934] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:20:0) after 5m33s
[17:10:07.066] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:20:0)
[17:10:07.094] Build status: STARTED
[17:10:07.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:21:0) after 4m36s
[17:14:43.155] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:21:0)
[17:14:43.172] Build status: STARTED
[17:14:43.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:22:0) after 8m46s
[17:23:29.241] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:22:0)
[17:23:29.259] Build status: STARTED
[17:23:29.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:23:0) after 7m52s
[17:31:21.273] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:23:0)
[17:31:21.289] Build status: STARTED
[17:31:21.289] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:24:0) after 2m32s
[17:33:53.320] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:24:0)
[17:33:53.335] Build status: STARTED
[17:33:53.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:25:0) after 2m32s
[17:36:25.348] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:25:0)
[17:36:25.366] Build status: STARTED
[17:36:25.366] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:26:0) after 5m59s
[17:42:24.597] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:26:0)
[17:42:24.617] Build status: STARTED
[17:42:24.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:27:0) after 4m40s
[17:47:04.665] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:27:0)
[17:47:04.684] Build status: STARTED
[17:47:04.684] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:28:0) after 5m53s
[17:52:57.797] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:28:0)
[17:52:57.814] Build status: STARTED
[17:52:57.814] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:29:0) after 6m22s
[17:59:19.879] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:29:0)
[17:59:19.898] Build status: STARTED
[17:59:19.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:30:0) after 7m51s
[18:07:10.905] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:30:0)
[18:07:10.924] Build status: STARTED
[18:07:10.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:31:0) after 9m27s
[18:16:38.006] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:31:0)
[18:16:38.033] Build status: STARTED
[18:16:38.033] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:32:0) after 6m13s
[18:22:51.106] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:32:0)
[18:22:51.150] Build status: STARTED
[18:22:51.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:33:0) after 8m25s
[18:31:16.172] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:33:0)
[18:31:16.192] Build status: STARTED
[18:31:16.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:34:0) after 6m46s
[18:38:02.206] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:34:0)
[18:38:02.225] Build status: STARTED
[18:38:02.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:35:0) after 7m49s
[18:45:51.244] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:35:0)
[18:45:51.270] Build status: STARTED
[18:45:51.270] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:36:0) after 8m29s
[18:54:20.376] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:36:0)
[18:54:20.408] Build status: STARTED
[18:54:20.408] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:37:0) after 9m16s
[19:03:36.460] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:37:0)
[19:03:36.476] Build status: STARTED
[19:03:36.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:38:0) after 7m21s
[19:10:57.538] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:38:0)
[19:10:57.573] Build status: STARTED
[19:10:57.573] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:39:0) after 7m53s
[19:18:50.757] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:39:0)
[19:18:50.773] Build status: STARTED
[19:18:50.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:40:0) after 1m19s
[19:20:09.926] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:40:0)
[19:20:09.945] Build status: STARTED
[19:20:09.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:41:0) after 5m3s
[19:25:12.963] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:41:0)
[19:25:12.982] Build status: STARTED
[19:25:12.982] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:42:0) after 7m46s
[19:32:59.003] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:42:0)
[19:32:59.039] Build status: STARTED
[19:32:59.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:43:0) after 2m23s
[19:35:22.097] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:43:0)
[19:35:22.118] Build status: STARTED
[19:35:22.118] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:44:0) after 6m51s
[19:42:13.124] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:44:0)
[19:42:13.144] Build status: STARTED
[19:42:13.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:45:0) after 6m44s
[19:48:57.356] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:45:0)
[19:48:57.377] Build status: STARTED
[19:48:57.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:46:0) after 3m26s
[19:52:23.538] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:46:0)
[19:52:23.553] Build status: STARTED
[19:52:23.553] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:47:0) after 1m37s
[19:54:00.683] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:47:0)
[19:54:00.773] Build status: STARTED
[19:54:00.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:48:0) after 1m42s
[19:55:42.806] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:48:0)
[19:55:42.834] Build status: STARTED
[19:55:42.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:49:0) after 6m51s
[20:02:33.849] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:49:0)
[20:02:33.867] Build status: STARTED
[20:02:33.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:50:0) after 5m45s
[20:08:18.979] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:50:0)
[20:08:19.000] Build status: STARTED
[20:08:19.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:51:0) after 2m49s
[20:11:08.068] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:51:0)
[20:11:08.097] Build status: STARTED
[20:11:08.097] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:52:0) after 6m12s
[20:17:20.225] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:52:0)
[20:17:20.243] Build status: STARTED
[20:17:20.243] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:53:0) after 4m3s
[20:21:23.355] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:53:0)
[20:21:23.428] Build status: STARTED
[20:21:23.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:54:0) after 5m18s
[20:26:41.447] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:54:0)
[20:26:41.474] Build status: STARTED
[20:26:41.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:55:0) after 9m20s
[20:36:01.490] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:55:0)
[20:36:01.510] Build status: STARTED
[20:36:01.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:56:0) after 2m53s
[20:38:54.592] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:56:0)
[20:38:54.622] Build status: STARTED
[20:38:54.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:57:0) after 7m46s
[20:46:40.690] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:57:0)
[20:46:40.709] Build status: STARTED
[20:46:40.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:58:0) after 9m19s
[20:55:59.725] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:58:0)
[20:55:59.741] Build status: STARTED
[20:55:59.741] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:59:0) after 4m53s
[21:00:53.039] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:59:0)
[21:00:53.063] Build status: STARTED
[21:00:53.063] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:60:0) after 5m16s
[21:06:09.084] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:60:0)
[21:06:09.136] Build status: STARTED
[21:06:09.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:61:0) after 1m7s
[21:07:16.246] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:61:0)
[21:07:16.265] Build status: STARTED
[21:07:16.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:62:0) after 2m55s
[21:10:11.328] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:62:0)
[21:10:11.359] Build status: STARTED
[21:10:11.359] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:63:0) after 6m47s
[21:16:58.386] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:63:0)
[21:16:58.403] Build status: STARTED
[21:16:58.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:64:0) after 6m39s
[21:23:37.516] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:64:0)
[21:23:37.551] Build status: STARTED
[21:23:37.551] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:65:0) after 3m39s
[21:27:16.573] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:65:0)
[21:27:16.591] Build status: STARTED
[21:27:16.591] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:66:0) after 9m18s
[21:36:34.612] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:66:0)
[21:36:34.636] Build status: STARTED
[21:36:34.636] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842081801648172304:67:0) after 3m56s
[21:39:36.689] Received PubSub notification, asking Buildbucket for the build status
[21:39:36.730] Build:
{
"id": "8676651739581444881",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21705,
"createdBy": "project:chromium",
"createTime": "2026-07-10T15:25:05.767187255Z",
"startTime": "2026-07-10T15:26:11.367146876Z",
"endTime": "2026-07-10T21:39:07.923316279Z",
"updateTime": "2026-07-10T21:39:07.923316279Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8e01df3d3559a3e9ad0f06f7dc4264da64b8fa92",
"ref": "refs/heads/main"
}
}
}
[21:39:36.730] Invocation finished in 6h14m32.316858839s with status FAILED