[06:41:04.586] New invocation is queued and will start shortly
[06:41:05.847] Starting the invocation (attempt 1)
[06:41:05.871] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/12ca094981d4a17ec8fab90e8cb7e76d2e9aff8a"
[06:41:05.872] Popped gitiles commit info from properties and tags
[06:41:05.872] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[06:41:05.872] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[06:41:05.872] Buildbucket request:
{
"requestId": "8842658350499885088",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "12ca094981d4a17ec8fab90e8cb7e76d2e9aff8a",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842658350499885088"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[06:41:06.382] Scheduled build:
{
"id": "8677228288435398001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21495,
"createdBy": "project:chromium",
"createTime": "2026-07-04T06:41:05.933080216Z",
"updateTime": "2026-07-04T06:41:05.933080216Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "12ca094981d4a17ec8fab90e8cb7e76d2e9aff8a",
"ref": "refs/heads/main"
}
},
"canary": true
}
[06:41:06.382] Task URL: https://cr-buildbucket.appspot.com/build/8677228288435398001
[06:41:06.382] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:2:0) after 4m22s
[06:41:07.323] Received PubSub notification, asking Buildbucket for the build status
[06:41:07.341] Build status: SCHEDULED
[06:42:23.808] Received PubSub notification, asking Buildbucket for the build status
[06:42:23.829] Build status: STARTED
[06:45:28.434] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:2:0)
[06:45:28.453] Build status: STARTED
[06:45:28.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:5:0) after 5m53s
[06:51:21.467] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:5:0)
[06:51:21.505] Build status: STARTED
[06:51:21.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:6:0) after 7m59s
[06:59:20.635] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:6:0)
[06:59:20.835] Build status: STARTED
[06:59:20.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:7:0) after 6m58s
[07:06:18.847] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:7:0)
[07:06:18.986] Build status: STARTED
[07:06:18.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:8:0) after 2m36s
[07:08:55.080] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:8:0)
[07:08:55.098] Build status: STARTED
[07:08:55.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:9:0) after 3m16s
[07:12:11.192] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:9:0)
[07:12:11.208] Build status: STARTED
[07:12:11.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:10:0) after 4m54s
[07:17:05.323] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:10:0)
[07:17:05.384] Build status: STARTED
[07:17:05.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:11:0) after 5m17s
[07:22:22.535] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:11:0)
[07:22:22.589] Build status: STARTED
[07:22:22.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:12:0) after 2m25s
[07:24:47.602] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:12:0)
[07:24:47.621] Build status: STARTED
[07:24:47.621] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:13:0) after 9m38s
[07:34:25.966] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:13:0)
[07:34:25.990] Build status: STARTED
[07:34:25.990] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:14:0) after 6m12s
[07:40:38.123] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:14:0)
[07:40:38.148] Build status: STARTED
[07:40:38.148] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:15:0) after 1m37s
[07:42:15.260] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:15:0)
[07:42:15.284] Build status: STARTED
[07:42:15.284] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:16:0) after 1m12s
[07:43:27.298] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:16:0)
[07:43:27.321] Build status: STARTED
[07:43:27.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:17:0) after 9m21s
[07:52:48.591] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:17:0)
[07:52:48.608] Build status: STARTED
[07:52:48.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:18:0) after 2m57s
[07:55:45.746] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:18:0)
[07:55:45.767] Build status: STARTED
[07:55:45.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:19:0) after 8m52s
[08:04:37.964] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:19:0)
[08:04:38.172] Build status: STARTED
[08:04:38.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:20:0) after 6m51s
[08:11:29.263] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:20:0)
[08:11:29.283] Build status: STARTED
[08:11:29.283] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:21:0) after 2m51s
[08:14:20.298] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:21:0)
[08:14:20.326] Build status: STARTED
[08:14:20.326] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:22:0) after 6m4s
[08:20:24.387] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:22:0)
[08:20:24.406] Build status: STARTED
[08:20:24.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:23:0) after 6m12s
[08:26:36.419] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:23:0)
[08:26:36.435] Build status: STARTED
[08:26:36.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:24:0) after 8m27s
[08:35:03.708] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:24:0)
[08:35:03.728] Build status: STARTED
[08:35:03.728] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:25:0) after 7m33s
[08:42:36.877] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:25:0)
[08:42:36.896] Build status: STARTED
[08:42:36.896] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:26:0) after 3m56s
[08:46:32.977] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:26:0)
[08:46:33.043] Build status: STARTED
[08:46:33.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:27:0) after 3m32s
[08:50:05.231] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:27:0)
[08:50:05.258] Build status: STARTED
[08:50:05.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:28:0) after 6m36s
[08:56:41.870] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:28:0)
[08:56:41.887] Build status: STARTED
[08:56:41.887] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:29:0) after 5m44s
[09:02:25.901] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:29:0)
[09:02:25.919] Build status: STARTED
[09:02:25.919] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:30:0) after 1m48s
[09:04:13.927] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:30:0)
[09:04:13.945] Build status: STARTED
[09:04:13.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:31:0) after 4m48s
[09:09:01.999] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:31:0)
[09:09:02.015] Build status: STARTED
[09:09:02.015] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:32:0) after 9m48s
[09:18:50.028] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:32:0)
[09:18:50.046] Build status: STARTED
[09:18:50.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:33:0) after 7m51s
[09:26:41.034] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:33:0)
[09:26:41.094] Build status: STARTED
[09:26:41.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:34:0) after 9m23s
[09:36:04.110] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:34:0)
[09:36:04.167] Build status: STARTED
[09:36:04.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:35:0) after 5m45s
[09:41:49.234] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:35:0)
[09:41:49.253] Build status: STARTED
[09:41:49.253] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:36:0) after 8m39s
[09:50:29.115] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:36:0)
[09:50:29.193] Build status: STARTED
[09:50:29.194] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:37:0) after 2m0s
[09:52:29.290] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:37:0)
[09:52:29.348] Build status: STARTED
[09:52:29.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:38:0) after 9m27s
[10:01:56.362] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:38:0)
[10:01:56.384] Build status: STARTED
[10:01:56.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:39:0) after 3m18s
[10:05:14.539] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:39:0)
[10:05:14.567] Build status: STARTED
[10:05:14.567] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:40:0) after 4m59s
[10:10:13.582] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:40:0)
[10:10:13.597] Build status: STARTED
[10:10:13.597] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:41:0) after 1m14s
[10:11:27.703] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:41:0)
[10:11:27.724] Build status: STARTED
[10:11:27.724] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:42:0) after 7m57s
[10:19:24.956] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:42:0)
[10:19:24.970] Build status: STARTED
[10:19:24.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:43:0) after 1m22s
[10:20:47.146] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:43:0)
[10:20:47.170] Build status: STARTED
[10:20:47.170] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:44:0) after 7m18s
[10:28:05.292] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:44:0)
[10:28:05.309] Build status: STARTED
[10:28:05.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:45:0) after 6m45s
[10:34:50.324] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:45:0)
[10:34:50.370] Build status: STARTED
[10:34:50.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:46:0) after 6m8s
[10:40:58.388] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:46:0)
[10:40:58.418] Build status: STARTED
[10:40:58.418] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:47:0) after 3m23s
[10:44:21.638] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:47:0)
[10:44:21.658] Build status: STARTED
[10:44:21.658] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:48:0) after 4m35s
[10:48:56.674] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:48:0)
[10:48:56.703] Build status: STARTED
[10:48:56.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:49:0) after 1m6s
[10:50:02.723] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:49:0)
[10:50:02.739] Build status: STARTED
[10:50:02.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:50:0) after 4m56s
[10:54:58.776] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:50:0)
[10:54:58.792] Build status: STARTED
[10:54:58.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:51:0) after 8m17s
[11:03:15.875] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:51:0)
[11:03:15.907] Build status: STARTED
[11:03:15.907] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:52:0) after 3m13s
[11:06:29.067] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:52:0)
[11:06:29.083] Build status: STARTED
[11:06:29.083] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:53:0) after 4m25s
[11:10:54.063] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:53:0)
[11:10:54.088] Build status: STARTED
[11:10:54.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:54:0) after 5m10s
[11:16:04.104] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:54:0)
[11:16:04.126] Build status: STARTED
[11:16:04.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:55:0) after 3m38s
[11:19:42.523] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:55:0)
[11:19:42.544] Build status: STARTED
[11:19:42.544] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:56:0) after 7m45s
[11:27:27.554] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:56:0)
[11:27:27.570] Build status: STARTED
[11:27:27.570] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:57:0) after 9m14s
[11:36:41.826] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:57:0)
[11:36:41.848] Build status: STARTED
[11:36:41.848] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:58:0) after 6m5s
[11:42:46.951] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:58:0)
[11:42:47.028] Build status: STARTED
[11:42:47.028] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:59:0) after 1m37s
[11:44:24.026] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:59:0)
[11:44:24.044] Build status: STARTED
[11:44:24.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:60:0) after 6m7s
[11:50:31.124] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:60:0)
[11:50:31.151] Build status: STARTED
[11:50:31.151] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:61:0) after 5m6s
[11:55:37.165] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:61:0)
[11:55:37.179] Build status: STARTED
[11:55:37.179] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:62:0) after 8m33s
[12:04:10.499] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:62:0)
[12:04:10.534] Build status: STARTED
[12:04:10.534] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:63:0) after 2m4s
[12:06:14.742] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:63:0)
[12:06:14.763] Build status: STARTED
[12:06:14.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842658350499885088:64:0) after 8m12s
[12:08:30.771] Received PubSub notification, asking Buildbucket for the build status
[12:08:30.788] Build:
{
"id": "8677228288435398001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21495,
"createdBy": "project:chromium",
"createTime": "2026-07-04T06:41:05.933080216Z",
"startTime": "2026-07-04T06:42:23.504520185Z",
"endTime": "2026-07-04T12:08:30.433147889Z",
"updateTime": "2026-07-04T12:08:30.433147889Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "12ca094981d4a17ec8fab90e8cb7e76d2e9aff8a",
"ref": "refs/heads/main"
}
},
"canary": true
}
[12:08:30.788] Invocation finished in 5h27m26.207692836s with status FAILED