[13:32:01.441] New invocation is queued and will start shortly
[13:32:02.527] Starting the invocation (attempt 1)
[13:32:02.575] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/31de92478587a03700ef31aeb4a8da44aedc4a56"
[13:32:02.575] Popped gitiles commit info from properties and tags
[13:32:02.576] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[13:32:02.576] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[13:32:02.576] Buildbucket request:
{
"requestId": "8857580995444113376",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "31de92478587a03700ef31aeb4a8da44aedc4a56",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857580995444113376"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[13:32:03.124] Scheduled build:
{
"id": "8692150933431002577",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 19400,
"createdBy": "project:chromium",
"createTime": "2026-01-20T13:32:02.668070186Z",
"updateTime": "2026-01-20T13:32:02.668070186Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "31de92478587a03700ef31aeb4a8da44aedc4a56",
"ref": "refs/heads/main"
}
}
}
[13:32:03.124] Task URL: https://cr-buildbucket.appspot.com/build/8692150933431002577
[13:32:03.124] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:2:0) after 8m58s
[13:32:03.251] Received PubSub notification, asking Buildbucket for the build status
[13:32:03.283] Build status: SCHEDULED
[13:33:07.373] Received PubSub notification, asking Buildbucket for the build status
[13:33:07.389] Build status: STARTED
[13:41:01.171] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:2:0)
[13:41:01.202] Build status: STARTED
[13:41:01.202] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:5:0) after 2m19s
[13:43:20.231] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:5:0)
[13:43:20.243] Build status: STARTED
[13:43:20.243] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:6:0) after 5m2s
[13:48:22.259] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:6:0)
[13:48:22.277] Build status: STARTED
[13:48:22.277] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:7:0) after 3m20s
[13:51:42.339] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:7:0)
[13:51:42.356] Build status: STARTED
[13:51:42.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:8:0) after 7m43s
[13:59:25.374] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:8:0)
[13:59:25.388] Build status: STARTED
[13:59:25.388] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:9:0) after 2m19s
[14:01:44.495] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:9:0)
[14:01:44.595] Build status: STARTED
[14:01:44.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:10:0) after 3m37s
[14:05:21.712] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:10:0)
[14:05:21.729] Build status: STARTED
[14:05:21.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:11:0) after 6m2s
[14:11:23.770] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:11:0)
[14:11:23.786] Build status: STARTED
[14:11:23.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:12:0) after 2m5s
[14:13:28.755] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:12:0)
[14:13:28.773] Build status: STARTED
[14:13:28.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:13:0) after 7m1s
[14:20:29.795] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:13:0)
[14:20:29.812] Build status: STARTED
[14:20:29.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:14:0) after 1m53s
[14:22:22.940] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:14:0)
[14:22:22.970] Build status: STARTED
[14:22:22.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:15:0) after 6m32s
[14:28:55.012] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:15:0)
[14:28:55.036] Build status: STARTED
[14:28:55.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:16:0) after 4m40s
[14:33:35.019] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:16:0)
[14:33:35.035] Build status: STARTED
[14:33:35.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:17:0) after 3m17s
[14:36:52.086] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:17:0)
[14:36:52.161] Build status: STARTED
[14:36:52.161] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:18:0) after 3m42s
[14:40:34.278] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:18:0)
[14:40:34.310] Build status: STARTED
[14:40:34.310] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:19:0) after 7m50s
[14:48:24.365] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:19:0)
[14:48:24.378] Build status: STARTED
[14:48:24.378] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:20:0) after 2m32s
[14:50:56.447] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:20:0)
[14:50:56.463] Build status: STARTED
[14:50:56.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:21:0) after 7m28s
[14:58:24.478] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:21:0)
[14:58:24.494] Build status: STARTED
[14:58:24.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:22:0) after 6m18s
[15:04:42.598] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:22:0)
[15:04:42.618] Build status: STARTED
[15:04:42.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:23:0) after 7m28s
[15:12:10.654] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:23:0)
[15:12:10.674] Build status: STARTED
[15:12:10.674] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:24:0) after 2m4s
[15:14:14.701] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:24:0)
[15:14:14.728] Build status: STARTED
[15:14:14.728] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:25:0) after 2m26s
[15:16:40.744] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:25:0)
[15:16:40.771] Build status: STARTED
[15:16:40.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:26:0) after 1m52s
[15:18:32.841] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:26:0)
[15:18:32.854] Build status: STARTED
[15:18:32.855] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:27:0) after 3m23s
[15:21:55.909] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:27:0)
[15:21:55.938] Build status: STARTED
[15:21:55.938] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:28:0) after 5m31s
[15:27:26.955] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:28:0)
[15:27:26.972] Build status: STARTED
[15:27:26.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:29:0) after 7m3s
[15:34:30.009] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:29:0)
[15:34:30.033] Build status: STARTED
[15:34:30.033] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:30:0) after 8m34s
[15:43:04.051] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:30:0)
[15:43:04.071] Build status: STARTED
[15:43:04.071] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:31:0) after 4m35s
[15:47:39.090] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:31:0)
[15:47:39.119] Build status: STARTED
[15:47:39.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:32:0) after 1m26s
[15:49:05.114] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:32:0)
[15:49:05.139] Build status: STARTED
[15:49:05.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:33:0) after 9m34s
[15:58:39.143] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:33:0)
[15:58:39.157] Build status: STARTED
[15:58:39.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:34:0) after 7m10s
[16:05:49.192] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:34:0)
[16:05:49.230] Build status: STARTED
[16:05:49.230] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:35:0) after 6m11s
[16:12:00.248] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:35:0)
[16:12:00.272] Build status: STARTED
[16:12:00.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:36:0) after 2m41s
[16:14:41.377] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:36:0)
[16:14:41.393] Build status: STARTED
[16:14:41.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:37:0) after 3m10s
[16:17:51.441] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:37:0)
[16:17:51.458] Build status: STARTED
[16:17:51.458] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:38:0) after 6m4s
[16:23:55.452] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:38:0)
[16:23:55.468] Build status: STARTED
[16:23:55.468] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:39:0) after 4m2s
[16:27:57.658] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:39:0)
[16:27:57.672] Build status: STARTED
[16:27:57.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:40:0) after 6m30s
[16:34:27.641] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:40:0)
[16:34:27.659] Build status: STARTED
[16:34:27.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:41:0) after 2m23s
[16:36:50.680] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:41:0)
[16:36:50.710] Build status: STARTED
[16:36:50.710] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:42:0) after 3m27s
[16:40:17.780] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:42:0)
[16:40:17.797] Build status: STARTED
[16:40:17.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:43:0) after 5m57s
[16:46:14.808] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:43:0)
[16:46:14.826] Build status: STARTED
[16:46:14.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:44:0) after 2m44s
[16:48:58.842] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:44:0)
[16:48:58.890] Build status: STARTED
[16:48:58.890] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:45:0) after 5m56s
[16:54:54.898] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:45:0)
[16:54:54.910] Build status: STARTED
[16:54:54.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:46:0) after 5m37s
[17:00:31.970] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:46:0)
[17:00:32.025] Build status: STARTED
[17:00:32.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:47:0) after 5m53s
[17:06:25.045] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:47:0)
[17:06:25.068] Build status: STARTED
[17:06:25.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:48:0) after 5m30s
[17:11:55.167] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:48:0)
[17:11:55.185] Build status: STARTED
[17:11:55.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:49:0) after 4m56s
[17:16:51.208] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:49:0)
[17:16:51.223] Build status: STARTED
[17:16:51.223] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:50:0) after 9m16s
[17:26:07.415] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:50:0)
[17:26:07.486] Build status: STARTED
[17:26:07.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:51:0) after 6m8s
[17:32:15.551] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:51:0)
[17:32:15.568] Build status: STARTED
[17:32:15.568] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:52:0) after 6m55s
[17:39:10.603] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:52:0)
[17:39:10.640] Build status: STARTED
[17:39:10.640] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:53:0) after 4m8s
[17:43:18.771] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:53:0)
[17:43:18.787] Build status: STARTED
[17:43:18.787] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:54:0) after 2m22s
[17:45:40.809] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:54:0)
[17:45:40.840] Build status: STARTED
[17:45:40.840] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:55:0) after 3m45s
[17:49:25.856] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:55:0)
[17:49:25.902] Build status: STARTED
[17:49:25.902] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:56:0) after 5m18s
[17:54:43.921] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:56:0)
[17:54:43.950] Build status: STARTED
[17:54:43.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:57:0) after 5m41s
[18:00:24.946] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:57:0)
[18:00:24.970] Build status: STARTED
[18:00:24.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:58:0) after 3m38s
[18:04:02.988] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:58:0)
[18:04:03.008] Build status: STARTED
[18:04:03.008] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:59:0) after 5m40s
[18:09:43.027] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:59:0)
[18:09:43.045] Build status: STARTED
[18:09:43.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:60:0) after 2m52s
[18:12:35.065] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:60:0)
[18:12:35.249] Build status: STARTED
[18:12:35.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:61:0) after 6m59s
[18:19:34.325] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:61:0)
[18:19:34.345] Build status: STARTED
[18:19:34.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:62:0) after 1m41s
[18:21:15.370] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:62:0)
[18:21:15.399] Build status: STARTED
[18:21:15.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:63:0) after 1m14s
[18:22:29.393] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:63:0)
[18:22:29.408] Build status: STARTED
[18:22:29.408] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:64:0) after 3m59s
[18:26:28.537] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:64:0)
[18:26:28.558] Build status: STARTED
[18:26:28.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857580995444113376:65:0) after 5m48s
[18:32:05.831] Received PubSub notification, asking Buildbucket for the build status
[18:32:05.845] Build:
{
"id": "8692150933431002577",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 19400,
"createdBy": "project:chromium",
"createTime": "2026-01-20T13:32:02.668070186Z",
"startTime": "2026-01-20T13:33:07.194194389Z",
"endTime": "2026-01-20T18:32:05.471928083Z",
"updateTime": "2026-01-20T18:32:05.471928083Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "31de92478587a03700ef31aeb4a8da44aedc4a56",
"ref": "refs/heads/main"
}
}
}
[18:32:05.845] Invocation finished in 5h0m4.480982042s with status FAILED