[11:29:22.609] New invocation is queued and will start shortly
[11:29:23.719] Starting the invocation (attempt 1)
[11:29:23.743] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/51a4f193d4f524473d3efc7373fa9fed2f775753"
[11:29:23.743] Popped gitiles commit info from properties and tags
[11:29:23.743] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:29:23.743] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:29:23.743] Buildbucket request:
{
"requestId": "8833489918600754112",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "51a4f193d4f524473d3efc7373fa9fed2f775753",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8833489918600754112"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:29:24.689] Scheduled build:
{
"id": "8668059856436085585",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 23273,
"createdBy": "project:chromium",
"createTime": "2026-10-13T11:29:24.052081512Z",
"updateTime": "2026-10-13T11:29:24.052081512Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "51a4f193d4f524473d3efc7373fa9fed2f775753",
"ref": "refs/heads/main"
}
},
"canary": true
}
[11:29:24.689] Task URL: https://cr-buildbucket.appspot.com/build/8668059856436085585
[11:29:24.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:2:0) after 3m44s
[11:29:25.771] Received PubSub notification, asking Buildbucket for the build status
[11:29:25.802] Build status: SCHEDULED
[11:30:31.688] Received PubSub notification, asking Buildbucket for the build status
[11:30:31.709] Build status: STARTED
[11:33:08.679] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:2:0)
[11:33:08.698] Build status: STARTED
[11:33:08.698] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:5:0) after 2m24s
[11:35:32.717] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:5:0)
[11:35:32.734] Build status: STARTED
[11:35:32.734] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:6:0) after 1m14s
[11:36:46.806] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:6:0)
[11:36:46.824] Build status: STARTED
[11:36:46.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:7:0) after 3m40s
[11:40:26.848] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:7:0)
[11:40:26.864] Build status: STARTED
[11:40:26.864] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:8:0) after 2m54s
[11:43:20.894] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:8:0)
[11:43:20.931] Build status: STARTED
[11:43:20.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:9:0) after 9m54s
[11:53:14.969] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:9:0)
[11:53:15.002] Build status: STARTED
[11:53:15.002] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:10:0) after 3m47s
[11:57:02.056] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:10:0)
[11:57:02.156] Build status: STARTED
[11:57:02.156] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:11:0) after 1m3s
[11:58:05.172] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:11:0)
[11:58:05.229] Build status: STARTED
[11:58:05.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:12:0) after 6m51s
[12:04:56.300] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:12:0)
[12:04:56.321] Build status: STARTED
[12:04:56.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:13:0) after 4m57s
[12:09:53.431] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:13:0)
[12:09:53.499] Build status: STARTED
[12:09:53.499] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:14:0) after 6m18s
[12:16:11.617] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:14:0)
[12:16:11.641] Build status: STARTED
[12:16:11.641] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:15:0) after 5m57s
[12:22:08.659] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:15:0)
[12:22:08.682] Build status: STARTED
[12:22:08.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:16:0) after 6m28s
[12:28:36.751] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:16:0)
[12:28:36.776] Build status: STARTED
[12:28:36.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:17:0) after 4m31s
[12:33:07.842] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:17:0)
[12:33:07.870] Build status: STARTED
[12:33:07.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:18:0) after 9m17s
[12:42:24.930] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:18:0)
[12:42:25.009] Build status: STARTED
[12:42:25.009] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:19:0) after 9m9s
[12:51:33.994] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:19:0)
[12:51:34.022] Build status: STARTED
[12:51:34.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:20:0) after 7m22s
[12:58:56.039] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:20:0)
[12:58:56.057] Build status: STARTED
[12:58:56.057] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:21:0) after 9m34s
[13:08:30.078] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:21:0)
[13:08:30.100] Build status: STARTED
[13:08:30.100] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:22:0) after 2m4s
[13:10:34.156] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:22:0)
[13:10:34.179] Build status: STARTED
[13:10:34.179] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:23:0) after 5m16s
[13:15:50.249] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:23:0)
[13:15:50.271] Build status: STARTED
[13:15:50.271] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:24:0) after 1m23s
[13:17:13.328] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:24:0)
[13:17:13.344] Build status: STARTED
[13:17:13.344] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:25:0) after 5m21s
[13:22:34.365] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:25:0)
[13:22:34.380] Build status: STARTED
[13:22:34.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:26:0) after 3m15s
[13:25:49.509] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:26:0)
[13:25:49.544] Build status: STARTED
[13:25:49.544] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:27:0) after 8m42s
[13:34:31.575] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:27:0)
[13:34:32.832] Build status: STARTED
[13:34:32.832] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:28:0) after 1m51s
[13:36:24.059] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:28:0)
[13:36:24.139] Build status: STARTED
[13:36:24.140] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:29:0) after 1m59s
[13:38:23.166] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:29:0)
[13:38:23.187] Build status: STARTED
[13:38:23.187] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:30:0) after 1m21s
[13:39:44.304] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:30:0)
[13:39:44.329] Build status: STARTED
[13:39:44.329] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:31:0) after 5m56s
[13:45:40.376] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:31:0)
[13:45:40.398] Build status: STARTED
[13:45:40.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:32:0) after 3m47s
[13:49:27.497] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:32:0)
[13:49:27.532] Build status: STARTED
[13:49:27.532] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:33:0) after 1m46s
[13:51:13.529] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:33:0)
[13:51:13.598] Build status: STARTED
[13:51:13.598] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:34:0) after 6m41s
[13:57:54.615] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:34:0)
[13:57:54.640] Build status: STARTED
[13:57:54.640] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:35:0) after 6m15s
[14:04:09.659] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:35:0)
[14:04:09.686] Build status: STARTED
[14:04:09.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:36:0) after 9m51s
[14:14:00.879] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:36:0)
[14:14:00.900] Build status: STARTED
[14:14:00.900] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:37:0) after 6m15s
[14:20:15.938] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:37:0)
[14:20:15.964] Build status: STARTED
[14:20:15.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:38:0) after 4m24s
[14:24:39.987] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:38:0)
[14:24:40.006] Build status: STARTED
[14:24:40.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:39:0) after 3m11s
[14:27:51.009] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:39:0)
[14:27:51.072] Build status: STARTED
[14:27:51.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:40:0) after 9m42s
[14:37:33.080] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:40:0)
[14:37:33.123] Build status: STARTED
[14:37:33.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:41:0) after 3m54s
[14:41:27.107] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:41:0)
[14:41:27.127] Build status: STARTED
[14:41:27.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:42:0) after 5m17s
[14:46:44.168] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:42:0)
[14:46:44.188] Build status: STARTED
[14:46:44.188] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:43:0) after 5m49s
[14:52:33.245] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:43:0)
[14:52:33.267] Build status: STARTED
[14:52:33.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:44:0) after 7m37s
[15:00:10.355] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:44:0)
[15:00:10.391] Build status: STARTED
[15:00:10.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:45:0) after 5m1s
[15:05:11.442] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:45:0)
[15:05:11.463] Build status: STARTED
[15:05:11.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:46:0) after 8m20s
[15:13:31.520] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:46:0)
[15:13:31.539] Build status: STARTED
[15:13:31.539] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:47:0) after 3m37s
[15:17:08.535] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:47:0)
[15:17:08.558] Build status: STARTED
[15:17:08.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:48:0) after 2m55s
[15:20:03.693] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:48:0)
[15:20:03.714] Build status: STARTED
[15:20:03.714] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8833489918600754112:49:0) after 6m25s
[15:22:57.113] Received PubSub notification, asking Buildbucket for the build status
[15:22:57.140] Build:
{
"id": "8668059856436085585",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 23273,
"createdBy": "project:chromium",
"createTime": "2026-10-13T11:29:24.052081512Z",
"startTime": "2026-10-13T11:30:30.201668765Z",
"endTime": "2026-10-13T15:22:56.516802377Z",
"updateTime": "2026-10-13T15:22:56.516802377Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "51a4f193d4f524473d3efc7373fa9fed2f775753",
"ref": "refs/heads/main"
}
},
"canary": true
}
[15:22:57.140] Invocation finished in 3h53m34.53638344s with status FAILED