[10:45:55.038] New invocation is queued and will start shortly
[10:45:56.089] Starting the invocation (attempt 1)
[10:45:56.116] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d07c2b8ad0064c5928c295e5a6d5de0d3fc0ca04"
[10:45:56.116] Popped gitiles commit info from properties and tags
[10:45:56.116] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[10:45:56.117] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[10:45:56.117] Buildbucket request:
{
"requestId": "8846901003863570848",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d07c2b8ad0064c5928c295e5a6d5de0d3fc0ca04",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8846901003863570848"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[10:45:56.728] Scheduled build:
{
"id": "8681470941879617553",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20635,
"createdBy": "project:chromium",
"createTime": "2026-05-18T10:45:56.310296640Z",
"updateTime": "2026-05-18T10:45:56.310296640Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d07c2b8ad0064c5928c295e5a6d5de0d3fc0ca04",
"ref": "refs/heads/main"
}
},
"canary": true
}
[10:45:56.728] Task URL: https://cr-buildbucket.appspot.com/build/8681470941879617553
[10:45:56.728] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:2:0) after 4m58s
[10:45:57.718] Received PubSub notification, asking Buildbucket for the build status
[10:45:57.740] Build status: SCHEDULED
[10:46:58.894] Received PubSub notification, asking Buildbucket for the build status
[10:46:58.915] Build status: STARTED
[10:50:54.951] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:2:0)
[10:50:54.977] Build status: STARTED
[10:50:54.977] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:5:0) after 2m29s
[10:53:23.993] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:5:0)
[10:53:24.018] Build status: STARTED
[10:53:24.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:6:0) after 4m17s
[10:57:41.133] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:6:0)
[10:57:41.208] Build status: STARTED
[10:57:41.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:7:0) after 8m16s
[11:05:57.456] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:7:0)
[11:05:57.475] Build status: STARTED
[11:05:57.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:8:0) after 4m21s
[11:10:18.491] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:8:0)
[11:10:18.517] Build status: STARTED
[11:10:18.517] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:9:0) after 5m44s
[11:16:02.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:9:0)
[11:16:02.610] Build status: STARTED
[11:16:02.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:10:0) after 7m23s
[11:23:25.627] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:10:0)
[11:23:25.704] Build status: STARTED
[11:23:25.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:11:0) after 5m38s
[11:29:03.831] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:11:0)
[11:29:03.849] Build status: STARTED
[11:29:03.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:12:0) after 5m12s
[11:34:15.863] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:12:0)
[11:34:15.926] Build status: STARTED
[11:34:15.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:13:0) after 9m47s
[11:44:03.042] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:13:0)
[11:44:03.105] Build status: STARTED
[11:44:03.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:14:0) after 9m57s
[11:54:00.120] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:14:0)
[11:54:00.141] Build status: STARTED
[11:54:00.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:15:0) after 4m41s
[11:58:41.308] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:15:0)
[11:58:41.323] Build status: STARTED
[11:58:41.323] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:16:0) after 6m8s
[12:04:49.388] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:16:0)
[12:04:49.405] Build status: STARTED
[12:04:49.405] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:17:0) after 9m34s
[12:14:23.500] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:17:0)
[12:14:23.518] Build status: STARTED
[12:14:23.518] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:18:0) after 5m19s
[12:19:42.614] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:18:0)
[12:19:42.633] Build status: STARTED
[12:19:42.633] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:19:0) after 1m42s
[12:21:24.722] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:19:0)
[12:21:24.821] Build status: STARTED
[12:21:24.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:20:0) after 4m42s
[12:26:07.006] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:20:0)
[12:26:07.031] Build status: STARTED
[12:26:07.031] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:21:0) after 5m37s
[12:31:44.075] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:21:0)
[12:31:44.094] Build status: STARTED
[12:31:44.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:22:0) after 1m5s
[12:32:49.111] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:22:0)
[12:32:49.134] Build status: STARTED
[12:32:49.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:23:0) after 7m23s
[12:40:12.211] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:23:0)
[12:40:12.228] Build status: STARTED
[12:40:12.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:24:0) after 7m55s
[12:48:07.461] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:24:0)
[12:48:07.484] Build status: STARTED
[12:48:07.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:25:0) after 4m12s
[12:52:19.513] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:25:0)
[12:52:19.537] Build status: STARTED
[12:52:19.537] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:26:0) after 1m39s
[12:53:58.529] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:26:0)
[12:53:58.543] Build status: STARTED
[12:53:58.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:27:0) after 2m57s
[12:56:55.560] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:27:0)
[12:56:55.576] Build status: STARTED
[12:56:55.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:28:0) after 3m38s
[13:00:33.634] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:28:0)
[13:00:33.652] Build status: STARTED
[13:00:33.652] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:29:0) after 8m14s
[13:08:47.668] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:29:0)
[13:08:47.689] Build status: STARTED
[13:08:47.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:30:0) after 2m41s
[13:11:28.702] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:30:0)
[13:11:28.721] Build status: STARTED
[13:11:28.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:31:0) after 7m14s
[13:18:42.738] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:31:0)
[13:18:42.775] Build status: STARTED
[13:18:42.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:32:0) after 2m50s
[13:21:32.890] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:32:0)
[13:21:32.909] Build status: STARTED
[13:21:32.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:33:0) after 7m43s
[13:29:16.076] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:33:0)
[13:29:16.095] Build status: STARTED
[13:29:16.095] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:34:0) after 3m43s
[13:32:59.111] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:34:0)
[13:32:59.138] Build status: STARTED
[13:32:59.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:35:0) after 5m19s
[13:38:18.157] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:35:0)
[13:38:18.219] Build status: STARTED
[13:38:18.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:36:0) after 7m19s
[13:45:37.239] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:36:0)
[13:45:37.267] Build status: STARTED
[13:45:37.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:37:0) after 1m53s
[13:47:30.288] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:37:0)
[13:47:30.306] Build status: STARTED
[13:47:30.306] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:38:0) after 3m4s
[13:50:34.769] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:38:0)
[13:50:34.786] Build status: STARTED
[13:50:34.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:39:0) after 3m14s
[13:53:48.838] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:39:0)
[13:53:48.855] Build status: STARTED
[13:53:48.855] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:40:0) after 8m24s
[14:02:12.887] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:40:0)
[14:02:12.926] Build status: STARTED
[14:02:12.927] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:41:0) after 9m23s
[14:11:35.945] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:41:0)
[14:11:36.050] Build status: STARTED
[14:11:36.051] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:42:0) after 2m10s
[14:13:46.558] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:42:0)
[14:13:46.593] Build status: STARTED
[14:13:46.593] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:43:0) after 2m36s
[14:16:22.621] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:43:0)
[14:16:22.636] Build status: STARTED
[14:16:22.636] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:44:0) after 6m30s
[14:22:52.649] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:44:0)
[14:22:52.676] Build status: STARTED
[14:22:52.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:45:0) after 3m37s
[14:26:29.814] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:45:0)
[14:26:30.151] Build status: STARTED
[14:26:30.151] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:46:0) after 2m2s
[14:28:32.169] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:46:0)
[14:28:32.186] Build status: STARTED
[14:28:32.186] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:47:0) after 4m34s
[14:33:06.226] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:47:0)
[14:33:06.246] Build status: STARTED
[14:33:06.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:48:0) after 3m44s
[14:36:50.315] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:48:0)
[14:36:50.392] Build status: STARTED
[14:36:50.392] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:49:0) after 1m14s
[14:38:04.409] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:49:0)
[14:38:04.510] Build status: STARTED
[14:38:04.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:50:0) after 5m11s
[14:43:15.728] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:50:0)
[14:43:15.822] Build status: STARTED
[14:43:15.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:51:0) after 8m2s
[14:51:17.895] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:51:0)
[14:51:17.912] Build status: STARTED
[14:51:17.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:52:0) after 3m20s
[14:54:38.013] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:52:0)
[14:54:38.094] Build status: STARTED
[14:54:38.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:53:0) after 7m14s
[15:01:52.105] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:53:0)
[15:01:52.121] Build status: STARTED
[15:01:52.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:54:0) after 1m7s
[15:02:59.156] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:54:0)
[15:02:59.175] Build status: STARTED
[15:02:59.175] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:55:0) after 7m21s
[15:10:20.244] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:55:0)
[15:10:20.264] Build status: STARTED
[15:10:20.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:56:0) after 7m52s
[15:18:12.232] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:56:0)
[15:18:12.259] Build status: STARTED
[15:18:12.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:57:0) after 3m39s
[15:21:51.309] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:57:0)
[15:21:51.365] Build status: STARTED
[15:21:51.365] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:58:0) after 8m54s
[15:30:45.383] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:58:0)
[15:30:45.407] Build status: STARTED
[15:30:45.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:59:0) after 3m57s
[15:34:42.437] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:59:0)
[15:34:42.461] Build status: STARTED
[15:34:42.461] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:60:0) after 7m25s
[15:42:07.478] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:60:0)
[15:42:07.505] Build status: STARTED
[15:42:07.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:61:0) after 5m7s
[15:47:14.576] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:61:0)
[15:47:14.592] Build status: STARTED
[15:47:14.592] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:62:0) after 3m31s
[15:50:45.610] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:62:0)
[15:50:45.694] Build status: STARTED
[15:50:45.694] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:63:0) after 8m12s
[15:58:57.762] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:63:0)
[15:58:57.783] Build status: STARTED
[15:58:57.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:64:0) after 8m57s
[16:07:54.800] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:64:0)
[16:07:54.818] Build status: STARTED
[16:07:54.818] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:65:0) after 3m0s
[16:10:54.885] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:65:0)
[16:10:54.908] Build status: STARTED
[16:10:54.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8846901003863570848:66:0) after 5m19s
[16:12:15.329] Received PubSub notification, asking Buildbucket for the build status
[16:12:15.406] Build:
{
"id": "8681470941879617553",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 20635,
"createdBy": "project:chromium",
"createTime": "2026-05-18T10:45:56.310296640Z",
"startTime": "2026-05-18T10:46:58.638674612Z",
"endTime": "2026-05-18T16:12:15.017747745Z",
"updateTime": "2026-05-18T16:12:15.017747745Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d07c2b8ad0064c5928c295e5a6d5de0d3fc0ca04",
"ref": "refs/heads/main"
}
},
"canary": true
}
[16:12:15.406] Invocation finished in 5h26m20.372392764s with status FAILED