[20:43:03.669] New invocation is queued and will start shortly
[20:43:04.723] Starting the invocation (attempt 1)
[20:43:04.752] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/7367ac98ed6c59f1b66e0ebe3fa459db0603fae7"
[20:43:04.752] Popped gitiles commit info from properties and tags
[20:43:04.753] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[20:43:04.753] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[20:43:04.753] Buildbucket request:
{
"requestId": "8842695974371724880",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7367ac98ed6c59f1b66e0ebe3fa459db0603fae7",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842695974371724880"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[20:43:05.266] Scheduled build:
{
"id": "8677265912470163505",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21492,
"createdBy": "project:chromium",
"createTime": "2026-07-03T20:43:04.857753281Z",
"updateTime": "2026-07-03T20:43:04.857753281Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7367ac98ed6c59f1b66e0ebe3fa459db0603fae7",
"ref": "refs/heads/main"
}
},
"canary": true
}
[20:43:05.266] Task URL: https://cr-buildbucket.appspot.com/build/8677265912470163505
[20:43:05.266] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:2:0) after 8m53s
[20:43:05.466] Received PubSub notification, asking Buildbucket for the build status
[20:43:05.516] Build status: SCHEDULED
[20:44:26.144] Received PubSub notification, asking Buildbucket for the build status
[20:44:26.169] Build status: STARTED
[20:51:58.295] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:2:0)
[20:51:58.321] Build status: STARTED
[20:51:58.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:5:0) after 4m12s
[20:56:10.400] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:5:0)
[20:56:10.419] Build status: STARTED
[20:56:10.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:6:0) after 2m55s
[20:59:05.433] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:6:0)
[20:59:05.484] Build status: STARTED
[20:59:05.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:7:0) after 9m5s
[21:08:10.563] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:7:0)
[21:08:10.588] Build status: STARTED
[21:08:10.588] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:8:0) after 5m51s
[21:14:01.567] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:8:0)
[21:14:01.591] Build status: STARTED
[21:14:01.592] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:9:0) after 9m36s
[21:23:38.140] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:9:0)
[21:23:38.269] Build status: STARTED
[21:23:38.269] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:10:0) after 2m54s
[21:26:32.319] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:10:0)
[21:26:32.337] Build status: STARTED
[21:26:32.337] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:11:0) after 5m4s
[21:31:36.425] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:11:0)
[21:31:36.446] Build status: STARTED
[21:31:36.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:12:0) after 4m44s
[21:36:20.468] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:12:0)
[21:36:20.486] Build status: STARTED
[21:36:20.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:13:0) after 8m46s
[21:45:06.644] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:13:0)
[21:45:06.709] Build status: STARTED
[21:45:06.710] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:14:0) after 1m37s
[21:46:43.807] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:14:0)
[21:46:43.824] Build status: STARTED
[21:46:43.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:15:0) after 1m6s
[21:47:49.861] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:15:0)
[21:47:49.883] Build status: STARTED
[21:47:49.883] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:16:0) after 4m8s
[21:51:57.900] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:16:0)
[21:51:57.919] Build status: STARTED
[21:51:57.919] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:17:0) after 8m10s
[22:00:07.939] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:17:0)
[22:00:07.965] Build status: STARTED
[22:00:07.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:18:0) after 5m58s
[22:06:05.981] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:18:0)
[22:06:06.037] Build status: STARTED
[22:06:06.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:19:0) after 8m15s
[22:14:21.090] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:19:0)
[22:14:21.115] Build status: STARTED
[22:14:21.115] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:20:0) after 1m12s
[22:15:33.185] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:20:0)
[22:15:33.246] Build status: STARTED
[22:15:33.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:21:0) after 6m6s
[22:21:39.283] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:21:0)
[22:21:39.299] Build status: STARTED
[22:21:39.299] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:22:0) after 4m5s
[22:25:44.518] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:22:0)
[22:25:44.535] Build status: STARTED
[22:25:44.535] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:23:0) after 6m3s
[22:31:47.941] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:23:0)
[22:31:47.965] Build status: STARTED
[22:31:47.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:24:0) after 6m50s
[22:38:38.420] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:24:0)
[22:38:38.490] Build status: STARTED
[22:38:38.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:25:0) after 6m42s
[22:45:20.762] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:25:0)
[22:45:20.787] Build status: STARTED
[22:45:20.787] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:26:0) after 7m9s
[22:52:29.808] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:26:0)
[22:52:29.849] Build status: STARTED
[22:52:29.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:27:0) after 8m25s
[23:00:54.939] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:27:0)
[23:00:54.997] Build status: STARTED
[23:00:54.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:28:0) after 3m58s
[23:04:53.232] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:28:0)
[23:04:53.266] Build status: STARTED
[23:04:53.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:29:0) after 4m16s
[23:09:09.382] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:29:0)
[23:09:09.407] Build status: STARTED
[23:09:09.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:30:0) after 7m26s
[23:16:35.466] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:30:0)
[23:16:35.510] Build status: STARTED
[23:16:35.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:31:0) after 1m11s
[23:17:46.592] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:31:0)
[23:17:46.609] Build status: STARTED
[23:17:46.609] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:32:0) after 4m9s
[23:21:55.629] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:32:0)
[23:21:55.854] Build status: STARTED
[23:21:55.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:33:0) after 2m17s
[23:24:12.957] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:33:0)
[23:24:12.976] Build status: STARTED
[23:24:12.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:34:0) after 9m35s
[23:33:48.414] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:34:0)
[23:33:48.435] Build status: STARTED
[23:33:48.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:35:0) after 2m45s
[23:36:33.451] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:35:0)
[23:36:33.470] Build status: STARTED
[23:36:33.470] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:36:0) after 9m25s
[23:45:58.627] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:36:0)
[23:45:58.653] Build status: STARTED
[23:45:58.653] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:37:0) after 4m17s
[23:50:15.954] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:37:0)
[23:50:15.997] Build status: STARTED
[23:50:15.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:38:0) after 1m57s
[23:52:13.004] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:38:0)
[23:52:13.023] Build status: STARTED
[23:52:13.023] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:39:0) after 9m4s
[00:01:17.040] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:39:0)
[00:01:17.059] Build status: STARTED
[00:01:17.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:40:0) after 4m52s
[00:06:09.082] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:40:0)
[00:06:09.139] Build status: STARTED
[00:06:09.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:41:0) after 1m54s
[00:08:03.158] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:41:0)
[00:08:03.203] Build status: STARTED
[00:08:03.203] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:42:0) after 4m16s
[00:12:19.220] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:42:0)
[00:12:19.241] Build status: STARTED
[00:12:19.241] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:43:0) after 6m20s
[00:18:39.258] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:43:0)
[00:18:39.273] Build status: STARTED
[00:18:39.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:44:0) after 2m6s
[00:20:45.818] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:44:0)
[00:20:45.838] Build status: STARTED
[00:20:45.838] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:45:0) after 1m27s
[00:22:13.273] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:45:0)
[00:22:13.343] Build status: STARTED
[00:22:13.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:46:0) after 5m5s
[00:27:18.361] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:46:0)
[00:27:18.378] Build status: STARTED
[00:27:18.378] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:47:0) after 1m52s
[00:29:10.394] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:47:0)
[00:29:10.409] Build status: STARTED
[00:29:10.410] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:48:0) after 1m2s
[00:30:12.506] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:48:0)
[00:30:12.521] Build status: STARTED
[00:30:12.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:49:0) after 1m5s
[00:31:17.854] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:49:0)
[00:31:17.914] Build status: STARTED
[00:31:17.914] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:50:0) after 4m45s
[00:36:02.950] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:50:0)
[00:36:02.970] Build status: STARTED
[00:36:02.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:51:0) after 6m4s
[00:42:07.426] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:51:0)
[00:42:07.458] Build status: STARTED
[00:42:07.458] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:52:0) after 5m32s
[00:47:39.582] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:52:0)
[00:47:39.640] Build status: STARTED
[00:47:39.640] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:53:0) after 9m2s
[00:56:42.375] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:53:0)
[00:56:42.448] Build status: STARTED
[00:56:42.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:54:0) after 6m55s
[01:03:37.707] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:54:0)
[01:03:37.722] Build status: STARTED
[01:03:37.722] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:55:0) after 5m10s
[01:08:47.946] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:55:0)
[01:08:47.963] Build status: STARTED
[01:08:47.963] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:56:0) after 8m32s
[01:17:20.101] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:56:0)
[01:17:20.182] Build status: STARTED
[01:17:20.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:57:0) after 8m31s
[01:25:51.551] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:57:0)
[01:25:51.608] Build status: STARTED
[01:25:51.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:58:0) after 7m49s
[01:33:40.733] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:58:0)
[01:33:40.757] Build status: STARTED
[01:33:40.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:59:0) after 1m20s
[01:35:00.855] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:59:0)
[01:35:00.880] Build status: STARTED
[01:35:00.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:60:0) after 7m16s
[01:42:16.922] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:60:0)
[01:42:16.941] Build status: STARTED
[01:42:16.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8842695974371724880:61:0) after 6m59s
[01:43:40.460] Received PubSub notification, asking Buildbucket for the build status
[01:43:40.506] Build:
{
"id": "8677265912470163505",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 21492,
"createdBy": "project:chromium",
"createTime": "2026-07-03T20:43:04.857753281Z",
"startTime": "2026-07-03T20:44:15.804262829Z",
"endTime": "2026-07-04T01:43:30.506201643Z",
"updateTime": "2026-07-04T01:43:30.506201643Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7367ac98ed6c59f1b66e0ebe3fa459db0603fae7",
"ref": "refs/heads/main"
}
},
"canary": true
}
[01:43:40.506] Invocation finished in 5h0m36.846506335s with status FAILED