[03:43:25.456] New invocation is queued and will start shortly
[03:43:26.944] Starting the invocation (attempt 1)
[03:43:26.974] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d32b1d3c10ebb1f94d6cdb03a6215676c7fd238a"
[03:43:26.974] Popped gitiles commit info from properties and tags
[03:43:26.974] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[03:43:26.975] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[03:43:26.975] Buildbucket request:
{
"requestId": "8867130708340973600",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d32b1d3c10ebb1f94d6cdb03a6215676c7fd238a",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867130708340973600"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[03:43:28.328] Scheduled build:
{
"id": "8701700645657988737",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 18116,
"createdBy": "project:chromium",
"createTime": "2025-10-07T03:43:27.388942038Z",
"updateTime": "2025-10-07T03:43:27.388942038Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d32b1d3c10ebb1f94d6cdb03a6215676c7fd238a",
"ref": "refs/heads/main"
}
}
}
[03:43:28.339] Task URL: https://cr-buildbucket.appspot.com/build/8701700645657988737
[03:43:28.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:2:0) after 8m33s
[03:43:42.810] Received PubSub notification, asking Buildbucket for the build status
[03:43:42.836] Build status: SCHEDULED
[03:44:07.650] Received PubSub notification, asking Buildbucket for the build status
[03:44:07.667] Build status: STARTED
[03:52:01.355] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:2:0)
[03:52:01.384] Build status: STARTED
[03:52:01.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:5:0) after 9m3s
[04:01:04.405] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:5:0)
[04:01:04.420] Build status: STARTED
[04:01:04.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:6:0) after 5m28s
[04:06:32.441] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:6:0)
[04:06:32.470] Build status: STARTED
[04:06:32.470] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:7:0) after 2m48s
[04:09:20.516] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:7:0)
[04:09:20.541] Build status: STARTED
[04:09:20.541] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:8:0) after 2m26s
[04:11:46.557] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:8:0)
[04:11:46.571] Build status: STARTED
[04:11:46.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:9:0) after 3m46s
[04:15:32.587] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:9:0)
[04:15:32.617] Build status: STARTED
[04:15:32.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:10:0) after 1m22s
[04:16:54.633] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:10:0)
[04:16:54.665] Build status: STARTED
[04:16:54.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:11:0) after 8m55s
[04:25:49.670] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:11:0)
[04:25:49.686] Build status: STARTED
[04:25:49.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:12:0) after 7m24s
[04:33:13.970] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:12:0)
[04:33:13.985] Build status: STARTED
[04:33:13.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:13:0) after 5m41s
[04:38:55.001] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:13:0)
[04:38:55.015] Build status: STARTED
[04:38:55.015] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:14:0) after 9m35s
[04:48:30.634] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:14:0)
[04:48:30.649] Build status: STARTED
[04:48:30.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:15:0) after 4m24s
[04:52:54.681] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:15:0)
[04:52:54.695] Build status: STARTED
[04:52:54.695] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:16:0) after 2m12s
[04:55:06.711] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:16:0)
[04:55:06.729] Build status: STARTED
[04:55:06.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:17:0) after 9m22s
[05:04:28.752] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:17:0)
[05:04:28.776] Build status: STARTED
[05:04:28.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:18:0) after 5m42s
[05:10:10.844] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:18:0)
[05:10:10.894] Build status: STARTED
[05:10:10.894] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:19:0) after 7m19s
[05:17:29.884] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:19:0)
[05:17:29.953] Build status: STARTED
[05:17:29.953] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:20:0) after 1m6s
[05:18:35.968] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:20:0)
[05:18:35.981] Build status: STARTED
[05:18:35.981] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:21:0) after 8m54s
[05:27:30.148] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:21:0)
[05:27:30.164] Build status: STARTED
[05:27:30.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:22:0) after 6m30s
[05:34:00.191] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:22:0)
[05:34:00.221] Build status: STARTED
[05:34:00.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:23:0) after 8m24s
[05:42:24.321] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:23:0)
[05:42:24.336] Build status: STARTED
[05:42:24.336] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:24:0) after 4m14s
[05:46:38.371] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:24:0)
[05:46:38.394] Build status: STARTED
[05:46:38.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:25:0) after 3m39s
[05:50:17.515] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:25:0)
[05:50:17.535] Build status: STARTED
[05:50:17.535] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:26:0) after 4m55s
[05:55:12.550] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:26:0)
[05:55:12.568] Build status: STARTED
[05:55:12.568] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:27:0) after 6m18s
[06:01:30.639] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:27:0)
[06:01:30.668] Build status: STARTED
[06:01:30.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:28:0) after 3m44s
[06:05:14.748] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:28:0)
[06:05:14.772] Build status: STARTED
[06:05:14.772] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:29:0) after 5m3s
[06:10:17.792] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:29:0)
[06:10:17.834] Build status: STARTED
[06:10:17.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:30:0) after 7m45s
[06:18:02.894] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:30:0)
[06:18:02.910] Build status: STARTED
[06:18:02.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:31:0) after 7m10s
[06:25:12.999] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:31:0)
[06:25:13.012] Build status: STARTED
[06:25:13.012] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:32:0) after 8m57s
[06:34:10.080] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:32:0)
[06:34:10.092] Build status: STARTED
[06:34:10.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:33:0) after 8m48s
[06:42:58.109] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:33:0)
[06:42:58.166] Build status: STARTED
[06:42:58.166] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:34:0) after 9m48s
[06:52:46.187] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:34:0)
[06:52:46.213] Build status: STARTED
[06:52:46.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:35:0) after 3m2s
[06:55:48.232] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:35:0)
[06:55:48.248] Build status: STARTED
[06:55:48.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:36:0) after 4m44s
[07:00:32.271] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:36:0)
[07:00:32.285] Build status: STARTED
[07:00:32.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:37:0) after 2m38s
[07:03:10.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:37:0)
[07:03:10.433] Build status: STARTED
[07:03:10.433] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:38:0) after 9m19s
[07:12:29.649] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:38:0)
[07:12:29.667] Build status: STARTED
[07:12:29.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:39:0) after 6m12s
[07:18:41.686] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:39:0)
[07:18:41.704] Build status: STARTED
[07:18:41.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:40:0) after 4m10s
[07:22:51.804] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:40:0)
[07:22:51.824] Build status: STARTED
[07:22:51.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:41:0) after 3m5s
[07:25:56.893] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:41:0)
[07:25:56.916] Build status: STARTED
[07:25:56.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:42:0) after 9m11s
[07:35:08.027] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:42:0)
[07:35:08.048] Build status: STARTED
[07:35:08.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:43:0) after 5m27s
[07:40:35.275] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:43:0)
[07:40:35.289] Build status: STARTED
[07:40:35.289] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:44:0) after 9m25s
[07:50:00.306] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:44:0)
[07:50:00.325] Build status: STARTED
[07:50:00.325] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:45:0) after 9m0s
[07:59:00.366] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:45:0)
[07:59:00.385] Build status: STARTED
[07:59:00.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:46:0) after 6m57s
[08:05:57.598] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:46:0)
[08:05:57.689] Build status: STARTED
[08:05:57.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:47:0) after 3m4s
[08:09:01.706] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:47:0)
[08:09:01.725] Build status: STARTED
[08:09:01.725] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:48:0) after 1m16s
[08:10:17.699] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:48:0)
[08:10:17.723] Build status: STARTED
[08:10:17.723] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:49:0) after 1m51s
[08:12:08.742] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:49:0)
[08:12:08.817] Build status: STARTED
[08:12:08.817] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:50:0) after 7m17s
[08:19:25.785] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:50:0)
[08:19:25.798] Build status: STARTED
[08:19:25.799] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:51:0) after 6m5s
[08:25:30.779] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:51:0)
[08:25:30.799] Build status: STARTED
[08:25:30.799] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:52:0) after 9m39s
[08:35:09.817] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:52:0)
[08:35:09.835] Build status: STARTED
[08:35:09.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:53:0) after 2m58s
[08:38:07.840] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:53:0)
[08:38:07.855] Build status: STARTED
[08:38:07.855] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:54:0) after 7m25s
[08:45:32.881] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:54:0)
[08:45:32.940] Build status: STARTED
[08:45:32.940] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:55:0) after 6m50s
[08:52:22.953] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:55:0)
[08:52:22.966] Build status: STARTED
[08:52:22.966] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:56:0) after 1m44s
[08:54:06.990] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:56:0)
[08:54:07.006] Build status: STARTED
[08:54:07.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:57:0) after 7m50s
[09:01:57.029] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:57:0)
[09:01:57.066] Build status: STARTED
[09:01:57.066] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867130708340973600:58:0) after 9m12s
[09:04:35.753] Received PubSub notification, asking Buildbucket for the build status
[09:04:35.814] Build:
{
"id": "8701700645657988737",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 18116,
"createdBy": "project:chromium",
"createTime": "2025-10-07T03:43:27.388942038Z",
"startTime": "2025-10-07T03:43:52.641251316Z",
"endTime": "2025-10-07T09:04:34.852113059Z",
"updateTime": "2025-10-07T09:04:34.852113059Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d32b1d3c10ebb1f94d6cdb03a6215676c7fd238a",
"ref": "refs/heads/main"
}
}
}
[09:04:35.814] Invocation finished in 5h21m10.36882488s with status FAILED