[01:17:44.942] New invocation is queued and will start shortly
[01:17:48.457] Starting the invocation (attempt 1)
[01:17:48.483] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d651b58dbc184f8c89454390ef3cfbaa8a3c87db"
[01:17:48.483] Popped gitiles commit info from properties and tags
[01:17:48.483] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:17:48.483] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:17:48.483] Buildbucket request:
{
"requestId": "8846846152959932528",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d651b58dbc184f8c89454390ef3cfbaa8a3c87db",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8846846152959932528"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:17:49.216] Scheduled build:
{
"id": "8681416088477461297",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2691,
"createdBy": "project:chromium",
"createTime": "2026-05-19T01:17:48.591499111Z",
"updateTime": "2026-05-19T01:17:48.591499111Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d651b58dbc184f8c89454390ef3cfbaa8a3c87db",
"ref": "refs/heads/main"
}
}
}
[01:17:49.216] Task URL: https://cr-buildbucket.appspot.com/build/8681416088477461297
[01:17:49.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:2:0) after 1m50s
[01:17:50.242] Received PubSub notification, asking Buildbucket for the build status
[01:17:50.305] Build status: SCHEDULED
[01:17:52.817] Received PubSub notification, asking Buildbucket for the build status
[01:17:52.832] Build status: STARTED
[01:19:39.233] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:2:0)
[01:19:39.290] Build status: STARTED
[01:19:39.290] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:5:0) after 9m12s
[01:28:51.313] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:5:0)
[01:28:51.336] Build status: STARTED
[01:28:51.336] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:6:0) after 1m31s
[01:30:22.332] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:6:0)
[01:30:22.371] Build status: STARTED
[01:30:22.371] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:7:0) after 7m46s
[01:38:08.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:7:0)
[01:38:08.565] Build status: STARTED
[01:38:08.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:8:0) after 5m58s
[01:44:06.558] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:8:0)
[01:44:06.640] Build status: STARTED
[01:44:06.640] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:9:0) after 1m51s
[01:45:57.665] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:9:0)
[01:45:57.680] Build status: STARTED
[01:45:57.680] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:10:0) after 3m26s
[01:49:23.694] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:10:0)
[01:49:23.711] Build status: STARTED
[01:49:23.711] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:11:0) after 7m55s
[01:57:18.874] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:11:0)
[01:57:18.892] Build status: STARTED
[01:57:18.892] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:12:0) after 7m36s
[02:04:54.957] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:12:0)
[02:04:54.973] Build status: STARTED
[02:04:54.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:13:0) after 9m26s
[02:14:20.987] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:13:0)
[02:14:21.006] Build status: STARTED
[02:14:21.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:14:0) after 5m37s
[02:19:58.021] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:14:0)
[02:19:58.043] Build status: STARTED
[02:19:58.043] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:15:0) after 5m38s
[02:25:36.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:15:0)
[02:25:36.159] Build status: STARTED
[02:25:36.159] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:16:0) after 1m56s
[02:27:32.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:16:0)
[02:27:32.248] Build status: STARTED
[02:27:32.248] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:17:0) after 2m55s
[02:30:27.282] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:17:0)
[02:30:27.303] Build status: STARTED
[02:30:27.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:18:0) after 6m43s
[02:37:10.323] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:18:0)
[02:37:10.343] Build status: STARTED
[02:37:10.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:19:0) after 7m15s
[02:44:25.371] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:19:0)
[02:44:25.401] Build status: STARTED
[02:44:25.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:20:0) after 7m27s
[02:51:52.437] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:20:0)
[02:51:52.454] Build status: STARTED
[02:51:52.454] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:21:0) after 6m42s
[02:58:34.472] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:21:0)
[02:58:34.494] Build status: STARTED
[02:58:34.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:22:0) after 4m50s
[03:03:24.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:22:0)
[03:03:24.543] Build status: STARTED
[03:03:24.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:23:0) after 5m8s
[03:08:32.599] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:23:0)
[03:08:32.617] Build status: STARTED
[03:08:32.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:24:0) after 2m20s
[03:10:52.633] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:24:0)
[03:10:52.655] Build status: STARTED
[03:10:52.655] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:25:0) after 8m23s
[03:19:15.669] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:25:0)
[03:19:15.715] Build status: STARTED
[03:19:15.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:26:0) after 2m57s
[03:22:12.796] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:26:0)
[03:22:12.824] Build status: STARTED
[03:22:12.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:27:0) after 9m13s
[03:31:25.839] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:27:0)
[03:31:25.861] Build status: STARTED
[03:31:25.861] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:28:0) after 3m16s
[03:34:41.888] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:28:0)
[03:34:41.911] Build status: STARTED
[03:34:41.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:29:0) after 2m55s
[03:37:36.933] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:29:0)
[03:37:36.963] Build status: STARTED
[03:37:36.963] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:30:0) after 6m50s
[03:44:27.034] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:30:0)
[03:44:27.061] Build status: STARTED
[03:44:27.061] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:31:0) after 1m7s
[03:45:34.078] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:31:0)
[03:45:34.097] Build status: STARTED
[03:45:34.097] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:32:0) after 3m20s
[03:48:54.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:32:0)
[03:48:54.136] Build status: STARTED
[03:48:54.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:33:0) after 2m8s
[03:51:02.202] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:33:0)
[03:51:02.218] Build status: STARTED
[03:51:02.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:34:0) after 8m6s
[03:59:08.314] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:34:0)
[03:59:08.332] Build status: STARTED
[03:59:08.332] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:35:0) after 6m16s
[04:05:24.449] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:35:0)
[04:05:24.474] Build status: STARTED
[04:05:24.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:36:0) after 9m51s
[04:15:15.585] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:36:0)
[04:15:15.608] Build status: STARTED
[04:15:15.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:37:0) after 1m49s
[04:17:04.623] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:37:0)
[04:17:04.638] Build status: STARTED
[04:17:04.638] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:38:0) after 9m44s
[04:26:48.658] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:38:0)
[04:26:48.673] Build status: STARTED
[04:26:48.673] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:39:0) after 5m10s
[04:31:58.726] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:39:0)
[04:31:58.745] Build status: STARTED
[04:31:58.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:40:0) after 9m32s
[04:41:30.765] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:40:0)
[04:41:30.785] Build status: STARTED
[04:41:30.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:41:0) after 3m59s
[04:45:29.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:41:0)
[04:45:29.872] Build status: STARTED
[04:45:29.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:42:0) after 4m18s
[04:49:47.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:42:0)
[04:49:47.914] Build status: STARTED
[04:49:47.914] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846846152959932528:43:0) after 9m50s
[04:57:50.686] Received PubSub notification, asking Buildbucket for the build status
[04:57:50.704] Build:
{
"id": "8681416088477461297",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2691,
"createdBy": "project:chromium",
"createTime": "2026-05-19T01:17:48.591499111Z",
"startTime": "2026-05-19T01:17:52.623181626Z",
"endTime": "2026-05-19T04:57:50.371664425Z",
"updateTime": "2026-05-19T04:57:50.371664425Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d651b58dbc184f8c89454390ef3cfbaa8a3c87db",
"ref": "refs/heads/main"
}
}
}
[04:57:50.704] Invocation finished in 3h40m5.772277182s with status FAILED