[05:33:31.337] New invocation is queued and will start shortly
[05:33:32.773] Starting the invocation (attempt 1)
[05:33:32.801] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/9316f80032419e211a317779d0fc2e8a1b676a35"
[05:33:32.801] Popped gitiles commit info from properties and tags
[05:33:32.801] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[05:33:32.801] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[05:33:32.801] Buildbucket request:
{
"requestId": "8841213049178075264",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9316f80032419e211a317779d0fc2e8a1b676a35",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841213049178075264"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[05:33:33.721] Scheduled build:
{
"id": "8675782986872088817",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3812,
"createdBy": "project:chromium",
"createTime": "2026-07-20T05:33:32.913012866Z",
"updateTime": "2026-07-20T05:33:32.913012866Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9316f80032419e211a317779d0fc2e8a1b676a35",
"ref": "refs/heads/main"
}
}
}
[05:33:33.721] Task URL: https://cr-buildbucket.appspot.com/build/8675782986872088817
[05:33:33.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:2:0) after 8m8s
[05:33:34.409] Received PubSub notification, asking Buildbucket for the build status
[05:33:34.423] Build status: SCHEDULED
[05:33:37.154] Received PubSub notification, asking Buildbucket for the build status
[05:33:37.185] Build status: STARTED
[05:41:41.768] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:2:0)
[05:41:41.790] Build status: STARTED
[05:41:41.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:5:0) after 8m20s
[05:50:01.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:5:0)
[05:50:01.821] Build status: STARTED
[05:50:01.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:6:0) after 6m15s
[05:56:16.856] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:6:0)
[05:56:16.877] Build status: STARTED
[05:56:16.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:7:0) after 4m57s
[06:01:13.894] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:7:0)
[06:01:13.920] Build status: STARTED
[06:01:13.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:8:0) after 5m44s
[06:06:57.939] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:8:0)
[06:06:57.962] Build status: STARTED
[06:06:57.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:9:0) after 8m27s
[06:15:25.026] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:9:0)
[06:15:25.046] Build status: STARTED
[06:15:25.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:10:0) after 5m20s
[06:20:45.022] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:10:0)
[06:20:45.038] Build status: STARTED
[06:20:45.038] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:11:0) after 8m49s
[06:29:34.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:11:0)
[06:29:34.124] Build status: STARTED
[06:29:34.124] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:12:0) after 5m51s
[06:35:25.151] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:12:0)
[06:35:25.168] Build status: STARTED
[06:35:25.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:13:0) after 5m38s
[06:41:03.184] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:13:0)
[06:41:03.201] Build status: STARTED
[06:41:03.201] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:14:0) after 2m36s
[06:43:39.232] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:14:0)
[06:43:39.250] Build status: STARTED
[06:43:39.250] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:15:0) after 4m49s
[06:48:28.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:15:0)
[06:48:28.404] Build status: STARTED
[06:48:28.404] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:16:0) after 1m58s
[06:50:26.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:16:0)
[06:50:26.480] Build status: STARTED
[06:50:26.480] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:17:0) after 4m50s
[06:55:17.080] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:17:0)
[06:55:17.098] Build status: STARTED
[06:55:17.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:18:0) after 5m54s
[07:01:11.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:18:0)
[07:01:11.717] Build status: STARTED
[07:01:11.717] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:19:0) after 9m45s
[07:10:56.987] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:19:0)
[07:10:57.019] Build status: STARTED
[07:10:57.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:20:0) after 6m47s
[07:17:44.048] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:20:0)
[07:17:44.067] Build status: STARTED
[07:17:44.067] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:21:0) after 7m56s
[07:25:40.081] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:21:0)
[07:25:40.098] Build status: STARTED
[07:25:40.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:22:0) after 1m0s
[07:26:40.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:22:0)
[07:26:40.262] Build status: STARTED
[07:26:40.262] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:23:0) after 1m27s
[07:28:07.380] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:23:0)
[07:28:07.458] Build status: STARTED
[07:28:07.458] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:24:0) after 2m35s
[07:30:42.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:24:0)
[07:30:42.545] Build status: STARTED
[07:30:42.545] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:25:0) after 3m39s
[07:34:21.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:25:0)
[07:34:21.631] Build status: STARTED
[07:34:21.631] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:26:0) after 8m51s
[07:43:12.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:26:0)
[07:43:12.665] Build status: STARTED
[07:43:12.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:27:0) after 1m27s
[07:44:39.694] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:27:0)
[07:44:39.720] Build status: STARTED
[07:44:39.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:28:0) after 2m42s
[07:47:21.772] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:28:0)
[07:47:21.792] Build status: STARTED
[07:47:21.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:29:0) after 4m56s
[07:52:17.914] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:29:0)
[07:52:17.941] Build status: STARTED
[07:52:17.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:30:0) after 1m32s
[07:53:49.955] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:30:0)
[07:53:50.025] Build status: STARTED
[07:53:50.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:31:0) after 6m28s
[08:00:18.084] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:31:0)
[08:00:18.113] Build status: STARTED
[08:00:18.113] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:32:0) after 1m58s
[08:02:16.191] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:32:0)
[08:02:16.211] Build status: STARTED
[08:02:16.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:33:0) after 9m23s
[08:11:39.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:33:0)
[08:11:39.335] Build status: STARTED
[08:11:39.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:34:0) after 5m58s
[08:17:37.356] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:34:0)
[08:17:37.374] Build status: STARTED
[08:17:37.374] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:35:0) after 2m30s
[08:20:07.405] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:35:0)
[08:20:07.434] Build status: STARTED
[08:20:07.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:36:0) after 5m43s
[08:25:50.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:36:0)
[08:25:50.475] Build status: STARTED
[08:25:50.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:37:0) after 6m31s
[08:32:21.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:37:0)
[08:32:21.517] Build status: STARTED
[08:32:21.517] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:38:0) after 5m56s
[08:38:17.858] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:38:0)
[08:38:17.874] Build status: STARTED
[08:38:17.874] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:39:0) after 4m51s
[08:43:09.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:39:0)
[08:43:09.103] Build status: STARTED
[08:43:09.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:40:0) after 1m11s
[08:44:20.684] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:40:0)
[08:44:20.707] Build status: STARTED
[08:44:20.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:41:0) after 3m16s
[08:47:36.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:41:0)
[08:47:36.787] Build status: STARTED
[08:47:36.788] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:42:0) after 7m1s
[08:54:37.803] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:42:0)
[08:54:37.833] Build status: STARTED
[08:54:37.833] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:43:0) after 4m51s
[08:59:28.851] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:43:0)
[08:59:28.866] Build status: STARTED
[08:59:28.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:44:0) after 4m10s
[09:03:38.863] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:44:0)
[09:03:38.882] Build status: STARTED
[09:03:38.882] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:45:0) after 4m56s
[09:08:34.935] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:45:0)
[09:08:34.952] Build status: STARTED
[09:08:34.952] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:46:0) after 3m34s
[09:12:08.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:46:0)
[09:12:09.012] Build status: STARTED
[09:12:09.012] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:47:0) after 1m12s
[09:13:21.010] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:47:0)
[09:13:21.028] Build status: STARTED
[09:13:21.028] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:48:0) after 6m32s
[09:19:53.048] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:48:0)
[09:19:53.105] Build status: STARTED
[09:19:53.105] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:49:0) after 1m11s
[09:21:04.143] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:49:0)
[09:21:04.184] Build status: STARTED
[09:21:04.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:50:0) after 3m55s
[09:24:59.155] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:50:0)
[09:24:59.177] Build status: STARTED
[09:24:59.177] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:51:0) after 6m40s
[09:31:39.196] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:51:0)
[09:31:39.215] Build status: STARTED
[09:31:39.215] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:52:0) after 5m34s
[09:37:13.230] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:52:0)
[09:37:13.302] Build status: STARTED
[09:37:13.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:53:0) after 9m44s
[09:46:57.431] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:53:0)
[09:46:57.446] Build status: STARTED
[09:46:57.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:54:0) after 6m21s
[09:53:18.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:54:0)
[09:53:18.497] Build status: STARTED
[09:53:18.497] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:55:0) after 4m13s
[09:57:31.517] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:55:0)
[09:57:31.534] Build status: STARTED
[09:57:31.534] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:56:0) after 3m21s
[10:00:52.598] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:56:0)
[10:00:52.689] Build status: STARTED
[10:00:52.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:57:0) after 3m30s
[10:04:22.705] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:57:0)
[10:04:22.729] Build status: STARTED
[10:04:22.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:58:0) after 3m20s
[10:07:42.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:58:0)
[10:07:42.784] Build status: STARTED
[10:07:42.784] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:59:0) after 6m44s
[10:14:26.849] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:59:0)
[10:14:26.867] Build status: STARTED
[10:14:26.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:60:0) after 6m26s
[10:20:52.988] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:60:0)
[10:20:53.006] Build status: STARTED
[10:20:53.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:61:0) after 6m41s
[10:27:34.021] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:61:0)
[10:27:34.036] Build status: STARTED
[10:27:34.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:62:0) after 3m13s
[10:30:47.136] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:62:0)
[10:30:47.158] Build status: STARTED
[10:30:47.158] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841213049178075264:63:0) after 3m50s
[10:33:14.949] Received PubSub notification, asking Buildbucket for the build status
[10:33:14.980] Build:
{
"id": "8675782986872088817",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3812,
"createdBy": "project:chromium",
"createTime": "2026-07-20T05:33:32.913012866Z",
"startTime": "2026-07-20T05:33:36.857978849Z",
"endTime": "2026-07-20T10:32:55.387633470Z",
"updateTime": "2026-07-20T10:32:55.387633470Z",
"status": "INFRA_FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9316f80032419e211a317779d0fc2e8a1b676a35",
"ref": "refs/heads/main"
}
}
}
[10:33:14.980] Invocation finished in 4h59m43.648829789s with status FAILED