[12:11:01.099] New invocation is queued and will start shortly
[12:11:05.548] Starting the invocation (attempt 1)
[12:11:05.587] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/c95cef757aed2b7e6e8bd0bce3c4a6028be29520"
[12:11:05.588] Popped gitiles commit info from properties and tags
[12:11:05.588] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[12:11:05.588] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[12:11:05.588] Buildbucket request:
{
"requestId": "8842909383250836208",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c95cef757aed2b7e6e8bd0bce3c4a6028be29520",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842909383250836208"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[12:11:06.139] Scheduled build:
{
"id": "8677479317814243233",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3566,
"createdBy": "project:chromium",
"createTime": "2026-07-01T12:11:05.662159999Z",
"updateTime": "2026-07-01T12:11:05.662159999Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c95cef757aed2b7e6e8bd0bce3c4a6028be29520",
"ref": "refs/heads/main"
}
}
}
[12:11:06.139] Task URL: https://cr-buildbucket.appspot.com/build/8677479317814243233
[12:11:06.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:2:0) after 1m27s
[12:11:07.122] Received PubSub notification, asking Buildbucket for the build status
[12:11:07.143] Build status: SCHEDULED
[12:11:48.604] Received PubSub notification, asking Buildbucket for the build status
[12:11:48.619] Build status: STARTED
[12:12:33.162] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:2:0)
[12:12:33.191] Build status: STARTED
[12:12:33.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:5:0) after 6m42s
[12:19:15.179] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:5:0)
[12:19:15.204] Build status: STARTED
[12:19:15.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:6:0) after 7m30s
[12:26:45.273] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:6:0)
[12:26:45.298] Build status: STARTED
[12:26:45.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:7:0) after 6m2s
[12:32:47.277] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:7:0)
[12:32:47.295] Build status: STARTED
[12:32:47.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:8:0) after 6m39s
[12:39:26.355] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:8:0)
[12:39:26.375] Build status: STARTED
[12:39:26.375] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:9:0) after 5m37s
[12:45:03.393] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:9:0)
[12:45:03.423] Build status: STARTED
[12:45:03.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:10:0) after 3m26s
[12:48:29.450] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:10:0)
[12:48:29.478] Build status: STARTED
[12:48:29.478] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:11:0) after 3m3s
[12:51:32.504] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:11:0)
[12:51:32.548] Build status: STARTED
[12:51:32.548] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:12:0) after 1m49s
[12:53:21.541] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:12:0)
[12:53:21.577] Build status: STARTED
[12:53:21.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:13:0) after 6m19s
[12:59:40.648] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:13:0)
[12:59:40.708] Build status: STARTED
[12:59:40.708] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:14:0) after 2m21s
[13:02:01.754] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:14:0)
[13:02:01.786] Build status: STARTED
[13:02:01.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:15:0) after 5m31s
[13:07:32.859] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:15:0)
[13:07:32.908] Build status: STARTED
[13:07:32.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:16:0) after 8m23s
[13:15:55.949] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:16:0)
[13:15:55.967] Build status: STARTED
[13:15:55.967] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:17:0) after 5m56s
[13:21:52.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:17:0)
[13:21:52.060] Build status: STARTED
[13:21:52.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:18:0) after 4m41s
[13:26:33.093] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:18:0)
[13:26:33.125] Build status: STARTED
[13:26:33.125] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:19:0) after 1m35s
[13:28:08.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:19:0)
[13:28:08.181] Build status: STARTED
[13:28:08.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:20:0) after 4m11s
[13:32:19.200] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:20:0)
[13:32:19.216] Build status: STARTED
[13:32:19.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:21:0) after 4m10s
[13:36:29.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:21:0)
[13:36:29.356] Build status: STARTED
[13:36:29.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:22:0) after 6m53s
[13:43:22.426] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:22:0)
[13:43:22.492] Build status: STARTED
[13:43:22.492] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:23:0) after 1m15s
[13:44:37.512] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:23:0)
[13:44:37.540] Build status: STARTED
[13:44:37.540] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:24:0) after 2m35s
[13:47:12.575] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:24:0)
[13:47:12.603] Build status: STARTED
[13:47:12.603] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:25:0) after 1m36s
[13:48:48.703] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:25:0)
[13:48:48.726] Build status: STARTED
[13:48:48.727] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:26:0) after 3m6s
[13:51:54.748] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:26:0)
[13:51:54.842] Build status: STARTED
[13:51:54.842] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:27:0) after 1m24s
[13:53:18.822] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:27:0)
[13:53:18.847] Build status: STARTED
[13:53:18.847] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:28:0) after 2m41s
[13:55:59.868] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:28:0)
[13:55:59.888] Build status: STARTED
[13:55:59.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:29:0) after 7m11s
[14:03:10.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:29:0)
[14:03:10.949] Build status: STARTED
[14:03:10.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:30:0) after 4m20s
[14:07:31.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:30:0)
[14:07:31.038] Build status: STARTED
[14:07:31.038] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:31:0) after 9m47s
[14:17:18.058] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:31:0)
[14:17:18.119] Build status: STARTED
[14:17:18.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:32:0) after 8m20s
[14:25:38.111] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:32:0)
[14:25:38.135] Build status: STARTED
[14:25:38.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:33:0) after 8m1s
[14:33:39.156] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:33:0)
[14:33:39.182] Build status: STARTED
[14:33:39.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:34:0) after 2m59s
[14:36:38.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:34:0)
[14:36:38.424] Build status: STARTED
[14:36:38.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:35:0) after 4m28s
[14:41:06.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:35:0)
[14:41:06.522] Build status: STARTED
[14:41:06.523] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:36:0) after 3m19s
[14:44:25.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:36:0)
[14:44:25.647] Build status: STARTED
[14:44:25.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:37:0) after 6m43s
[14:51:08.790] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:37:0)
[14:51:08.807] Build status: STARTED
[14:51:08.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:38:0) after 8m15s
[14:59:23.826] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:38:0)
[14:59:23.855] Build status: STARTED
[14:59:23.855] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:39:0) after 9m24s
[15:08:47.885] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:39:0)
[15:08:47.972] Build status: STARTED
[15:08:47.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:40:0) after 1m46s
[15:10:34.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:40:0)
[15:10:34.117] Build status: STARTED
[15:10:34.117] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:41:0) after 5m22s
[15:15:56.167] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:41:0)
[15:15:56.188] Build status: STARTED
[15:15:56.188] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:42:0) after 5m21s
[15:21:17.344] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:42:0)
[15:21:17.376] Build status: STARTED
[15:21:17.376] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:43:0) after 1m0s
[15:22:17.415] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:43:0)
[15:22:17.431] Build status: STARTED
[15:22:17.431] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:44:0) after 4m43s
[15:27:00.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:44:0)
[15:27:00.469] Build status: STARTED
[15:27:00.469] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:45:0) after 5m17s
[15:32:17.544] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:45:0)
[15:32:17.570] Build status: STARTED
[15:32:17.570] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:46:0) after 3m37s
[15:35:54.685] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:46:0)
[15:35:54.723] Build status: STARTED
[15:35:54.723] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:47:0) after 4m55s
[15:40:49.747] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:47:0)
[15:40:49.998] Build status: STARTED
[15:40:49.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:48:0) after 2m36s
[15:43:26.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:48:0)
[15:43:26.071] Build status: STARTED
[15:43:26.071] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:49:0) after 5m9s
[15:48:35.119] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:49:0)
[15:48:35.138] Build status: STARTED
[15:48:35.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:50:0) after 1m56s
[15:50:31.190] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:50:0)
[15:50:31.208] Build status: STARTED
[15:50:31.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:51:0) after 2m26s
[15:52:57.284] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:51:0)
[15:52:57.303] Build status: STARTED
[15:52:57.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:52:0) after 8m27s
[16:01:24.331] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:52:0)
[16:01:24.354] Build status: STARTED
[16:01:24.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:53:0) after 2m45s
[16:04:09.421] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:53:0)
[16:04:09.444] Build status: STARTED
[16:04:09.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:54:0) after 6m54s
[16:11:03.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:54:0)
[16:11:03.671] Build status: STARTED
[16:11:03.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:55:0) after 1m6s
[16:12:09.827] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:55:0)
[16:12:09.849] Build status: STARTED
[16:12:09.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:56:0) after 2m44s
[16:14:53.867] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:56:0)
[16:14:53.888] Build status: STARTED
[16:14:53.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:57:0) after 1m36s
[16:16:29.909] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:57:0)
[16:16:29.932] Build status: STARTED
[16:16:29.932] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:58:0) after 8m20s
[16:24:49.950] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:58:0)
[16:24:49.979] Build status: STARTED
[16:24:49.979] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:59:0) after 2m8s
[16:26:58.029] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:59:0)
[16:26:58.047] Build status: STARTED
[16:26:58.047] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:60:0) after 4m43s
[16:31:41.068] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:60:0)
[16:31:41.086] Build status: STARTED
[16:31:41.086] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:61:0) after 5m16s
[16:36:57.220] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:61:0)
[16:36:57.246] Build status: STARTED
[16:36:57.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:62:0) after 8m58s
[16:45:55.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:62:0)
[16:45:55.294] Build status: STARTED
[16:45:55.322] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:63:0) after 4m17s
[16:50:12.394] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:63:0)
[16:50:12.417] Build status: STARTED
[16:50:12.417] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:64:0) after 8m33s
[16:58:45.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:64:0)
[16:58:45.540] Build status: STARTED
[16:58:45.540] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:65:0) after 9m54s
[17:08:39.513] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:65:0)
[17:08:39.629] Build status: STARTED
[17:08:39.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842909383250836208:66:0) after 8m51s
[17:09:20.845] Received PubSub notification, asking Buildbucket for the build status
[17:09:20.866] Build:
{
"id": "8677479317814243233",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3566,
"createdBy": "project:chromium",
"createTime": "2026-07-01T12:11:05.662159999Z",
"startTime": "2026-07-01T12:11:48.400359981Z",
"endTime": "2026-07-01T17:09:20.567972311Z",
"updateTime": "2026-07-01T17:09:20.567972311Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c95cef757aed2b7e6e8bd0bce3c4a6028be29520",
"ref": "refs/heads/main"
}
}
}
[17:09:20.867] Invocation finished in 4h58m19.77403714s with status FAILED