[19:11:58.713] New invocation is queued and will start shortly
[19:11:59.777] Starting the invocation (attempt 1)
[19:11:59.802] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e908606526f3af6658f010d82c6703afb6a88521"
[19:11:59.802] Popped gitiles commit info from properties and tags
[19:11:59.802] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:11:59.802] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:11:59.803] Buildbucket request:
{
"requestId": "8842248719958765008",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e908606526f3af6658f010d82c6703afb6a88521",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842248719958765008"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:12:00.349] Scheduled build:
{
"id": "8676818658082530145",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3680,
"createdBy": "project:chromium",
"createTime": "2026-07-08T19:11:59.880741563Z",
"updateTime": "2026-07-08T19:11:59.880741563Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e908606526f3af6658f010d82c6703afb6a88521",
"ref": "refs/heads/main"
}
}
}
[19:12:00.349] Task URL: https://cr-buildbucket.appspot.com/build/8676818658082530145
[19:12:00.349] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:2:0) after 1m6s
[19:12:01.367] Received PubSub notification, asking Buildbucket for the build status
[19:12:01.388] Build status: SCHEDULED
[19:12:03.726] Received PubSub notification, asking Buildbucket for the build status
[19:12:03.751] Build status: STARTED
[19:13:06.436] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:2:0)
[19:13:06.460] Build status: STARTED
[19:13:06.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:5:0) after 7m38s
[19:20:44.532] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:5:0)
[19:20:44.563] Build status: STARTED
[19:20:44.564] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:6:0) after 4m21s
[19:25:05.584] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:6:0)
[19:25:05.601] Build status: STARTED
[19:25:05.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:7:0) after 9m23s
[19:34:28.659] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:7:0)
[19:34:28.689] Build status: STARTED
[19:34:28.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:8:0) after 2m23s
[19:36:51.716] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:8:0)
[19:36:51.783] Build status: STARTED
[19:36:51.783] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:9:0) after 9m47s
[19:46:38.813] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:9:0)
[19:46:38.831] Build status: STARTED
[19:46:38.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:10:0) after 4m25s
[19:51:03.990] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:10:0)
[19:51:04.115] Build status: STARTED
[19:51:04.115] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:11:0) after 2m23s
[19:53:27.287] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:11:0)
[19:53:27.305] Build status: STARTED
[19:53:27.305] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:12:0) after 4m30s
[19:57:57.302] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:12:0)
[19:57:57.334] Build status: STARTED
[19:57:57.334] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:13:0) after 1m15s
[19:59:12.351] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:13:0)
[19:59:12.406] Build status: STARTED
[19:59:12.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:14:0) after 1m8s
[20:00:20.657] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:14:0)
[20:00:20.677] Build status: STARTED
[20:00:20.677] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:15:0) after 7m22s
[20:07:42.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:15:0)
[20:07:42.773] Build status: STARTED
[20:07:42.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:16:0) after 8m19s
[20:16:01.911] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:16:0)
[20:16:01.932] Build status: STARTED
[20:16:01.932] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:17:0) after 7m4s
[20:23:05.972] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:17:0)
[20:23:05.991] Build status: STARTED
[20:23:05.992] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:18:0) after 1m43s
[20:24:49.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:18:0)
[20:24:49.197] Build status: STARTED
[20:24:49.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:19:0) after 7m44s
[20:32:33.442] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:19:0)
[20:32:33.470] Build status: STARTED
[20:32:33.470] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:20:0) after 5m22s
[20:37:55.532] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:20:0)
[20:37:55.549] Build status: STARTED
[20:37:55.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:21:0) after 2m9s
[20:40:04.649] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:21:0)
[20:40:04.731] Build status: STARTED
[20:40:04.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:22:0) after 9m39s
[20:49:43.885] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:22:0)
[20:49:43.909] Build status: STARTED
[20:49:43.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:23:0) after 3m43s
[20:53:26.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:23:0)
[20:53:26.973] Build status: STARTED
[20:53:26.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:24:0) after 2m4s
[20:55:31.080] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:24:0)
[20:55:31.099] Build status: STARTED
[20:55:31.099] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:25:0) after 5m41s
[21:01:12.239] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:25:0)
[21:01:12.254] Build status: STARTED
[21:01:12.254] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:26:0) after 4m32s
[21:05:44.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:26:0)
[21:05:44.298] Build status: STARTED
[21:05:44.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:27:0) after 5m18s
[21:11:02.465] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:27:0)
[21:11:02.519] Build status: STARTED
[21:11:02.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:28:0) after 3m30s
[21:14:32.591] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:28:0)
[21:14:32.610] Build status: STARTED
[21:14:32.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:29:0) after 9m47s
[21:24:19.718] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:29:0)
[21:24:19.738] Build status: STARTED
[21:24:19.738] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:30:0) after 9m51s
[21:34:10.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:30:0)
[21:34:10.813] Build status: STARTED
[21:34:10.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:31:0) after 1m38s
[21:35:48.832] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:31:0)
[21:35:48.867] Build status: STARTED
[21:35:48.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:32:0) after 1m55s
[21:37:43.939] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:32:0)
[21:37:43.964] Build status: STARTED
[21:37:43.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:33:0) after 3m47s
[21:41:30.992] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:33:0)
[21:41:31.063] Build status: STARTED
[21:41:31.063] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:34:0) after 3m45s
[21:45:16.218] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:34:0)
[21:45:16.298] Build status: STARTED
[21:45:16.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:35:0) after 2m40s
[21:47:56.345] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:35:0)
[21:47:56.370] Build status: STARTED
[21:47:56.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:36:0) after 2m31s
[21:50:27.632] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:36:0)
[21:50:27.676] Build status: STARTED
[21:50:27.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:37:0) after 1m20s
[21:51:47.757] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:37:0)
[21:51:47.775] Build status: STARTED
[21:51:47.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:38:0) after 2m29s
[21:54:16.817] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:38:0)
[21:54:16.849] Build status: STARTED
[21:54:16.849] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:39:0) after 3m23s
[21:57:39.842] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:39:0)
[21:57:39.949] Build status: STARTED
[21:57:39.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:40:0) after 1m3s
[21:58:42.935] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:40:0)
[21:58:42.954] Build status: STARTED
[21:58:42.954] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:41:0) after 6m1s
[22:04:44.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:41:0)
[22:04:44.043] Build status: STARTED
[22:04:44.043] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:42:0) after 3m58s
[22:08:42.024] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:42:0)
[22:08:42.042] Build status: STARTED
[22:08:42.042] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:43:0) after 7m16s
[22:15:58.064] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:43:0)
[22:15:58.095] Build status: STARTED
[22:15:58.095] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:44:0) after 7m13s
[22:23:11.170] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:44:0)
[22:23:11.192] Build status: STARTED
[22:23:11.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:45:0) after 8m8s
[22:31:19.256] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:45:0)
[22:31:19.339] Build status: STARTED
[22:31:19.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:46:0) after 1m8s
[22:32:27.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:46:0)
[22:32:27.447] Build status: STARTED
[22:32:27.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:47:0) after 6m33s
[22:39:00.465] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:47:0)
[22:39:00.635] Build status: STARTED
[22:39:00.635] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:48:0) after 1m19s
[22:40:19.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:48:0)
[22:40:19.725] Build status: STARTED
[22:40:19.725] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:49:0) after 8m24s
[22:48:43.745] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:49:0)
[22:48:43.766] Build status: STARTED
[22:48:43.766] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:50:0) after 8m41s
[22:57:24.809] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:50:0)
[22:57:24.827] Build status: STARTED
[22:57:24.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:51:0) after 6m4s
[23:03:28.924] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:51:0)
[23:03:28.947] Build status: STARTED
[23:03:28.947] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:52:0) after 4m41s
[23:08:10.034] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:52:0)
[23:08:10.050] Build status: STARTED
[23:08:10.050] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:53:0) after 6m4s
[23:14:14.163] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:53:0)
[23:14:14.236] Build status: STARTED
[23:14:14.236] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:54:0) after 4m27s
[23:18:41.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:54:0)
[23:18:41.274] Build status: STARTED
[23:18:41.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:55:0) after 2m31s
[23:21:12.463] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:55:0)
[23:21:12.511] Build status: STARTED
[23:21:12.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:56:0) after 3m18s
[23:24:30.599] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:56:0)
[23:24:30.760] Build status: STARTED
[23:24:30.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:57:0) after 3m17s
[23:27:47.833] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:57:0)
[23:27:47.851] Build status: STARTED
[23:27:47.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:58:0) after 5m50s
[23:33:38.025] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:58:0)
[23:33:38.044] Build status: STARTED
[23:33:38.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:59:0) after 5m38s
[23:39:16.176] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:59:0)
[23:39:16.192] Build status: STARTED
[23:39:16.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:60:0) after 7m19s
[23:46:35.361] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:60:0)
[23:46:35.384] Build status: STARTED
[23:46:35.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:61:0) after 3m23s
[23:49:58.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:61:0)
[23:49:58.533] Build status: STARTED
[23:49:58.533] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:62:0) after 7m53s
[23:57:51.567] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:62:0)
[23:57:51.671] Build status: STARTED
[23:57:51.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:63:0) after 5m36s
[00:03:27.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:63:0)
[00:03:27.786] Build status: STARTED
[00:03:27.786] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:64:0) after 5m26s
[00:08:53.965] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:64:0)
[00:08:54.026] Build status: STARTED
[00:08:54.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:65:0) after 1m43s
[00:10:37.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:65:0)
[00:10:37.112] Build status: STARTED
[00:10:37.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:66:0) after 2m30s
[00:13:07.192] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:66:0)
[00:13:07.238] Build status: STARTED
[00:13:07.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842248719958765008:67:0) after 5m53s
[00:14:55.499] Received PubSub notification, asking Buildbucket for the build status
[00:14:55.518] Build:
{
"id": "8676818658082530145",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3680,
"createdBy": "project:chromium",
"createTime": "2026-07-08T19:11:59.880741563Z",
"startTime": "2026-07-08T19:12:03.490678610Z",
"endTime": "2026-07-09T00:14:55.196774755Z",
"updateTime": "2026-07-09T00:14:55.196774755Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e908606526f3af6658f010d82c6703afb6a88521",
"ref": "refs/heads/main"
}
}
}
[00:14:55.518] Invocation finished in 5h2m56.811224206s with status FAILED