[21:52:12.716] New invocation is queued and will start shortly
[21:52:13.773] Starting the invocation (attempt 1)
[21:52:13.804] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/947d6009fe2755da32e410112752ad319aa9eccf"
[21:52:13.804] Popped gitiles commit info from properties and tags
[21:52:13.804] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:52:13.804] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:52:13.804] Buildbucket request:
{
"requestId": "8857549526266624848",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "947d6009fe2755da32e410112752ad319aa9eccf",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857549526266624848"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:52:14.402] Scheduled build:
{
"id": "8692119464419641729",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1216,
"createdBy": "project:chromium",
"createTime": "2026-01-20T21:52:13.856854165Z",
"updateTime": "2026-01-20T21:52:13.856854165Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "947d6009fe2755da32e410112752ad319aa9eccf",
"ref": "refs/heads/main"
}
},
"canary": true
}
[21:52:14.402] Task URL: https://cr-buildbucket.appspot.com/build/8692119464419641729
[21:52:14.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:2:0) after 9m54s
[21:52:15.391] Received PubSub notification, asking Buildbucket for the build status
[21:52:15.444] Build status: SCHEDULED
[21:52:17.707] Received PubSub notification, asking Buildbucket for the build status
[21:52:17.768] Build status: STARTED
[22:02:08.446] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:2:0)
[22:02:08.582] Build status: STARTED
[22:02:08.583] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:5:0) after 2m35s
[22:04:43.698] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:5:0)
[22:04:43.730] Build status: STARTED
[22:04:43.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:6:0) after 3m6s
[22:07:49.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:6:0)
[22:07:49.764] Build status: STARTED
[22:07:49.764] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:7:0) after 2m16s
[22:10:05.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:7:0)
[22:10:05.811] Build status: STARTED
[22:10:05.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:8:0) after 3m52s
[22:13:57.840] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:8:0)
[22:13:57.870] Build status: STARTED
[22:13:57.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:9:0) after 7m26s
[22:21:23.958] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:9:0)
[22:21:23.972] Build status: STARTED
[22:21:23.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:10:0) after 5m7s
[22:26:30.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:10:0)
[22:26:31.006] Build status: STARTED
[22:26:31.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:11:0) after 8m45s
[22:35:16.078] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:11:0)
[22:35:16.096] Build status: STARTED
[22:35:16.096] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:12:0) after 2m17s
[22:37:33.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:12:0)
[22:37:33.218] Build status: STARTED
[22:37:33.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:13:0) after 6m58s
[22:44:31.244] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:13:0)
[22:44:31.263] Build status: STARTED
[22:44:31.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:14:0) after 1m44s
[22:46:15.349] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:14:0)
[22:46:15.407] Build status: STARTED
[22:46:15.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:15:0) after 7m29s
[22:53:44.426] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:15:0)
[22:53:44.443] Build status: STARTED
[22:53:44.443] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:16:0) after 7m14s
[23:00:58.485] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:16:0)
[23:00:58.503] Build status: STARTED
[23:00:58.504] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:17:0) after 8m52s
[23:09:50.520] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:17:0)
[23:09:50.543] Build status: STARTED
[23:09:50.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:18:0) after 8m57s
[23:18:47.642] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:18:0)
[23:18:47.655] Build status: STARTED
[23:18:47.655] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:19:0) after 9m55s
[23:28:42.678] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:19:0)
[23:28:42.705] Build status: STARTED
[23:28:42.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:20:0) after 2m14s
[23:30:56.737] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:20:0)
[23:30:56.755] Build status: STARTED
[23:30:56.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:21:0) after 2m32s
[23:33:28.774] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:21:0)
[23:33:28.835] Build status: STARTED
[23:33:28.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:22:0) after 6m53s
[23:40:21.954] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:22:0)
[23:40:22.026] Build status: STARTED
[23:40:22.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:23:0) after 7m45s
[23:48:07.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:23:0)
[23:48:07.069] Build status: STARTED
[23:48:07.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:24:0) after 7m3s
[23:55:10.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:24:0)
[23:55:10.149] Build status: STARTED
[23:55:10.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:25:0) after 4m23s
[23:59:33.170] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:25:0)
[23:59:33.185] Build status: STARTED
[23:59:33.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:26:0) after 4m23s
[00:03:56.195] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:26:0)
[00:03:56.211] Build status: STARTED
[00:03:56.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:27:0) after 3m13s
[00:07:09.226] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:27:0)
[00:07:09.249] Build status: STARTED
[00:07:09.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:28:0) after 4m29s
[00:11:38.360] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:28:0)
[00:11:38.423] Build status: STARTED
[00:11:38.423] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:29:0) after 1m58s
[00:13:36.491] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:29:0)
[00:13:36.504] Build status: STARTED
[00:13:36.504] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:30:0) after 4m53s
[00:18:29.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:30:0)
[00:18:29.648] Build status: STARTED
[00:18:29.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:31:0) after 2m52s
[00:21:21.672] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:31:0)
[00:21:21.716] Build status: STARTED
[00:21:21.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:32:0) after 3m13s
[00:24:34.782] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:32:0)
[00:24:34.807] Build status: STARTED
[00:24:34.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:33:0) after 5m58s
[00:30:32.841] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:33:0)
[00:30:32.857] Build status: STARTED
[00:30:32.857] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:34:0) after 7m16s
[00:37:48.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:34:0)
[00:37:48.898] Build status: STARTED
[00:37:48.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:35:0) after 6m58s
[00:44:46.955] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:35:0)
[00:44:46.966] Build status: STARTED
[00:44:46.966] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:36:0) after 6m43s
[00:51:30.154] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:36:0)
[00:51:30.171] Build status: STARTED
[00:51:30.171] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:37:0) after 3m6s
[00:54:36.176] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:37:0)
[00:54:36.200] Build status: STARTED
[00:54:36.200] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:38:0) after 4m44s
[00:59:20.183] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:38:0)
[00:59:20.213] Build status: STARTED
[00:59:20.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:39:0) after 8m19s
[01:07:39.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:39:0)
[01:07:39.331] Build status: STARTED
[01:07:39.332] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:40:0) after 1m16s
[01:08:55.346] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:40:0)
[01:08:55.369] Build status: STARTED
[01:08:55.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:41:0) after 1m36s
[01:10:31.406] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:41:0)
[01:10:31.456] Build status: STARTED
[01:10:31.456] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:42:0) after 8m37s
[01:19:08.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:42:0)
[01:19:08.495] Build status: STARTED
[01:19:08.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:43:0) after 3m38s
[01:22:46.605] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:43:0)
[01:22:46.617] Build status: STARTED
[01:22:46.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:44:0) after 2m2s
[01:24:48.633] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:44:0)
[01:24:48.647] Build status: STARTED
[01:24:48.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:45:0) after 2m11s
[01:26:59.674] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:45:0)
[01:26:59.695] Build status: STARTED
[01:26:59.695] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:46:0) after 9m37s
[01:36:36.812] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:46:0)
[01:36:36.836] Build status: STARTED
[01:36:36.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:47:0) after 7m54s
[01:44:30.990] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:47:0)
[01:44:31.008] Build status: STARTED
[01:44:31.008] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:48:0) after 4m49s
[01:49:20.086] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:48:0)
[01:49:20.103] Build status: STARTED
[01:49:20.103] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:49:0) after 6m18s
[01:55:38.122] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:49:0)
[01:55:38.138] Build status: STARTED
[01:55:38.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:50:0) after 4m23s
[02:00:01.169] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:50:0)
[02:00:01.187] Build status: STARTED
[02:00:01.188] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:51:0) after 8m45s
[02:08:46.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:51:0)
[02:08:46.231] Build status: STARTED
[02:08:46.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:52:0) after 5m50s
[02:14:36.324] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:52:0)
[02:14:36.339] Build status: STARTED
[02:14:36.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:53:0) after 5m58s
[02:20:34.362] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:53:0)
[02:20:34.387] Build status: STARTED
[02:20:34.387] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:54:0) after 9m54s
[02:30:28.495] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:54:0)
[02:30:28.509] Build status: STARTED
[02:30:28.509] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:55:0) after 8m5s
[02:38:33.490] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:55:0)
[02:38:33.618] Build status: STARTED
[02:38:33.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:56:0) after 8m23s
[02:46:56.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:56:0)
[02:46:56.798] Build status: STARTED
[02:46:56.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:57:0) after 3m44s
[02:50:40.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:57:0)
[02:50:40.834] Build status: STARTED
[02:50:40.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:58:0) after 8m24s
[02:59:04.852] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:58:0)
[02:59:04.870] Build status: STARTED
[02:59:04.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857549526266624848:59:0) after 8m17s
[03:06:18.639] Received PubSub notification, asking Buildbucket for the build status
[03:06:18.657] Build:
{
"id": "8692119464419641729",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1216,
"createdBy": "project:chromium",
"createTime": "2026-01-20T21:52:13.856854165Z",
"startTime": "2026-01-20T21:52:17.485187116Z",
"endTime": "2026-01-21T03:06:16.787934Z",
"updateTime": "2026-01-21T03:06:16.787934Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "947d6009fe2755da32e410112752ad319aa9eccf",
"ref": "refs/heads/main"
}
},
"canary": true
}
[03:06:18.657] Invocation finished in 5h14m5.945656614s with status FAILED