[01:59:22.284] New invocation is queued and will start shortly
[01:59:23.337] Starting the invocation (attempt 1)
[01:59:23.363] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/8273f28dc9952b7f5158deaff479ab7b2bc54dad"
[01:59:23.363] Popped gitiles commit info from properties and tags
[01:59:23.363] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:59:23.363] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:59:23.363] Buildbucket request:
{
"requestId": "8857624573302084432",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8273f28dc9952b7f5158deaff479ab7b2bc54dad",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857624573302084432"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:59:23.791] Scheduled build:
{
"id": "8692194511467373985",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1212,
"createdBy": "project:chromium",
"createTime": "2026-01-20T01:59:23.414798199Z",
"updateTime": "2026-01-20T01:59:23.414798199Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8273f28dc9952b7f5158deaff479ab7b2bc54dad",
"ref": "refs/heads/main"
}
}
}
[01:59:23.791] Task URL: https://cr-buildbucket.appspot.com/build/8692194511467373985
[01:59:23.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:2:0) after 9m21s
[01:59:24.751] Received PubSub notification, asking Buildbucket for the build status
[01:59:24.772] Build status: SCHEDULED
[01:59:27.115] Received PubSub notification, asking Buildbucket for the build status
[01:59:27.134] Build status: STARTED
[02:08:44.922] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:2:0)
[02:08:44.939] Build status: STARTED
[02:08:44.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:5:0) after 9m29s
[02:18:14.063] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:5:0)
[02:18:14.079] Build status: STARTED
[02:18:14.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:6:0) after 3m26s
[02:21:40.099] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:6:0)
[02:21:40.116] Build status: STARTED
[02:21:40.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:7:0) after 2m56s
[02:24:36.136] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:7:0)
[02:24:36.154] Build status: STARTED
[02:24:36.154] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:8:0) after 5m31s
[02:30:07.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:8:0)
[02:30:07.274] Build status: STARTED
[02:30:07.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:9:0) after 5m10s
[02:35:17.327] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:9:0)
[02:35:17.345] Build status: STARTED
[02:35:17.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:10:0) after 9m54s
[02:45:11.470] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:10:0)
[02:45:11.554] Build status: STARTED
[02:45:11.554] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:11:0) after 5m33s
[02:50:44.654] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:11:0)
[02:50:44.680] Build status: STARTED
[02:50:44.680] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:12:0) after 7m19s
[02:58:03.853] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:12:0)
[02:58:03.876] Build status: STARTED
[02:58:03.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:13:0) after 8m50s
[03:06:53.978] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:13:0)
[03:06:53.996] Build status: STARTED
[03:06:53.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:14:0) after 4m51s
[03:11:45.114] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:14:0)
[03:11:45.135] Build status: STARTED
[03:11:45.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:15:0) after 3m48s
[03:15:33.154] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:15:0)
[03:15:33.172] Build status: STARTED
[03:15:33.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:16:0) after 1m10s
[03:16:43.189] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:16:0)
[03:16:43.321] Build status: STARTED
[03:16:43.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:17:0) after 2m45s
[03:19:28.467] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:17:0)
[03:19:28.485] Build status: STARTED
[03:19:28.485] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:18:0) after 7m45s
[03:27:13.756] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:18:0)
[03:27:13.771] Build status: STARTED
[03:27:13.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:19:0) after 4m2s
[03:31:15.800] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:19:0)
[03:31:15.815] Build status: STARTED
[03:31:15.815] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:20:0) after 9m32s
[03:40:47.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:20:0)
[03:40:47.901] Build status: STARTED
[03:40:47.901] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:21:0) after 8m43s
[03:49:30.908] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:21:0)
[03:49:31.039] Build status: STARTED
[03:49:31.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:22:0) after 8m37s
[03:58:08.056] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:22:0)
[03:58:08.089] Build status: STARTED
[03:58:08.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:23:0) after 1m45s
[03:59:53.104] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:23:0)
[03:59:53.147] Build status: STARTED
[03:59:53.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:24:0) after 3m45s
[04:03:38.262] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:24:0)
[04:03:38.277] Build status: STARTED
[04:03:38.277] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:25:0) after 2m18s
[04:05:56.346] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:25:0)
[04:05:56.383] Build status: STARTED
[04:05:56.383] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:26:0) after 5m49s
[04:11:45.459] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:26:0)
[04:11:45.483] Build status: STARTED
[04:11:45.483] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:27:0) after 9m46s
[04:21:31.500] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:27:0)
[04:21:31.514] Build status: STARTED
[04:21:31.514] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:28:0) after 8m10s
[04:29:41.644] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:28:0)
[04:29:41.663] Build status: STARTED
[04:29:41.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:29:0) after 5m17s
[04:34:59.268] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:29:0)
[04:34:59.287] Build status: STARTED
[04:34:59.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:30:0) after 9m2s
[04:44:01.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:30:0)
[04:44:01.433] Build status: STARTED
[04:44:01.433] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:31:0) after 7m51s
[04:51:52.841] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:31:0)
[04:51:53.087] Build status: STARTED
[04:51:53.087] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:32:0) after 4m14s
[04:56:07.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:32:0)
[04:56:07.123] Build status: STARTED
[04:56:07.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:33:0) after 2m36s
[04:58:43.239] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:33:0)
[04:58:43.268] Build status: STARTED
[04:58:43.268] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:34:0) after 2m35s
[05:01:18.448] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:34:0)
[05:01:18.464] Build status: STARTED
[05:01:18.464] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:35:0) after 6m7s
[05:07:25.527] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:35:0)
[05:07:25.547] Build status: STARTED
[05:07:25.547] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:36:0) after 1m47s
[05:09:12.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:36:0)
[05:09:12.577] Build status: STARTED
[05:09:12.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:37:0) after 9m30s
[05:18:42.756] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:37:0)
[05:18:42.768] Build status: STARTED
[05:18:42.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:38:0) after 7m23s
[05:26:05.836] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:38:0)
[05:26:05.877] Build status: STARTED
[05:26:05.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:39:0) after 8m57s
[05:35:03.044] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:39:0)
[05:35:03.328] Build status: STARTED
[05:35:03.328] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:40:0) after 1m49s
[05:36:52.546] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:40:0)
[05:36:52.557] Build status: STARTED
[05:36:52.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:41:0) after 9m58s
[05:46:50.932] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:41:0)
[05:46:50.950] Build status: STARTED
[05:46:50.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:42:0) after 7m18s
[05:54:09.458] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:42:0)
[05:54:09.475] Build status: STARTED
[05:54:09.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:43:0) after 3m21s
[05:57:30.490] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:43:0)
[05:57:30.507] Build status: STARTED
[05:57:30.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:44:0) after 7m20s
[06:04:51.036] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:44:0)
[06:04:51.053] Build status: STARTED
[06:04:51.053] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:45:0) after 1m48s
[06:06:39.252] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:45:0)
[06:06:39.276] Build status: STARTED
[06:06:39.276] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:46:0) after 1m16s
[06:07:55.466] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:46:0)
[06:07:55.488] Build status: STARTED
[06:07:55.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:47:0) after 5m37s
[06:13:32.602] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:47:0)
[06:13:32.617] Build status: STARTED
[06:13:32.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:48:0) after 2m16s
[06:15:48.802] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:48:0)
[06:15:48.815] Build status: STARTED
[06:15:48.815] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:49:0) after 5m24s
[06:21:12.884] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:49:0)
[06:21:12.898] Build status: STARTED
[06:21:12.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:50:0) after 3m59s
[06:25:12.025] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:50:0)
[06:25:12.039] Build status: STARTED
[06:25:12.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:51:0) after 3m36s
[06:28:48.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:51:0)
[06:28:48.487] Build status: STARTED
[06:28:48.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:52:0) after 2m44s
[06:31:32.573] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:52:0)
[06:31:32.586] Build status: STARTED
[06:31:32.586] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:53:0) after 5m25s
[06:36:57.601] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:53:0)
[06:36:57.615] Build status: STARTED
[06:36:57.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:54:0) after 3m25s
[06:40:22.673] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:54:0)
[06:40:22.736] Build status: STARTED
[06:40:22.736] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:55:0) after 9m33s
[06:49:56.433] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:55:0)
[06:49:56.621] Build status: STARTED
[06:49:56.621] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:56:0) after 3m30s
[06:53:26.629] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:56:0)
[06:53:26.644] Build status: STARTED
[06:53:26.644] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:57:0) after 5m50s
[06:59:16.673] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:57:0)
[06:59:16.713] Build status: STARTED
[06:59:16.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:58:0) after 3m22s
[07:02:38.729] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:58:0)
[07:02:38.757] Build status: STARTED
[07:02:38.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:59:0) after 5m52s
[07:08:30.819] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:59:0)
[07:08:30.837] Build status: STARTED
[07:08:30.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:60:0) after 2m41s
[07:11:11.917] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:60:0)
[07:11:11.942] Build status: STARTED
[07:11:11.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:61:0) after 2m11s
[07:13:22.963] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:61:0)
[07:13:22.979] Build status: STARTED
[07:13:22.979] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8857624573302084432:62:0) after 5m50s
[07:14:53.592] Received PubSub notification, asking Buildbucket for the build status
[07:14:53.607] Build:
{
"id": "8692194511467373985",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 1212,
"createdBy": "project:chromium",
"createTime": "2026-01-20T01:59:23.414798199Z",
"startTime": "2026-01-20T01:59:26.969384411Z",
"endTime": "2026-01-20T07:14:53.258990511Z",
"updateTime": "2026-01-20T07:14:53.258990511Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8273f28dc9952b7f5158deaff479ab7b2bc54dad",
"ref": "refs/heads/main"
}
}
}
[07:14:53.607] Invocation finished in 5h15m31.327372958s with status FAILED