[07:04:58.556] New invocation is queued and will start shortly
[07:04:59.635] Starting the invocation (attempt 1)
[07:04:59.660] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/547d3e000b79fe0ff0f97ddf2cdd2b922a08f1b0"
[07:04:59.660] Popped gitiles commit info from properties and tags
[07:04:59.660] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:04:59.660] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:04:59.661] Buildbucket request:
{
"requestId": "8848726843620599008",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "547d3e000b79fe0ff0f97ddf2cdd2b922a08f1b0",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8848726843620599008"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:05:00.426] Scheduled build:
{
"id": "8683296781755510481",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2489,
"createdBy": "project:chromium",
"createTime": "2026-04-28T07:04:59.714616364Z",
"updateTime": "2026-04-28T07:04:59.714616364Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "547d3e000b79fe0ff0f97ddf2cdd2b922a08f1b0",
"ref": "refs/heads/main"
}
}
}
[07:05:00.426] Task URL: https://cr-buildbucket.appspot.com/build/8683296781755510481
[07:05:00.426] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:2:0) after 6m21s
[07:05:01.333] Received PubSub notification, asking Buildbucket for the build status
[07:05:01.379] Build status: SCHEDULED
[07:05:43.566] Received PubSub notification, asking Buildbucket for the build status
[07:05:43.623] Build status: STARTED
[07:11:21.447] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:2:0)
[07:11:21.464] Build status: STARTED
[07:11:21.464] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:5:0) after 6m4s
[07:17:25.485] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:5:0)
[07:17:25.511] Build status: STARTED
[07:17:25.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:6:0) after 5m59s
[07:23:24.529] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:6:0)
[07:23:24.550] Build status: STARTED
[07:23:24.550] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:7:0) after 2m59s
[07:26:23.659] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:7:0)
[07:26:23.676] Build status: STARTED
[07:26:23.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:8:0) after 5m2s
[07:31:25.691] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:8:0)
[07:31:25.712] Build status: STARTED
[07:31:25.712] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:9:0) after 5m40s
[07:37:05.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:9:0)
[07:37:05.745] Build status: STARTED
[07:37:05.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:10:0) after 3m36s
[07:40:41.724] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:10:0)
[07:40:41.751] Build status: STARTED
[07:40:41.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:11:0) after 7m16s
[07:47:57.729] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:11:0)
[07:47:57.752] Build status: STARTED
[07:47:57.752] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:12:0) after 4m18s
[07:52:15.818] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:12:0)
[07:52:15.837] Build status: STARTED
[07:52:15.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:13:0) after 1m21s
[07:53:36.854] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:13:0)
[07:53:36.875] Build status: STARTED
[07:53:36.875] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:14:0) after 7m22s
[08:00:58.890] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:14:0)
[08:00:58.911] Build status: STARTED
[08:00:58.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:15:0) after 2m25s
[08:03:23.936] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:15:0)
[08:03:23.950] Build status: STARTED
[08:03:23.951] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:16:0) after 1m53s
[08:05:17.082] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:16:0)
[08:05:17.106] Build status: STARTED
[08:05:17.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:17:0) after 4m51s
[08:10:08.127] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:17:0)
[08:10:08.150] Build status: STARTED
[08:10:08.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:18:0) after 4m4s
[08:14:12.167] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:18:0)
[08:14:12.190] Build status: STARTED
[08:14:12.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:19:0) after 4m4s
[08:18:16.206] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:19:0)
[08:18:16.248] Build status: STARTED
[08:18:16.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:20:0) after 2m29s
[08:20:45.317] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:20:0)
[08:20:45.336] Build status: STARTED
[08:20:45.336] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:21:0) after 1m56s
[08:22:41.349] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:21:0)
[08:22:41.365] Build status: STARTED
[08:22:41.365] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:22:0) after 1m2s
[08:23:43.443] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:22:0)
[08:23:43.500] Build status: STARTED
[08:23:43.501] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:23:0) after 5m47s
[08:29:30.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:23:0)
[08:29:30.604] Build status: STARTED
[08:29:30.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:24:0) after 6m26s
[08:35:56.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:24:0)
[08:35:56.658] Build status: STARTED
[08:35:56.658] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:25:0) after 2m35s
[08:38:31.725] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:25:0)
[08:38:31.743] Build status: STARTED
[08:38:31.743] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:26:0) after 5m56s
[08:44:27.810] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:26:0)
[08:44:27.861] Build status: STARTED
[08:44:27.861] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:27:0) after 7m18s
[08:51:45.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:27:0)
[08:51:45.951] Build status: STARTED
[08:51:45.951] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:28:0) after 5m44s
[08:57:29.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:28:0)
[08:57:29.987] Build status: STARTED
[08:57:29.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:29:0) after 9m0s
[09:06:30.104] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:29:0)
[09:06:30.126] Build status: STARTED
[09:06:30.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:30:0) after 5m45s
[09:12:15.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:30:0)
[09:12:15.312] Build status: STARTED
[09:12:15.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:31:0) after 9m10s
[09:21:25.385] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:31:0)
[09:21:25.399] Build status: STARTED
[09:21:25.399] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:32:0) after 4m52s
[09:26:17.427] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:32:0)
[09:26:17.442] Build status: STARTED
[09:26:17.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:33:0) after 1m29s
[09:27:46.472] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:33:0)
[09:27:46.490] Build status: STARTED
[09:27:46.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:34:0) after 1m26s
[09:29:12.508] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:34:0)
[09:29:12.547] Build status: STARTED
[09:29:12.547] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:35:0) after 6m15s
[09:35:27.689] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:35:0)
[09:35:27.707] Build status: STARTED
[09:35:27.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:36:0) after 2m58s
[09:38:25.775] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:36:0)
[09:38:25.794] Build status: STARTED
[09:38:25.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:37:0) after 5m11s
[09:43:36.910] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:37:0)
[09:43:36.927] Build status: STARTED
[09:43:36.927] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:38:0) after 8m56s
[09:52:32.899] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:38:0)
[09:52:32.924] Build status: STARTED
[09:52:32.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:39:0) after 9m7s
[10:01:39.993] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:39:0)
[10:01:41.054] Build status: STARTED
[10:01:41.054] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:40:0) after 6m58s
[10:08:39.071] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:40:0)
[10:08:39.085] Build status: STARTED
[10:08:39.085] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:41:0) after 2m2s
[10:10:41.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:41:0)
[10:10:41.157] Build status: STARTED
[10:10:41.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:42:0) after 2m4s
[10:12:45.173] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:42:0)
[10:12:45.300] Build status: STARTED
[10:12:45.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:43:0) after 1m44s
[10:14:29.316] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:43:0)
[10:14:29.332] Build status: STARTED
[10:14:29.332] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:44:0) after 6m22s
[10:20:51.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:44:0)
[10:20:51.401] Build status: STARTED
[10:20:51.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:45:0) after 1m59s
[10:22:50.417] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:45:0)
[10:22:50.436] Build status: STARTED
[10:22:50.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:46:0) after 8m7s
[10:30:57.458] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:46:0)
[10:30:57.493] Build status: STARTED
[10:30:57.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:47:0) after 4m43s
[10:35:40.510] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:47:0)
[10:35:40.541] Build status: STARTED
[10:35:40.541] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:48:0) after 8m2s
[10:43:42.564] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:48:0)
[10:43:42.582] Build status: STARTED
[10:43:42.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:49:0) after 3m23s
[10:47:05.600] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:49:0)
[10:47:05.625] Build status: STARTED
[10:47:05.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:50:0) after 4m51s
[10:51:56.694] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:50:0)
[10:51:56.714] Build status: STARTED
[10:51:56.714] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:51:0) after 8m8s
[11:00:04.817] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:51:0)
[11:00:04.834] Build status: STARTED
[11:00:04.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:52:0) after 2m22s
[11:02:26.855] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:52:0)
[11:02:26.995] Build status: STARTED
[11:02:26.995] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8848726843620599008:53:0) after 5m23s
[11:04:43.871] Received PubSub notification, asking Buildbucket for the build status
[11:04:43.985] Build:
{
"id": "8683296781755510481",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2489,
"createdBy": "project:chromium",
"createTime": "2026-04-28T07:04:59.714616364Z",
"startTime": "2026-04-28T07:05:43.264630790Z",
"endTime": "2026-04-28T11:04:43.472288420Z",
"updateTime": "2026-04-28T11:04:43.472288420Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "547d3e000b79fe0ff0f97ddf2cdd2b922a08f1b0",
"ref": "refs/heads/main"
}
}
}
[11:04:43.985] Invocation finished in 3h59m45.433408095s with status FAILED