[07:06:30.549] New invocation is queued and will start shortly
[07:06:31.685] Starting the invocation (attempt 1)
[07:06:31.713] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/962b8490d3a53845f4bd48a58581b256dfc67418"
[07:06:31.713] Popped gitiles commit info from properties and tags
[07:06:31.713] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:06:31.714] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:06:31.714] Buildbucket request:
{
"requestId": "8849995104696392864",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "962b8490d3a53845f4bd48a58581b256dfc67418",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8849995104696392864"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:06:32.477] Scheduled build:
{
"id": "8684565042731240673",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2141,
"createdBy": "project:chromium",
"createTime": "2026-04-14T07:06:31.795107090Z",
"updateTime": "2026-04-14T07:06:31.795107090Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "962b8490d3a53845f4bd48a58581b256dfc67418",
"ref": "refs/heads/main"
}
}
}
[07:06:32.477] Task URL: https://cr-buildbucket.appspot.com/build/8684565042731240673
[07:06:32.477] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:2:0) after 9m32s
[07:06:33.255] Received PubSub notification, asking Buildbucket for the build status
[07:06:33.279] Build status: SCHEDULED
[07:06:35.714] Received PubSub notification, asking Buildbucket for the build status
[07:06:35.731] Build status: STARTED
[07:16:04.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:2:0)
[07:16:04.557] Build status: STARTED
[07:16:04.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:5:0) after 6m28s
[07:22:32.555] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:5:0)
[07:22:32.622] Build status: STARTED
[07:22:32.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:6:0) after 2m39s
[07:25:11.639] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:6:0)
[07:25:11.676] Build status: STARTED
[07:25:11.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:7:0) after 9m22s
[07:34:33.693] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:7:0)
[07:34:33.712] Build status: STARTED
[07:34:33.712] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:8:0) after 3m42s
[07:38:15.784] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:8:0)
[07:38:15.804] Build status: STARTED
[07:38:15.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:9:0) after 9m33s
[07:47:48.823] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:9:0)
[07:47:48.839] Build status: STARTED
[07:47:48.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:10:0) after 5m36s
[07:53:24.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:10:0)
[07:53:24.903] Build status: STARTED
[07:53:24.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:11:0) after 3m19s
[07:56:43.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:11:0)
[07:56:43.948] Build status: STARTED
[07:56:43.948] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:12:0) after 2m54s
[07:59:37.949] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:12:0)
[07:59:38.066] Build status: STARTED
[07:59:38.066] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:13:0) after 4m9s
[08:03:47.122] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:13:0)
[08:03:47.143] Build status: STARTED
[08:03:47.143] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:14:0) after 6m19s
[08:10:06.147] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:14:0)
[08:10:06.167] Build status: STARTED
[08:10:06.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:15:0) after 9m12s
[08:19:18.261] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:15:0)
[08:19:18.285] Build status: STARTED
[08:19:18.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:16:0) after 7m52s
[08:27:10.306] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:16:0)
[08:27:10.331] Build status: STARTED
[08:27:10.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:17:0) after 7m22s
[08:34:32.358] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:17:0)
[08:34:32.381] Build status: STARTED
[08:34:32.381] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:18:0) after 2m16s
[08:36:48.449] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:18:0)
[08:36:48.532] Build status: STARTED
[08:36:48.532] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:19:0) after 2m16s
[08:39:04.576] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:19:0)
[08:39:04.592] Build status: STARTED
[08:39:04.592] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:20:0) after 4m0s
[08:43:04.753] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:20:0)
[08:43:04.769] Build status: STARTED
[08:43:04.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:21:0) after 3m0s
[08:46:04.842] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:21:0)
[08:46:04.867] Build status: STARTED
[08:46:04.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:22:0) after 6m34s
[08:52:38.953] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:22:0)
[08:52:38.973] Build status: STARTED
[08:52:38.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:23:0) after 8m56s
[09:01:35.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:23:0)
[09:01:35.025] Build status: STARTED
[09:01:35.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:24:0) after 4m11s
[09:05:46.044] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:24:0)
[09:05:46.068] Build status: STARTED
[09:05:46.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:25:0) after 2m45s
[09:08:31.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:25:0)
[09:08:31.199] Build status: STARTED
[09:08:31.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:26:0) after 1m52s
[09:10:23.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:26:0)
[09:10:23.240] Build status: STARTED
[09:10:23.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:27:0) after 7m0s
[09:17:23.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:27:0)
[09:17:23.272] Build status: STARTED
[09:17:23.272] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:28:0) after 8m19s
[09:25:42.309] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:28:0)
[09:25:42.344] Build status: STARTED
[09:25:42.344] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:29:0) after 9m8s
[09:34:50.376] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:29:0)
[09:34:50.606] Build status: STARTED
[09:34:50.606] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:30:0) after 9m18s
[09:44:08.626] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:30:0)
[09:44:08.646] Build status: STARTED
[09:44:08.646] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:31:0) after 1m26s
[09:45:34.664] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:31:0)
[09:45:34.687] Build status: STARTED
[09:45:34.687] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:32:0) after 1m34s
[09:47:08.712] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:32:0)
[09:47:08.733] Build status: STARTED
[09:47:08.733] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:33:0) after 6m43s
[09:53:51.811] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:33:0)
[09:53:51.827] Build status: STARTED
[09:53:51.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:34:0) after 4m7s
[09:57:58.863] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:34:0)
[09:57:58.881] Build status: STARTED
[09:57:58.881] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:35:0) after 3m28s
[10:01:26.898] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:35:0)
[10:01:26.920] Build status: STARTED
[10:01:26.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:36:0) after 9m57s
[10:11:24.038] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:36:0)
[10:11:24.092] Build status: STARTED
[10:11:24.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:37:0) after 4m3s
[10:15:27.257] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:37:0)
[10:15:27.302] Build status: STARTED
[10:15:27.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:38:0) after 8m15s
[10:23:42.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:38:0)
[10:23:42.373] Build status: STARTED
[10:23:42.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:39:0) after 4m8s
[10:27:50.415] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:39:0)
[10:27:50.435] Build status: STARTED
[10:27:50.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:40:0) after 3m9s
[10:30:59.469] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:40:0)
[10:30:59.491] Build status: STARTED
[10:30:59.491] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:41:0) after 7m30s
[10:38:29.550] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:41:0)
[10:38:29.591] Build status: STARTED
[10:38:29.592] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:42:0) after 4m41s
[10:43:10.696] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:42:0)
[10:43:10.720] Build status: STARTED
[10:43:10.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:43:0) after 7m42s
[10:50:52.739] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:43:0)
[10:50:52.765] Build status: STARTED
[10:50:52.765] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:44:0) after 8m53s
[10:59:45.955] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:44:0)
[10:59:45.979] Build status: STARTED
[10:59:45.979] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8849995104696392864:45:0) after 8m41s
[11:04:22.182] Received PubSub notification, asking Buildbucket for the build status
[11:04:22.199] Build:
{
"id": "8684565042731240673",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2141,
"createdBy": "project:chromium",
"createTime": "2026-04-14T07:06:31.795107090Z",
"startTime": "2026-04-14T07:06:35.563767599Z",
"endTime": "2026-04-14T11:04:21.927916367Z",
"updateTime": "2026-04-14T11:04:21.927916367Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "962b8490d3a53845f4bd48a58581b256dfc67418",
"ref": "refs/heads/main"
}
}
}
[11:04:22.199] Invocation finished in 3h57m51.661234066s with status FAILED