[11:58:54.589] New invocation is queued and will start shortly
[11:58:56.034] Starting the invocation (attempt 1)
[11:58:56.062] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/bf82d1fd78e1fd3e95832b00ebad8dae442466db"
[11:58:56.062] Popped gitiles commit info from properties and tags
[11:58:56.062] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:58:56.062] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:58:56.062] Buildbucket request:
{
"requestId": "8842457160221064976",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bf82d1fd78e1fd3e95832b00ebad8dae442466db",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842457160221064976"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:58:57.158] Scheduled build:
{
"id": "8677027097908645729",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3628,
"createdBy": "project:chromium",
"createTime": "2026-07-06T11:58:56.171589011Z",
"updateTime": "2026-07-06T11:58:56.171589011Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bf82d1fd78e1fd3e95832b00ebad8dae442466db",
"ref": "refs/heads/main"
}
}
}
[11:58:57.158] Task URL: https://cr-buildbucket.appspot.com/build/8677027097908645729
[11:58:57.158] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:2:0) after 5m28s
[11:58:58.065] Received PubSub notification, asking Buildbucket for the build status
[11:58:58.085] Build status: SCHEDULED
[11:59:01.168] Received PubSub notification, asking Buildbucket for the build status
[11:59:01.325] Build status: STARTED
[12:04:25.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:2:0)
[12:04:25.192] Build status: STARTED
[12:04:25.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:5:0) after 3m52s
[12:08:17.238] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:5:0)
[12:08:17.260] Build status: STARTED
[12:08:17.260] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:6:0) after 7m35s
[12:15:52.339] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:6:0)
[12:15:52.401] Build status: STARTED
[12:15:52.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:7:0) after 9m22s
[12:25:14.416] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:7:0)
[12:25:14.434] Build status: STARTED
[12:25:14.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:8:0) after 2m3s
[12:27:17.885] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:8:0)
[12:27:17.904] Build status: STARTED
[12:27:17.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:9:0) after 3m12s
[12:30:29.921] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:9:0)
[12:30:29.939] Build status: STARTED
[12:30:29.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:10:0) after 5m28s
[12:35:57.960] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:10:0)
[12:35:57.977] Build status: STARTED
[12:35:57.977] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:11:0) after 8m47s
[12:44:45.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:11:0)
[12:44:45.186] Build status: STARTED
[12:44:45.186] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:12:0) after 9m34s
[12:54:19.469] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:12:0)
[12:54:19.483] Build status: STARTED
[12:54:19.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:13:0) after 7m44s
[13:02:03.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:13:0)
[13:02:03.660] Build status: STARTED
[13:02:03.660] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:14:0) after 6m12s
[13:08:15.819] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:14:0)
[13:08:15.845] Build status: STARTED
[13:08:15.845] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:15:0) after 3m54s
[13:12:09.861] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:15:0)
[13:12:09.909] Build status: STARTED
[13:12:09.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:16:0) after 5m22s
[13:17:31.966] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:16:0)
[13:17:31.998] Build status: STARTED
[13:17:31.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:17:0) after 3m59s
[13:21:31.115] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:17:0)
[13:21:31.133] Build status: STARTED
[13:21:31.133] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:18:0) after 5m4s
[13:26:35.209] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:18:0)
[13:26:35.235] Build status: STARTED
[13:26:35.235] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:19:0) after 9m40s
[13:36:15.249] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:19:0)
[13:36:15.291] Build status: STARTED
[13:36:15.291] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:20:0) after 4m24s
[13:40:39.364] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:20:0)
[13:40:39.383] Build status: STARTED
[13:40:39.383] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:21:0) after 5m40s
[13:46:19.401] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:21:0)
[13:46:19.432] Build status: STARTED
[13:46:19.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:22:0) after 8m51s
[13:55:10.502] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:22:0)
[13:55:10.526] Build status: STARTED
[13:55:10.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:23:0) after 4m56s
[14:00:06.556] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:23:0)
[14:00:06.629] Build status: STARTED
[14:00:06.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:24:0) after 7m5s
[14:07:11.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:24:0)
[14:07:11.757] Build status: STARTED
[14:07:11.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:25:0) after 3m15s
[14:10:26.815] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:25:0)
[14:10:26.832] Build status: STARTED
[14:10:26.832] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:26:0) after 6m51s
[14:17:17.848] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:26:0)
[14:17:17.867] Build status: STARTED
[14:17:17.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:27:0) after 9m24s
[14:26:41.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:27:0)
[14:26:41.917] Build status: STARTED
[14:26:41.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:28:0) after 8m55s
[14:35:36.933] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:28:0)
[14:35:36.959] Build status: STARTED
[14:35:36.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:29:0) after 6m21s
[14:41:58.073] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:29:0)
[14:41:58.110] Build status: STARTED
[14:41:58.110] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:30:0) after 6m14s
[14:48:12.126] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:30:0)
[14:48:12.147] Build status: STARTED
[14:48:12.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:31:0) after 5m9s
[14:53:21.593] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:31:0)
[14:53:21.610] Build status: STARTED
[14:53:21.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:32:0) after 6m7s
[14:59:28.608] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:32:0)
[14:59:28.641] Build status: STARTED
[14:59:28.641] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:33:0) after 8m26s
[15:07:54.617] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:33:0)
[15:07:54.677] Build status: STARTED
[15:07:54.677] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:34:0) after 9m5s
[15:16:59.694] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:34:0)
[15:16:59.709] Build status: STARTED
[15:16:59.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:35:0) after 4m50s
[15:21:49.745] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:35:0)
[15:21:49.763] Build status: STARTED
[15:21:49.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:36:0) after 3m15s
[15:25:04.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:36:0)
[15:25:04.983] Build status: STARTED
[15:25:04.983] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:37:0) after 6m16s
[15:31:21.084] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:37:0)
[15:31:21.100] Build status: STARTED
[15:31:21.100] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:38:0) after 5m27s
[15:36:48.130] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:38:0)
[15:36:48.145] Build status: STARTED
[15:36:48.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:39:0) after 3m39s
[15:40:27.182] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:39:0)
[15:40:27.273] Build status: STARTED
[15:40:27.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:40:0) after 3m14s
[15:43:41.309] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:40:0)
[15:43:41.327] Build status: STARTED
[15:43:41.327] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:41:0) after 9m20s
[15:53:01.343] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:41:0)
[15:53:01.378] Build status: STARTED
[15:53:01.378] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:42:0) after 9m53s
[16:02:54.582] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:42:0)
[16:02:54.602] Build status: STARTED
[16:02:54.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:43:0) after 4m2s
[16:06:56.614] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:43:0)
[16:06:56.633] Build status: STARTED
[16:06:56.633] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:44:0) after 6m45s
[16:13:41.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:44:0)
[16:13:41.957] Build status: STARTED
[16:13:41.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:45:0) after 3m3s
[16:16:45.144] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:45:0)
[16:16:45.163] Build status: STARTED
[16:16:45.163] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:46:0) after 2m7s
[16:18:52.261] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:46:0)
[16:18:52.285] Build status: STARTED
[16:18:52.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:47:0) after 8m56s
[16:27:48.263] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:47:0)
[16:27:48.279] Build status: STARTED
[16:27:48.279] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:48:0) after 9m12s
[16:37:00.344] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:48:0)
[16:37:00.360] Build status: STARTED
[16:37:00.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:49:0) after 9m53s
[16:46:53.405] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:49:0)
[16:46:53.883] Build status: STARTED
[16:46:53.883] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8842457160221064976:50:0) after 9m5s
[16:53:09.401] Received PubSub notification, asking Buildbucket for the build status
[16:53:09.466] Build:
{
"id": "8677027097908645729",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3628,
"createdBy": "project:chromium",
"createTime": "2026-07-06T11:58:56.171589011Z",
"startTime": "2026-07-06T11:59:00.970461767Z",
"endTime": "2026-07-06T16:53:09.073453042Z",
"updateTime": "2026-07-06T16:53:09.073453042Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bf82d1fd78e1fd3e95832b00ebad8dae442466db",
"ref": "refs/heads/main"
}
}
}
[16:53:09.466] Invocation finished in 4h54m14.884237215s with status FAILED