[16:58:25.437] New invocation is queued and will start shortly
[16:58:26.702] Starting the invocation (attempt 1)
[16:58:26.729] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/45ad50904316ca9e0dab847cff3fe86cc2c0bd53"
[16:58:26.729] Popped gitiles commit info from properties and tags
[16:58:26.729] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[16:58:26.729] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[16:58:26.730] Buildbucket request:
{
"requestId": "8841079361924929264",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "45ad50904316ca9e0dab847cff3fe86cc2c0bd53",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841079361924929264"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[16:58:27.207] Scheduled build:
{
"id": "8675649299856639697",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3899,
"createdBy": "project:chromium",
"createTime": "2026-07-21T16:58:26.786748416Z",
"updateTime": "2026-07-21T16:58:26.786748416Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "45ad50904316ca9e0dab847cff3fe86cc2c0bd53",
"ref": "refs/heads/main"
}
}
}
[16:58:27.207] Task URL: https://cr-buildbucket.appspot.com/build/8675649299856639697
[16:58:27.207] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:2:0) after 3m30s
[16:58:28.242] Received PubSub notification, asking Buildbucket for the build status
[16:58:28.266] Build status: SCHEDULED
[16:58:31.217] Received PubSub notification, asking Buildbucket for the build status
[16:58:31.290] Build status: STARTED
[17:01:57.307] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:2:0)
[17:01:57.330] Build status: STARTED
[17:01:57.330] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:5:0) after 2m30s
[17:04:27.349] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:5:0)
[17:04:27.370] Build status: STARTED
[17:04:27.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:6:0) after 8m15s
[17:12:42.524] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:6:0)
[17:12:42.549] Build status: STARTED
[17:12:42.549] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:7:0) after 1m14s
[17:13:56.569] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:7:0)
[17:13:56.589] Build status: STARTED
[17:13:56.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:8:0) after 2m15s
[17:16:11.657] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:8:0)
[17:16:11.730] Build status: STARTED
[17:16:11.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:9:0) after 5m14s
[17:21:25.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:9:0)
[17:21:25.774] Build status: STARTED
[17:21:25.774] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:10:0) after 2m38s
[17:24:03.813] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:10:0)
[17:24:03.835] Build status: STARTED
[17:24:03.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:11:0) after 4m19s
[17:28:22.852] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:11:0)
[17:28:22.869] Build status: STARTED
[17:28:22.869] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:12:0) after 1m54s
[17:30:16.884] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:12:0)
[17:30:16.949] Build status: STARTED
[17:30:16.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:13:0) after 5m21s
[17:35:38.016] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:13:0)
[17:35:38.032] Build status: STARTED
[17:35:38.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:14:0) after 3m28s
[17:39:06.069] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:14:0)
[17:39:06.090] Build status: STARTED
[17:39:06.090] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:15:0) after 2m46s
[17:41:52.160] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:15:0)
[17:41:52.187] Build status: STARTED
[17:41:52.187] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:16:0) after 9m31s
[17:51:23.260] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:16:0)
[17:51:23.359] Build status: STARTED
[17:51:23.359] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:17:0) after 5m16s
[17:56:39.380] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:17:0)
[17:56:39.398] Build status: STARTED
[17:56:39.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:18:0) after 3m9s
[17:59:48.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:18:0)
[17:59:48.436] Build status: STARTED
[17:59:48.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:19:0) after 8m37s
[18:08:25.459] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:19:0)
[18:08:25.481] Build status: STARTED
[18:08:25.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:20:0) after 6m23s
[18:14:48.506] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:20:0)
[18:14:48.611] Build status: STARTED
[18:14:48.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:21:0) after 9m11s
[18:23:59.765] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:21:0)
[18:23:59.792] Build status: STARTED
[18:23:59.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:22:0) after 4m11s
[18:28:10.973] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:22:0)
[18:28:10.996] Build status: STARTED
[18:28:10.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:23:0) after 9m48s
[18:37:59.026] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:23:0)
[18:37:59.065] Build status: STARTED
[18:37:59.065] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:24:0) after 3m16s
[18:41:15.264] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:24:0)
[18:41:15.319] Build status: STARTED
[18:41:15.319] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:25:0) after 4m3s
[18:45:18.361] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:25:0)
[18:45:18.399] Build status: STARTED
[18:45:18.399] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:26:0) after 5m25s
[18:50:43.416] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:26:0)
[18:50:43.435] Build status: STARTED
[18:50:43.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:27:0) after 8m52s
[18:59:35.459] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:27:0)
[18:59:35.486] Build status: STARTED
[18:59:35.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:28:0) after 3m9s
[19:02:44.503] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:28:0)
[19:02:44.519] Build status: STARTED
[19:02:44.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:29:0) after 8m59s
[19:11:43.635] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:29:0)
[19:11:43.667] Build status: STARTED
[19:11:43.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:30:0) after 2m25s
[19:14:08.685] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:30:0)
[19:14:08.708] Build status: STARTED
[19:14:08.708] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:31:0) after 5m21s
[19:19:29.743] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:31:0)
[19:19:29.768] Build status: STARTED
[19:19:29.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:32:0) after 6m52s
[19:26:21.837] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:32:0)
[19:26:21.865] Build status: STARTED
[19:26:21.865] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:33:0) after 4m39s
[19:31:00.923] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:33:0)
[19:31:00.957] Build status: STARTED
[19:31:00.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:34:0) after 8m51s
[19:39:52.058] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:34:0)
[19:39:52.078] Build status: STARTED
[19:39:52.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:35:0) after 1m10s
[19:41:02.136] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:35:0)
[19:41:02.168] Build status: STARTED
[19:41:02.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:36:0) after 9m59s
[19:51:01.188] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:36:0)
[19:51:01.250] Build status: STARTED
[19:51:01.250] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:37:0) after 5m41s
[19:56:42.262] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:37:0)
[19:56:42.316] Build status: STARTED
[19:56:42.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:38:0) after 9m17s
[20:05:59.339] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:38:0)
[20:05:59.363] Build status: STARTED
[20:05:59.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:39:0) after 3m4s
[20:09:03.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:39:0)
[20:09:03.405] Build status: STARTED
[20:09:03.405] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:40:0) after 9m49s
[20:18:52.428] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:40:0)
[20:18:52.449] Build status: STARTED
[20:18:52.450] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:41:0) after 9m40s
[20:28:32.496] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:41:0)
[20:28:32.512] Build status: STARTED
[20:28:32.513] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:42:0) after 9m15s
[20:37:47.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:42:0)
[20:37:47.602] Build status: STARTED
[20:37:47.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:43:0) after 7m55s
[20:45:42.655] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:43:0)
[20:45:42.689] Build status: STARTED
[20:45:42.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:44:0) after 3m33s
[20:49:15.728] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:44:0)
[20:49:15.749] Build status: STARTED
[20:49:15.750] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:45:0) after 9m53s
[20:59:08.754] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:45:0)
[20:59:08.794] Build status: STARTED
[20:59:08.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:46:0) after 9m46s
[21:08:54.868] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:46:0)
[21:08:54.903] Build status: STARTED
[21:08:54.903] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:47:0) after 7m12s
[21:16:06.919] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:47:0)
[21:16:06.941] Build status: STARTED
[21:16:06.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:48:0) after 2m6s
[21:18:13.106] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:48:0)
[21:18:13.132] Build status: STARTED
[21:18:13.132] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:49:0) after 3m3s
[21:21:16.153] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:49:0)
[21:21:16.191] Build status: STARTED
[21:21:16.191] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:50:0) after 5m50s
[21:27:06.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:50:0)
[21:27:06.331] Build status: STARTED
[21:27:06.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:51:0) after 3m42s
[21:30:48.393] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:51:0)
[21:30:48.516] Build status: STARTED
[21:30:48.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:52:0) after 5m30s
[21:36:18.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:52:0)
[21:36:18.567] Build status: STARTED
[21:36:18.567] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:53:0) after 5m2s
[21:41:20.592] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:53:0)
[21:41:20.627] Build status: STARTED
[21:41:20.627] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:54:0) after 4m43s
[21:46:03.632] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:54:0)
[21:46:03.713] Build status: STARTED
[21:46:03.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:55:0) after 5m47s
[21:51:50.728] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:55:0)
[21:51:50.747] Build status: STARTED
[21:51:50.747] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8841079361924929264:56:0) after 9m46s
[21:58:25.783] Received PubSub notification, asking Buildbucket for the build status
[21:58:25.802] Build:
{
"id": "8675649299856639697",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 3899,
"createdBy": "project:chromium",
"createTime": "2026-07-21T16:58:26.786748416Z",
"startTime": "2026-07-21T16:58:30.961012031Z",
"endTime": "2026-07-21T21:58:25.530234929Z",
"updateTime": "2026-07-21T21:58:25.530234929Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "45ad50904316ca9e0dab847cff3fe86cc2c0bd53",
"ref": "refs/heads/main"
}
}
}
[21:58:25.802] Invocation finished in 5h0m0.371972759s with status FAILED