[17:22:31.472] New invocation is queued and will start shortly
[17:22:32.917] Starting the invocation (attempt 1)
[17:22:32.943] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/bc1007a6a3125f80f50e18c36d5cea744453a53d"
[17:22:32.943] Popped gitiles commit info from properties and tags
[17:22:32.943] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[17:22:32.943] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[17:22:32.943] Buildbucket request:
{
"requestId": "8850681124081381344",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bc1007a6a3125f80f50e18c36d5cea744453a53d",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850681124081381344"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[17:22:33.808] Scheduled build:
{
"id": "8685251061817401905",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2403,
"createdBy": "project:chromium",
"createTime": "2026-04-06T17:22:33.012821175Z",
"updateTime": "2026-04-06T17:22:33.012821175Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bc1007a6a3125f80f50e18c36d5cea744453a53d",
"ref": "refs/heads/main"
}
}
}
[17:22:33.808] Task URL: https://cr-buildbucket.appspot.com/build/8685251061817401905
[17:22:33.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:2:0) after 6m22s
[17:22:34.710] Received PubSub notification, asking Buildbucket for the build status
[17:22:34.731] Build status: SCHEDULED
[17:23:21.216] Received PubSub notification, asking Buildbucket for the build status
[17:23:21.234] Build status: STARTED
[17:28:55.876] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:2:0)
[17:28:55.905] Build status: STARTED
[17:28:55.905] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:5:0) after 9m9s
[17:38:05.033] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:5:0)
[17:38:05.049] Build status: STARTED
[17:38:05.049] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:6:0) after 7m3s
[17:45:08.069] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:6:0)
[17:45:08.088] Build status: STARTED
[17:45:08.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:7:0) after 4m10s
[17:49:18.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:7:0)
[17:49:18.233] Build status: STARTED
[17:49:18.233] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:8:0) after 2m44s
[17:52:02.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:8:0)
[17:52:02.260] Build status: STARTED
[17:52:02.260] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:9:0) after 1m51s
[17:53:53.366] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:9:0)
[17:53:53.383] Build status: STARTED
[17:53:53.383] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:10:0) after 2m21s
[17:56:14.400] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:10:0)
[17:56:14.415] Build status: STARTED
[17:56:14.415] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:11:0) after 5m53s
[18:02:07.439] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:11:0)
[18:02:07.457] Build status: STARTED
[18:02:07.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:12:0) after 8m46s
[18:10:53.475] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:12:0)
[18:10:53.557] Build status: STARTED
[18:10:53.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:13:0) after 9m3s
[18:19:56.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:13:0)
[18:19:56.593] Build status: STARTED
[18:19:56.593] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:14:0) after 8m1s
[18:27:57.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:14:0)
[18:27:57.651] Build status: STARTED
[18:27:57.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:15:0) after 7m40s
[18:35:37.672] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:15:0)
[18:35:37.686] Build status: STARTED
[18:35:37.686] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:16:0) after 6m38s
[18:42:15.757] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:16:0)
[18:42:15.797] Build status: STARTED
[18:42:15.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:17:0) after 5m16s
[18:47:31.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:17:0)
[18:47:31.882] Build status: STARTED
[18:47:31.882] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:18:0) after 4m11s
[18:51:42.948] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:18:0)
[18:51:42.962] Build status: STARTED
[18:51:42.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:19:0) after 3m10s
[18:54:53.029] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:19:0)
[18:54:53.071] Build status: STARTED
[18:54:53.071] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:20:0) after 3m40s
[18:58:33.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:20:0)
[18:58:33.070] Build status: STARTED
[18:58:33.070] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:21:0) after 2m16s
[19:00:49.159] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:21:0)
[19:00:49.212] Build status: STARTED
[19:00:49.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:22:0) after 2m0s
[19:02:49.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:22:0)
[19:02:49.210] Build status: STARTED
[19:02:49.210] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:23:0) after 9m15s
[19:12:04.228] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:23:0)
[19:12:04.245] Build status: STARTED
[19:12:04.245] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:24:0) after 4m8s
[19:16:12.271] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:24:0)
[19:16:12.292] Build status: STARTED
[19:16:12.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:25:0) after 8m45s
[19:24:57.321] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:25:0)
[19:24:57.340] Build status: STARTED
[19:24:57.340] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:26:0) after 7m26s
[19:32:23.357] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:26:0)
[19:32:23.423] Build status: STARTED
[19:32:23.423] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:27:0) after 2m33s
[19:34:56.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:27:0)
[19:34:56.557] Build status: STARTED
[19:34:56.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:28:0) after 2m24s
[19:37:20.580] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:28:0)
[19:37:20.599] Build status: STARTED
[19:37:20.599] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:29:0) after 2m28s
[19:39:48.662] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:29:0)
[19:39:48.680] Build status: STARTED
[19:39:48.680] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:30:0) after 3m54s
[19:43:42.767] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:30:0)
[19:43:42.846] Build status: STARTED
[19:43:42.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:31:0) after 3m44s
[19:47:26.956] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:31:0)
[19:47:26.974] Build status: STARTED
[19:47:26.974] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:32:0) after 6m35s
[19:54:01.986] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:32:0)
[19:54:02.048] Build status: STARTED
[19:54:02.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:33:0) after 8m31s
[20:02:33.104] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:33:0)
[20:02:33.135] Build status: STARTED
[20:02:33.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:34:0) after 5m43s
[20:08:16.129] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:34:0)
[20:08:16.147] Build status: STARTED
[20:08:16.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:35:0) after 2m13s
[20:10:29.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:35:0)
[20:10:29.232] Build status: STARTED
[20:10:29.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:36:0) after 5m21s
[20:15:50.222] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:36:0)
[20:15:50.238] Build status: STARTED
[20:15:50.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:37:0) after 9m59s
[20:25:49.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:37:0)
[20:25:49.279] Build status: STARTED
[20:25:49.279] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:38:0) after 3m8s
[20:28:57.296] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:38:0)
[20:28:57.313] Build status: STARTED
[20:28:57.313] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:39:0) after 9m33s
[20:38:30.579] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:39:0)
[20:38:30.661] Build status: STARTED
[20:38:30.661] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:40:0) after 8m43s
[20:47:13.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:40:0)
[20:47:13.707] Build status: STARTED
[20:47:13.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:41:0) after 2m53s
[20:50:06.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:41:0)
[20:50:06.744] Build status: STARTED
[20:50:06.744] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:42:0) after 3m48s
[20:53:54.761] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:42:0)
[20:53:54.776] Build status: STARTED
[20:53:54.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:43:0) after 9m7s
[21:03:01.930] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:43:0)
[21:03:01.950] Build status: STARTED
[21:03:01.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:44:0) after 2m9s
[21:05:10.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:44:0)
[21:05:10.978] Build status: STARTED
[21:05:10.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:45:0) after 1m28s
[21:06:39.108] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:45:0)
[21:06:39.129] Build status: STARTED
[21:06:39.129] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:46:0) after 2m13s
[21:08:52.146] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:46:0)
[21:08:52.169] Build status: STARTED
[21:08:52.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:47:0) after 6m12s
[21:15:04.259] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:47:0)
[21:15:04.322] Build status: STARTED
[21:15:04.323] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:48:0) after 5m9s
[21:20:13.382] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:48:0)
[21:20:13.401] Build status: STARTED
[21:20:13.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:49:0) after 4m25s
[21:24:38.418] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:49:0)
[21:24:38.432] Build status: STARTED
[21:24:38.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:50:0) after 6m39s
[21:31:17.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:50:0)
[21:31:17.461] Build status: STARTED
[21:31:17.461] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:51:0) after 4m9s
[21:35:26.613] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:51:0)
[21:35:26.626] Build status: STARTED
[21:35:26.626] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:52:0) after 7m14s
[21:42:40.642] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:52:0)
[21:42:40.718] Build status: STARTED
[21:42:40.718] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:53:0) after 1m34s
[21:44:14.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:53:0)
[21:44:14.846] Build status: STARTED
[21:44:14.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:54:0) after 3m57s
[21:48:11.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:54:0)
[21:48:11.887] Build status: STARTED
[21:48:11.888] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:55:0) after 2m6s
[21:50:17.967] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:55:0)
[21:50:17.985] Build status: STARTED
[21:50:17.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:56:0) after 5m31s
[21:55:49.108] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:56:0)
[21:55:49.139] Build status: STARTED
[21:55:49.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:57:0) after 5m5s
[22:00:54.208] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:57:0)
[22:00:54.232] Build status: STARTED
[22:00:54.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850681124081381344:58:0) after 7m9s
[22:05:10.531] Received PubSub notification, asking Buildbucket for the build status
[22:05:10.596] Build:
{
"id": "8685251061817401905",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2403,
"createdBy": "project:chromium",
"createTime": "2026-04-06T17:22:33.012821175Z",
"startTime": "2026-04-06T17:23:16.552570530Z",
"endTime": "2026-04-06T22:05:09.948627500Z",
"updateTime": "2026-04-06T22:05:09.948627500Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "bc1007a6a3125f80f50e18c36d5cea744453a53d",
"ref": "refs/heads/main"
}
}
}
[22:05:10.596] Invocation finished in 4h42m39.127395691s with status FAILED