[02:15:29.068] New invocation is queued and will start shortly
[02:15:30.210] Starting the invocation (attempt 1)
[02:15:30.235] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/a348a486958f8fdeadb5993e8283f796d7979104"
[02:15:30.235] Popped gitiles commit info from properties and tags
[02:15:30.235] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[02:15:30.235] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[02:15:30.235] Buildbucket request:
{
"requestId": "8851553562815417792",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a348a486958f8fdeadb5993e8283f796d7979104",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851553562815417792"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[02:15:30.797] Scheduled build:
{
"id": "8686123500861225041",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2354,
"createdBy": "project:chromium",
"createTime": "2026-03-28T02:15:30.303759882Z",
"updateTime": "2026-03-28T02:15:30.303759882Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a348a486958f8fdeadb5993e8283f796d7979104",
"ref": "refs/heads/main"
}
}
}
[02:15:30.797] Task URL: https://cr-buildbucket.appspot.com/build/8686123500861225041
[02:15:30.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:2:0) after 2m31s
[02:15:31.837] Received PubSub notification, asking Buildbucket for the build status
[02:15:31.856] Build status: SCHEDULED
[02:16:11.391] Received PubSub notification, asking Buildbucket for the build status
[02:16:11.427] Build status: STARTED
[02:18:01.810] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:2:0)
[02:18:01.826] Build status: STARTED
[02:18:01.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:5:0) after 5m1s
[02:23:02.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:5:0)
[02:23:02.950] Build status: STARTED
[02:23:02.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:6:0) after 9m14s
[02:32:16.966] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:6:0)
[02:32:16.984] Build status: STARTED
[02:32:16.984] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:7:0) after 3m23s
[02:35:40.009] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:7:0)
[02:35:40.039] Build status: STARTED
[02:35:40.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:8:0) after 4m43s
[02:40:23.058] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:8:0)
[02:40:23.078] Build status: STARTED
[02:40:23.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:9:0) after 9m3s
[02:49:26.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:9:0)
[02:49:26.118] Build status: STARTED
[02:49:26.118] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:10:0) after 6m36s
[02:56:02.236] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:10:0)
[02:56:02.265] Build status: STARTED
[02:56:02.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:11:0) after 5m37s
[03:01:39.301] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:11:0)
[03:01:39.332] Build status: STARTED
[03:01:39.332] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:12:0) after 3m26s
[03:05:05.335] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:12:0)
[03:05:05.386] Build status: STARTED
[03:05:05.386] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:13:0) after 2m7s
[03:07:12.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:13:0)
[03:07:12.471] Build status: STARTED
[03:07:12.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:14:0) after 2m37s
[03:09:49.488] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:14:0)
[03:09:49.594] Build status: STARTED
[03:09:49.594] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:15:0) after 8m54s
[03:18:43.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:15:0)
[03:18:43.633] Build status: STARTED
[03:18:43.633] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:16:0) after 3m28s
[03:22:11.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:16:0)
[03:22:11.710] Build status: STARTED
[03:22:11.710] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:17:0) after 5m7s
[03:27:18.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:17:0)
[03:27:18.746] Build status: STARTED
[03:27:18.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:18:0) after 5m22s
[03:32:40.848] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:18:0)
[03:32:40.866] Build status: STARTED
[03:32:40.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:19:0) after 3m12s
[03:35:52.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:19:0)
[03:35:52.969] Build status: STARTED
[03:35:52.969] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:20:0) after 3m14s
[03:39:07.018] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:20:0)
[03:39:07.039] Build status: STARTED
[03:39:07.039] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:21:0) after 6m3s
[03:45:10.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:21:0)
[03:45:10.053] Build status: STARTED
[03:45:10.053] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:22:0) after 5m45s
[03:50:55.100] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:22:0)
[03:50:55.122] Build status: STARTED
[03:50:55.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:23:0) after 4m53s
[03:55:48.183] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:23:0)
[03:55:48.209] Build status: STARTED
[03:55:48.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:24:0) after 7m29s
[04:03:17.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:24:0)
[04:03:17.295] Build status: STARTED
[04:03:17.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:25:0) after 3m59s
[04:07:16.329] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:25:0)
[04:07:16.402] Build status: STARTED
[04:07:16.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:26:0) after 7m19s
[04:14:35.468] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:26:0)
[04:14:35.488] Build status: STARTED
[04:14:35.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:27:0) after 5m31s
[04:20:06.505] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:27:0)
[04:20:06.521] Build status: STARTED
[04:20:06.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:28:0) after 3m44s
[04:23:50.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:28:0)
[04:23:50.666] Build status: STARTED
[04:23:50.666] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:29:0) after 3m25s
[04:27:15.689] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:29:0)
[04:27:15.716] Build status: STARTED
[04:27:15.717] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:30:0) after 4m24s
[04:31:39.734] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:30:0)
[04:31:39.767] Build status: STARTED
[04:31:39.767] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:31:0) after 7m40s
[04:39:19.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:31:0)
[04:39:19.817] Build status: STARTED
[04:39:19.817] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:32:0) after 1m3s
[04:40:22.840] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:32:0)
[04:40:22.893] Build status: STARTED
[04:40:22.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:33:0) after 5m55s
[04:46:17.957] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:33:0)
[04:46:17.986] Build status: STARTED
[04:46:17.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:34:0) after 3m55s
[04:50:13.011] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:34:0)
[04:50:13.032] Build status: STARTED
[04:50:13.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:35:0) after 2m17s
[04:52:30.049] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:35:0)
[04:52:30.064] Build status: STARTED
[04:52:30.064] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:36:0) after 7m56s
[05:00:26.085] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:36:0)
[05:00:26.110] Build status: STARTED
[05:00:26.110] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:37:0) after 2m40s
[05:03:06.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:37:0)
[05:03:06.139] Build status: STARTED
[05:03:06.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:38:0) after 9m3s
[05:12:09.208] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:38:0)
[05:12:09.242] Build status: STARTED
[05:12:09.242] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:39:0) after 2m54s
[05:15:03.296] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:39:0)
[05:15:03.315] Build status: STARTED
[05:15:03.315] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:40:0) after 7m43s
[05:22:46.335] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:40:0)
[05:22:46.349] Build status: STARTED
[05:22:46.349] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:41:0) after 4m14s
[05:27:00.419] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:41:0)
[05:27:00.440] Build status: STARTED
[05:27:00.440] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:42:0) after 8m48s
[05:35:48.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:42:0)
[05:35:48.503] Build status: STARTED
[05:35:48.503] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:43:0) after 5m49s
[05:41:37.581] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:43:0)
[05:41:37.605] Build status: STARTED
[05:41:37.606] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:44:0) after 9m44s
[05:51:21.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:44:0)
[05:51:21.663] Build status: STARTED
[05:51:21.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:45:0) after 1m31s
[05:52:52.680] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:45:0)
[05:52:52.705] Build status: STARTED
[05:52:52.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:46:0) after 6m33s
[05:59:25.724] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:46:0)
[05:59:25.743] Build status: STARTED
[05:59:25.743] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:47:0) after 8m28s
[06:07:53.764] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:47:0)
[06:07:53.784] Build status: STARTED
[06:07:53.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:48:0) after 5m6s
[06:12:59.800] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:48:0)
[06:12:59.817] Build status: STARTED
[06:12:59.817] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:49:0) after 7m33s
[06:20:32.911] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:49:0)
[06:20:32.936] Build status: STARTED
[06:20:32.936] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:50:0) after 2m4s
[06:22:37.002] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:50:0)
[06:22:37.015] Build status: STARTED
[06:22:37.015] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:51:0) after 8m24s
[06:31:01.033] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:51:0)
[06:31:01.060] Build status: STARTED
[06:31:01.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:52:0) after 5m59s
[06:37:00.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:52:0)
[06:37:00.148] Build status: STARTED
[06:37:00.148] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:53:0) after 4m18s
[06:41:18.138] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:53:0)
[06:41:18.156] Build status: STARTED
[06:41:18.156] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:54:0) after 7m19s
[06:48:37.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:54:0)
[06:48:37.195] Build status: STARTED
[06:48:37.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:55:0) after 6m15s
[06:54:52.220] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:55:0)
[06:54:52.240] Build status: STARTED
[06:54:52.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851553562815417792:56:0) after 6m28s
[06:58:27.042] Received PubSub notification, asking Buildbucket for the build status
[06:58:27.060] Build:
{
"id": "8686123500861225041",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2354,
"createdBy": "project:chromium",
"createTime": "2026-03-28T02:15:30.303759882Z",
"startTime": "2026-03-28T02:16:11.180809861Z",
"endTime": "2026-03-28T06:58:26.351767670Z",
"updateTime": "2026-03-28T06:58:26.351767670Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a348a486958f8fdeadb5993e8283f796d7979104",
"ref": "refs/heads/main"
}
}
}
[06:58:27.060] Invocation finished in 4h42m58.004735248s with status FAILED