[05:49:57.155] New invocation is queued and will start shortly
[05:49:58.886] Starting the invocation (attempt 1)
[05:49:58.917] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/ee3568eb9ea18c756f071293e34f3e78acb26ab5"
[05:49:58.917] Popped gitiles commit info from properties and tags
[05:49:58.917] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[05:49:58.917] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[05:49:58.917] Buildbucket request:
{
"requestId": "8851358875707725872",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ee3568eb9ea18c756f071293e34f3e78acb26ab5",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851358875707725872"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[05:49:59.374] Scheduled build:
{
"id": "8685928813159139873",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2365,
"createdBy": "project:chromium",
"createTime": "2026-03-30T05:49:58.965515830Z",
"updateTime": "2026-03-30T05:49:58.965515830Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ee3568eb9ea18c756f071293e34f3e78acb26ab5",
"ref": "refs/heads/main"
}
}
}
[05:49:59.374] Task URL: https://cr-buildbucket.appspot.com/build/8685928813159139873
[05:49:59.374] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:2:0) after 6m47s
[05:50:00.342] Received PubSub notification, asking Buildbucket for the build status
[05:50:00.360] Build status: SCHEDULED
[05:50:38.630] Received PubSub notification, asking Buildbucket for the build status
[05:50:38.654] Build status: STARTED
[05:56:46.441] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:2:0)
[05:56:46.481] Build status: STARTED
[05:56:46.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:5:0) after 9m8s
[06:05:54.498] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:5:0)
[06:05:54.804] Build status: STARTED
[06:05:54.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:6:0) after 9m18s
[06:15:12.820] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:6:0)
[06:15:12.846] Build status: STARTED
[06:15:12.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:7:0) after 1m4s
[06:16:16.872] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:7:0)
[06:16:16.889] Build status: STARTED
[06:16:16.889] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:8:0) after 1m9s
[06:17:25.873] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:8:0)
[06:17:25.892] Build status: STARTED
[06:17:25.892] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:9:0) after 7m45s
[06:25:10.917] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:9:0)
[06:25:10.940] Build status: STARTED
[06:25:10.940] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:10:0) after 2m2s
[06:27:12.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:10:0)
[06:27:13.097] Build status: STARTED
[06:27:13.097] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:11:0) after 8m18s
[06:35:31.117] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:11:0)
[06:35:31.141] Build status: STARTED
[06:35:31.141] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:12:0) after 1m27s
[06:36:58.182] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:12:0)
[06:36:58.202] Build status: STARTED
[06:36:58.202] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:13:0) after 3m14s
[06:40:12.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:13:0)
[06:40:12.254] Build status: STARTED
[06:40:12.255] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:14:0) after 6m36s
[06:46:48.264] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:14:0)
[06:46:48.285] Build status: STARTED
[06:46:48.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:15:0) after 4m29s
[06:51:17.305] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:15:0)
[06:51:17.345] Build status: STARTED
[06:51:17.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:16:0) after 2m33s
[06:53:50.444] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:16:0)
[06:53:50.543] Build status: STARTED
[06:53:50.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:17:0) after 8m58s
[07:02:48.517] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:17:0)
[07:02:48.531] Build status: STARTED
[07:02:48.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:18:0) after 7m19s
[07:10:07.598] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:18:0)
[07:10:07.620] Build status: STARTED
[07:10:07.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:19:0) after 4m7s
[07:14:14.634] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:19:0)
[07:14:14.652] Build status: STARTED
[07:14:14.652] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:20:0) after 3m22s
[07:17:36.719] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:20:0)
[07:17:36.781] Build status: STARTED
[07:17:36.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:21:0) after 1m38s
[07:19:14.924] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:21:0)
[07:19:14.942] Build status: STARTED
[07:19:14.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:22:0) after 5m31s
[07:24:45.958] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:22:0)
[07:24:45.977] Build status: STARTED
[07:24:45.977] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:23:0) after 4m49s
[07:29:35.021] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:23:0)
[07:29:35.045] Build status: STARTED
[07:29:35.045] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:24:0) after 6m45s
[07:36:20.064] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:24:0)
[07:36:20.079] Build status: STARTED
[07:36:20.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:25:0) after 4m42s
[07:41:02.187] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:25:0)
[07:41:02.205] Build status: STARTED
[07:41:02.205] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:26:0) after 5m40s
[07:46:42.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:26:0)
[07:46:42.251] Build status: STARTED
[07:46:42.251] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:27:0) after 5m21s
[07:52:03.269] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:27:0)
[07:52:03.353] Build status: STARTED
[07:52:03.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:28:0) after 7m37s
[07:59:40.374] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:28:0)
[07:59:40.422] Build status: STARTED
[07:59:40.422] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:29:0) after 7m32s
[08:07:12.508] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:29:0)
[08:07:12.525] Build status: STARTED
[08:07:12.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:30:0) after 5m12s
[08:12:24.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:30:0)
[08:12:24.691] Build status: STARTED
[08:12:24.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:31:0) after 2m58s
[08:15:22.710] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:31:0)
[08:15:22.728] Build status: STARTED
[08:15:22.728] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:32:0) after 9m46s
[08:25:08.763] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:32:0)
[08:25:08.779] Build status: STARTED
[08:25:08.779] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:33:0) after 2m19s
[08:27:27.763] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:33:0)
[08:27:27.837] Build status: STARTED
[08:27:27.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:34:0) after 8m38s
[08:36:05.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:34:0)
[08:36:06.062] Build status: STARTED
[08:36:06.062] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:35:0) after 1m14s
[08:37:20.077] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:35:0)
[08:37:20.093] Build status: STARTED
[08:37:20.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:36:0) after 5m53s
[08:43:13.119] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:36:0)
[08:43:13.565] Build status: STARTED
[08:43:13.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:37:0) after 2m38s
[08:45:51.615] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:37:0)
[08:45:51.634] Build status: STARTED
[08:45:51.634] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:38:0) after 3m25s
[08:49:16.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:38:0)
[08:49:16.768] Build status: STARTED
[08:49:16.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:39:0) after 5m12s
[08:54:30.564] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:39:0)
[08:54:30.589] Build status: STARTED
[08:54:30.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:40:0) after 4m26s
[08:58:56.610] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:40:0)
[08:58:56.625] Build status: STARTED
[08:58:56.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:41:0) after 9m46s
[09:08:42.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:41:0)
[09:08:42.667] Build status: STARTED
[09:08:42.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:42:0) after 7m27s
[09:16:09.657] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:42:0)
[09:16:09.726] Build status: STARTED
[09:16:09.726] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:43:0) after 6m58s
[09:23:07.791] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:43:0)
[09:23:07.807] Build status: STARTED
[09:23:07.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:44:0) after 1m15s
[09:24:22.967] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:44:0)
[09:24:22.983] Build status: STARTED
[09:24:22.983] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:45:0) after 1m9s
[09:25:32.000] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:45:0)
[09:25:32.019] Build status: STARTED
[09:25:32.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:46:0) after 2m30s
[09:28:02.038] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:46:0)
[09:28:02.069] Build status: STARTED
[09:28:02.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:47:0) after 3m58s
[09:32:00.083] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:47:0)
[09:32:00.104] Build status: STARTED
[09:32:00.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:48:0) after 8m6s
[09:40:06.158] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:48:0)
[09:40:06.190] Build status: STARTED
[09:40:06.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:49:0) after 9m15s
[09:49:21.301] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:49:0)
[09:49:21.316] Build status: STARTED
[09:49:21.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:50:0) after 2m31s
[09:51:52.344] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:50:0)
[09:51:52.366] Build status: STARTED
[09:51:52.366] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:51:0) after 3m42s
[09:55:34.464] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:51:0)
[09:55:34.484] Build status: STARTED
[09:55:34.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:52:0) after 7m33s
[10:03:07.554] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:52:0)
[10:03:07.575] Build status: STARTED
[10:03:07.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:53:0) after 5m22s
[10:08:29.607] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:53:0)
[10:08:29.624] Build status: STARTED
[10:08:29.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:54:0) after 1m8s
[10:09:37.739] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:54:0)
[10:09:37.755] Build status: STARTED
[10:09:37.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:55:0) after 7m28s
[10:17:05.771] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:55:0)
[10:17:05.794] Build status: STARTED
[10:17:05.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:56:0) after 1m46s
[10:18:51.843] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:56:0)
[10:18:51.865] Build status: STARTED
[10:18:51.865] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:57:0) after 2m29s
[10:21:20.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:57:0)
[10:21:20.944] Build status: STARTED
[10:21:20.944] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:58:0) after 4m27s
[10:25:47.962] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:58:0)
[10:25:47.978] Build status: STARTED
[10:25:47.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851358875707725872:59:0) after 8m8s
[10:29:51.121] Received PubSub notification, asking Buildbucket for the build status
[10:29:51.141] Build:
{
"id": "8685928813159139873",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2365,
"createdBy": "project:chromium",
"createTime": "2026-03-30T05:49:58.965515830Z",
"startTime": "2026-03-30T05:50:38.398656797Z",
"endTime": "2026-03-30T10:29:50.878424687Z",
"updateTime": "2026-03-30T10:29:50.878424687Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ee3568eb9ea18c756f071293e34f3e78acb26ab5",
"ref": "refs/heads/main"
}
}
}
[10:29:51.141] Invocation finished in 4h39m53.990711393s with status FAILED