[15:17:26.095] New invocation is queued and will start shortly
[15:17:27.168] Starting the invocation (attempt 1)
[15:17:27.194] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/4fac6f01137cd470ed6b16f2b9c2a870a833c4be"
[15:17:27.194] Popped gitiles commit info from properties and tags
[15:17:27.194] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[15:17:27.194] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[15:17:27.194] Buildbucket request:
{
"requestId": "8849783024378719664",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "Mac ARM64 ASAN Release"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4fac6f01137cd470ed6b16f2b9c2a870a833c4be",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8849783024378719664"
},
{
"key": "scheduler_job_id",
"value": "chromium/Mac ARM64 ASAN Release"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[15:17:28.338] Scheduled build:
{
"id": "8684352962484073057",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "Mac ARM64 ASAN Release"
},
"number": 3278,
"createdBy": "project:chromium",
"createTime": "2026-04-16T15:17:27.280924717Z",
"updateTime": "2026-04-16T15:17:27.280924717Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4fac6f01137cd470ed6b16f2b9c2a870a833c4be",
"ref": "refs/heads/main"
}
}
}
[15:17:28.338] Task URL: https://cr-buildbucket.appspot.com/build/8684352962484073057
[15:17:28.338] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:2:0) after 5m54s
[15:17:28.789] Received PubSub notification, asking Buildbucket for the build status
[15:17:28.812] Build status: SCHEDULED
[15:23:22.357] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:2:0)
[15:23:22.379] Build status: SCHEDULED
[15:23:22.379] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:4:0) after 8m43s
[15:32:05.408] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:4:0)
[15:32:05.439] Build status: SCHEDULED
[15:32:05.439] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:5:0) after 6m30s
[15:38:35.546] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:5:0)
[15:38:35.635] Build status: SCHEDULED
[15:38:35.635] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:6:0) after 2m26s
[15:41:01.706] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:6:0)
[15:41:01.733] Build status: SCHEDULED
[15:41:01.733] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:7:0) after 4m57s
[15:45:58.772] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:7:0)
[15:45:58.792] Build status: SCHEDULED
[15:45:58.792] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:8:0) after 3m7s
[15:49:05.763] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:8:0)
[15:49:05.778] Build status: SCHEDULED
[15:49:05.778] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:9:0) after 6m6s
[15:55:11.795] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:9:0)
[15:55:11.829] Build status: SCHEDULED
[15:55:11.830] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:10:0) after 3m8s
[15:58:19.851] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:10:0)
[15:58:19.871] Build status: SCHEDULED
[15:58:19.871] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:11:0) after 7m30s
[16:05:50.076] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:11:0)
[16:05:50.122] Build status: SCHEDULED
[16:05:50.122] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:12:0) after 1m12s
[16:07:02.144] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:12:0)
[16:07:02.175] Build status: SCHEDULED
[16:07:02.175] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:13:0) after 6m45s
[16:13:47.219] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:13:0)
[16:13:47.238] Build status: SCHEDULED
[16:13:47.238] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:14:0) after 1m29s
[16:15:16.267] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:14:0)
[16:15:16.287] Build status: SCHEDULED
[16:15:16.287] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:15:0) after 4m44s
[16:20:00.287] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:15:0)
[16:20:00.302] Build status: SCHEDULED
[16:20:00.302] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:16:0) after 4m7s
[16:24:07.431] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:16:0)
[16:24:07.455] Build status: SCHEDULED
[16:24:07.455] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:17:0) after 3m59s
[16:28:06.524] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:17:0)
[16:28:06.540] Build status: SCHEDULED
[16:28:06.540] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:18:0) after 9m18s
[16:37:24.625] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:18:0)
[16:37:24.644] Build status: SCHEDULED
[16:37:24.644] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:19:0) after 8m55s
[16:46:19.668] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:19:0)
[16:46:19.685] Build status: SCHEDULED
[16:46:19.685] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:20:0) after 3m47s
[16:50:06.793] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:20:0)
[16:50:06.812] Build status: SCHEDULED
[16:50:06.812] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:21:0) after 3m24s
[16:53:30.832] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:21:0)
[16:53:30.851] Build status: SCHEDULED
[16:53:30.851] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:22:0) after 7m12s
[17:00:42.868] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:22:0)
[17:00:42.890] Build status: SCHEDULED
[17:00:42.890] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:23:0) after 7m37s
[17:08:19.930] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:23:0)
[17:08:19.946] Build status: SCHEDULED
[17:08:19.946] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:24:0) after 4m32s
[17:11:28.524] Received PubSub notification, asking Buildbucket for the build status
[17:11:28.546] Build status: STARTED
[17:12:52.142] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:24:0)
[17:12:52.164] Build status: STARTED
[17:12:52.164] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:26:0) after 8m42s
[17:21:34.202] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:26:0)
[17:21:34.284] Build status: STARTED
[17:21:34.284] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:27:0) after 6m11s
[17:27:45.307] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:27:0)
[17:27:45.328] Build status: STARTED
[17:27:45.328] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:28:0) after 2m3s
[17:29:48.362] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:28:0)
[17:29:48.380] Build status: STARTED
[17:29:48.380] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:29:0) after 2m53s
[17:32:41.431] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:29:0)
[17:32:41.460] Build status: STARTED
[17:32:41.460] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:30:0) after 6m10s
[17:38:51.731] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:30:0)
[17:38:51.748] Build status: STARTED
[17:38:51.748] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:31:0) after 7m55s
[17:46:46.800] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:31:0)
[17:46:46.833] Build status: STARTED
[17:46:46.833] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:32:0) after 8m18s
[17:55:04.956] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:32:0)
[17:55:04.975] Build status: STARTED
[17:55:04.975] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:33:0) after 6m59s
[18:02:03.992] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:33:0)
[18:02:04.036] Build status: STARTED
[18:02:04.036] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:34:0) after 4m22s
[18:06:26.107] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:34:0)
[18:06:26.138] Build status: STARTED
[18:06:26.139] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:35:0) after 4m25s
[18:10:51.413] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:35:0)
[18:10:51.457] Build status: STARTED
[18:10:51.457] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:36:0) after 3m29s
[18:14:20.526] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:36:0)
[18:14:20.547] Build status: STARTED
[18:14:20.547] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:37:0) after 4m33s
[18:18:53.648] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:37:0)
[18:18:53.665] Build status: STARTED
[18:18:53.665] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:38:0) after 7m16s
[18:26:09.706] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:38:0)
[18:26:09.729] Build status: STARTED
[18:26:09.729] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:39:0) after 5m55s
[18:32:04.766] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:39:0)
[18:32:04.796] Build status: STARTED
[18:32:04.796] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:40:0) after 7m22s
[18:39:26.824] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:40:0)
[18:39:26.839] Build status: STARTED
[18:39:26.839] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:41:0) after 5m25s
[18:44:51.892] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:41:0)
[18:44:51.908] Build status: STARTED
[18:44:51.908] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:42:0) after 2m54s
[18:47:45.973] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:42:0)
[18:47:45.992] Build status: STARTED
[18:47:45.992] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:43:0) after 3m30s
[18:51:16.018] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:43:0)
[18:51:16.033] Build status: STARTED
[18:51:16.033] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:44:0) after 6m30s
[18:57:46.106] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:44:0)
[18:57:46.129] Build status: STARTED
[18:57:46.129] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:45:0) after 8m13s
[19:05:59.220] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:45:0)
[19:05:59.319] Build status: STARTED
[19:05:59.319] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:46:0) after 6m55s
[19:12:54.317] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:46:0)
[19:12:54.347] Build status: STARTED
[19:12:54.347] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:47:0) after 4m55s
[19:17:49.498] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:47:0)
[19:17:49.516] Build status: STARTED
[19:17:49.516] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:48:0) after 9m49s
[19:27:38.537] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:48:0)
[19:27:38.585] Build status: STARTED
[19:27:38.586] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:49:0) after 6m34s
[19:34:12.603] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:49:0)
[19:34:12.641] Build status: STARTED
[19:34:12.641] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:50:0) after 1m11s
[19:35:23.654] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:50:0)
[19:35:23.688] Build status: STARTED
[19:35:23.688] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:51:0) after 8m59s
[19:44:22.707] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:51:0)
[19:44:22.773] Build status: STARTED
[19:44:22.773] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:52:0) after 8m10s
[19:52:32.792] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:52:0)
[19:52:32.813] Build status: STARTED
[19:52:32.813] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:53:0) after 1m39s
[19:54:11.885] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:53:0)
[19:54:11.901] Build status: STARTED
[19:54:11.901] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:54:0) after 7m53s
[20:02:04.971] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:54:0)
[20:02:05.005] Build status: STARTED
[20:02:05.005] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:55:0) after 5m46s
[20:07:51.029] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:55:0)
[20:07:51.052] Build status: STARTED
[20:07:51.052] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:56:0) after 3m27s
[20:11:18.096] Handling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:56:0)
[20:11:18.112] Build status: STARTED
[20:11:18.112] Scheduling timer "check-buildbucket-build-status" (chromium/Mac ARM64 ASAN Release:8849783024378719664:57:0) after 4m10s
[20:12:03.634] Received PubSub notification, asking Buildbucket for the build status
[20:12:03.688] Build:
{
"id": "8684352962484073057",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "Mac ARM64 ASAN Release"
},
"number": 3278,
"createdBy": "project:chromium",
"createTime": "2026-04-16T15:17:27.280924717Z",
"startTime": "2026-04-16T17:11:13.561274323Z",
"endTime": "2026-04-16T20:12:02.901491898Z",
"updateTime": "2026-04-16T20:12:02.901491898Z",
"status": "INFRA_FAILURE",
"statusDetails": {
"timeout": {
}
},
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "4fac6f01137cd470ed6b16f2b9c2a870a833c4be",
"ref": "refs/heads/main"
}
}
}
[20:12:03.688] Invocation finished in 4h54m37.598733367s with status FAILED