[15:07:45.792] New invocation is queued and will start shortly
[15:07:47.302] Starting the invocation (attempt 1)
[15:07:47.322] Ignoring gitiles_ref tag without the buildset tag
[15:07:47.322] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[15:07:47.322] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[15:07:47.322] Buildbucket request:
{
"requestId": "8851686169170028592",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "mac-ubsan-fyi-rel"
},
"properties": {
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851686169170028592"
},
{
"key": "scheduler_job_id",
"value": "chromium/mac-ubsan-fyi-rel"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[15:07:48.312] Scheduled build:
{
"id": "8686256106597196593",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "mac-ubsan-fyi-rel"
},
"number": 638,
"createdBy": "project:chromium",
"createTime": "2026-03-26T15:07:47.619306458Z",
"updateTime": "2026-03-26T15:07:47.619306458Z",
"status": "SCHEDULED",
"input": {
}
}
[15:07:48.312] Task URL: https://cr-buildbucket.appspot.com/build/8686256106597196593
[15:07:48.312] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:2:0) after 6m38s
[15:07:49.196] Received PubSub notification, asking Buildbucket for the build status
[15:07:49.216] Build status: SCHEDULED
[15:07:59.302] Received PubSub notification, asking Buildbucket for the build status
[15:07:59.317] Build status: STARTED
[15:14:26.433] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:2:0)
[15:14:26.449] Build status: STARTED
[15:14:26.449] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:5:0) after 9m51s
[15:24:17.528] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:5:0)
[15:24:17.540] Build status: STARTED
[15:24:17.540] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:6:0) after 9m59s
[15:34:16.652] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:6:0)
[15:34:16.674] Build status: STARTED
[15:34:16.674] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:7:0) after 6m24s
[15:40:40.776] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:7:0)
[15:40:40.796] Build status: STARTED
[15:40:40.796] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:8:0) after 1m0s
[15:41:40.818] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:8:0)
[15:41:40.839] Build status: STARTED
[15:41:40.839] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:9:0) after 7m27s
[15:49:07.851] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:9:0)
[15:49:07.870] Build status: STARTED
[15:49:07.870] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:10:0) after 2m35s
[15:51:42.883] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:10:0)
[15:51:43.016] Build status: STARTED
[15:51:43.016] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:11:0) after 2m56s
[15:54:39.124] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:11:0)
[15:54:39.180] Build status: STARTED
[15:54:39.180] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:12:0) after 1m10s
[15:55:49.258] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:12:0)
[15:55:49.289] Build status: STARTED
[15:55:49.289] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:13:0) after 5m24s
[16:01:13.406] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:13:0)
[16:01:13.475] Build status: STARTED
[16:01:13.475] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:14:0) after 2m41s
[16:03:54.518] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:14:0)
[16:03:54.536] Build status: STARTED
[16:03:54.536] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:15:0) after 7m56s
[16:11:50.566] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:15:0)
[16:11:50.652] Build status: STARTED
[16:11:50.652] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:16:0) after 1m2s
[16:12:52.668] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:16:0)
[16:12:52.729] Build status: STARTED
[16:12:52.729] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:17:0) after 2m58s
[16:15:50.818] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:17:0)
[16:15:50.894] Build status: STARTED
[16:15:50.894] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:18:0) after 3m24s
[16:19:14.907] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:18:0)
[16:19:14.928] Build status: STARTED
[16:19:14.928] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:19:0) after 6m8s
[16:25:22.955] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:19:0)
[16:25:22.994] Build status: STARTED
[16:25:22.994] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:20:0) after 7m51s
[16:33:14.008] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:20:0)
[16:33:14.030] Build status: STARTED
[16:33:14.030] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:21:0) after 6m48s
[16:40:02.050] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:21:0)
[16:40:02.090] Build status: STARTED
[16:40:02.090] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:22:0) after 7m55s
[16:47:57.110] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:22:0)
[16:47:57.265] Build status: STARTED
[16:47:57.265] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:23:0) after 5m2s
[16:52:59.618] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:23:0)
[16:52:59.681] Build status: STARTED
[16:52:59.681] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:24:0) after 2m41s
[16:55:40.919] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:24:0)
[16:55:40.943] Build status: STARTED
[16:55:40.943] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:25:0) after 8m2s
[17:03:43.011] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:25:0)
[17:03:43.038] Build status: STARTED
[17:03:43.038] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:26:0) after 5m53s
[17:09:36.056] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:26:0)
[17:09:36.178] Build status: STARTED
[17:09:36.178] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:27:0) after 6m49s
[17:16:25.219] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:27:0)
[17:16:25.237] Build status: STARTED
[17:16:25.237] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:28:0) after 8m33s
[17:24:58.250] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:28:0)
[17:24:58.268] Build status: STARTED
[17:24:58.268] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:29:0) after 2m43s
[17:27:41.299] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:29:0)
[17:27:41.318] Build status: STARTED
[17:27:41.318] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:30:0) after 3m49s
[17:31:30.490] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:30:0)
[17:31:30.517] Build status: STARTED
[17:31:30.518] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:31:0) after 6m40s
[17:38:10.576] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:31:0)
[17:38:10.594] Build status: STARTED
[17:38:10.594] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:32:0) after 6m34s
[17:44:44.658] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:32:0)
[17:44:44.682] Build status: STARTED
[17:44:44.682] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:33:0) after 3m23s
[17:48:07.652] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:33:0)
[17:48:07.669] Build status: STARTED
[17:48:07.669] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:34:0) after 7m59s
[17:56:06.686] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:34:0)
[17:56:06.759] Build status: STARTED
[17:56:06.759] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:35:0) after 6m40s
[18:02:46.897] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:35:0)
[18:02:46.929] Build status: STARTED
[18:02:46.929] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:36:0) after 5m4s
[18:07:50.961] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:36:0)
[18:07:50.978] Build status: STARTED
[18:07:50.978] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:37:0) after 3m40s
[18:11:31.272] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:37:0)
[18:11:31.292] Build status: STARTED
[18:11:31.292] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:38:0) after 7m31s
[18:19:02.430] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:38:0)
[18:19:02.458] Build status: STARTED
[18:19:02.458] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:39:0) after 4m52s
[18:23:54.549] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:39:0)
[18:23:54.569] Build status: STARTED
[18:23:54.569] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:40:0) after 5m28s
[18:29:22.553] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:40:0)
[18:29:22.578] Build status: STARTED
[18:29:22.578] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:41:0) after 6m34s
[18:35:56.644] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:41:0)
[18:35:56.670] Build status: STARTED
[18:35:56.670] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:42:0) after 2m10s
[18:38:06.727] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:42:0)
[18:38:06.743] Build status: STARTED
[18:38:06.743] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:43:0) after 3m42s
[18:41:48.764] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:43:0)
[18:41:48.781] Build status: STARTED
[18:41:48.781] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:44:0) after 4m8s
[18:45:56.796] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:44:0)
[18:45:56.893] Build status: STARTED
[18:45:56.893] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:45:0) after 9m4s
[18:55:01.001] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:45:0)
[18:55:01.017] Build status: STARTED
[18:55:01.017] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:46:0) after 2m32s
[18:57:33.038] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:46:0)
[18:57:33.059] Build status: STARTED
[18:57:33.059] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:47:0) after 9m6s
[19:06:39.244] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:47:0)
[19:06:39.261] Build status: STARTED
[19:06:39.261] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:48:0) after 4m16s
[19:10:55.289] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:48:0)
[19:10:55.314] Build status: STARTED
[19:10:55.314] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:49:0) after 6m13s
[19:17:08.417] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:49:0)
[19:17:08.441] Build status: STARTED
[19:17:08.441] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:50:0) after 1m18s
[19:18:26.537] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:50:0)
[19:18:26.553] Build status: STARTED
[19:18:26.553] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:51:0) after 2m30s
[19:20:56.678] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:51:0)
[19:20:56.707] Build status: STARTED
[19:20:56.707] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:52:0) after 7m6s
[19:28:02.753] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:52:0)
[19:28:02.779] Build status: STARTED
[19:28:02.779] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:53:0) after 8m4s
[19:36:06.826] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:53:0)
[19:36:06.954] Build status: STARTED
[19:36:06.954] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:54:0) after 8m44s
[19:44:51.158] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:54:0)
[19:44:51.197] Build status: STARTED
[19:44:51.197] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:55:0) after 7m15s
[19:52:06.423] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:55:0)
[19:52:06.438] Build status: STARTED
[19:52:06.438] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:56:0) after 9m44s
[20:01:50.525] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:56:0)
[20:01:50.546] Build status: STARTED
[20:01:50.546] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:57:0) after 2m42s
[20:04:32.546] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:57:0)
[20:04:32.563] Build status: STARTED
[20:04:32.563] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:58:0) after 9m18s
[20:13:50.612] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:58:0)
[20:13:50.626] Build status: STARTED
[20:13:50.626] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:59:0) after 1m32s
[20:15:22.843] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:59:0)
[20:15:22.895] Build status: STARTED
[20:15:22.895] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:60:0) after 3m45s
[20:19:07.949] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:60:0)
[20:19:07.965] Build status: STARTED
[20:19:07.965] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:61:0) after 6m2s
[20:25:10.067] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:61:0)
[20:25:10.132] Build status: STARTED
[20:25:10.132] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:62:0) after 3m10s
[20:28:20.181] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:62:0)
[20:28:20.222] Build status: STARTED
[20:28:20.222] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:63:0) after 5m13s
[20:33:33.263] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:63:0)
[20:33:33.286] Build status: STARTED
[20:33:33.286] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:64:0) after 4m24s
[20:37:57.305] Handling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:64:0)
[20:37:57.322] Build status: STARTED
[20:37:57.322] Scheduling timer "check-buildbucket-build-status" (chromium/mac-ubsan-fyi-rel:8851686169170028592:65:0) after 8m57s
[20:41:23.639] Received PubSub notification, asking Buildbucket for the build status
[20:41:23.660] Build:
{
"id": "8686256106597196593",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "mac-ubsan-fyi-rel"
},
"number": 638,
"createdBy": "project:chromium",
"createTime": "2026-03-26T15:07:47.619306458Z",
"startTime": "2026-03-26T15:07:58.944603412Z",
"endTime": "2026-03-26T20:41:23.213897967Z",
"updateTime": "2026-03-26T20:41:23.213897967Z",
"status": "FAILURE",
"input": {
}
}
[20:41:23.660] Invocation finished in 5h33m37.877656676s with status FAILED