[18:50:04.606] New invocation is queued and will start shortly
[18:50:04.606] Triggered by user:haowei@google.com
[18:50:05.705] Starting the invocation (attempt 1)
[18:50:05.734] Ignoring gitiles_ref tag without the buildset tag
[18:50:05.734] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[18:50:05.734] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[18:50:05.734] Buildbucket request:
{
"requestId": "8865261727365232304",
"builder": {
"project": "fuchsia",
"bucket": "prod",
"builder": "clang-ml-training"
},
"properties": {
"$recipe_engine/scheduler": {
"hostname": "luci-scheduler.appspot.com",
"invocation": "8865261727365232304",
"job": "fuchsia/prod-clang-ml-training",
"triggers": [
{
"id": "f89e8c2b969789f4",
"title": "Triggered via web UI",
"webui": {
}
}
]
}
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8865261727365232304"
},
{
"key": "scheduler_job_id",
"value": "fuchsia/prod-clang-ml-training"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[18:50:06.125] Scheduled build:
{
"id": "8699831665423601937",
"builder": {
"project": "fuchsia",
"bucket": "prod",
"builder": "clang-ml-training"
},
"createdBy": "project:fuchsia",
"createTime": "2025-10-27T18:50:05.835403884Z",
"updateTime": "2025-10-27T18:50:05.835403884Z",
"status": "SCHEDULED",
"input": {
}
}
[18:50:06.125] Task URL: https://cr-buildbucket.appspot.com/build/8699831665423601937
[18:50:06.125] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:2:0) after 7m17s
[18:50:07.067] Received PubSub notification, asking Buildbucket for the build status
[18:50:07.087] Build status: SCHEDULED
[18:50:09.473] Received PubSub notification, asking Buildbucket for the build status
[18:50:09.494] Build status: STARTED
[18:57:23.199] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:2:0)
[18:57:23.218] Build status: STARTED
[18:57:23.218] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:5:0) after 4m16s
[19:01:39.237] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:5:0)
[19:01:39.267] Build status: STARTED
[19:01:39.267] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:6:0) after 1m56s
[19:03:35.284] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:6:0)
[19:03:35.310] Build status: STARTED
[19:03:35.310] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:7:0) after 2m53s
[19:06:28.349] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:7:0)
[19:06:28.406] Build status: STARTED
[19:06:28.406] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:8:0) after 2m2s
[19:08:30.543] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:8:0)
[19:08:30.560] Build status: STARTED
[19:08:30.560] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:9:0) after 5m15s
[19:13:45.622] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:9:0)
[19:13:45.642] Build status: STARTED
[19:13:45.643] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:10:0) after 9m38s
[19:23:23.724] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:10:0)
[19:23:23.738] Build status: STARTED
[19:23:23.738] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:11:0) after 4m7s
[19:27:30.852] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:11:0)
[19:27:30.868] Build status: STARTED
[19:27:30.868] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:12:0) after 3m45s
[19:31:15.983] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:12:0)
[19:31:16.034] Build status: STARTED
[19:31:16.034] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:13:0) after 3m9s
[19:34:25.050] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:13:0)
[19:34:25.190] Build status: STARTED
[19:34:25.190] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:14:0) after 5m9s
[19:39:34.217] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:14:0)
[19:39:34.269] Build status: STARTED
[19:39:34.269] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:15:0) after 5m17s
[19:44:51.283] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:15:0)
[19:44:51.301] Build status: STARTED
[19:44:51.302] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:16:0) after 6m10s
[19:51:01.297] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:16:0)
[19:51:01.403] Build status: STARTED
[19:51:01.403] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:17:0) after 6m36s
[19:57:37.419] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:17:0)
[19:57:37.434] Build status: STARTED
[19:57:37.434] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:18:0) after 1m16s
[19:58:53.439] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:18:0)
[19:58:53.456] Build status: STARTED
[19:58:53.456] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:19:0) after 3m16s
[20:02:09.503] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:19:0)
[20:02:09.521] Build status: STARTED
[20:02:09.522] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:20:0) after 7m22s
[20:09:31.506] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:20:0)
[20:09:31.523] Build status: STARTED
[20:09:31.523] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:21:0) after 2m11s
[20:11:42.559] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:21:0)
[20:11:42.576] Build status: STARTED
[20:11:42.576] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:22:0) after 6m32s
[20:18:14.656] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:22:0)
[20:18:14.679] Build status: STARTED
[20:18:14.679] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:23:0) after 4m11s
[20:22:25.697] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:23:0)
[20:22:25.715] Build status: STARTED
[20:22:25.715] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:24:0) after 8m33s
[20:30:58.797] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:24:0)
[20:30:58.822] Build status: STARTED
[20:30:58.822] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:25:0) after 5m0s
[20:35:59.007] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:25:0)
[20:35:59.029] Build status: STARTED
[20:35:59.029] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:26:0) after 7m47s
[20:43:46.042] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:26:0)
[20:43:46.083] Build status: STARTED
[20:43:46.083] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:27:0) after 4m22s
[20:48:08.151] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:27:0)
[20:48:08.169] Build status: STARTED
[20:48:08.169] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:28:0) after 8m24s
[20:56:32.355] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:28:0)
[20:56:32.380] Build status: STARTED
[20:56:32.381] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:29:0) after 7m12s
[21:03:44.443] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:29:0)
[21:03:44.498] Build status: STARTED
[21:03:44.499] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:30:0) after 4m39s
[21:08:23.594] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:30:0)
[21:08:23.623] Build status: STARTED
[21:08:23.623] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:31:0) after 9m47s
[21:18:10.675] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:31:0)
[21:18:10.694] Build status: STARTED
[21:18:10.694] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:32:0) after 1m55s
[21:20:05.716] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:32:0)
[21:20:05.737] Build status: STARTED
[21:20:05.737] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:33:0) after 2m2s
[21:22:07.791] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:33:0)
[21:22:07.829] Build status: STARTED
[21:22:07.829] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:34:0) after 3m48s
[21:25:55.899] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:34:0)
[21:25:55.917] Build status: STARTED
[21:25:55.917] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:35:0) after 1m9s
[21:27:04.884] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:35:0)
[21:27:04.946] Build status: STARTED
[21:27:04.947] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:36:0) after 2m34s
[21:29:38.960] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:36:0)
[21:29:38.973] Build status: STARTED
[21:29:38.973] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:37:0) after 5m41s
[21:35:19.990] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:37:0)
[21:35:20.045] Build status: STARTED
[21:35:20.046] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:38:0) after 4m16s
[21:39:36.070] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:38:0)
[21:39:36.098] Build status: STARTED
[21:39:36.098] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:39:0) after 2m56s
[21:42:32.165] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:39:0)
[21:42:32.193] Build status: STARTED
[21:42:32.193] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:40:0) after 2m34s
[21:45:06.177] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:40:0)
[21:45:06.213] Build status: STARTED
[21:45:06.213] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:41:0) after 5m50s
[21:50:56.205] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:41:0)
[21:50:56.219] Build status: STARTED
[21:50:56.219] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:42:0) after 3m1s
[21:53:57.250] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:42:0)
[21:53:57.273] Build status: STARTED
[21:53:57.273] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:43:0) after 7m30s
[22:01:27.336] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:43:0)
[22:01:27.365] Build status: STARTED
[22:01:27.365] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:44:0) after 8m55s
[22:10:22.383] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:44:0)
[22:10:22.405] Build status: STARTED
[22:10:22.405] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:45:0) after 7m54s
[22:18:16.452] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:45:0)
[22:18:16.479] Build status: STARTED
[22:18:16.479] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:46:0) after 5m41s
[22:23:57.537] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:46:0)
[22:23:57.554] Build status: STARTED
[22:23:57.554] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:47:0) after 3m44s
[22:27:41.635] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:47:0)
[22:27:41.660] Build status: STARTED
[22:27:41.660] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:48:0) after 7m5s
[22:34:46.768] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:48:0)
[22:34:46.784] Build status: STARTED
[22:34:46.784] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:49:0) after 2m24s
[22:37:10.816] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:49:0)
[22:37:10.831] Build status: STARTED
[22:37:10.831] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:50:0) after 2m16s
[22:39:26.851] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:50:0)
[22:39:26.867] Build status: STARTED
[22:39:26.867] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:51:0) after 8m24s
[22:47:50.877] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:51:0)
[22:47:50.896] Build status: STARTED
[22:47:50.896] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:52:0) after 2m58s
[22:50:48.996] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:52:0)
[22:50:49.013] Build status: STARTED
[22:50:49.013] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:53:0) after 5m50s
[22:56:39.030] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:53:0)
[22:56:39.048] Build status: STARTED
[22:56:39.048] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:54:0) after 9m42s
[23:06:21.079] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:54:0)
[23:06:21.096] Build status: STARTED
[23:06:21.096] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:55:0) after 3m16s
[23:09:37.159] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:55:0)
[23:09:37.176] Build status: STARTED
[23:09:37.176] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:56:0) after 2m59s
[23:12:36.187] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:56:0)
[23:12:36.203] Build status: STARTED
[23:12:36.203] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:57:0) after 6m1s
[23:18:37.241] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:57:0)
[23:18:37.273] Build status: STARTED
[23:18:37.273] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:58:0) after 8m31s
[23:27:08.287] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:58:0)
[23:27:08.312] Build status: STARTED
[23:27:08.312] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:59:0) after 5m49s
[23:32:57.394] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:59:0)
[23:32:57.413] Build status: STARTED
[23:32:57.413] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:60:0) after 2m38s
[23:35:35.423] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:60:0)
[23:35:35.439] Build status: STARTED
[23:35:35.439] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:61:0) after 6m52s
[23:42:27.538] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:61:0)
[23:42:27.555] Build status: STARTED
[23:42:27.555] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:62:0) after 7m36s
[23:50:03.572] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:62:0)
[23:50:03.628] Build status: STARTED
[23:50:03.628] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:63:0) after 8m18s
[23:58:21.645] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:63:0)
[23:58:21.720] Build status: STARTED
[23:58:21.720] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:64:0) after 4m35s
[00:02:56.792] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:64:0)
[00:02:56.808] Build status: STARTED
[00:02:56.808] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:65:0) after 2m10s
[00:05:06.899] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:65:0)
[00:05:06.917] Build status: STARTED
[00:05:06.917] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:66:0) after 8m16s
[00:13:22.937] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:66:0)
[00:13:22.956] Build status: STARTED
[00:13:22.956] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:67:0) after 2m56s
[00:16:19.167] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:67:0)
[00:16:19.189] Build status: STARTED
[00:16:19.189] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:68:0) after 1m22s
[00:17:41.202] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:68:0)
[00:17:41.217] Build status: STARTED
[00:17:41.218] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:69:0) after 4m36s
[00:22:17.545] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:69:0)
[00:22:17.643] Build status: STARTED
[00:22:17.643] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:70:0) after 8m12s
[00:30:29.663] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:70:0)
[00:30:29.677] Build status: STARTED
[00:30:29.678] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:71:0) after 1m33s
[00:32:02.698] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:71:0)
[00:32:02.775] Build status: STARTED
[00:32:02.775] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:72:0) after 7m36s
[00:39:38.825] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:72:0)
[00:39:38.875] Build status: STARTED
[00:39:38.875] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:73:0) after 7m23s
[00:47:01.941] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:73:0)
[00:47:01.958] Build status: STARTED
[00:47:01.958] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:74:0) after 4m46s
[00:51:47.978] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:74:0)
[00:51:48.021] Build status: STARTED
[00:51:48.021] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:75:0) after 4m14s
[00:56:02.075] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:75:0)
[00:56:02.089] Build status: STARTED
[00:56:02.089] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:76:0) after 9m41s
[01:05:43.145] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:76:0)
[01:05:43.165] Build status: STARTED
[01:05:43.165] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:77:0) after 4m27s
[01:10:10.185] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:77:0)
[01:10:10.203] Build status: STARTED
[01:10:10.203] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:78:0) after 7m38s
[01:17:48.273] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:78:0)
[01:17:48.288] Build status: STARTED
[01:17:48.288] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:79:0) after 6m6s
[01:23:54.306] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:79:0)
[01:23:54.378] Build status: STARTED
[01:23:54.378] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:80:0) after 1m12s
[01:25:06.438] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:80:0)
[01:25:06.514] Build status: STARTED
[01:25:06.514] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:81:0) after 6m23s
[01:32:11.766] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:81:0)
[01:32:11.788] Build status: STARTED
[01:32:11.789] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:82:0) after 4m19s
[01:36:30.806] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:82:0)
[01:36:30.828] Build status: STARTED
[01:36:30.828] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:83:0) after 4m37s
[01:41:07.900] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:83:0)
[01:41:08.123] Build status: STARTED
[01:41:08.123] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:84:0) after 2m19s
[01:43:27.300] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:84:0)
[01:43:27.368] Build status: STARTED
[01:43:27.368] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:85:0) after 1m56s
[01:45:23.386] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:85:0)
[01:45:23.410] Build status: STARTED
[01:45:23.410] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:86:0) after 7m7s
[01:52:30.408] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:86:0)
[01:52:30.423] Build status: STARTED
[01:52:30.423] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:87:0) after 2m55s
[01:55:25.453] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:87:0)
[01:55:25.503] Build status: STARTED
[01:55:25.503] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:88:0) after 8m43s
[02:04:08.529] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:88:0)
[02:04:08.557] Build status: STARTED
[02:04:08.557] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:89:0) after 5m57s
[02:10:05.541] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:89:0)
[02:10:05.556] Build status: STARTED
[02:10:05.556] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:90:0) after 2m32s
[02:12:37.571] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:90:0)
[02:12:37.588] Build status: STARTED
[02:12:37.588] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:91:0) after 7m51s
[02:20:28.653] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:91:0)
[02:20:28.669] Build status: STARTED
[02:20:28.669] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:92:0) after 2m1s
[02:22:29.706] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:92:0)
[02:22:29.721] Build status: STARTED
[02:22:29.721] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:93:0) after 1m43s
[02:24:12.789] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:93:0)
[02:24:12.813] Build status: STARTED
[02:24:12.813] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:94:0) after 7m29s
[02:31:41.828] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:94:0)
[02:31:41.851] Build status: STARTED
[02:31:41.851] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:95:0) after 6m50s
[02:38:31.867] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:95:0)
[02:38:31.978] Build status: STARTED
[02:38:31.978] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:96:0) after 8m7s
[02:46:39.001] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:96:0)
[02:46:39.124] Build status: STARTED
[02:46:39.124] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:97:0) after 9m13s
[02:55:52.141] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:97:0)
[02:55:52.163] Build status: STARTED
[02:55:52.163] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:98:0) after 2m36s
[02:58:28.178] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:98:0)
[02:58:28.194] Build status: STARTED
[02:58:28.194] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:99:0) after 3m33s
[03:02:01.263] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:99:0)
[03:02:01.316] Build status: STARTED
[03:02:01.316] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:100:0) after 5m57s
[03:07:58.329] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:100:0)
[03:07:58.449] Build status: STARTED
[03:07:58.449] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:101:0) after 6m19s
[03:14:17.464] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:101:0)
[03:14:17.481] Build status: STARTED
[03:14:17.481] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:102:0) after 2m57s
[03:17:14.497] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:102:0)
[03:17:14.534] Build status: STARTED
[03:17:14.534] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:103:0) after 8m53s
[03:26:07.596] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:103:0)
[03:26:07.706] Build status: STARTED
[03:26:07.706] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:104:0) after 7m55s
[03:34:02.775] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:104:0)
[03:34:02.797] Build status: STARTED
[03:34:02.797] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:105:0) after 1m33s
[03:35:35.799] Handling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:105:0)
[03:35:35.816] Build status: STARTED
[03:35:35.817] Scheduling timer "check-buildbucket-build-status" (fuchsia/prod-clang-ml-training:8865261727365232304:106:0) after 1m12s
[03:35:59.873] Received PubSub notification, asking Buildbucket for the build status
[03:35:59.895] Build:
{
"id": "8699831665423601937",
"builder": {
"project": "fuchsia",
"bucket": "prod",
"builder": "clang-ml-training"
},
"createdBy": "project:fuchsia",
"createTime": "2025-10-27T18:50:05.835403884Z",
"startTime": "2025-10-27T18:50:09.327219791Z",
"endTime": "2025-10-28T03:35:59.489972900Z",
"updateTime": "2025-10-28T03:35:59.489972900Z",
"status": "INFRA_FAILURE",
"input": {
}
}
[03:35:59.895] Invocation finished in 8h45m55.294498852s with status FAILED