[20:40:28.898] New invocation is queued and will start shortly
[20:40:28.898] Triggered by user:rerunner@pigweed-service-accounts.iam.gserviceaccount.com
[20:40:29.968] Starting the invocation (attempt 1)
[20:40:29.996] Ignoring gitiles_ref tag without the buildset tag
[20:40:29.996] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[20:40:29.996] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[20:40:29.997] Buildbucket request:
{
"requestId": "8849219118239078160",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-rust"
},
"properties": {
"$recipe_engine/scheduler": {
"hostname": "luci-scheduler.appspot.com",
"invocation": "8849219118239078160",
"job": "pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust",
"triggers": [
{
"buildbucket": {
"tags": [
"parent_buildername:rerunner",
"user_agent:rerunner"
]
},
"id": "00704508-76c1-4a65-926d-3b12bd96cfd9",
"title": "rerunner/0",
"url": "https://cr-buildbucket.appspot.com/build/8683789296451822081"
}
]
}
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8849219118239078160"
},
{
"key": "scheduler_job_id",
"value": "pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust"
},
{
"key": "user_agent",
"value": "luci-scheduler"
},
{
"key": "parent_buildername",
"value": "rerunner"
},
{
"key": "user_agent",
"value": "rerunner"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[20:40:30.390] Scheduled build:
{
"id": "8683789056355473265",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-rust"
},
"createdBy": "project:pigweed",
"createTime": "2026-04-22T20:40:30.072796935Z",
"updateTime": "2026-04-22T20:40:30.072796935Z",
"status": "SCHEDULED",
"input": {
}
}
[20:40:30.390] Task URL: https://cr-buildbucket.appspot.com/build/8683789056355473265
[20:40:30.390] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:2:0) after 6m5s
[20:40:31.312] Received PubSub notification, asking Buildbucket for the build status
[20:40:31.335] Build status: SCHEDULED
[20:46:35.502] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:2:0)
[20:46:35.540] Build status: SCHEDULED
[20:46:35.540] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:4:0) after 4m49s
[20:51:24.553] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:4:0)
[20:51:24.611] Build status: SCHEDULED
[20:51:24.611] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:5:0) after 9m26s
[21:00:50.643] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:5:0)
[21:00:50.763] Build status: SCHEDULED
[21:00:50.763] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:6:0) after 2m32s
[21:03:22.775] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:6:0)
[21:03:22.816] Build status: SCHEDULED
[21:03:22.816] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:7:0) after 8m9s
[21:11:31.896] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:7:0)
[21:11:31.924] Build status: SCHEDULED
[21:11:31.924] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:8:0) after 1m28s
[21:12:59.937] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:8:0)
[21:12:59.987] Build status: SCHEDULED
[21:12:59.987] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:9:0) after 2m27s
[21:15:27.051] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:9:0)
[21:15:27.113] Build status: SCHEDULED
[21:15:27.113] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:10:0) after 6m58s
[21:22:25.294] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:10:0)
[21:22:25.362] Build status: SCHEDULED
[21:22:25.362] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:11:0) after 5m21s
[21:27:46.422] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:11:0)
[21:27:46.439] Build status: SCHEDULED
[21:27:46.439] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:12:0) after 6m18s
[21:34:04.477] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:12:0)
[21:34:04.499] Build status: SCHEDULED
[21:34:04.499] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:13:0) after 8m54s
[21:42:58.520] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:13:0)
[21:42:58.539] Build status: SCHEDULED
[21:42:58.540] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:14:0) after 2m46s
[21:45:44.551] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:14:0)
[21:45:44.566] Build status: SCHEDULED
[21:45:44.566] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:15:0) after 8m33s
[21:54:17.643] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:15:0)
[21:54:17.661] Build status: SCHEDULED
[21:54:17.661] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:16:0) after 9m49s
[22:04:06.777] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:16:0)
[22:04:06.797] Build status: SCHEDULED
[22:04:06.797] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:17:0) after 7m39s
[22:11:45.892] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:17:0)
[22:11:45.909] Build status: SCHEDULED
[22:11:45.909] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:18:0) after 4m6s
[22:15:51.924] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:18:0)
[22:15:51.942] Build status: SCHEDULED
[22:15:51.942] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:19:0) after 1m17s
[22:17:09.008] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:19:0)
[22:17:09.051] Build status: SCHEDULED
[22:17:09.051] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:20:0) after 7m3s
[22:24:12.051] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:20:0)
[22:24:12.068] Build status: SCHEDULED
[22:24:12.068] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:21:0) after 4m27s
[22:28:39.051] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:21:0)
[22:28:39.071] Build status: SCHEDULED
[22:28:39.071] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:22:0) after 6m42s
[22:35:21.082] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:22:0)
[22:35:21.099] Build status: SCHEDULED
[22:35:21.099] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:23:0) after 6m58s
[22:42:19.121] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:23:0)
[22:42:19.222] Build status: SCHEDULED
[22:42:19.222] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:24:0) after 5m55s
[22:48:14.457] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:24:0)
[22:48:14.510] Build status: SCHEDULED
[22:48:14.510] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:25:0) after 1m14s
[22:49:28.488] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:25:0)
[22:49:28.508] Build status: SCHEDULED
[22:49:28.508] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:26:0) after 8m30s
[22:57:58.780] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:26:0)
[22:57:58.796] Build status: SCHEDULED
[22:57:58.796] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:27:0) after 5m22s
[23:03:20.809] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:27:0)
[23:03:20.834] Build status: SCHEDULED
[23:03:20.834] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:28:0) after 1m54s
[23:05:14.873] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:28:0)
[23:05:14.896] Build status: SCHEDULED
[23:05:14.896] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:29:0) after 9m36s
[23:14:50.913] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:29:0)
[23:14:50.940] Build status: SCHEDULED
[23:14:50.940] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:30:0) after 6m25s
[23:21:15.978] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:30:0)
[23:21:15.998] Build status: SCHEDULED
[23:21:15.998] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:31:0) after 4m58s
[23:26:14.033] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:31:0)
[23:26:14.054] Build status: SCHEDULED
[23:26:14.054] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:32:0) after 1m8s
[23:27:22.074] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:32:0)
[23:27:22.159] Build status: SCHEDULED
[23:27:22.159] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:33:0) after 3m22s
[23:30:44.312] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:33:0)
[23:30:44.332] Build status: SCHEDULED
[23:30:44.332] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:34:0) after 1m4s
[23:31:48.422] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:34:0)
[23:31:48.440] Build status: SCHEDULED
[23:31:48.440] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:35:0) after 9m51s
[23:41:39.558] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:35:0)
[23:41:39.575] Build status: SCHEDULED
[23:41:39.575] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:36:0) after 7m55s
[23:49:34.638] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:36:0)
[23:49:34.653] Build status: SCHEDULED
[23:49:34.654] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:37:0) after 7m44s
[23:57:18.719] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:37:0)
[23:57:18.735] Build status: SCHEDULED
[23:57:18.735] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:38:0) after 1m5s
[23:58:23.790] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:38:0)
[23:58:23.809] Build status: SCHEDULED
[23:58:23.809] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:39:0) after 8m22s
[00:06:45.926] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:39:0)
[00:06:45.946] Build status: SCHEDULED
[00:06:45.946] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:40:0) after 2m18s
[00:09:03.942] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:40:0)
[00:09:03.959] Build status: SCHEDULED
[00:09:03.959] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:41:0) after 8m18s
[00:17:22.014] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:41:0)
[00:17:22.035] Build status: SCHEDULED
[00:17:22.035] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:42:0) after 5m50s
[00:23:12.089] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:42:0)
[00:23:12.147] Build status: SCHEDULED
[00:23:12.147] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:43:0) after 1m8s
[00:24:20.238] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:43:0)
[00:24:20.347] Build status: SCHEDULED
[00:24:20.347] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:44:0) after 8m12s
[00:30:25.747] Received PubSub notification, asking Buildbucket for the build status
[00:30:25.780] Build status: STARTED
[00:32:32.438] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:44:0)
[00:32:32.459] Build status: STARTED
[00:32:32.459] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-rust:8849219118239078160:46:0) after 8m30s
[00:38:29.104] Received PubSub notification, asking Buildbucket for the build status
[00:38:29.129] Build:
{
"id": "8683789056355473265",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-rust"
},
"createdBy": "project:pigweed",
"createTime": "2026-04-22T20:40:30.072796935Z",
"startTime": "2026-04-23T00:30:25.586632133Z",
"endTime": "2026-04-23T00:38:28.793336717Z",
"updateTime": "2026-04-23T00:38:28.793336717Z",
"status": "FAILURE",
"input": {
}
}
[00:38:29.129] Invocation finished in 3h58m0.237241887s with status FAILED