[17:39:01.136] New invocation is queued and will start shortly
[17:39:01.136] Triggered by user:rerunner@pigweed-service-accounts.iam.gserviceaccount.com
[17:39:04.110] Starting the invocation (attempt 1)
[17:39:04.143] Ignoring gitiles_ref tag without the buildset tag
[17:39:04.143] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[17:39:04.143] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[17:39:04.143] Buildbucket request:
{
"requestId": "8841076807911697008",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-host-cpp17-v9"
},
"properties": {
"$recipe_engine/scheduler": {
"hostname": "luci-scheduler.appspot.com",
"invocation": "8841076807911697008",
"job": "pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9",
"triggers": [
{
"buildbucket": {
"tags": [
"parent_buildername:rerunner",
"user_agent:rerunner"
]
},
"id": "074cab62-7776-4593-bd2d-ecb2be693451",
"title": "rerunner/0",
"url": "https://cr-buildbucket.appspot.com/build/8675647058342882721"
}
]
}
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841076807911697008"
},
{
"key": "scheduler_job_id",
"value": "pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9"
},
{
"key": "user_agent",
"value": "luci-scheduler"
},
{
"key": "parent_buildername",
"value": "rerunner"
},
{
"key": "user_agent",
"value": "rerunner"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[17:39:04.586] Scheduled build:
{
"id": "8675646744000560737",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-host-cpp17-v9"
},
"createdBy": "project:pigweed",
"createTime": "2026-07-21T17:39:04.241519246Z",
"updateTime": "2026-07-21T17:39:04.241519246Z",
"status": "SCHEDULED",
"input": {
}
}
[17:39:04.586] Task URL: https://cr-buildbucket.appspot.com/build/8675646744000560737
[17:39:04.586] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:2:0) after 8m5s
[17:39:05.575] Received PubSub notification, asking Buildbucket for the build status
[17:39:05.602] Build status: SCHEDULED
[17:47:09.664] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:2:0)
[17:47:09.688] Build status: SCHEDULED
[17:47:09.688] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:4:0) after 4m50s
[17:51:59.753] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:4:0)
[17:51:59.772] Build status: SCHEDULED
[17:51:59.772] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:5:0) after 4m0s
[17:55:59.888] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:5:0)
[17:55:59.952] Build status: SCHEDULED
[17:55:59.952] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:6:0) after 6m28s
[18:02:28.038] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:6:0)
[18:02:28.064] Build status: SCHEDULED
[18:02:28.064] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:7:0) after 9m18s
[18:11:46.050] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:7:0)
[18:11:46.084] Build status: SCHEDULED
[18:11:46.084] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:8:0) after 7m28s
[18:19:14.160] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:8:0)
[18:19:14.186] Build status: SCHEDULED
[18:19:14.187] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:9:0) after 7m3s
[18:26:17.356] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:9:0)
[18:26:17.377] Build status: SCHEDULED
[18:26:17.377] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:10:0) after 5m50s
[18:32:07.460] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:10:0)
[18:32:07.481] Build status: SCHEDULED
[18:32:07.481] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:11:0) after 1m50s
[18:33:57.526] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:11:0)
[18:33:57.577] Build status: SCHEDULED
[18:33:57.577] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:12:0) after 8m17s
[18:42:14.642] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:12:0)
[18:42:14.754] Build status: SCHEDULED
[18:42:14.754] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:13:0) after 3m9s
[18:45:23.770] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:13:0)
[18:45:23.800] Build status: SCHEDULED
[18:45:23.801] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:14:0) after 5m40s
[18:51:03.823] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:14:0)
[18:51:03.842] Build status: SCHEDULED
[18:51:03.842] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:15:0) after 3m23s
[18:54:26.955] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:15:0)
[18:54:26.998] Build status: SCHEDULED
[18:54:26.998] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:16:0) after 5m20s
[18:59:47.012] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:16:0)
[18:59:47.045] Build status: SCHEDULED
[18:59:47.045] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:17:0) after 9m4s
[19:08:51.076] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:17:0)
[19:08:51.099] Build status: SCHEDULED
[19:08:51.100] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:18:0) after 3m34s
[19:12:25.113] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:18:0)
[19:12:25.152] Build status: SCHEDULED
[19:12:25.152] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:19:0) after 3m58s
[19:16:23.168] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:19:0)
[19:16:23.188] Build status: SCHEDULED
[19:16:23.188] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:20:0) after 6m33s
[19:22:56.257] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:20:0)
[19:22:56.276] Build status: SCHEDULED
[19:22:56.277] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:21:0) after 8m19s
[19:31:15.280] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:21:0)
[19:31:15.300] Build status: SCHEDULED
[19:31:15.300] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:22:0) after 4m29s
[19:35:44.355] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:22:0)
[19:35:44.388] Build status: SCHEDULED
[19:35:44.388] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:23:0) after 3m24s
[19:39:08.403] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:23:0)
[19:39:08.479] Build status: SCHEDULED
[19:39:08.479] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:24:0) after 9m8s
[19:48:16.494] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:24:0)
[19:48:16.521] Build status: SCHEDULED
[19:48:16.521] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:25:0) after 2m5s
[19:50:21.505] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:25:0)
[19:50:21.525] Build status: SCHEDULED
[19:50:21.525] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:26:0) after 2m32s
[19:52:53.539] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:26:0)
[19:52:53.575] Build status: SCHEDULED
[19:52:53.575] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:27:0) after 3m37s
[19:56:30.589] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:27:0)
[19:56:30.615] Build status: SCHEDULED
[19:56:30.615] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:28:0) after 6m37s
[20:03:07.684] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:28:0)
[20:03:07.701] Build status: SCHEDULED
[20:03:07.701] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:29:0) after 1m29s
[20:04:36.719] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:29:0)
[20:04:36.819] Build status: SCHEDULED
[20:04:36.819] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:30:0) after 2m12s
[20:06:48.834] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:30:0)
[20:06:48.885] Build status: SCHEDULED
[20:06:48.885] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:31:0) after 3m36s
[20:10:25.064] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:31:0)
[20:10:25.092] Build status: SCHEDULED
[20:10:25.092] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:32:0) after 7m15s
[20:17:40.206] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:32:0)
[20:17:40.228] Build status: SCHEDULED
[20:17:40.228] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:33:0) after 2m18s
[20:19:58.248] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:33:0)
[20:19:58.267] Build status: SCHEDULED
[20:19:58.268] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:34:0) after 3m35s
[20:23:33.450] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:34:0)
[20:23:33.471] Build status: SCHEDULED
[20:23:33.471] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:35:0) after 3m28s
[20:27:01.489] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:35:0)
[20:27:01.508] Build status: SCHEDULED
[20:27:01.508] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:36:0) after 2m28s
[20:29:29.510] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:36:0)
[20:29:29.530] Build status: SCHEDULED
[20:29:29.530] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:37:0) after 3m52s
[20:33:21.737] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:37:0)
[20:33:21.756] Build status: SCHEDULED
[20:33:21.757] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:38:0) after 6m36s
[20:39:57.884] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:38:0)
[20:39:57.911] Build status: SCHEDULED
[20:39:57.911] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:39:0) after 7m41s
[20:46:19.957] Received PubSub notification, asking Buildbucket for the build status
[20:46:20.126] Build status: STARTED
[20:47:39.176] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:39:0)
[20:47:39.198] Build status: STARTED
[20:47:39.198] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:41:0) after 4m18s
[20:51:57.288] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:41:0)
[20:51:57.326] Build status: STARTED
[20:51:57.326] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-arm-bazel-host-cpp17-v9:8841076807911697008:42:0) after 4m23s
[20:53:21.783] Received PubSub notification, asking Buildbucket for the build status
[20:53:21.816] Build:
{
"id": "8675646744000560737",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-arm-bazel-host-cpp17-v9"
},
"createdBy": "project:pigweed",
"createTime": "2026-07-21T17:39:04.241519246Z",
"startTime": "2026-07-21T20:46:07.498151497Z",
"endTime": "2026-07-21T20:53:21.481742346Z",
"updateTime": "2026-07-21T20:53:21.481742346Z",
"status": "FAILURE",
"input": {
}
}
[20:53:21.816] Invocation finished in 3h14m20.688606901s with status FAILED