[01:45:31.534] New invocation is queued and will start shortly
[01:45:31.534] Triggered by user:rerunner@pigweed-service-accounts.iam.gserviceaccount.com
[01:45:32.849] Starting the invocation (attempt 1)
[01:45:32.873] Ignoring gitiles_ref tag without the buildset tag
[01:45:32.873] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:45:32.873] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:45:32.873] Buildbucket request:
{
"requestId": "8867953498580559584",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-x86-bazel-sanitizers-tsan"
},
"properties": {
"$recipe_engine/scheduler": {
"hostname": "luci-scheduler.appspot.com",
"invocation": "8867953498580559584",
"job": "pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan",
"triggers": [
{
"buildbucket": {
"tags": [
"parent_buildername:rerunner",
"user_agent:rerunner"
]
},
"id": "2c51fa61-c66e-465a-90b3-2d1efeb2b93d",
"title": "rerunner/0",
"url": "https://cr-buildbucket.appspot.com/build/8702523615458121985"
}
]
}
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867953498580559584"
},
{
"key": "scheduler_job_id",
"value": "pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
},
{
"key": "parent_buildername",
"value": "rerunner"
},
{
"key": "user_agent",
"value": "rerunner"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:45:33.750] Scheduled build:
{
"id": "8702523436332693009",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-x86-bazel-sanitizers-tsan"
},
"createdBy": "project:pigweed",
"createTime": "2025-09-28T01:45:33.054709028Z",
"updateTime": "2025-09-28T01:45:33.054709028Z",
"status": "SCHEDULED",
"input": {
}
}
[01:45:33.750] Task URL: https://cr-buildbucket.appspot.com/build/8702523436332693009
[01:45:33.750] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:2:0) after 7m34s
[01:46:08.245] Received PubSub notification, asking Buildbucket for the build status
[01:46:08.267] Build status: SCHEDULED
[01:53:07.764] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:2:0)
[01:53:07.842] Build status: SCHEDULED
[01:53:07.843] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:4:0) after 8m29s
[02:01:36.955] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:4:0)
[02:01:37.159] Build status: SCHEDULED
[02:01:37.159] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:5:0) after 6m37s
[02:08:14.541] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:5:0)
[02:08:14.569] Build status: SCHEDULED
[02:08:14.569] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:6:0) after 2m15s
[02:10:29.943] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:6:0)
[02:10:29.958] Build status: SCHEDULED
[02:10:29.958] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:7:0) after 4m30s
[02:15:00.330] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:7:0)
[02:15:00.408] Build status: SCHEDULED
[02:15:00.408] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:8:0) after 8m54s
[02:23:54.428] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:8:0)
[02:23:54.498] Build status: SCHEDULED
[02:23:54.498] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:9:0) after 2m7s
[02:26:01.566] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:9:0)
[02:26:01.590] Build status: SCHEDULED
[02:26:01.590] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:10:0) after 6m6s
[02:32:07.625] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:10:0)
[02:32:07.645] Build status: SCHEDULED
[02:32:07.645] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:11:0) after 3m55s
[02:36:02.843] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:11:0)
[02:36:02.862] Build status: SCHEDULED
[02:36:02.862] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:12:0) after 3m33s
[02:39:35.884] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:12:0)
[02:39:35.910] Build status: SCHEDULED
[02:39:35.910] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:13:0) after 2m24s
[02:41:59.988] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:13:0)
[02:42:00.021] Build status: SCHEDULED
[02:42:00.021] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:14:0) after 3m39s
[02:45:39.035] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:14:0)
[02:45:39.055] Build status: SCHEDULED
[02:45:39.055] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:15:0) after 7m58s
[02:53:37.612] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:15:0)
[02:53:37.659] Build status: SCHEDULED
[02:53:37.659] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:16:0) after 7m57s
[03:01:34.752] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:16:0)
[03:01:34.781] Build status: SCHEDULED
[03:01:34.781] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:17:0) after 3m11s
[03:04:46.054] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:17:0)
[03:04:46.121] Build status: SCHEDULED
[03:04:46.121] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:18:0) after 2m39s
[03:07:25.134] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:18:0)
[03:07:25.156] Build status: SCHEDULED
[03:07:25.156] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:19:0) after 6m36s
[03:14:01.296] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:19:0)
[03:14:01.321] Build status: SCHEDULED
[03:14:01.322] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:20:0) after 1m58s
[03:15:59.383] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:20:0)
[03:15:59.399] Build status: SCHEDULED
[03:15:59.399] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:21:0) after 2m24s
[03:18:23.411] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:21:0)
[03:18:23.431] Build status: SCHEDULED
[03:18:23.431] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:22:0) after 9m5s
[03:27:28.552] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:22:0)
[03:27:28.577] Build status: SCHEDULED
[03:27:28.577] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:23:0) after 3m38s
[03:31:06.592] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:23:0)
[03:31:06.611] Build status: SCHEDULED
[03:31:06.611] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:24:0) after 5m8s
[03:36:14.725] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:24:0)
[03:36:14.757] Build status: SCHEDULED
[03:36:14.757] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:25:0) after 5m36s
[03:41:50.889] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:25:0)
[03:41:50.906] Build status: SCHEDULED
[03:41:50.906] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:26:0) after 3m34s
[03:45:24.921] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:26:0)
[03:45:24.941] Build status: SCHEDULED
[03:45:24.941] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:27:0) after 2m35s
[03:48:00.110] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:27:0)
[03:48:00.123] Build status: SCHEDULED
[03:48:00.123] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:28:0) after 2m23s
[03:50:23.349] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:28:0)
[03:50:23.367] Build status: SCHEDULED
[03:50:23.367] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:29:0) after 1m35s
[03:51:58.526] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:29:0)
[03:51:58.548] Build status: SCHEDULED
[03:51:58.548] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:30:0) after 2m39s
[03:54:37.586] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:30:0)
[03:54:37.600] Build status: SCHEDULED
[03:54:37.600] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:31:0) after 8m40s
[04:03:17.612] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:31:0)
[04:03:17.633] Build status: SCHEDULED
[04:03:17.633] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:32:0) after 5m55s
[04:09:12.660] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:32:0)
[04:09:12.675] Build status: SCHEDULED
[04:09:12.675] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:33:0) after 4m43s
[04:13:56.107] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:33:0)
[04:13:56.122] Build status: SCHEDULED
[04:13:56.122] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:34:0) after 5m50s
[04:19:46.239] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:34:0)
[04:19:46.260] Build status: SCHEDULED
[04:19:46.260] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:35:0) after 8m16s
[04:28:02.285] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:35:0)
[04:28:02.302] Build status: SCHEDULED
[04:28:02.302] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:36:0) after 9m57s
[04:37:59.373] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:36:0)
[04:37:59.391] Build status: SCHEDULED
[04:37:59.391] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:37:0) after 3m25s
[04:41:24.401] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:37:0)
[04:41:24.429] Build status: SCHEDULED
[04:41:24.429] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:38:0) after 7m40s
[04:49:04.467] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:38:0)
[04:49:04.484] Build status: SCHEDULED
[04:49:04.484] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:39:0) after 1m31s
[04:50:35.735] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:39:0)
[04:50:35.752] Build status: SCHEDULED
[04:50:35.752] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:40:0) after 2m58s
[04:53:34.038] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:40:0)
[04:53:34.214] Build status: SCHEDULED
[04:53:34.214] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:41:0) after 2m0s
[04:55:34.389] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:41:0)
[04:55:34.407] Build status: SCHEDULED
[04:55:34.407] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:42:0) after 8m14s
[05:03:48.403] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:42:0)
[05:03:48.416] Build status: SCHEDULED
[05:03:48.416] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:43:0) after 6m28s
[05:10:16.569] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:43:0)
[05:10:16.592] Build status: SCHEDULED
[05:10:16.592] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:44:0) after 7m58s
[05:18:14.608] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:44:0)
[05:18:14.626] Build status: SCHEDULED
[05:18:14.626] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:45:0) after 2m54s
[05:21:08.645] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:45:0)
[05:21:08.661] Build status: SCHEDULED
[05:21:08.661] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:46:0) after 1m36s
[05:22:44.724] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:46:0)
[05:22:44.771] Build status: SCHEDULED
[05:22:44.771] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:47:0) after 7m35s
[05:30:19.955] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:47:0)
[05:30:20.082] Build status: SCHEDULED
[05:30:20.082] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:48:0) after 4m19s
[05:34:39.120] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:48:0)
[05:34:39.274] Build status: SCHEDULED
[05:34:39.274] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:49:0) after 7m40s
[05:42:19.295] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:49:0)
[05:42:19.312] Build status: SCHEDULED
[05:42:19.312] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:50:0) after 5m35s
[05:47:54.385] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:50:0)
[05:47:54.402] Build status: SCHEDULED
[05:47:54.402] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:51:0) after 3m48s
[05:51:42.522] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:51:0)
[05:51:42.538] Build status: SCHEDULED
[05:51:42.539] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:52:0) after 3m4s
[05:54:46.554] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:52:0)
[05:54:46.568] Build status: SCHEDULED
[05:54:46.568] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:53:0) after 3m4s
[05:57:50.639] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:53:0)
[05:57:50.657] Build status: SCHEDULED
[05:57:50.657] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:54:0) after 1m37s
[05:59:27.676] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:54:0)
[05:59:27.767] Build status: SCHEDULED
[05:59:27.767] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:55:0) after 2m52s
[06:02:19.994] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:55:0)
[06:02:20.013] Build status: SCHEDULED
[06:02:20.013] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:56:0) after 7m19s
[06:09:39.146] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:56:0)
[06:09:39.228] Build status: SCHEDULED
[06:09:39.228] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:57:0) after 4m22s
[06:14:01.485] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:57:0)
[06:14:01.507] Build status: SCHEDULED
[06:14:01.507] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:58:0) after 9m5s
[06:23:06.623] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:58:0)
[06:23:06.648] Build status: SCHEDULED
[06:23:06.648] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:59:0) after 2m19s
[06:25:25.659] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:59:0)
[06:25:25.674] Build status: SCHEDULED
[06:25:25.674] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:60:0) after 4m41s
[06:30:06.899] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:60:0)
[06:30:06.922] Build status: SCHEDULED
[06:30:06.922] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:61:0) after 1m37s
[06:31:44.076] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:61:0)
[06:31:44.236] Build status: SCHEDULED
[06:31:44.236] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:62:0) after 3m50s
[06:35:34.259] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:62:0)
[06:35:34.276] Build status: SCHEDULED
[06:35:34.276] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:63:0) after 8m10s
[06:43:44.315] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:63:0)
[06:43:44.337] Build status: SCHEDULED
[06:43:44.337] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:64:0) after 8m19s
[06:52:03.475] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:64:0)
[06:52:03.514] Build status: SCHEDULED
[06:52:03.514] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:65:0) after 7m36s
[06:59:39.531] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:65:0)
[06:59:39.544] Build status: SCHEDULED
[06:59:39.545] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:66:0) after 4m12s
[07:03:51.832] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:66:0)
[07:03:51.848] Build status: SCHEDULED
[07:03:51.848] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:67:0) after 4m7s
[07:07:58.989] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:67:0)
[07:07:59.007] Build status: SCHEDULED
[07:07:59.007] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:68:0) after 1m56s
[07:09:55.107] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:68:0)
[07:09:55.180] Build status: SCHEDULED
[07:09:55.180] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:69:0) after 8m31s
[07:18:26.196] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:69:0)
[07:18:26.231] Build status: SCHEDULED
[07:18:26.231] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:70:0) after 3m58s
[07:22:24.635] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:70:0)
[07:22:24.654] Build status: SCHEDULED
[07:22:24.654] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:71:0) after 7m14s
[07:29:38.670] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:71:0)
[07:29:38.685] Build status: SCHEDULED
[07:29:38.685] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:72:0) after 2m42s
[07:32:20.754] Handling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:72:0)
[07:32:20.781] Build status: SCHEDULED
[07:32:20.781] Scheduling timer "check-buildbucket-build-status" (pigweed/pigweed.dev.ci-pigweed-mac-x86-bazel-sanitizers-tsan:8867953498580559584:73:0) after 9m3s
[07:34:37.524] Received PubSub notification, asking Buildbucket for the build status
[07:34:37.564] Build status: STARTED
[07:39:19.508] Received PubSub notification, asking Buildbucket for the build status
[07:39:19.529] Build:
{
"id": "8702523436332693009",
"builder": {
"project": "pigweed",
"bucket": "pigweed.dev.ci",
"builder": "pigweed-mac-x86-bazel-sanitizers-tsan"
},
"createdBy": "project:pigweed",
"createTime": "2025-09-28T01:45:33.054709028Z",
"startTime": "2025-09-28T07:33:27.861543921Z",
"endTime": "2025-09-28T07:39:19.341194214Z",
"updateTime": "2025-09-28T07:39:19.341194214Z",
"status": "FAILURE",
"input": {
}
}
[07:39:19.529] Invocation finished in 5h53m48.00382027s with status FAILED