[11:32:37.114] New invocation is queued and will start shortly
[11:32:38.226] Starting the invocation (attempt 1)
[11:32:38.252] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/235a8c20eda34dd8f8dfa283485ed765a8c72556"
[11:32:38.252] Popped gitiles commit info from properties and tags
[11:32:38.252] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:32:38.252] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:32:38.252] Buildbucket request:
{
"requestId": "8851790301860589584",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "235a8c20eda34dd8f8dfa283485ed765a8c72556",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851790301860589584"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMac"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:32:38.826] Scheduled build:
{
"id": "8686360239925722065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40039,
"createdBy": "project:chromium",
"createTime": "2026-03-25T11:32:38.338099172Z",
"updateTime": "2026-03-25T11:32:38.338099172Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "235a8c20eda34dd8f8dfa283485ed765a8c72556",
"ref": "refs/heads/main"
}
}
}
[11:32:38.826] Task URL: https://cr-buildbucket.appspot.com/build/8686360239925722065
[11:32:38.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:2:0) after 1m18s
[11:33:10.188] Received PubSub notification, asking Buildbucket for the build status
[11:33:10.269] Build status: STARTED
[11:33:43.767] Received PubSub notification, asking Buildbucket for the build status
[11:33:43.788] Build status: STARTED
[11:33:57.042] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:2:0)
[11:33:57.075] Build status: STARTED
[11:33:57.075] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:5:0) after 9m44s
[11:43:41.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:5:0)
[11:43:41.179] Build status: STARTED
[11:43:41.179] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:6:0) after 3m39s
[11:47:20.437] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:6:0)
[11:47:20.477] Build status: STARTED
[11:47:20.477] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:7:0) after 5m35s
[11:52:55.543] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:7:0)
[11:52:55.556] Build status: STARTED
[11:52:55.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:8:0) after 6m43s
[11:59:38.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:8:0)
[11:59:38.709] Build status: STARTED
[11:59:38.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:9:0) after 8m15s
[12:07:54.113] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:9:0)
[12:07:54.131] Build status: STARTED
[12:07:54.131] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:10:0) after 8m22s
[12:16:16.153] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:10:0)
[12:16:16.204] Build status: STARTED
[12:16:16.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:11:0) after 8m10s
[12:24:26.381] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:11:0)
[12:24:26.398] Build status: STARTED
[12:24:26.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:12:0) after 9m2s
[12:33:28.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:12:0)
[12:33:28.623] Build status: STARTED
[12:33:28.623] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:13:0) after 2m42s
[12:36:10.730] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:13:0)
[12:36:10.751] Build status: STARTED
[12:36:10.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:14:0) after 2m41s
[12:38:51.769] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:14:0)
[12:38:51.791] Build status: STARTED
[12:38:51.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:15:0) after 9m6s
[12:47:57.906] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:15:0)
[12:47:57.941] Build status: STARTED
[12:47:57.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:16:0) after 1m45s
[12:49:43.465] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:16:0)
[12:49:43.481] Build status: STARTED
[12:49:43.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:17:0) after 8m39s
[12:58:22.502] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:17:0)
[12:58:22.520] Build status: STARTED
[12:58:22.520] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:18:0) after 1m54s
[13:00:16.498] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:18:0)
[13:00:16.617] Build status: STARTED
[13:00:16.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:19:0) after 6m50s
[13:07:06.640] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:19:0)
[13:07:06.660] Build status: STARTED
[13:07:06.660] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:20:0) after 4m37s
[13:11:43.720] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:20:0)
[13:11:43.738] Build status: STARTED
[13:11:43.738] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:21:0) after 6m9s
[13:17:52.851] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:21:0)
[13:17:52.920] Build status: STARTED
[13:17:52.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:22:0) after 3m44s
[13:21:37.088] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:22:0)
[13:21:37.138] Build status: STARTED
[13:21:37.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:23:0) after 7m22s
[13:28:59.337] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:23:0)
[13:28:59.360] Build status: STARTED
[13:28:59.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:24:0) after 4m43s
[13:33:42.395] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:24:0)
[13:33:42.452] Build status: STARTED
[13:33:42.452] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:25:0) after 4m29s
[13:38:11.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:25:0)
[13:38:11.500] Build status: STARTED
[13:38:11.500] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:26:0) after 7m53s
[13:46:04.617] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:26:0)
[13:46:04.643] Build status: STARTED
[13:46:04.643] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:27:0) after 1m11s
[13:47:15.665] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:27:0)
[13:47:15.688] Build status: STARTED
[13:47:15.688] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:28:0) after 3m41s
[13:50:56.672] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:28:0)
[13:50:56.719] Build status: STARTED
[13:50:56.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:29:0) after 6m16s
[13:57:12.758] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:29:0)
[13:57:12.794] Build status: STARTED
[13:57:12.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:30:0) after 1m6s
[13:58:18.824] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:30:0)
[13:58:18.844] Build status: STARTED
[13:58:18.845] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:31:0) after 2m21s
[14:00:39.866] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:31:0)
[14:00:39.880] Build status: STARTED
[14:00:39.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:32:0) after 9m9s
[14:09:48.951] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:32:0)
[14:09:48.973] Build status: STARTED
[14:09:48.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:33:0) after 2m58s
[14:12:46.963] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:33:0)
[14:12:46.983] Build status: STARTED
[14:12:46.983] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:34:0) after 4m28s
[14:17:15.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:34:0)
[14:17:15.067] Build status: STARTED
[14:17:15.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:35:0) after 2m27s
[14:19:42.167] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:35:0)
[14:19:42.184] Build status: STARTED
[14:19:42.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:36:0) after 2m17s
[14:21:59.254] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:36:0)
[14:21:59.278] Build status: STARTED
[14:21:59.278] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:37:0) after 3m24s
[14:25:23.293] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:37:0)
[14:25:23.311] Build status: STARTED
[14:25:23.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:38:0) after 9m53s
[14:35:16.330] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:38:0)
[14:35:16.560] Build status: STARTED
[14:35:16.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:39:0) after 7m7s
[14:42:23.748] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:39:0)
[14:42:23.768] Build status: STARTED
[14:42:23.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:40:0) after 3m17s
[14:45:40.824] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:40:0)
[14:45:40.845] Build status: STARTED
[14:45:40.845] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:41:0) after 8m27s
[14:54:07.965] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:41:0)
[14:54:07.993] Build status: STARTED
[14:54:07.993] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:42:0) after 9m31s
[15:03:39.013] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:42:0)
[15:03:39.057] Build status: STARTED
[15:03:39.057] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:43:0) after 3m33s
[15:07:12.097] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:43:0)
[15:07:12.206] Build status: STARTED
[15:07:12.207] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:44:0) after 3m59s
[15:11:11.227] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:44:0)
[15:11:11.256] Build status: STARTED
[15:11:11.256] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:45:0) after 5m19s
[15:16:30.275] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:45:0)
[15:16:30.292] Build status: STARTED
[15:16:30.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:46:0) after 2m57s
[15:19:27.424] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:46:0)
[15:19:27.439] Build status: STARTED
[15:19:27.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:47:0) after 7m21s
[15:26:48.417] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:47:0)
[15:26:48.479] Build status: STARTED
[15:26:48.479] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:48:0) after 2m50s
[15:29:38.566] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:48:0)
[15:29:38.612] Build status: STARTED
[15:29:38.612] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:49:0) after 8m56s
[15:38:34.624] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:49:0)
[15:38:34.662] Build status: STARTED
[15:38:34.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:50:0) after 5m55s
[15:44:29.710] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:50:0)
[15:44:29.729] Build status: STARTED
[15:44:29.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:51:0) after 2m0s
[15:46:29.774] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:51:0)
[15:46:29.819] Build status: STARTED
[15:46:29.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:52:0) after 7m51s
[15:54:20.890] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:52:0)
[15:54:21.000] Build status: STARTED
[15:54:21.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:53:0) after 3m20s
[15:57:41.017] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:53:0)
[15:57:41.033] Build status: STARTED
[15:57:41.033] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:54:0) after 9m44s
[16:07:25.101] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:54:0)
[16:07:25.119] Build status: STARTED
[16:07:25.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:55:0) after 7m6s
[16:14:31.136] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:55:0)
[16:14:31.155] Build status: STARTED
[16:14:31.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:56:0) after 5m4s
[16:19:35.174] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:56:0)
[16:19:35.199] Build status: STARTED
[16:19:35.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:57:0) after 9m0s
[16:28:35.219] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:57:0)
[16:28:35.238] Build status: STARTED
[16:28:35.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:58:0) after 1m41s
[16:30:16.242] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:58:0)
[16:30:16.263] Build status: STARTED
[16:30:16.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:59:0) after 9m52s
[16:40:08.382] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:59:0)
[16:40:08.407] Build status: STARTED
[16:40:08.407] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:60:0) after 7m14s
[16:47:22.502] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:60:0)
[16:47:22.519] Build status: STARTED
[16:47:22.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:61:0) after 2m22s
[16:49:44.547] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:61:0)
[16:49:44.572] Build status: STARTED
[16:49:44.605] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:62:0) after 1m59s
[16:51:43.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:62:0)
[16:51:43.637] Build status: STARTED
[16:51:43.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:63:0) after 5m57s
[16:57:40.696] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:63:0)
[16:57:40.712] Build status: STARTED
[16:57:40.712] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:64:0) after 9m10s
[17:06:50.748] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:64:0)
[17:06:50.794] Build status: STARTED
[17:06:50.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:65:0) after 5m51s
[17:12:41.862] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:65:0)
[17:12:41.921] Build status: STARTED
[17:12:41.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:66:0) after 4m27s
[17:17:09.033] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:66:0)
[17:17:09.056] Build status: STARTED
[17:17:09.056] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:67:0) after 3m42s
[17:20:51.055] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:67:0)
[17:20:51.080] Build status: STARTED
[17:20:51.080] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:68:0) after 1m57s
[17:22:48.112] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:68:0)
[17:22:48.128] Build status: STARTED
[17:22:48.128] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851790301860589584:69:0) after 8m29s
[17:30:20.493] Received PubSub notification, asking Buildbucket for the build status
[17:30:20.513] Build:
{
"id": "8686360239925722065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40039,
"createdBy": "project:chromium",
"createTime": "2026-03-25T11:32:38.338099172Z",
"startTime": "2026-03-25T11:33:07.823855751Z",
"endTime": "2026-03-25T17:30:20.170294547Z",
"updateTime": "2026-03-25T17:30:20.170294547Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "235a8c20eda34dd8f8dfa283485ed765a8c72556",
"ref": "refs/heads/main"
}
}
}
[17:30:20.513] Invocation finished in 5h57m43.404266916s with status FAILED