[21:33:04.373] New invocation is queued and will start shortly
[21:33:05.616] Starting the invocation (attempt 1)
[21:33:05.641] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/b3cd2fff2711fce32930ca15ef47c9740ff909bf"
[21:33:05.641] Popped gitiles commit info from properties and tags
[21:33:05.641] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:33:05.641] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:33:05.641] Buildbucket request:
{
"requestId": "8851390136675219904",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b3cd2fff2711fce32930ca15ef47c9740ff909bf",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851390136675219904"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMac"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:33:06.607] Scheduled build:
{
"id": "8685960074593116753",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40104,
"createdBy": "project:chromium",
"createTime": "2026-03-29T21:33:05.738313920Z",
"updateTime": "2026-03-29T21:33:05.738313920Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b3cd2fff2711fce32930ca15ef47c9740ff909bf",
"ref": "refs/heads/main"
}
}
}
[21:33:06.607] Task URL: https://cr-buildbucket.appspot.com/build/8685960074593116753
[21:33:06.607] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:2:0) after 9m39s
[21:33:07.401] Received PubSub notification, asking Buildbucket for the build status
[21:33:07.418] Build status: SCHEDULED
[21:33:47.032] Received PubSub notification, asking Buildbucket for the build status
[21:33:47.051] Build status: STARTED
[21:42:45.704] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:2:0)
[21:42:45.726] Build status: STARTED
[21:42:45.726] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:5:0) after 9m31s
[21:52:16.850] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:5:0)
[21:52:16.867] Build status: STARTED
[21:52:16.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:6:0) after 4m0s
[21:56:17.116] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:6:0)
[21:56:17.133] Build status: STARTED
[21:56:17.133] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:7:0) after 1m14s
[21:57:31.291] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:7:0)
[21:57:31.309] Build status: STARTED
[21:57:31.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:8:0) after 5m30s
[22:03:01.683] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:8:0)
[22:03:01.700] Build status: STARTED
[22:03:01.700] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:9:0) after 5m4s
[22:08:05.717] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:9:0)
[22:08:05.731] Build status: STARTED
[22:08:05.731] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:10:0) after 3m47s
[22:11:52.829] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:10:0)
[22:11:52.849] Build status: STARTED
[22:11:52.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:11:0) after 8m7s
[22:19:59.967] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:11:0)
[22:19:59.979] Build status: STARTED
[22:19:59.979] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:12:0) after 5m34s
[22:25:34.119] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:12:0)
[22:25:34.149] Build status: STARTED
[22:25:34.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:13:0) after 5m49s
[22:31:23.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:13:0)
[22:31:23.247] Build status: STARTED
[22:31:23.247] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:14:0) after 2m43s
[22:34:06.328] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:14:0)
[22:34:06.356] Build status: STARTED
[22:34:06.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:15:0) after 6m12s
[22:40:18.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:15:0)
[22:40:18.386] Build status: STARTED
[22:40:18.386] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:16:0) after 5m27s
[22:45:45.442] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:16:0)
[22:45:45.458] Build status: STARTED
[22:45:45.459] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:17:0) after 8m55s
[22:54:41.070] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:17:0)
[22:54:41.088] Build status: STARTED
[22:54:41.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:18:0) after 5m22s
[23:00:03.108] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:18:0)
[23:00:03.123] Build status: STARTED
[23:00:03.124] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:19:0) after 8m8s
[23:08:11.197] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:19:0)
[23:08:11.225] Build status: STARTED
[23:08:11.225] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:20:0) after 5m51s
[23:14:02.983] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:20:0)
[23:14:03.003] Build status: STARTED
[23:14:03.003] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:21:0) after 2m13s
[23:16:15.981] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:21:0)
[23:16:16.029] Build status: STARTED
[23:16:16.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:22:0) after 1m19s
[23:17:35.118] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:22:0)
[23:17:35.144] Build status: STARTED
[23:17:35.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:23:0) after 1m41s
[23:19:16.177] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:23:0)
[23:19:16.197] Build status: STARTED
[23:19:16.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:24:0) after 1m0s
[23:20:16.221] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:24:0)
[23:20:16.238] Build status: STARTED
[23:20:16.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:25:0) after 5m20s
[23:25:36.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:25:0)
[23:25:36.601] Build status: STARTED
[23:25:36.601] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:26:0) after 8m54s
[23:34:30.784] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:26:0)
[23:34:30.800] Build status: STARTED
[23:34:30.800] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:27:0) after 3m16s
[23:37:46.834] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:27:0)
[23:37:46.853] Build status: STARTED
[23:37:46.853] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:28:0) after 9m32s
[23:47:18.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:28:0)
[23:47:18.986] Build status: STARTED
[23:47:18.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:29:0) after 8m17s
[23:55:36.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:29:0)
[23:55:36.571] Build status: STARTED
[23:55:36.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:30:0) after 9m57s
[00:05:34.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:30:0)
[00:05:34.035] Build status: STARTED
[00:05:34.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:31:0) after 6m19s
[00:11:53.138] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:31:0)
[00:11:53.249] Build status: STARTED
[00:11:53.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:32:0) after 3m27s
[00:15:20.268] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:32:0)
[00:15:20.297] Build status: STARTED
[00:15:20.297] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:33:0) after 6m52s
[00:22:12.317] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:33:0)
[00:22:12.348] Build status: STARTED
[00:22:12.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:34:0) after 2m26s
[00:24:38.457] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:34:0)
[00:24:38.486] Build status: STARTED
[00:24:38.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:35:0) after 9m3s
[00:33:41.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:35:0)
[00:33:41.562] Build status: STARTED
[00:33:41.562] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:36:0) after 2m22s
[00:36:03.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:36:0)
[00:36:03.946] Build status: STARTED
[00:36:03.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:37:0) after 1m12s
[00:37:16.185] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:37:0)
[00:37:16.204] Build status: STARTED
[00:37:16.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:38:0) after 1m5s
[00:38:21.645] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:38:0)
[00:38:21.716] Build status: STARTED
[00:38:21.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:39:0) after 8m32s
[00:46:53.926] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:39:0)
[00:46:53.942] Build status: STARTED
[00:46:53.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:40:0) after 1m27s
[00:48:21.092] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:40:0)
[00:48:21.107] Build status: STARTED
[00:48:21.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:41:0) after 4m41s
[00:53:02.346] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:41:0)
[00:53:02.360] Build status: STARTED
[00:53:02.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:42:0) after 9m36s
[01:02:38.383] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:42:0)
[01:02:38.402] Build status: STARTED
[01:02:38.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:43:0) after 7m56s
[01:10:34.439] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:43:0)
[01:10:34.463] Build status: STARTED
[01:10:34.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:44:0) after 1m38s
[01:12:12.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:44:0)
[01:12:12.536] Build status: STARTED
[01:12:12.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:45:0) after 1m58s
[01:14:10.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:45:0)
[01:14:10.577] Build status: STARTED
[01:14:10.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:46:0) after 5m59s
[01:20:09.595] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:46:0)
[01:20:09.664] Build status: STARTED
[01:20:09.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:47:0) after 5m0s
[01:25:09.677] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:47:0)
[01:25:09.696] Build status: STARTED
[01:25:09.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:48:0) after 5m54s
[01:31:03.716] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:48:0)
[01:31:03.742] Build status: STARTED
[01:31:03.742] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:49:0) after 3m36s
[01:34:39.996] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:49:0)
[01:34:40.024] Build status: STARTED
[01:34:40.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:50:0) after 8m44s
[01:43:24.062] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:50:0)
[01:43:24.113] Build status: STARTED
[01:43:24.113] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:51:0) after 4m25s
[01:47:49.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:51:0)
[01:47:49.409] Build status: STARTED
[01:47:49.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:52:0) after 1m59s
[01:49:48.430] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:52:0)
[01:49:48.455] Build status: STARTED
[01:49:48.455] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:53:0) after 2m22s
[01:52:10.539] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:53:0)
[01:52:10.578] Build status: STARTED
[01:52:10.579] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:54:0) after 1m58s
[01:54:08.595] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:54:0)
[01:54:08.686] Build status: STARTED
[01:54:08.687] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:55:0) after 5m54s
[02:00:02.876] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:55:0)
[02:00:02.899] Build status: STARTED
[02:00:02.899] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:56:0) after 2m39s
[02:02:41.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:56:0)
[02:02:41.999] Build status: STARTED
[02:02:41.999] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:57:0) after 6m25s
[02:09:07.060] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:57:0)
[02:09:07.078] Build status: STARTED
[02:09:07.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:58:0) after 8m49s
[02:17:56.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:58:0)
[02:17:56.133] Build status: STARTED
[02:17:56.133] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:59:0) after 3m56s
[02:21:52.433] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:59:0)
[02:21:52.546] Build status: STARTED
[02:21:52.547] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:60:0) after 3m0s
[02:24:52.522] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:60:0)
[02:24:52.536] Build status: STARTED
[02:24:52.537] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:61:0) after 8m18s
[02:33:10.650] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:61:0)
[02:33:10.667] Build status: STARTED
[02:33:10.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:62:0) after 9m21s
[02:42:31.680] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:62:0)
[02:42:31.694] Build status: STARTED
[02:42:31.695] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:63:0) after 3m33s
[02:46:04.761] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:63:0)
[02:46:04.826] Build status: STARTED
[02:46:04.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:64:0) after 5m36s
[02:51:40.874] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:64:0)
[02:51:40.893] Build status: STARTED
[02:51:40.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:65:0) after 9m51s
[03:01:31.914] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:65:0)
[03:01:31.931] Build status: STARTED
[03:01:31.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:66:0) after 3m15s
[03:04:47.050] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:66:0)
[03:04:47.127] Build status: STARTED
[03:04:47.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:67:0) after 3m47s
[03:08:34.172] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:67:0)
[03:08:34.192] Build status: STARTED
[03:08:34.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:68:0) after 8m37s
[03:17:11.209] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:68:0)
[03:17:11.228] Build status: STARTED
[03:17:11.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:69:0) after 1m56s
[03:19:07.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:69:0)
[03:19:07.264] Build status: STARTED
[03:19:07.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:70:0) after 8m19s
[03:27:26.339] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:70:0)
[03:27:26.359] Build status: STARTED
[03:27:26.359] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851390136675219904:71:0) after 3m57s
[03:28:30.680] Received PubSub notification, asking Buildbucket for the build status
[03:28:30.789] Build:
{
"id": "8685960074593116753",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40104,
"createdBy": "project:chromium",
"createTime": "2026-03-29T21:33:05.738313920Z",
"startTime": "2026-03-29T21:33:46.642969623Z",
"endTime": "2026-03-30T03:28:30.367842516Z",
"updateTime": "2026-03-30T03:28:30.367842516Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "b3cd2fff2711fce32930ca15ef47c9740ff909bf",
"ref": "refs/heads/main"
}
}
}
[03:28:30.789] Invocation finished in 5h55m26.420621002s with status FAILED