[03:48:20.893] New invocation is queued and will start shortly
[03:48:21.944] Starting the invocation (attempt 1)
[03:48:21.971] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/93710b35c0d382ad217f81624761ca30c1a0d833"
[03:48:21.971] Popped gitiles commit info from properties and tags
[03:48:21.971] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[03:48:21.971] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[03:48:21.971] Buildbucket request:
{
"requestId": "8851638317293827328",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "93710b35c0d382ad217f81624761ca30c1a0d833",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851638317293827328"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMac"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[03:48:22.410] Scheduled build:
{
"id": "8686208255444571009",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40093,
"createdBy": "project:chromium",
"createTime": "2026-03-27T03:48:22.034024097Z",
"updateTime": "2026-03-27T03:48:22.034024097Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "93710b35c0d382ad217f81624761ca30c1a0d833",
"ref": "refs/heads/main"
}
}
}
[03:48:22.411] Task URL: https://cr-buildbucket.appspot.com/build/8686208255444571009
[03:48:22.411] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:2:0) after 2m15s
[03:48:23.372] Received PubSub notification, asking Buildbucket for the build status
[03:48:23.385] Build status: SCHEDULED
[03:49:04.841] Received PubSub notification, asking Buildbucket for the build status
[03:49:04.859] Build status: STARTED
[03:50:37.554] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:2:0)
[03:50:37.581] Build status: STARTED
[03:50:37.581] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:5:0) after 4m21s
[03:54:58.762] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:5:0)
[03:54:58.787] Build status: STARTED
[03:54:58.787] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:6:0) after 8m25s
[04:03:23.912] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:6:0)
[04:03:24.002] Build status: STARTED
[04:03:24.002] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:7:0) after 8m10s
[04:11:34.042] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:7:0)
[04:11:34.061] Build status: STARTED
[04:11:34.061] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:8:0) after 2m33s
[04:14:07.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:8:0)
[04:14:07.122] Build status: STARTED
[04:14:07.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:9:0) after 6m50s
[04:20:57.141] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:9:0)
[04:20:57.217] Build status: STARTED
[04:20:57.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:10:0) after 6m7s
[04:27:04.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:10:0)
[04:27:04.424] Build status: STARTED
[04:27:04.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:11:0) after 7m42s
[04:34:46.494] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:11:0)
[04:34:46.513] Build status: STARTED
[04:34:46.513] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:12:0) after 9m30s
[04:44:16.537] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:12:0)
[04:44:16.558] Build status: STARTED
[04:44:16.558] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:13:0) after 3m45s
[04:48:01.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:13:0)
[04:48:01.595] Build status: STARTED
[04:48:01.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:14:0) after 2m9s
[04:50:10.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:14:0)
[04:50:10.636] Build status: STARTED
[04:50:10.636] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:15:0) after 4m52s
[04:55:02.739] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:15:0)
[04:55:02.776] Build status: STARTED
[04:55:02.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:16:0) after 9m54s
[05:04:56.793] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:16:0)
[05:04:56.822] Build status: STARTED
[05:04:56.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:17:0) after 2m12s
[05:07:08.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:17:0)
[05:07:08.968] Build status: STARTED
[05:07:08.968] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:18:0) after 6m6s
[05:13:14.987] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:18:0)
[05:13:15.005] Build status: STARTED
[05:13:15.005] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:19:0) after 5m9s
[05:18:24.131] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:19:0)
[05:18:24.171] Build status: STARTED
[05:18:24.171] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:20:0) after 8m2s
[05:26:26.192] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:20:0)
[05:26:26.220] Build status: STARTED
[05:26:26.220] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:21:0) after 5m38s
[05:32:04.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:21:0)
[05:32:04.265] Build status: STARTED
[05:32:04.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:22:0) after 5m27s
[05:37:31.337] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:22:0)
[05:37:31.353] Build status: STARTED
[05:37:31.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:23:0) after 2m9s
[05:39:40.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:23:0)
[05:39:40.387] Build status: STARTED
[05:39:40.387] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:24:0) after 4m17s
[05:43:57.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:24:0)
[05:43:57.581] Build status: STARTED
[05:43:57.581] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:25:0) after 7m43s
[05:51:40.683] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:25:0)
[05:51:40.701] Build status: STARTED
[05:51:40.701] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:26:0) after 1m46s
[05:53:26.804] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:26:0)
[05:53:26.825] Build status: STARTED
[05:53:26.825] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:27:0) after 7m36s
[06:01:02.903] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:27:0)
[06:01:02.922] Build status: STARTED
[06:01:02.922] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:28:0) after 5m24s
[06:06:27.055] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:28:0)
[06:06:27.074] Build status: STARTED
[06:06:27.074] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:29:0) after 5m50s
[06:12:17.082] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:29:0)
[06:12:17.102] Build status: STARTED
[06:12:17.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:30:0) after 1m5s
[06:13:22.262] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:30:0)
[06:13:22.287] Build status: STARTED
[06:13:22.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:31:0) after 2m22s
[06:15:44.338] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:31:0)
[06:15:44.355] Build status: STARTED
[06:15:44.356] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:32:0) after 6m57s
[06:22:41.427] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:32:0)
[06:22:41.519] Build status: STARTED
[06:22:41.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:33:0) after 3m1s
[06:25:42.685] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:33:0)
[06:25:42.715] Build status: STARTED
[06:25:42.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:34:0) after 9m54s
[06:35:36.890] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:34:0)
[06:35:36.964] Build status: STARTED
[06:35:36.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:35:0) after 5m3s
[06:40:39.961] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:35:0)
[06:40:39.978] Build status: STARTED
[06:40:39.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:36:0) after 5m4s
[06:45:43.999] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:36:0)
[06:45:44.036] Build status: STARTED
[06:45:44.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:37:0) after 3m26s
[06:49:10.054] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:37:0)
[06:49:10.072] Build status: STARTED
[06:49:10.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:38:0) after 3m58s
[06:53:08.093] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:38:0)
[06:53:08.112] Build status: STARTED
[06:53:08.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:39:0) after 4m28s
[06:57:36.129] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:39:0)
[06:57:36.175] Build status: STARTED
[06:57:36.175] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:40:0) after 8m12s
[07:05:48.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:40:0)
[07:05:48.221] Build status: STARTED
[07:05:48.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:41:0) after 5m24s
[07:11:12.428] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:41:0)
[07:11:12.473] Build status: STARTED
[07:11:12.473] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:42:0) after 7m17s
[07:18:29.685] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:42:0)
[07:18:29.705] Build status: STARTED
[07:18:29.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:43:0) after 6m44s
[07:25:13.776] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:43:0)
[07:25:13.792] Build status: STARTED
[07:25:13.793] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:44:0) after 3m48s
[07:29:01.815] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:44:0)
[07:29:01.836] Build status: STARTED
[07:29:01.836] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:45:0) after 2m32s
[07:31:33.870] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:45:0)
[07:31:33.892] Build status: STARTED
[07:31:33.892] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:46:0) after 9m43s
[07:41:16.914] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:46:0)
[07:41:16.933] Build status: STARTED
[07:41:16.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:47:0) after 4m44s
[07:46:01.116] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:47:0)
[07:46:01.135] Build status: STARTED
[07:46:01.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:48:0) after 4m1s
[07:50:02.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:48:0)
[07:50:02.197] Build status: STARTED
[07:50:02.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:49:0) after 5m21s
[07:55:23.318] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:49:0)
[07:55:23.339] Build status: STARTED
[07:55:23.340] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:50:0) after 5m44s
[08:01:07.345] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:50:0)
[08:01:07.412] Build status: STARTED
[08:01:07.412] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:51:0) after 7m38s
[08:08:45.438] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:51:0)
[08:08:45.459] Build status: STARTED
[08:08:45.459] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:52:0) after 4m14s
[08:12:59.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:52:0)
[08:12:59.505] Build status: STARTED
[08:12:59.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:53:0) after 2m19s
[08:15:18.534] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:53:0)
[08:15:18.551] Build status: STARTED
[08:15:18.551] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:54:0) after 2m40s
[08:17:58.598] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:54:0)
[08:17:58.699] Build status: STARTED
[08:17:58.699] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:55:0) after 1m3s
[08:19:01.840] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:55:0)
[08:19:01.863] Build status: STARTED
[08:19:01.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:56:0) after 6m50s
[08:25:51.882] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:56:0)
[08:25:51.907] Build status: STARTED
[08:25:51.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:57:0) after 4m38s
[08:30:29.894] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:57:0)
[08:30:29.911] Build status: STARTED
[08:30:29.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:58:0) after 8m19s
[08:38:48.942] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:58:0)
[08:38:48.959] Build status: STARTED
[08:38:48.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:59:0) after 8m40s
[08:47:29.107] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:59:0)
[08:47:29.123] Build status: STARTED
[08:47:29.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:60:0) after 9m3s
[08:56:32.257] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:60:0)
[08:56:32.295] Build status: STARTED
[08:56:32.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:61:0) after 6m50s
[09:03:22.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:61:0)
[09:03:22.335] Build status: STARTED
[09:03:22.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:62:0) after 4m45s
[09:08:07.431] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:62:0)
[09:08:07.448] Build status: STARTED
[09:08:07.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:63:0) after 3m21s
[09:11:28.610] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:63:0)
[09:11:28.651] Build status: STARTED
[09:11:28.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:64:0) after 6m41s
[09:18:09.671] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:64:0)
[09:18:09.693] Build status: STARTED
[09:18:09.693] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:65:0) after 3m1s
[09:21:10.747] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:65:0)
[09:21:10.766] Build status: STARTED
[09:21:10.766] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:66:0) after 2m11s
[09:23:21.834] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:66:0)
[09:23:21.876] Build status: STARTED
[09:23:21.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:67:0) after 7m49s
[09:31:10.927] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:67:0)
[09:31:10.953] Build status: STARTED
[09:31:10.953] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:68:0) after 7m8s
[09:38:18.976] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:68:0)
[09:38:19.013] Build status: STARTED
[09:38:19.013] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:69:0) after 3m16s
[09:41:35.033] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:69:0)
[09:41:35.104] Build status: STARTED
[09:41:35.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:70:0) after 2m41s
[09:44:16.191] Handling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:70:0)
[09:44:16.258] Build status: STARTED
[09:44:16.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMac:8851638317293827328:71:0) after 5m30s
[09:47:50.251] Received PubSub notification, asking Buildbucket for the build status
[09:47:50.273] Build:
{
"id": "8686208255444571009",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMac"
},
"number": 40093,
"createdBy": "project:chromium",
"createTime": "2026-03-27T03:48:22.034024097Z",
"startTime": "2026-03-27T03:49:04.623007708Z",
"endTime": "2026-03-27T09:47:49.955449752Z",
"updateTime": "2026-03-27T09:47:49.955449752Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "93710b35c0d382ad217f81624761ca30c1a0d833",
"ref": "refs/heads/main"
}
}
}
[09:47:50.273] Invocation finished in 5h59m29.385958271s with status FAILED