[19:15:14.537] New invocation is queued and will start shortly
[19:15:16.096] Starting the invocation (attempt 1)
[19:15:16.128] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/adb1b6d6a383ff379b6a6542ea9ab9651d6029b8"
[19:15:16.128] Popped gitiles commit info from properties and tags
[19:15:16.128] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:15:16.128] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:15:16.128] Buildbucket request:
{
"requestId": "8841433141926592240",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "adb1b6d6a383ff379b6a6542ea9ab9651d6029b8",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841433141926592240"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:15:17.019] Scheduled build:
{
"id": "8676003079374063937",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31519,
"createdBy": "project:chromium",
"createTime": "2026-07-17T19:15:16.347220081Z",
"updateTime": "2026-07-17T19:15:16.347220081Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "adb1b6d6a383ff379b6a6542ea9ab9651d6029b8",
"ref": "refs/heads/main"
}
}
}
[19:15:17.019] Task URL: https://cr-buildbucket.appspot.com/build/8676003079374063937
[19:15:17.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:2:0) after 1m51s
[19:15:18.001] Received PubSub notification, asking Buildbucket for the build status
[19:15:18.032] Build status: SCHEDULED
[19:16:33.749] Received PubSub notification, asking Buildbucket for the build status
[19:16:33.849] Build status: STARTED
[19:17:08.039] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:2:0)
[19:17:08.057] Build status: STARTED
[19:17:08.057] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:5:0) after 1m23s
[19:18:31.225] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:5:0)
[19:18:31.252] Build status: STARTED
[19:18:31.252] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:6:0) after 2m15s
[19:20:46.655] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:6:0)
[19:20:46.748] Build status: STARTED
[19:20:46.748] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:7:0) after 1m41s
[19:22:27.884] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:7:0)
[19:22:27.923] Build status: STARTED
[19:22:27.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:8:0) after 4m37s
[19:27:04.962] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:8:0)
[19:27:04.980] Build status: STARTED
[19:27:04.980] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:9:0) after 3m35s
[19:30:39.975] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:9:0)
[19:30:40.032] Build status: STARTED
[19:30:40.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:10:0) after 9m33s
[19:40:13.055] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:10:0)
[19:40:13.089] Build status: STARTED
[19:40:13.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:11:0) after 8m48s
[19:49:01.247] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:11:0)
[19:49:01.265] Build status: STARTED
[19:49:01.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:12:0) after 4m16s
[19:53:17.387] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:12:0)
[19:53:17.434] Build status: STARTED
[19:53:17.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:13:0) after 2m49s
[19:56:06.523] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:13:0)
[19:56:06.540] Build status: STARTED
[19:56:06.541] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:14:0) after 8m36s
[20:04:42.562] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:14:0)
[20:04:42.691] Build status: STARTED
[20:04:42.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:15:0) after 3m17s
[20:07:59.681] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:15:0)
[20:07:59.759] Build status: STARTED
[20:07:59.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:16:0) after 7m24s
[20:15:23.783] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:16:0)
[20:15:23.903] Build status: STARTED
[20:15:23.903] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:17:0) after 3m30s
[20:18:53.920] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:17:0)
[20:18:53.937] Build status: STARTED
[20:18:53.937] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:18:0) after 4m6s
[20:23:00.069] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:18:0)
[20:23:00.089] Build status: STARTED
[20:23:00.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:19:0) after 5m10s
[20:28:10.363] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:19:0)
[20:28:10.379] Build status: STARTED
[20:28:10.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:20:0) after 4m10s
[20:32:20.446] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:20:0)
[20:32:20.585] Build status: STARTED
[20:32:20.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:21:0) after 2m10s
[20:34:30.630] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:21:0)
[20:34:30.649] Build status: STARTED
[20:34:30.650] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:22:0) after 2m59s
[20:37:29.687] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:22:0)
[20:37:29.707] Build status: STARTED
[20:37:29.708] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:23:0) after 4m54s
[20:42:23.745] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:23:0)
[20:42:23.781] Build status: STARTED
[20:42:23.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:24:0) after 4m46s
[20:47:09.768] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:24:0)
[20:47:09.827] Build status: STARTED
[20:47:09.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:25:0) after 1m47s
[20:48:56.851] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:25:0)
[20:48:56.872] Build status: STARTED
[20:48:56.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:26:0) after 6m12s
[20:55:08.874] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:26:0)
[20:55:08.893] Build status: STARTED
[20:55:08.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:27:0) after 7m10s
[21:02:18.940] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:27:0)
[21:02:18.960] Build status: STARTED
[21:02:18.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:28:0) after 4m41s
[21:06:59.996] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:28:0)
[21:07:00.019] Build status: STARTED
[21:07:00.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:29:0) after 8m42s
[21:15:42.043] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:29:0)
[21:15:42.116] Build status: STARTED
[21:15:42.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:30:0) after 6m46s
[21:22:28.144] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:30:0)
[21:22:28.169] Build status: STARTED
[21:22:28.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:31:0) after 9m29s
[21:31:57.186] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:31:0)
[21:31:57.233] Build status: STARTED
[21:31:57.233] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:32:0) after 1m33s
[21:33:30.217] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:32:0)
[21:33:30.614] Build status: STARTED
[21:33:30.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:33:0) after 5m54s
[21:39:24.740] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:33:0)
[21:39:24.768] Build status: STARTED
[21:39:24.768] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:34:0) after 2m11s
[21:41:35.816] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:34:0)
[21:41:35.841] Build status: STARTED
[21:41:35.841] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:35:0) after 7m46s
[21:49:21.867] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:35:0)
[21:49:21.887] Build status: STARTED
[21:49:21.887] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:36:0) after 9m20s
[21:58:41.940] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:36:0)
[21:58:41.959] Build status: STARTED
[21:58:41.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:37:0) after 8m28s
[22:07:10.046] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:37:0)
[22:07:10.119] Build status: STARTED
[22:07:10.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:38:0) after 6m54s
[22:14:04.168] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:38:0)
[22:14:04.247] Build status: STARTED
[22:14:04.247] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:39:0) after 3m41s
[22:17:45.345] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:39:0)
[22:17:45.380] Build status: STARTED
[22:17:45.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:40:0) after 4m1s
[22:21:46.450] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:40:0)
[22:21:46.469] Build status: STARTED
[22:21:46.469] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:41:0) after 2m51s
[22:24:37.491] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:41:0)
[22:24:37.514] Build status: STARTED
[22:24:37.514] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:42:0) after 1m47s
[22:26:24.638] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:42:0)
[22:26:24.673] Build status: STARTED
[22:26:24.673] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:43:0) after 6m57s
[22:33:21.682] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:43:0)
[22:33:21.700] Build status: STARTED
[22:33:21.700] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:44:0) after 7m28s
[22:40:49.719] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:44:0)
[22:40:49.741] Build status: STARTED
[22:40:49.741] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:45:0) after 3m8s
[22:43:57.728] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:45:0)
[22:43:57.753] Build status: STARTED
[22:43:57.753] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:46:0) after 5m42s
[22:49:39.777] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:46:0)
[22:49:39.793] Build status: STARTED
[22:49:39.793] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841433141926592240:47:0) after 6m45s
[22:51:08.165] Received PubSub notification, asking Buildbucket for the build status
[22:51:08.212] Build:
{
"id": "8676003079374063937",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31519,
"createdBy": "project:chromium",
"createTime": "2026-07-17T19:15:16.347220081Z",
"startTime": "2026-07-17T19:16:29.594935193Z",
"endTime": "2026-07-17T22:51:07.770394446Z",
"updateTime": "2026-07-17T22:51:07.770394446Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "adb1b6d6a383ff379b6a6542ea9ab9651d6029b8",
"ref": "refs/heads/main"
}
}
}
[22:51:08.212] Invocation finished in 3h35m53.683283314s with status FAILED