[14:50:20.594] New invocation is queued and will start shortly
[14:50:22.662] Starting the invocation (attempt 1)
[14:50:22.693] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/8a0fa60baa74612e761dc45cd01a7e1ef3e94fa0"
[14:50:22.693] Popped gitiles commit info from properties and tags
[14:50:22.693] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[14:50:22.693] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[14:50:22.694] Buildbucket request:
{
"requestId": "8867179346389711680",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8a0fa60baa74612e761dc45cd01a7e1ef3e94fa0",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867179346389711680"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[14:50:23.214] Scheduled build:
{
"id": "8701749283467262225",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27183,
"createdBy": "project:chromium",
"createTime": "2025-10-06T14:50:22.758397702Z",
"updateTime": "2025-10-06T14:50:22.758397702Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8a0fa60baa74612e761dc45cd01a7e1ef3e94fa0",
"ref": "refs/heads/main"
}
}
}
[14:50:23.214] Task URL: https://cr-buildbucket.appspot.com/build/8701749283467262225
[14:50:23.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:2:0) after 9m54s
[14:50:24.265] Received PubSub notification, asking Buildbucket for the build status
[14:50:24.279] Build status: SCHEDULED
[14:51:29.022] Received PubSub notification, asking Buildbucket for the build status
[14:51:29.047] Build status: STARTED
[15:00:17.267] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:2:0)
[15:00:17.286] Build status: STARTED
[15:00:17.286] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:5:0) after 6m37s
[15:06:54.309] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:5:0)
[15:06:54.433] Build status: STARTED
[15:06:54.433] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:6:0) after 2m34s
[15:09:28.497] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:6:0)
[15:09:28.529] Build status: STARTED
[15:09:28.529] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:7:0) after 5m27s
[15:14:55.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:7:0)
[15:14:55.609] Build status: STARTED
[15:14:55.609] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:8:0) after 4m26s
[15:19:35.809] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:8:0)
[15:19:35.834] Build status: STARTED
[15:19:35.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:9:0) after 3m28s
[15:23:03.884] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:9:0)
[15:23:03.909] Build status: STARTED
[15:23:03.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:10:0) after 3m53s
[15:26:56.928] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:10:0)
[15:26:56.958] Build status: STARTED
[15:26:56.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:11:0) after 2m15s
[15:29:12.072] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:11:0)
[15:29:12.088] Build status: STARTED
[15:29:12.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:12:0) after 9m40s
[15:38:52.152] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:12:0)
[15:38:52.213] Build status: STARTED
[15:38:52.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:13:0) after 8m47s
[15:47:39.354] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:13:0)
[15:47:39.368] Build status: STARTED
[15:47:39.368] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:14:0) after 5m17s
[15:52:56.398] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:14:0)
[15:52:56.429] Build status: STARTED
[15:52:56.429] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:15:0) after 8m17s
[16:01:13.481] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:15:0)
[16:01:13.555] Build status: STARTED
[16:01:13.555] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:16:0) after 2m3s
[16:03:16.618] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:16:0)
[16:03:16.635] Build status: STARTED
[16:03:16.635] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:17:0) after 7m32s
[16:10:48.684] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:17:0)
[16:10:49.166] Build status: STARTED
[16:10:49.166] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:18:0) after 2m33s
[16:13:22.227] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:18:0)
[16:13:22.239] Build status: STARTED
[16:13:22.239] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:19:0) after 6m23s
[16:19:45.324] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:19:0)
[16:19:45.339] Build status: STARTED
[16:19:45.339] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:20:0) after 2m15s
[16:22:00.551] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:20:0)
[16:22:00.570] Build status: STARTED
[16:22:00.570] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:21:0) after 2m52s
[16:24:52.605] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:21:0)
[16:24:52.622] Build status: STARTED
[16:24:52.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:22:0) after 7m45s
[16:32:37.687] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:22:0)
[16:32:37.705] Build status: STARTED
[16:32:37.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:23:0) after 8m30s
[16:41:07.827] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:23:0)
[16:41:07.891] Build status: STARTED
[16:41:07.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:24:0) after 1m50s
[16:42:58.012] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:24:0)
[16:42:58.040] Build status: STARTED
[16:42:58.040] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:25:0) after 5m5s
[16:48:03.106] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:25:0)
[16:48:03.123] Build status: STARTED
[16:48:03.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:26:0) after 1m20s
[16:49:23.107] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:26:0)
[16:49:23.136] Build status: STARTED
[16:49:23.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:27:0) after 6m24s
[16:55:47.157] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:27:0)
[16:55:47.172] Build status: STARTED
[16:55:47.173] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:28:0) after 2m16s
[16:58:03.195] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:28:0)
[16:58:03.212] Build status: STARTED
[16:58:03.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:29:0) after 3m51s
[17:01:54.244] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:29:0)
[17:01:54.264] Build status: STARTED
[17:01:54.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:30:0) after 7m8s
[17:09:02.283] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:30:0)
[17:09:02.326] Build status: STARTED
[17:09:02.326] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:31:0) after 3m4s
[17:12:06.341] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:31:0)
[17:12:06.361] Build status: STARTED
[17:12:06.361] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:32:0) after 7m49s
[17:19:55.378] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:32:0)
[17:19:55.401] Build status: STARTED
[17:19:55.401] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:33:0) after 6m44s
[17:26:39.427] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:33:0)
[17:26:39.442] Build status: STARTED
[17:26:39.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:34:0) after 7m43s
[17:34:22.503] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:34:0)
[17:34:22.595] Build status: STARTED
[17:34:22.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:35:0) after 9m56s
[17:44:18.658] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:35:0)
[17:44:18.674] Build status: STARTED
[17:44:18.674] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:36:0) after 2m25s
[17:46:43.690] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:36:0)
[17:46:43.704] Build status: STARTED
[17:46:43.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:37:0) after 3m7s
[17:49:50.679] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:37:0)
[17:49:50.697] Build status: STARTED
[17:49:50.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:38:0) after 8m29s
[17:58:19.724] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:38:0)
[17:58:19.740] Build status: STARTED
[17:58:19.740] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:39:0) after 7m28s
[18:05:47.805] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:39:0)
[18:05:47.826] Build status: STARTED
[18:05:47.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:40:0) after 2m8s
[18:07:55.960] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:40:0)
[18:07:55.987] Build status: STARTED
[18:07:55.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:41:0) after 4m5s
[18:12:01.004] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:41:0)
[18:12:01.026] Build status: STARTED
[18:12:01.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:42:0) after 1m44s
[18:13:45.015] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:42:0)
[18:13:45.032] Build status: STARTED
[18:13:45.033] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:43:0) after 1m24s
[18:15:09.034] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:43:0)
[18:15:09.111] Build status: STARTED
[18:15:09.111] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:44:0) after 9m28s
[18:24:37.128] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:44:0)
[18:24:37.144] Build status: STARTED
[18:24:37.144] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:45:0) after 3m17s
[18:27:54.212] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:45:0)
[18:27:54.234] Build status: STARTED
[18:27:54.234] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:46:0) after 7m51s
[18:35:45.346] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:46:0)
[18:35:45.360] Build status: STARTED
[18:35:45.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:47:0) after 4m9s
[18:39:54.376] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:47:0)
[18:39:54.403] Build status: STARTED
[18:39:54.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:48:0) after 2m29s
[18:42:23.421] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:48:0)
[18:42:23.436] Build status: STARTED
[18:42:23.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:49:0) after 2m13s
[18:44:36.484] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:49:0)
[18:44:36.545] Build status: STARTED
[18:44:36.545] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:50:0) after 7m3s
[18:51:39.569] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:50:0)
[18:51:39.589] Build status: STARTED
[18:51:39.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:51:0) after 7m58s
[18:59:37.673] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:51:0)
[18:59:37.688] Build status: STARTED
[18:59:37.688] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:52:0) after 6m27s
[19:06:04.712] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:52:0)
[19:06:04.799] Build status: STARTED
[19:06:04.799] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:53:0) after 7m15s
[19:13:19.819] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:53:0)
[19:13:19.835] Build status: STARTED
[19:13:19.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:54:0) after 7m37s
[19:20:56.862] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:54:0)
[19:20:56.878] Build status: STARTED
[19:20:56.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:55:0) after 5m8s
[19:26:04.991] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:55:0)
[19:26:05.007] Build status: STARTED
[19:26:05.007] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:56:0) after 6m49s
[19:32:54.205] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:56:0)
[19:32:54.221] Build status: STARTED
[19:32:54.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:57:0) after 7m45s
[19:40:39.250] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:57:0)
[19:40:39.274] Build status: STARTED
[19:40:39.274] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:58:0) after 2m28s
[19:43:07.340] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:58:0)
[19:43:07.396] Build status: STARTED
[19:43:07.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:59:0) after 8m48s
[19:51:55.572] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:59:0)
[19:51:55.587] Build status: STARTED
[19:51:55.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:60:0) after 8m56s
[20:00:51.611] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:60:0)
[20:00:51.642] Build status: STARTED
[20:00:51.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:61:0) after 1m51s
[20:02:42.701] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:61:0)
[20:02:42.745] Build status: STARTED
[20:02:42.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:62:0) after 2m19s
[20:05:01.797] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:62:0)
[20:05:01.843] Build status: STARTED
[20:05:01.843] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:63:0) after 4m52s
[20:09:53.863] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:63:0)
[20:09:53.880] Build status: STARTED
[20:09:53.880] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867179346389711680:64:0) after 3m57s
[20:13:25.881] Received PubSub notification, asking Buildbucket for the build status
[20:13:25.896] Build:
{
"id": "8701749283467262225",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27183,
"createdBy": "project:chromium",
"createTime": "2025-10-06T14:50:22.758397702Z",
"startTime": "2025-10-06T14:51:28.860205567Z",
"endTime": "2025-10-06T20:13:25.068849957Z",
"updateTime": "2025-10-06T20:13:25.068849957Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "8a0fa60baa74612e761dc45cd01a7e1ef3e94fa0",
"ref": "refs/heads/main"
}
}
}
[20:13:25.896] Invocation finished in 5h23m5.308364652s with status FAILED