[11:04:26.159] New invocation is queued and will start shortly
[11:04:27.255] Starting the invocation (attempt 1)
[11:04:27.280] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/500d79833eb107684c744a44524b9de7002a6beb"
[11:04:27.280] Popped gitiles commit info from properties and tags
[11:04:27.280] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:04:27.281] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:04:27.281] Buildbucket request:
{
"requestId": "8843638347979631584",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "500d79833eb107684c744a44524b9de7002a6beb",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8843638347979631584"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:04:27.810] Scheduled build:
{
"id": "8678208286072764657",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31141,
"createdBy": "project:chromium",
"createTime": "2026-06-23T11:04:27.356889497Z",
"updateTime": "2026-06-23T11:04:27.356889497Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "500d79833eb107684c744a44524b9de7002a6beb",
"ref": "refs/heads/main"
}
}
}
[11:04:27.810] Task URL: https://cr-buildbucket.appspot.com/build/8678208286072764657
[11:04:27.810] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:2:0) after 8m30s
[11:04:28.737] Received PubSub notification, asking Buildbucket for the build status
[11:04:28.756] Build status: SCHEDULED
[11:05:45.542] Received PubSub notification, asking Buildbucket for the build status
[11:05:45.558] Build status: STARTED
[11:12:57.929] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:2:0)
[11:12:58.002] Build status: STARTED
[11:12:58.002] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:5:0) after 7m26s
[11:20:24.115] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:5:0)
[11:20:24.150] Build status: STARTED
[11:20:24.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:6:0) after 7m4s
[11:27:28.173] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:6:0)
[11:27:28.198] Build status: STARTED
[11:27:28.198] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:7:0) after 6m20s
[11:33:48.214] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:7:0)
[11:33:48.231] Build status: STARTED
[11:33:48.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:8:0) after 5m2s
[11:38:50.248] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:8:0)
[11:38:50.265] Build status: STARTED
[11:38:50.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:9:0) after 2m39s
[11:41:29.322] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:9:0)
[11:41:29.342] Build status: STARTED
[11:41:29.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:10:0) after 4m5s
[11:45:34.388] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:10:0)
[11:45:34.411] Build status: STARTED
[11:45:34.411] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:11:0) after 4m7s
[11:49:41.427] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:11:0)
[11:49:41.542] Build status: STARTED
[11:49:41.542] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:12:0) after 4m11s
[11:53:52.602] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:12:0)
[11:53:52.622] Build status: STARTED
[11:53:52.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:13:0) after 4m25s
[11:58:17.619] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:13:0)
[11:58:17.637] Build status: STARTED
[11:58:17.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:14:0) after 5m26s
[12:03:43.705] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:14:0)
[12:03:43.722] Build status: STARTED
[12:03:43.722] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:15:0) after 7m4s
[12:10:47.759] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:15:0)
[12:10:47.779] Build status: STARTED
[12:10:47.779] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:16:0) after 3m0s
[12:13:47.855] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:16:0)
[12:13:47.875] Build status: STARTED
[12:13:47.875] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:17:0) after 3m7s
[12:16:54.856] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:17:0)
[12:16:54.873] Build status: STARTED
[12:16:54.873] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:18:0) after 7m30s
[12:24:24.922] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:18:0)
[12:24:24.944] Build status: STARTED
[12:24:24.944] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:19:0) after 2m32s
[12:26:57.002] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:19:0)
[12:26:57.027] Build status: STARTED
[12:26:57.027] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:20:0) after 6m54s
[12:33:51.044] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:20:0)
[12:33:51.059] Build status: STARTED
[12:33:51.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:21:0) after 2m43s
[12:36:34.129] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:21:0)
[12:36:34.237] Build status: STARTED
[12:36:34.237] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:22:0) after 3m3s
[12:39:37.446] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:22:0)
[12:39:37.571] Build status: STARTED
[12:39:37.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:23:0) after 8m50s
[12:48:27.602] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:23:0)
[12:48:27.696] Build status: STARTED
[12:48:27.696] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:24:0) after 1m52s
[12:50:19.718] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:24:0)
[12:50:19.737] Build status: STARTED
[12:50:19.737] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:25:0) after 9m15s
[12:59:34.839] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:25:0)
[12:59:34.855] Build status: STARTED
[12:59:34.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:26:0) after 7m13s
[13:06:47.831] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:26:0)
[13:06:47.846] Build status: STARTED
[13:06:47.846] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:27:0) after 2m42s
[13:09:29.894] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:27:0)
[13:09:29.910] Build status: STARTED
[13:09:29.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:28:0) after 1m35s
[13:11:04.886] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:28:0)
[13:11:04.997] Build status: STARTED
[13:11:04.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:29:0) after 2m56s
[13:14:01.083] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:29:0)
[13:14:01.101] Build status: STARTED
[13:14:01.101] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:30:0) after 3m45s
[13:17:46.285] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:30:0)
[13:17:46.369] Build status: STARTED
[13:17:46.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:31:0) after 2m11s
[13:19:57.421] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:31:0)
[13:19:57.448] Build status: STARTED
[13:19:57.449] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:32:0) after 6m51s
[13:26:48.556] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:32:0)
[13:26:48.613] Build status: STARTED
[13:26:48.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:33:0) after 9m10s
[13:35:58.630] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:33:0)
[13:35:58.648] Build status: STARTED
[13:35:58.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:34:0) after 7m0s
[13:42:58.670] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:34:0)
[13:42:58.695] Build status: STARTED
[13:42:58.695] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:35:0) after 8m21s
[13:51:19.715] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:35:0)
[13:51:19.738] Build status: STARTED
[13:51:19.738] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:36:0) after 9m53s
[14:01:12.807] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:36:0)
[14:01:12.873] Build status: STARTED
[14:01:12.873] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:37:0) after 8m22s
[14:09:34.890] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:37:0)
[14:09:34.909] Build status: STARTED
[14:09:34.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:38:0) after 4m25s
[14:13:59.977] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:38:0)
[14:14:00.000] Build status: STARTED
[14:14:00.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:39:0) after 3m30s
[14:17:30.037] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:39:0)
[14:17:30.069] Build status: STARTED
[14:17:30.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:40:0) after 8m37s
[14:26:07.087] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:40:0)
[14:26:07.102] Build status: STARTED
[14:26:07.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:41:0) after 8m18s
[14:34:25.171] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:41:0)
[14:34:25.211] Build status: STARTED
[14:34:25.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:42:0) after 1m5s
[14:35:30.277] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:42:0)
[14:35:30.295] Build status: STARTED
[14:35:30.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:43:0) after 4m26s
[14:39:56.345] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:43:0)
[14:39:56.365] Build status: STARTED
[14:39:56.365] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:44:0) after 6m55s
[14:46:51.498] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:44:0)
[14:46:51.519] Build status: STARTED
[14:46:51.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:45:0) after 8m36s
[14:55:27.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:45:0)
[14:55:27.614] Build status: STARTED
[14:55:27.614] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8843638347979631584:46:0) after 3m48s
[14:58:28.295] Received PubSub notification, asking Buildbucket for the build status
[14:58:28.332] Build:
{
"id": "8678208286072764657",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31141,
"createdBy": "project:chromium",
"createTime": "2026-06-23T11:04:27.356889497Z",
"startTime": "2026-06-23T11:05:44.898280312Z",
"endTime": "2026-06-23T14:58:27.972204042Z",
"updateTime": "2026-06-23T14:58:27.972204042Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "500d79833eb107684c744a44524b9de7002a6beb",
"ref": "refs/heads/main"
}
}
}
[14:58:28.332] Invocation finished in 3h54m2.177601073s with status FAILED