[21:17:28.792] New invocation is queued and will start shortly
[21:17:32.690] Starting the invocation (attempt 1)
[21:17:32.716] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/153676948fbe8f12a60243fd70beb30abf3222d9"
[21:17:32.716] Popped gitiles commit info from properties and tags
[21:17:32.716] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:17:32.716] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:17:32.716] Buildbucket request:
{
"requestId": "8851844102535801328",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "153676948fbe8f12a60243fd70beb30abf3222d9",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851844102535801328"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:17:33.385] Scheduled build:
{
"id": "8686414037677182369",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 29865,
"createdBy": "project:chromium",
"createTime": "2026-03-24T21:17:32.804035835Z",
"updateTime": "2026-03-24T21:17:32.804035835Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "153676948fbe8f12a60243fd70beb30abf3222d9",
"ref": "refs/heads/main"
}
}
}
[21:17:33.385] Task URL: https://cr-buildbucket.appspot.com/build/8686414037677182369
[21:17:33.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:2:0) after 9m54s
[21:17:34.318] Received PubSub notification, asking Buildbucket for the build status
[21:17:34.334] Build status: SCHEDULED
[21:18:37.509] Received PubSub notification, asking Buildbucket for the build status
[21:18:37.543] Build status: STARTED
[21:27:27.420] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:2:0)
[21:27:27.440] Build status: STARTED
[21:27:27.440] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:5:0) after 9m34s
[21:37:01.635] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:5:0)
[21:37:01.709] Build status: STARTED
[21:37:01.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:6:0) after 2m46s
[21:39:47.729] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:6:0)
[21:39:47.746] Build status: STARTED
[21:39:47.747] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:7:0) after 7m4s
[21:46:51.773] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:7:0)
[21:46:51.792] Build status: STARTED
[21:46:51.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:8:0) after 2m28s
[21:49:19.817] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:8:0)
[21:49:19.854] Build status: STARTED
[21:49:19.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:9:0) after 8m11s
[21:57:30.888] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:9:0)
[21:57:30.905] Build status: STARTED
[21:57:30.905] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:10:0) after 1m17s
[21:58:47.931] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:10:0)
[21:58:47.945] Build status: STARTED
[21:58:47.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:11:0) after 5m16s
[22:04:04.105] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:11:0)
[22:04:04.126] Build status: STARTED
[22:04:04.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:12:0) after 2m13s
[22:06:17.213] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:12:0)
[22:06:17.229] Build status: STARTED
[22:06:17.229] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:13:0) after 4m16s
[22:10:33.247] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:13:0)
[22:10:33.273] Build status: STARTED
[22:10:33.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:14:0) after 2m1s
[22:12:34.295] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:14:0)
[22:12:34.311] Build status: STARTED
[22:12:34.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:15:0) after 2m46s
[22:15:20.330] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:15:0)
[22:15:20.417] Build status: STARTED
[22:15:20.417] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:16:0) after 6m10s
[22:21:30.438] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:16:0)
[22:21:30.455] Build status: STARTED
[22:21:30.455] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:17:0) after 3m24s
[22:24:54.479] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:17:0)
[22:24:54.501] Build status: STARTED
[22:24:54.501] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:18:0) after 7m0s
[22:31:54.522] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:18:0)
[22:31:54.537] Build status: STARTED
[22:31:54.537] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:19:0) after 1m49s
[22:33:43.597] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:19:0)
[22:33:43.638] Build status: STARTED
[22:33:43.639] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:20:0) after 8m14s
[22:41:57.661] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:20:0)
[22:41:57.702] Build status: STARTED
[22:41:57.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:21:0) after 4m44s
[22:46:41.729] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:21:0)
[22:46:41.751] Build status: STARTED
[22:46:41.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:22:0) after 7m2s
[22:53:43.769] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:22:0)
[22:53:43.784] Build status: STARTED
[22:53:43.784] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:23:0) after 9m44s
[23:03:27.875] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:23:0)
[23:03:27.892] Build status: STARTED
[23:03:27.892] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:24:0) after 9m14s
[23:12:41.981] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:24:0)
[23:12:42.000] Build status: STARTED
[23:12:42.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:25:0) after 9m33s
[23:22:15.022] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:25:0)
[23:22:15.104] Build status: STARTED
[23:22:15.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:26:0) after 6m58s
[23:29:13.121] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:26:0)
[23:29:13.221] Build status: STARTED
[23:29:13.221] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:27:0) after 6m10s
[23:35:23.227] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:27:0)
[23:35:23.263] Build status: STARTED
[23:35:23.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:28:0) after 4m25s
[23:39:48.287] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:28:0)
[23:39:48.304] Build status: STARTED
[23:39:48.304] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:29:0) after 9m40s
[23:49:28.398] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:29:0)
[23:49:28.412] Build status: STARTED
[23:49:28.412] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:30:0) after 1m44s
[23:51:12.479] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:30:0)
[23:51:12.511] Build status: STARTED
[23:51:12.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:31:0) after 3m41s
[23:54:53.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:31:0)
[23:54:53.709] Build status: STARTED
[23:54:53.709] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:32:0) after 1m39s
[23:56:32.777] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:32:0)
[23:56:32.794] Build status: STARTED
[23:56:32.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:33:0) after 1m9s
[23:57:41.781] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:33:0)
[23:57:41.797] Build status: STARTED
[23:57:41.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:34:0) after 6m1s
[00:03:42.909] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:34:0)
[00:03:42.927] Build status: STARTED
[00:03:42.927] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:35:0) after 6m9s
[00:09:51.960] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:35:0)
[00:09:51.983] Build status: STARTED
[00:09:51.983] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:36:0) after 3m35s
[00:13:27.013] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:36:0)
[00:13:27.044] Build status: STARTED
[00:13:27.044] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:37:0) after 8m3s
[00:21:30.046] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:37:0)
[00:21:30.063] Build status: STARTED
[00:21:30.063] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:38:0) after 3m34s
[00:25:04.278] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:38:0)
[00:25:04.330] Build status: STARTED
[00:25:04.330] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:39:0) after 2m43s
[00:27:47.349] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:39:0)
[00:27:47.389] Build status: STARTED
[00:27:47.389] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:40:0) after 9m28s
[00:37:15.503] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:40:0)
[00:37:15.524] Build status: STARTED
[00:37:15.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:41:0) after 8m53s
[00:46:08.637] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:41:0)
[00:46:08.657] Build status: STARTED
[00:46:08.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:42:0) after 6m59s
[00:53:07.744] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:42:0)
[00:53:07.757] Build status: STARTED
[00:53:07.757] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:43:0) after 4m9s
[00:57:16.836] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:43:0)
[00:57:16.867] Build status: STARTED
[00:57:16.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:44:0) after 4m52s
[01:02:08.935] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:44:0)
[01:02:08.953] Build status: STARTED
[01:02:08.953] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:45:0) after 8m54s
[01:11:02.973] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:45:0)
[01:11:03.018] Build status: STARTED
[01:11:03.018] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:46:0) after 4m34s
[01:15:36.992] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:46:0)
[01:15:37.056] Build status: STARTED
[01:15:37.056] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:47:0) after 7m9s
[01:22:46.211] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:47:0)
[01:22:46.237] Build status: STARTED
[01:22:46.237] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8851844102535801328:48:0) after 6m8s
[01:23:10.066] Received PubSub notification, asking Buildbucket for the build status
[01:23:10.133] Build:
{
"id": "8686414037677182369",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 29865,
"createdBy": "project:chromium",
"createTime": "2026-03-24T21:17:32.804035835Z",
"startTime": "2026-03-24T21:18:37.289776011Z",
"endTime": "2026-03-25T01:23:09.790295832Z",
"updateTime": "2026-03-25T01:23:09.790295832Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "153676948fbe8f12a60243fd70beb30abf3222d9",
"ref": "refs/heads/main"
}
}
}
[01:23:10.133] Invocation finished in 4h5m41.346322288s with status FAILED