[11:04:56.257] New invocation is queued and will start shortly
[11:04:57.382] Starting the invocation (attempt 1)
[11:04:57.408] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/961170ae7de8a6505b22be7e0a6d4f4a463e73be"
[11:04:57.408] Popped gitiles commit info from properties and tags
[11:04:57.408] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[11:04:57.408] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[11:04:57.408] Buildbucket request:
{
"requestId": "8841463989224513920",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "961170ae7de8a6505b22be7e0a6d4f4a463e73be",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841463989224513920"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[11:04:57.932] Scheduled build:
{
"id": "8676033927285087825",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31517,
"createdBy": "project:chromium",
"createTime": "2026-07-17T11:04:57.486099745Z",
"updateTime": "2026-07-17T11:04:57.486099745Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "961170ae7de8a6505b22be7e0a6d4f4a463e73be",
"ref": "refs/heads/main"
}
}
}
[11:04:57.932] Task URL: https://cr-buildbucket.appspot.com/build/8676033927285087825
[11:04:57.932] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:2:0) after 6m30s
[11:04:58.872] Received PubSub notification, asking Buildbucket for the build status
[11:04:58.888] Build status: SCHEDULED
[11:06:15.728] Received PubSub notification, asking Buildbucket for the build status
[11:06:15.757] Build status: STARTED
[11:11:27.949] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:2:0)
[11:11:27.982] Build status: STARTED
[11:11:27.982] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:5:0) after 4m45s
[11:16:13.214] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:5:0)
[11:16:13.246] Build status: STARTED
[11:16:13.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:6:0) after 5m27s
[11:21:40.284] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:6:0)
[11:21:40.315] Build status: STARTED
[11:21:40.315] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:7:0) after 3m48s
[11:25:28.370] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:7:0)
[11:25:28.413] Build status: STARTED
[11:25:28.413] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:8:0) after 7m54s
[11:33:22.560] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:8:0)
[11:33:22.585] Build status: STARTED
[11:33:22.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:9:0) after 3m38s
[11:37:00.623] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:9:0)
[11:37:00.650] Build status: STARTED
[11:37:00.650] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:10:0) after 7m20s
[11:44:20.747] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:10:0)
[11:44:20.778] Build status: STARTED
[11:44:20.778] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:11:0) after 3m58s
[11:48:18.801] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:11:0)
[11:48:18.821] Build status: STARTED
[11:48:18.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:12:0) after 5m58s
[11:54:16.836] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:12:0)
[11:54:16.866] Build status: STARTED
[11:54:16.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:13:0) after 8m22s
[12:02:38.968] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:13:0)
[12:02:38.987] Build status: STARTED
[12:02:38.987] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:14:0) after 2m16s
[12:04:55.119] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:14:0)
[12:04:55.145] Build status: STARTED
[12:04:55.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:15:0) after 9m9s
[12:14:04.226] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:15:0)
[12:14:04.252] Build status: STARTED
[12:14:04.252] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:16:0) after 3m25s
[12:17:29.340] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:16:0)
[12:17:29.363] Build status: STARTED
[12:17:29.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:17:0) after 1m18s
[12:18:47.384] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:17:0)
[12:18:47.423] Build status: STARTED
[12:18:47.423] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:18:0) after 9m4s
[12:27:51.550] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:18:0)
[12:27:51.629] Build status: STARTED
[12:27:51.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:19:0) after 5m43s
[12:33:34.715] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:19:0)
[12:33:34.770] Build status: STARTED
[12:33:34.770] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:20:0) after 9m1s
[12:42:35.803] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:20:0)
[12:42:35.820] Build status: STARTED
[12:42:35.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:21:0) after 2m16s
[12:44:51.855] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:21:0)
[12:44:51.878] Build status: STARTED
[12:44:51.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:22:0) after 2m16s
[12:47:07.894] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:22:0)
[12:47:07.919] Build status: STARTED
[12:47:07.919] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:23:0) after 2m20s
[12:49:28.222] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:23:0)
[12:49:28.244] Build status: STARTED
[12:49:28.244] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:24:0) after 3m58s
[12:53:26.262] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:24:0)
[12:53:26.348] Build status: STARTED
[12:53:26.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:25:0) after 2m44s
[12:56:10.336] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:25:0)
[12:56:10.352] Build status: STARTED
[12:56:10.352] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:26:0) after 3m50s
[13:00:00.448] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:26:0)
[13:00:00.468] Build status: STARTED
[13:00:00.468] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:27:0) after 3m51s
[13:03:51.621] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:27:0)
[13:03:51.637] Build status: STARTED
[13:03:51.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:28:0) after 6m2s
[13:09:53.670] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:28:0)
[13:09:53.693] Build status: STARTED
[13:09:53.693] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:29:0) after 9m54s
[13:19:47.790] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:29:0)
[13:19:47.811] Build status: STARTED
[13:19:47.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:30:0) after 9m31s
[13:29:19.471] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:30:0)
[13:29:19.492] Build status: STARTED
[13:29:19.492] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:31:0) after 7m30s
[13:36:49.608] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:31:0)
[13:36:49.622] Build status: STARTED
[13:36:49.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:32:0) after 9m47s
[13:46:36.639] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:32:0)
[13:46:36.663] Build status: STARTED
[13:46:36.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:33:0) after 1m16s
[13:47:52.643] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:33:0)
[13:47:52.658] Build status: STARTED
[13:47:52.658] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:34:0) after 8m14s
[13:56:06.676] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:34:0)
[13:56:06.720] Build status: STARTED
[13:56:06.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:35:0) after 6m8s
[14:02:14.750] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:35:0)
[14:02:14.769] Build status: STARTED
[14:02:14.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:36:0) after 1m13s
[14:03:27.789] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:36:0)
[14:03:27.807] Build status: STARTED
[14:03:27.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:37:0) after 1m54s
[14:05:21.853] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:37:0)
[14:05:21.878] Build status: STARTED
[14:05:21.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:38:0) after 2m4s
[14:07:25.986] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:38:0)
[14:07:26.006] Build status: STARTED
[14:07:26.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:39:0) after 6m7s
[14:13:33.087] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:39:0)
[14:13:33.135] Build status: STARTED
[14:13:33.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:40:0) after 1m49s
[14:15:22.373] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:40:0)
[14:15:22.393] Build status: STARTED
[14:15:22.393] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:41:0) after 8m27s
[14:23:49.406] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:41:0)
[14:23:49.451] Build status: STARTED
[14:23:49.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:42:0) after 8m7s
[14:31:56.602] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:42:0)
[14:31:56.620] Build status: STARTED
[14:31:56.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:43:0) after 4m30s
[14:36:26.640] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:43:0)
[14:36:26.657] Build status: STARTED
[14:36:26.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:44:0) after 4m17s
[14:40:43.737] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:44:0)
[14:40:43.760] Build status: STARTED
[14:40:43.760] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:45:0) after 5m59s
[14:46:42.837] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:45:0)
[14:46:42.856] Build status: STARTED
[14:46:42.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:46:0) after 5m12s
[14:51:54.877] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:46:0)
[14:51:54.901] Build status: STARTED
[14:51:54.901] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:47:0) after 3m10s
[14:55:05.014] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:47:0)
[14:55:05.036] Build status: STARTED
[14:55:05.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8841463989224513920:48:0) after 6m18s
[14:58:27.673] Received PubSub notification, asking Buildbucket for the build status
[14:58:27.703] Build:
{
"id": "8676033927285087825",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 31517,
"createdBy": "project:chromium",
"createTime": "2026-07-17T11:04:57.486099745Z",
"startTime": "2026-07-17T11:06:15.515512321Z",
"endTime": "2026-07-17T14:58:25.995100772Z",
"updateTime": "2026-07-17T14:58:25.995100772Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "961170ae7de8a6505b22be7e0a6d4f4a463e73be",
"ref": "refs/heads/main"
}
}
}
[14:58:27.703] Invocation finished in 3h53m31.45049724s with status FAILED