[21:34:36.205] New invocation is queued and will start shortly
[21:34:37.330] Starting the invocation (attempt 1)
[21:34:37.359] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/db596e4e0730af77a265ade44f031387618810b6"
[21:34:37.359] Popped gitiles commit info from properties and tags
[21:34:37.359] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:34:37.360] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:34:37.360] Buildbucket request:
{
"requestId": "8887357036047979856",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "db596e4e0730af77a265ade44f031387618810b6",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8887357036047979856"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:34:37.960] Scheduled build:
{
"id": "8721926974121377489",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 24636,
"createdBy": "project:chromium",
"createTime": "2025-02-25T21:34:37.417604784Z",
"updateTime": "2025-02-25T21:34:37.417604784Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "db596e4e0730af77a265ade44f031387618810b6",
"ref": "refs/heads/main"
}
}
}
[21:34:37.960] Task URL: https://cr-buildbucket.appspot.com/build/8721926974121377489
[21:34:37.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:2:0) after 8m5s
[21:34:43.723] Received PubSub notification, asking Buildbucket for the build status
[21:34:43.749] Build status: SCHEDULED
[21:35:31.900] Received PubSub notification, asking Buildbucket for the build status
[21:35:32.124] Build status: STARTED
[21:42:43.057] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:2:0)
[21:42:43.124] Build status: STARTED
[21:42:43.125] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:5:0) after 3m56s
[21:46:39.175] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:5:0)
[21:46:39.197] Build status: STARTED
[21:46:39.197] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:6:0) after 6m48s
[21:53:27.232] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:6:0)
[21:53:27.257] Build status: STARTED
[21:53:27.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:7:0) after 4m13s
[21:57:40.335] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:7:0)
[21:57:40.368] Build status: STARTED
[21:57:40.368] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:8:0) after 6m50s
[22:04:30.386] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:8:0)
[22:04:30.418] Build status: STARTED
[22:04:30.418] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:9:0) after 4m39s
[22:09:09.529] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:9:0)
[22:09:09.620] Build status: STARTED
[22:09:09.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:10:0) after 7m26s
[22:16:35.621] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:10:0)
[22:16:35.642] Build status: STARTED
[22:16:35.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:11:0) after 3m43s
[22:20:18.672] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:11:0)
[22:20:18.694] Build status: STARTED
[22:20:18.694] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:12:0) after 6m19s
[22:26:37.760] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:12:0)
[22:26:37.781] Build status: STARTED
[22:26:37.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:13:0) after 3m12s
[22:29:49.776] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:13:0)
[22:29:49.791] Build status: STARTED
[22:29:49.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:14:0) after 4m50s
[22:34:39.873] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:14:0)
[22:34:39.908] Build status: STARTED
[22:34:39.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:15:0) after 5m43s
[22:40:22.938] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:15:0)
[22:40:22.959] Build status: STARTED
[22:40:22.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:16:0) after 8m47s
[22:49:09.979] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:16:0)
[22:49:10.006] Build status: STARTED
[22:49:10.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:17:0) after 2m54s
[22:52:04.044] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:17:0)
[22:52:04.082] Build status: STARTED
[22:52:04.082] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:18:0) after 1m15s
[22:53:19.166] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:18:0)
[22:53:19.181] Build status: STARTED
[22:53:19.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:19:0) after 3m30s
[22:56:49.166] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:19:0)
[22:56:49.187] Build status: STARTED
[22:56:49.187] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:20:0) after 6m22s
[23:03:11.217] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:20:0)
[23:03:11.239] Build status: STARTED
[23:03:11.239] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:21:0) after 4m22s
[23:07:33.269] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:21:0)
[23:07:33.283] Build status: STARTED
[23:07:33.283] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:22:0) after 4m23s
[23:11:56.306] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:22:0)
[23:11:56.354] Build status: STARTED
[23:11:56.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:23:0) after 1m42s
[23:13:38.373] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:23:0)
[23:13:38.393] Build status: STARTED
[23:13:38.393] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:24:0) after 4m43s
[23:18:21.462] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:24:0)
[23:18:21.522] Build status: STARTED
[23:18:21.522] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:25:0) after 5m49s
[23:24:10.575] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:25:0)
[23:24:10.602] Build status: STARTED
[23:24:10.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:26:0) after 8m13s
[23:32:23.630] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:26:0)
[23:32:23.652] Build status: STARTED
[23:32:23.652] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:27:0) after 9m8s
[23:41:31.779] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:27:0)
[23:41:31.796] Build status: STARTED
[23:41:31.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:28:0) after 8m13s
[23:49:44.821] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:28:0)
[23:49:44.872] Build status: STARTED
[23:49:44.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:29:0) after 2m18s
[23:52:03.045] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:29:0)
[23:52:03.065] Build status: STARTED
[23:52:03.065] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:30:0) after 7m40s
[23:59:43.093] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:30:0)
[23:59:43.131] Build status: STARTED
[23:59:43.131] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:31:0) after 2m16s
[00:01:59.204] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:31:0)
[00:01:59.239] Build status: STARTED
[00:01:59.239] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:32:0) after 1m33s
[00:03:32.272] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:32:0)
[00:03:32.285] Build status: STARTED
[00:03:32.286] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:33:0) after 4m38s
[00:08:10.261] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:33:0)
[00:08:10.292] Build status: STARTED
[00:08:10.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:34:0) after 3m43s
[00:11:53.366] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:34:0)
[00:11:53.380] Build status: STARTED
[00:11:53.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:35:0) after 3m31s
[00:15:24.422] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:35:0)
[00:15:24.441] Build status: STARTED
[00:15:24.441] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:36:0) after 4m48s
[00:20:12.472] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:36:0)
[00:20:12.551] Build status: STARTED
[00:20:12.551] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:37:0) after 1m47s
[00:21:59.647] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:37:0)
[00:21:59.664] Build status: STARTED
[00:21:59.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:38:0) after 8m6s
[00:30:05.760] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:38:0)
[00:30:05.781] Build status: STARTED
[00:30:05.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:39:0) after 6m42s
[00:36:47.841] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:39:0)
[00:36:47.861] Build status: STARTED
[00:36:47.861] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:40:0) after 5m43s
[00:42:30.893] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:40:0)
[00:42:30.908] Build status: STARTED
[00:42:30.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:41:0) after 6m45s
[00:49:15.939] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:41:0)
[00:49:15.960] Build status: STARTED
[00:49:15.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:42:0) after 7m52s
[00:57:08.058] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:42:0)
[00:57:08.074] Build status: STARTED
[00:57:08.074] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:43:0) after 6m30s
[01:03:38.132] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:43:0)
[01:03:38.154] Build status: STARTED
[01:03:38.154] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:44:0) after 8m52s
[01:12:30.232] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:44:0)
[01:12:30.257] Build status: STARTED
[01:12:30.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:45:0) after 4m21s
[01:16:51.369] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:45:0)
[01:16:51.442] Build status: STARTED
[01:16:51.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:46:0) after 2m48s
[01:19:39.469] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:46:0)
[01:19:39.485] Build status: STARTED
[01:19:39.485] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:47:0) after 9m8s
[01:28:47.533] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:47:0)
[01:28:47.552] Build status: STARTED
[01:28:47.552] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:48:0) after 5m7s
[01:33:54.601] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:48:0)
[01:33:54.620] Build status: STARTED
[01:33:54.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:49:0) after 2m38s
[01:36:32.650] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:49:0)
[01:36:32.667] Build status: STARTED
[01:36:32.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:50:0) after 5m14s
[01:41:46.677] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:50:0)
[01:41:46.697] Build status: STARTED
[01:41:46.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:51:0) after 4m56s
[01:46:42.776] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:51:0)
[01:46:42.810] Build status: STARTED
[01:46:42.810] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:52:0) after 4m46s
[01:51:28.837] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:52:0)
[01:51:28.918] Build status: STARTED
[01:51:28.918] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:53:0) after 9m23s
[02:00:51.945] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:53:0)
[02:00:51.962] Build status: STARTED
[02:00:51.962] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:54:0) after 8m34s
[02:09:26.099] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:54:0)
[02:09:26.135] Build status: STARTED
[02:09:26.135] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:55:0) after 4m11s
[02:13:37.169] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:55:0)
[02:13:37.185] Build status: STARTED
[02:13:37.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:56:0) after 5m26s
[02:19:03.217] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:56:0)
[02:19:03.231] Build status: STARTED
[02:19:03.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:57:0) after 2m21s
[02:21:24.311] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:57:0)
[02:21:24.333] Build status: STARTED
[02:21:24.333] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:58:0) after 4m12s
[02:25:36.358] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:58:0)
[02:25:36.377] Build status: STARTED
[02:25:36.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:59:0) after 8m43s
[02:34:19.492] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:59:0)
[02:34:19.505] Build status: STARTED
[02:34:19.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:60:0) after 5m46s
[02:40:05.522] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:60:0)
[02:40:05.542] Build status: STARTED
[02:40:05.542] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:61:0) after 1m54s
[02:41:59.610] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:61:0)
[02:41:59.634] Build status: STARTED
[02:41:59.634] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:62:0) after 1m8s
[02:43:07.662] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:62:0)
[02:43:07.678] Build status: STARTED
[02:43:07.678] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:63:0) after 1m1s
[02:44:08.671] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:63:0)
[02:44:08.691] Build status: STARTED
[02:44:08.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:64:0) after 9m17s
[02:53:25.883] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:64:0)
[02:53:25.900] Build status: STARTED
[02:53:25.900] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:65:0) after 2m34s
[02:55:59.981] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:65:0)
[02:56:00.053] Build status: STARTED
[02:56:00.053] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:66:0) after 5m57s
[03:01:57.080] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:66:0)
[03:01:57.120] Build status: STARTED
[03:01:57.120] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8887357036047979856:67:0) after 9m52s
[03:09:16.401] Received PubSub notification, asking Buildbucket for the build status
[03:09:16.416] Build:
{
"id": "8721926974121377489",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 24636,
"createdBy": "project:chromium",
"createTime": "2025-02-25T21:34:37.417604784Z",
"startTime": "2025-02-25T21:35:31.716217258Z",
"endTime": "2025-02-26T03:09:16.043444227Z",
"updateTime": "2025-02-26T03:09:16.043444227Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "db596e4e0730af77a265ade44f031387618810b6",
"ref": "refs/heads/main"
}
}
}
[03:09:16.416] Invocation finished in 5h34m40.220054263s with status FAILED