[03:35:39.269] New invocation is queued and will start shortly
[03:35:40.417] Starting the invocation (attempt 1)
[03:35:40.450] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/2838d14cffbafdf205da43e2b80dc93b481d5e02"
[03:35:40.451] Popped gitiles commit info from properties and tags
[03:35:40.451] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[03:35:40.451] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[03:35:40.451] Buildbucket request:
{
"requestId": "8867221794138680656",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "2838d14cffbafdf205da43e2b80dc93b481d5e02",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867221794138680656"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[03:35:40.949] Scheduled build:
{
"id": "8701791732168489649",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27181,
"createdBy": "project:chromium",
"createTime": "2025-10-06T03:35:40.522394951Z",
"updateTime": "2025-10-06T03:35:40.522394951Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "2838d14cffbafdf205da43e2b80dc93b481d5e02",
"ref": "refs/heads/main"
}
}
}
[03:35:40.949] Task URL: https://cr-buildbucket.appspot.com/build/8701791732168489649
[03:35:40.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:2:0) after 1m5s
[03:35:41.894] Received PubSub notification, asking Buildbucket for the build status
[03:35:41.943] Build status: SCHEDULED
[03:36:27.079] Received PubSub notification, asking Buildbucket for the build status
[03:36:27.096] Build status: STARTED
[03:36:45.968] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:2:0)
[03:36:45.986] Build status: STARTED
[03:36:45.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:5:0) after 4m15s
[03:41:01.125] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:5:0)
[03:41:01.142] Build status: STARTED
[03:41:01.142] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:6:0) after 2m41s
[03:43:42.120] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:6:0)
[03:43:42.143] Build status: STARTED
[03:43:42.143] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:7:0) after 5m23s
[03:49:05.161] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:7:0)
[03:49:05.177] Build status: STARTED
[03:49:05.177] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:8:0) after 4m21s
[03:53:26.202] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:8:0)
[03:53:26.220] Build status: STARTED
[03:53:26.220] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:9:0) after 4m37s
[03:58:03.327] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:9:0)
[03:58:03.349] Build status: STARTED
[03:58:03.349] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:10:0) after 3m56s
[04:01:59.418] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:10:0)
[04:01:59.455] Build status: STARTED
[04:01:59.455] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:11:0) after 4m31s
[04:06:30.475] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:11:0)
[04:06:30.498] Build status: STARTED
[04:06:30.498] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:12:0) after 2m27s
[04:08:57.605] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:12:0)
[04:08:57.624] Build status: STARTED
[04:08:57.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:13:0) after 7m33s
[04:16:30.716] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:13:0)
[04:16:30.775] Build status: STARTED
[04:16:30.775] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:14:0) after 8m15s
[04:24:45.910] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:14:0)
[04:24:45.981] Build status: STARTED
[04:24:45.981] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:15:0) after 7m49s
[04:32:35.052] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:15:0)
[04:32:35.116] Build status: STARTED
[04:32:35.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:16:0) after 3m30s
[04:36:05.304] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:16:0)
[04:36:05.319] Build status: STARTED
[04:36:05.319] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:17:0) after 4m1s
[04:40:06.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:17:0)
[04:40:06.624] Build status: STARTED
[04:40:06.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:18:0) after 6m48s
[04:46:54.848] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:18:0)
[04:46:54.918] Build status: STARTED
[04:46:54.918] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:19:0) after 7m51s
[04:54:45.965] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:19:0)
[04:54:46.025] Build status: STARTED
[04:54:46.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:20:0) after 9m46s
[05:04:32.077] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:20:0)
[05:04:32.092] Build status: STARTED
[05:04:32.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:21:0) after 8m48s
[05:13:20.296] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:21:0)
[05:13:20.330] Build status: STARTED
[05:13:20.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:22:0) after 6m23s
[05:19:43.365] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:22:0)
[05:19:43.391] Build status: STARTED
[05:19:43.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:23:0) after 4m41s
[05:24:24.365] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:23:0)
[05:24:24.382] Build status: STARTED
[05:24:24.382] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:24:0) after 5m25s
[05:29:49.402] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:24:0)
[05:29:49.422] Build status: STARTED
[05:29:49.422] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:25:0) after 4m1s
[05:33:50.437] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:25:0)
[05:33:50.460] Build status: STARTED
[05:33:50.460] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:26:0) after 6m56s
[05:40:46.477] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:26:0)
[05:40:46.637] Build status: STARTED
[05:40:46.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:27:0) after 4m28s
[05:45:14.805] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:27:0)
[05:45:14.822] Build status: STARTED
[05:45:14.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:28:0) after 4m9s
[05:49:23.847] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:28:0)
[05:49:23.866] Build status: STARTED
[05:49:23.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:29:0) after 8m25s
[05:57:48.914] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:29:0)
[05:57:48.928] Build status: STARTED
[05:57:48.928] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:30:0) after 4m43s
[06:02:31.949] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:30:0)
[06:02:31.971] Build status: STARTED
[06:02:31.971] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:31:0) after 7m21s
[06:09:53.006] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:31:0)
[06:09:53.102] Build status: STARTED
[06:09:53.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:32:0) after 8m10s
[06:18:03.224] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:32:0)
[06:18:03.238] Build status: STARTED
[06:18:03.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:33:0) after 1m0s
[06:19:03.335] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:33:0)
[06:19:03.358] Build status: STARTED
[06:19:03.358] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:34:0) after 1m34s
[06:20:37.536] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:34:0)
[06:20:37.553] Build status: STARTED
[06:20:37.553] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:35:0) after 7m43s
[06:28:20.623] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:35:0)
[06:28:20.651] Build status: STARTED
[06:28:20.652] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:36:0) after 3m19s
[06:31:39.876] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:36:0)
[06:31:39.900] Build status: STARTED
[06:31:39.900] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:37:0) after 8m11s
[06:39:51.182] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:37:0)
[06:39:51.219] Build status: STARTED
[06:39:51.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:38:0) after 9m39s
[06:49:30.452] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:38:0)
[06:49:30.480] Build status: STARTED
[06:49:30.480] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:39:0) after 4m59s
[06:54:29.549] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:39:0)
[06:54:29.565] Build status: STARTED
[06:54:29.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:40:0) after 6m5s
[07:00:34.578] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:40:0)
[07:00:34.631] Build status: STARTED
[07:00:34.631] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:41:0) after 7m34s
[07:08:08.642] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:41:0)
[07:08:08.656] Build status: STARTED
[07:08:08.656] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:42:0) after 8m55s
[07:17:03.776] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:42:0)
[07:17:03.795] Build status: STARTED
[07:17:03.795] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:43:0) after 9m39s
[07:26:42.812] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:43:0)
[07:26:42.826] Build status: STARTED
[07:26:42.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:44:0) after 3m10s
[07:29:52.843] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:44:0)
[07:29:52.864] Build status: STARTED
[07:29:52.864] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:45:0) after 3m34s
[07:33:27.099] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:45:0)
[07:33:27.114] Build status: STARTED
[07:33:27.114] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:46:0) after 3m25s
[07:36:52.466] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:46:0)
[07:36:52.484] Build status: STARTED
[07:36:52.484] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:47:0) after 4m2s
[07:40:54.500] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:47:0)
[07:40:54.519] Build status: STARTED
[07:40:54.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:48:0) after 8m12s
[07:49:06.574] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:48:0)
[07:49:06.649] Build status: STARTED
[07:49:06.650] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:49:0) after 6m41s
[07:55:47.667] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:49:0)
[07:55:47.684] Build status: STARTED
[07:55:47.684] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:50:0) after 8m27s
[08:04:14.701] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:50:0)
[08:04:14.715] Build status: STARTED
[08:04:14.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:51:0) after 8m42s
[08:12:56.726] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:51:0)
[08:12:56.875] Build status: STARTED
[08:12:56.875] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:52:0) after 4m20s
[08:17:17.014] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:52:0)
[08:17:17.034] Build status: STARTED
[08:17:17.034] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:53:0) after 3m1s
[08:20:18.057] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:53:0)
[08:20:18.112] Build status: STARTED
[08:20:18.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:54:0) after 9m23s
[08:29:41.177] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:54:0)
[08:29:41.240] Build status: STARTED
[08:29:41.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:55:0) after 2m34s
[08:32:15.306] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:55:0)
[08:32:15.342] Build status: STARTED
[08:32:15.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:56:0) after 8m50s
[08:41:05.358] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:56:0)
[08:41:05.387] Build status: STARTED
[08:41:05.387] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:57:0) after 6m20s
[08:47:25.403] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:57:0)
[08:47:25.416] Build status: STARTED
[08:47:25.416] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:58:0) after 8m34s
[08:55:59.432] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:58:0)
[08:55:59.452] Build status: STARTED
[08:55:59.452] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867221794138680656:59:0) after 7m35s
[08:56:37.296] Received PubSub notification, asking Buildbucket for the build status
[08:56:37.315] Build:
{
"id": "8701791732168489649",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27181,
"createdBy": "project:chromium",
"createTime": "2025-10-06T03:35:40.522394951Z",
"startTime": "2025-10-06T03:36:26.582594536Z",
"endTime": "2025-10-06T08:56:36.782997774Z",
"updateTime": "2025-10-06T08:56:36.782997774Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "2838d14cffbafdf205da43e2b80dc93b481d5e02",
"ref": "refs/heads/main"
}
}
}
[08:56:37.315] Invocation finished in 5h20m58.055122508s with status FAILED