[06:56:08.796] New invocation is queued and will start shortly
[06:56:09.883] Starting the invocation (attempt 1)
[06:56:09.914] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/44058065daec929c7f5ef594acb9494c1be81f37"
[06:56:09.914] Popped gitiles commit info from properties and tags
[06:56:09.914] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[06:56:09.914] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[06:56:09.914] Buildbucket request:
{
"requestId": "8867118583305646160",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "44058065daec929c7f5ef594acb9494c1be81f37",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867118583305646160"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[06:56:10.342] Scheduled build:
{
"id": "8701688521413535105",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27186,
"createdBy": "project:chromium",
"createTime": "2025-10-07T06:56:09.968385660Z",
"updateTime": "2025-10-07T06:56:09.968385660Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "44058065daec929c7f5ef594acb9494c1be81f37",
"ref": "refs/heads/main"
}
},
"canary": true
}
[06:56:10.342] Task URL: https://cr-buildbucket.appspot.com/build/8701688521413535105
[06:56:10.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:2:0) after 6m31s
[06:56:11.243] Received PubSub notification, asking Buildbucket for the build status
[06:56:11.281] Build status: SCHEDULED
[06:57:10.187] Received PubSub notification, asking Buildbucket for the build status
[06:57:10.200] Build status: STARTED
[07:02:41.360] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:2:0)
[07:02:41.474] Build status: STARTED
[07:02:41.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:5:0) after 2m31s
[07:05:12.592] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:5:0)
[07:05:12.610] Build status: STARTED
[07:05:12.610] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:6:0) after 5m2s
[07:10:14.626] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:6:0)
[07:10:14.641] Build status: STARTED
[07:10:14.641] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:7:0) after 3m25s
[07:13:39.659] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:7:0)
[07:13:39.830] Build status: STARTED
[07:13:39.830] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:8:0) after 5m59s
[07:19:38.845] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:8:0)
[07:19:38.861] Build status: STARTED
[07:19:38.861] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:9:0) after 4m13s
[07:23:51.916] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:9:0)
[07:23:51.943] Build status: STARTED
[07:23:51.943] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:10:0) after 3m10s
[07:27:02.035] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:10:0)
[07:27:02.050] Build status: STARTED
[07:27:02.050] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:11:0) after 6m19s
[07:33:21.072] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:11:0)
[07:33:21.093] Build status: STARTED
[07:33:21.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:12:0) after 2m15s
[07:35:36.144] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:12:0)
[07:35:36.168] Build status: STARTED
[07:35:36.168] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:13:0) after 3m24s
[07:39:00.186] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:13:0)
[07:39:00.208] Build status: STARTED
[07:39:00.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:14:0) after 6m50s
[07:45:50.264] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:14:0)
[07:45:50.502] Build status: STARTED
[07:45:50.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:15:0) after 6m7s
[07:51:57.485] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:15:0)
[07:51:57.509] Build status: STARTED
[07:51:57.509] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:16:0) after 2m29s
[07:54:26.526] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:16:0)
[07:54:26.543] Build status: STARTED
[07:54:26.543] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:17:0) after 5m15s
[07:59:41.585] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:17:0)
[07:59:41.602] Build status: STARTED
[07:59:41.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:18:0) after 4m51s
[08:04:32.620] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:18:0)
[08:04:32.639] Build status: STARTED
[08:04:32.639] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:19:0) after 7m47s
[08:12:19.725] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:19:0)
[08:12:19.744] Build status: STARTED
[08:12:19.744] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:20:0) after 9m44s
[08:22:03.838] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:20:0)
[08:22:03.855] Build status: STARTED
[08:22:03.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:21:0) after 5m51s
[08:27:54.970] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:21:0)
[08:27:54.986] Build status: STARTED
[08:27:54.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:22:0) after 5m57s
[08:33:52.022] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:22:0)
[08:33:52.062] Build status: STARTED
[08:33:52.062] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:23:0) after 6m30s
[08:40:22.050] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:23:0)
[08:40:22.278] Build status: STARTED
[08:40:22.278] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:24:0) after 5m9s
[08:45:31.292] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:24:0)
[08:45:31.331] Build status: STARTED
[08:45:31.331] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:25:0) after 3m40s
[08:49:11.348] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:25:0)
[08:49:11.364] Build status: STARTED
[08:49:11.364] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:26:0) after 6m54s
[08:56:05.396] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:26:0)
[08:56:05.424] Build status: STARTED
[08:56:05.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:27:0) after 3m38s
[08:59:43.509] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:27:0)
[08:59:43.526] Build status: STARTED
[08:59:43.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:28:0) after 1m34s
[09:01:17.590] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:28:0)
[09:01:17.619] Build status: STARTED
[09:01:17.619] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:29:0) after 9m3s
[09:10:20.679] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:29:0)
[09:10:20.697] Build status: STARTED
[09:10:20.697] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:30:0) after 5m20s
[09:15:40.679] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:30:0)
[09:15:40.698] Build status: STARTED
[09:15:40.698] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:31:0) after 8m34s
[09:24:14.716] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:31:0)
[09:24:14.736] Build status: STARTED
[09:24:14.737] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:32:0) after 9m36s
[09:33:50.806] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:32:0)
[09:33:50.820] Build status: STARTED
[09:33:50.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:33:0) after 2m24s
[09:36:14.841] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:33:0)
[09:36:15.171] Build status: STARTED
[09:36:15.171] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:34:0) after 9m27s
[09:45:42.191] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:34:0)
[09:45:42.208] Build status: STARTED
[09:45:42.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:35:0) after 7m15s
[09:52:57.239] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:35:0)
[09:52:57.259] Build status: STARTED
[09:52:57.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:36:0) after 3m34s
[09:56:31.275] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:36:0)
[09:56:31.291] Build status: STARTED
[09:56:31.291] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:37:0) after 8m53s
[10:05:24.323] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:37:0)
[10:05:24.496] Build status: STARTED
[10:05:24.496] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:38:0) after 8m3s
[10:13:27.479] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:38:0)
[10:13:27.494] Build status: STARTED
[10:13:27.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:39:0) after 9m25s
[10:22:52.545] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:39:0)
[10:22:52.582] Build status: STARTED
[10:22:52.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:40:0) after 3m11s
[10:26:03.597] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:40:0)
[10:26:03.618] Build status: STARTED
[10:26:03.618] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:41:0) after 1m24s
[10:27:27.727] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:41:0)
[10:27:27.742] Build status: STARTED
[10:27:27.742] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:42:0) after 9m11s
[10:36:38.775] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:42:0)
[10:36:38.796] Build status: STARTED
[10:36:38.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:43:0) after 5m54s
[10:42:32.900] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:43:0)
[10:42:32.916] Build status: STARTED
[10:42:32.916] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:44:0) after 4m35s
[10:47:07.936] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:44:0)
[10:47:07.951] Build status: STARTED
[10:47:07.951] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:45:0) after 4m41s
[10:51:49.066] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:45:0)
[10:51:49.156] Build status: STARTED
[10:51:49.156] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:46:0) after 3m5s
[10:54:54.195] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:46:0)
[10:54:54.213] Build status: STARTED
[10:54:54.213] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:47:0) after 7m28s
[11:02:22.311] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:47:0)
[11:02:22.345] Build status: STARTED
[11:02:22.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:48:0) after 9m8s
[11:11:30.387] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:48:0)
[11:11:30.428] Build status: STARTED
[11:11:30.428] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:49:0) after 5m44s
[11:17:14.550] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:49:0)
[11:17:14.597] Build status: STARTED
[11:17:14.597] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:50:0) after 3m46s
[11:21:01.016] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:50:0)
[11:21:01.031] Build status: STARTED
[11:21:01.031] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:51:0) after 9m18s
[11:30:19.271] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:51:0)
[11:30:19.441] Build status: STARTED
[11:30:19.441] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:52:0) after 8m24s
[11:38:43.808] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:52:0)
[11:38:43.821] Build status: STARTED
[11:38:43.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:53:0) after 5m54s
[11:44:37.881] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:53:0)
[11:44:37.896] Build status: STARTED
[11:44:37.896] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:54:0) after 9m0s
[11:53:38.035] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:54:0)
[11:53:38.054] Build status: STARTED
[11:53:38.054] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:55:0) after 9m38s
[12:03:16.098] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:55:0)
[12:03:16.118] Build status: STARTED
[12:03:16.118] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:56:0) after 5m52s
[12:09:08.145] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:56:0)
[12:09:08.164] Build status: STARTED
[12:09:08.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:57:0) after 9m9s
[12:18:17.183] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:57:0)
[12:18:17.201] Build status: STARTED
[12:18:17.201] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8867118583305646160:58:0) after 5m57s
[12:18:28.277] Received PubSub notification, asking Buildbucket for the build status
[12:18:28.292] Build:
{
"id": "8701688521413535105",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 27186,
"createdBy": "project:chromium",
"createTime": "2025-10-07T06:56:09.968385660Z",
"startTime": "2025-10-07T06:57:07.455820316Z",
"endTime": "2025-10-07T12:18:27.916055390Z",
"updateTime": "2025-10-07T12:18:27.916055390Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "44058065daec929c7f5ef594acb9494c1be81f37",
"ref": "refs/heads/main"
}
},
"canary": true
}
[12:18:28.292] Invocation finished in 5h22m19.512153524s with status FAILED