[01:40:08.831] New invocation is queued and will start shortly
[01:40:09.906] Starting the invocation (attempt 1)
[01:40:09.940] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/6f0a1e9f4424d44c2e06d825b4e528fceabb3b1f"
[01:40:09.940] Popped gitiles commit info from properties and tags
[01:40:09.940] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:40:09.940] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:40:09.940] Buildbucket request:
{
"requestId": "9009919283315598272",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6f0a1e9f4424d44c2e06d825b4e528fceabb3b1f",
"ref": "refs/heads/master"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "9009919283315598272"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:40:11.348] Scheduled build:
{
"id": "8844489221312352288",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 10004,
"createdBy": "project:chromium",
"createTime": "2021-06-14T01:40:10.110794Z",
"updateTime": "2021-06-14T01:40:11.157898Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6f0a1e9f4424d44c2e06d825b4e528fceabb3b1f",
"ref": "refs/heads/master"
}
}
}
[01:40:11.348] Task URL: https://cr-buildbucket.appspot.com/build/8844489221312352288
[01:40:11.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:2:0) after 7m50s
[01:41:23.648] Received PubSub notification, asking Buildbucket for the build status
[01:41:23.716] Build status: STARTED
[01:48:01.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:2:0)
[01:48:01.392] Build status: STARTED
[01:48:01.392] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:4:0) after 3m9s
[01:51:10.737] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:4:0)
[01:51:10.763] Build status: STARTED
[01:51:10.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:5:0) after 8m21s
[01:59:31.779] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:5:0)
[01:59:33.736] Build status: STARTED
[01:59:33.736] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:6:0) after 2m37s
[02:02:10.764] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:6:0)
[02:02:10.789] Build status: STARTED
[02:02:10.789] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:7:0) after 5m15s
[02:07:25.805] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:7:0)
[02:07:25.835] Build status: STARTED
[02:07:25.835] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:8:0) after 7m49s
[02:15:14.875] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:8:0)
[02:15:14.915] Build status: STARTED
[02:15:14.915] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:9:0) after 8m26s
[02:23:40.935] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:9:0)
[02:23:40.963] Build status: STARTED
[02:23:40.963] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:10:0) after 8m50s
[02:32:31.171] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:10:0)
[02:32:31.207] Build status: STARTED
[02:32:31.207] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:11:0) after 2m49s
[02:35:20.328] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:11:0)
[02:35:20.355] Build status: STARTED
[02:35:20.355] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:12:0) after 6m55s
[02:42:15.374] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:12:0)
[02:42:15.402] Build status: STARTED
[02:42:15.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:13:0) after 5m33s
[02:47:48.423] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:13:0)
[02:47:48.451] Build status: STARTED
[02:47:48.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:14:0) after 1m19s
[02:49:07.568] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:14:0)
[02:49:07.602] Build status: STARTED
[02:49:07.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:15:0) after 1m49s
[02:50:56.697] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:15:0)
[02:50:56.721] Build status: STARTED
[02:50:56.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:16:0) after 1m48s
[02:52:44.699] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:16:0)
[02:52:44.729] Build status: STARTED
[02:52:44.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:17:0) after 9m59s
[03:02:43.768] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:17:0)
[03:02:43.816] Build status: STARTED
[03:02:43.816] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:18:0) after 6m23s
[03:09:06.838] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:18:0)
[03:09:06.908] Build status: STARTED
[03:09:06.908] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:19:0) after 6m23s
[03:15:29.923] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:19:0)
[03:15:29.950] Build status: STARTED
[03:15:29.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:20:0) after 7m49s
[03:23:18.953] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:20:0)
[03:23:18.991] Build status: STARTED
[03:23:18.991] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:21:0) after 5m24s
[03:28:43.121] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:21:0)
[03:28:43.149] Build status: STARTED
[03:28:43.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:22:0) after 8m36s
[03:37:20.748] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:22:0)
[03:37:20.823] Build status: STARTED
[03:37:20.823] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:23:0) after 7m44s
[03:45:04.950] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:23:0)
[03:45:04.975] Build status: STARTED
[03:45:04.975] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:24:0) after 8m45s
[03:53:49.995] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:24:0)
[03:53:50.021] Build status: STARTED
[03:53:50.021] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:25:0) after 6m47s
[04:00:38.006] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:25:0)
[04:00:38.037] Build status: STARTED
[04:00:38.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:26:0) after 7m38s
[04:08:16.183] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:26:0)
[04:08:16.211] Build status: STARTED
[04:08:16.211] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:27:0) after 9m39s
[04:17:55.320] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:27:0)
[04:17:55.521] Build status: STARTED
[04:17:55.521] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:28:0) after 5m55s
[04:23:50.561] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:28:0)
[04:23:50.590] Build status: STARTED
[04:23:50.590] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:29:0) after 7m13s
[04:31:03.692] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:29:0)
[04:31:03.745] Build status: STARTED
[04:31:03.745] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:30:0) after 6m40s
[04:37:43.763] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:30:0)
[04:37:43.792] Build status: STARTED
[04:37:43.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:31:0) after 1m15s
[04:38:58.887] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:31:0)
[04:38:58.920] Build status: STARTED
[04:38:58.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:32:0) after 3m0s
[04:41:58.941] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:32:0)
[04:41:58.965] Build status: STARTED
[04:41:58.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:33:0) after 1m20s
[04:43:18.985] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:33:0)
[04:43:19.012] Build status: STARTED
[04:43:19.012] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:34:0) after 2m3s
[04:45:22.106] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:34:0)
[04:45:22.136] Build status: STARTED
[04:45:22.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:35:0) after 8m10s
[04:53:32.187] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:35:0)
[04:53:32.212] Build status: STARTED
[04:53:32.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:36:0) after 4m23s
[04:57:55.318] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:36:0)
[04:57:55.347] Build status: STARTED
[04:57:55.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:37:0) after 3m23s
[05:01:18.344] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:37:0)
[05:01:18.381] Build status: STARTED
[05:01:18.381] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:38:0) after 3m6s
[05:04:24.755] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:38:0)
[05:04:24.785] Build status: STARTED
[05:04:24.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:39:0) after 2m4s
[05:06:28.780] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:39:0)
[05:06:28.812] Build status: STARTED
[05:06:28.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:40:0) after 5m20s
[05:11:49.657] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:40:0)
[05:11:49.685] Build status: STARTED
[05:11:49.685] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:41:0) after 7m29s
[05:19:18.757] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:41:0)
[05:19:18.795] Build status: STARTED
[05:19:18.795] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:42:0) after 1m8s
[05:20:26.994] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:42:0)
[05:20:27.017] Build status: STARTED
[05:20:27.017] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:43:0) after 1m11s
[05:21:38.036] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:43:0)
[05:21:38.067] Build status: STARTED
[05:21:38.067] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:44:0) after 9m11s
[05:30:49.349] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:44:0)
[05:30:49.440] Build status: STARTED
[05:30:49.440] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:45:0) after 7m39s
[05:38:28.638] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:45:0)
[05:38:28.673] Build status: STARTED
[05:38:28.673] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:46:0) after 6m24s
[05:44:52.709] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:46:0)
[05:44:52.737] Build status: STARTED
[05:44:52.737] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:47:0) after 2m26s
[05:47:18.756] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:47:0)
[05:47:19.023] Build status: STARTED
[05:47:19.023] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:48:0) after 4m40s
[05:51:59.244] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:48:0)
[05:51:59.352] Build status: STARTED
[05:51:59.352] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:49:0) after 4m43s
[05:56:42.376] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:49:0)
[05:56:42.405] Build status: STARTED
[05:56:42.405] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:50:0) after 3m4s
[05:59:46.427] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:50:0)
[05:59:46.457] Build status: STARTED
[05:59:46.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:51:0) after 7m19s
[06:07:05.509] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:51:0)
[06:07:05.536] Build status: STARTED
[06:07:05.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:52:0) after 3m58s
[06:11:03.562] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:52:0)
[06:11:03.590] Build status: STARTED
[06:11:03.590] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:53:0) after 6m29s
[06:17:33.183] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:53:0)
[06:17:33.237] Build status: STARTED
[06:17:33.237] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:54:0) after 8m30s
[06:26:03.677] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:54:0)
[06:26:03.705] Build status: STARTED
[06:26:03.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:55:0) after 8m37s
[06:34:40.808] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:55:0)
[06:34:40.837] Build status: STARTED
[06:34:40.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:56:0) after 9m40s
[06:44:20.821] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:56:0)
[06:44:20.848] Build status: STARTED
[06:44:20.848] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:57:0) after 7m20s
[06:51:40.867] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:57:0)
[06:51:40.895] Build status: STARTED
[06:51:40.895] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:58:0) after 2m42s
[06:54:23.008] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:58:0)
[06:54:23.041] Build status: STARTED
[06:54:23.041] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:59:0) after 6m24s
[07:00:47.058] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:59:0)
[07:00:47.119] Build status: STARTED
[07:00:47.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:60:0) after 8m55s
[07:09:42.138] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:60:0)
[07:09:42.164] Build status: STARTED
[07:09:42.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:61:0) after 6m2s
[07:15:44.234] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:61:0)
[07:15:44.263] Build status: STARTED
[07:15:44.263] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:62:0) after 6m34s
[07:22:18.356] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:62:0)
[07:22:18.388] Build status: STARTED
[07:22:18.388] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009919283315598272:63:0) after 7m49s
[07:24:54.486] Received PubSub notification, asking Buildbucket for the build status
[07:24:54.537] Build:
{
"id": "8844489221312352288",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 10004,
"createdBy": "project:chromium",
"createTime": "2021-06-14T01:40:10.110794Z",
"startTime": "2021-06-14T01:41:22.857226Z",
"endTime": "2021-06-14T07:24:54.059537237Z",
"updateTime": "2021-06-14T07:24:54.059910922Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "6f0a1e9f4424d44c2e06d825b4e528fceabb3b1f",
"ref": "refs/heads/master"
}
}
}
[07:24:54.537] Invocation finished in 5h44m45.720364465s with status FAILED