[22:32:08.069] New invocation is queued and will start shortly
[22:32:09.216] Starting the invocation (attempt 1)
[22:32:09.251] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/5a3f9deda7d79e7e5ecaa8a66730a87348a3e854"
[22:32:09.251] Popped gitiles commit info from properties and tags
[22:32:09.251] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[22:32:09.252] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[22:32:09.252] Buildbucket request:
{
"requestId": "8973692325491527744",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5a3f9deda7d79e7e5ecaa8a66730a87348a3e854",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8973692325491527744"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[22:32:09.831] Scheduled build:
{
"id": "8808262263510528001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 13576,
"createdBy": "project:chromium",
"createTime": "2022-07-18T22:32:09.327610456Z",
"updateTime": "2022-07-18T22:32:09.327610456Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5a3f9deda7d79e7e5ecaa8a66730a87348a3e854",
"ref": "refs/heads/main"
}
}
}
[22:32:09.831] Task URL: https://cr-buildbucket.appspot.com/build/8808262263510528001
[22:32:09.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:2:0) after 4m51s
[22:34:39.319] Received PubSub notification, asking Buildbucket for the build status
[22:34:39.339] Build status: STARTED
[22:37:00.965] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:2:0)
[22:37:00.995] Build status: STARTED
[22:37:00.995] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:4:0) after 8m12s
[22:45:13.032] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:4:0)
[22:45:13.058] Build status: STARTED
[22:45:13.058] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:5:0) after 4m57s
[22:50:10.078] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:5:0)
[22:50:10.156] Build status: STARTED
[22:50:10.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:6:0) after 9m14s
[22:59:24.223] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:6:0)
[22:59:24.245] Build status: STARTED
[22:59:24.245] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:7:0) after 9m1s
[23:08:25.384] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:7:0)
[23:08:25.404] Build status: STARTED
[23:08:25.404] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:8:0) after 1m16s
[23:09:41.427] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:8:0)
[23:09:41.510] Build status: STARTED
[23:09:41.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:9:0) after 4m56s
[23:14:37.648] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:9:0)
[23:14:37.669] Build status: STARTED
[23:14:37.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:10:0) after 4m45s
[23:19:22.791] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:10:0)
[23:19:22.834] Build status: STARTED
[23:19:22.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:11:0) after 4m18s
[23:23:40.856] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:11:0)
[23:23:40.893] Build status: STARTED
[23:23:40.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:12:0) after 1m55s
[23:25:35.940] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:12:0)
[23:25:35.964] Build status: STARTED
[23:25:35.964] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:13:0) after 1m14s
[23:26:50.072] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:13:0)
[23:26:50.094] Build status: STARTED
[23:26:50.094] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:14:0) after 3m25s
[23:30:15.180] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:14:0)
[23:30:15.223] Build status: STARTED
[23:30:15.223] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:15:0) after 5m27s
[23:35:42.263] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:15:0)
[23:35:42.295] Build status: STARTED
[23:35:42.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:16:0) after 6m21s
[23:42:03.391] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:16:0)
[23:42:03.415] Build status: STARTED
[23:42:03.415] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:17:0) after 1m44s
[23:43:47.555] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:17:0)
[23:43:47.584] Build status: STARTED
[23:43:47.584] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:18:0) after 1m49s
[23:45:36.634] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:18:0)
[23:45:36.657] Build status: STARTED
[23:45:36.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:19:0) after 4m47s
[23:50:23.751] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:19:0)
[23:50:23.780] Build status: STARTED
[23:50:23.780] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:20:0) after 2m46s
[23:53:09.869] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:20:0)
[23:53:09.921] Build status: STARTED
[23:53:09.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:21:0) after 9m15s
[00:02:24.989] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:21:0)
[00:02:25.016] Build status: STARTED
[00:02:25.016] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:22:0) after 6m24s
[00:08:49.354] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:22:0)
[00:08:49.377] Build status: STARTED
[00:08:49.377] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:23:0) after 8m50s
[00:17:39.487] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:23:0)
[00:17:39.520] Build status: STARTED
[00:17:39.520] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:24:0) after 8m51s
[00:26:30.544] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:24:0)
[00:26:30.575] Build status: STARTED
[00:26:30.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:25:0) after 6m4s
[00:32:35.001] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:25:0)
[00:32:35.022] Build status: STARTED
[00:32:35.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:26:0) after 8m9s
[00:40:44.053] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:26:0)
[00:40:44.074] Build status: STARTED
[00:40:44.074] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:27:0) after 6m46s
[00:47:30.095] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:27:0)
[00:47:30.161] Build status: STARTED
[00:47:30.161] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:28:0) after 4m45s
[00:52:15.266] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:28:0)
[00:52:15.287] Build status: STARTED
[00:52:15.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:29:0) after 9m40s
[01:01:55.362] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:29:0)
[01:01:55.384] Build status: STARTED
[01:01:55.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:30:0) after 1m18s
[01:03:13.656] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:30:0)
[01:03:13.722] Build status: STARTED
[01:03:13.722] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:31:0) after 6m25s
[01:09:38.781] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:31:0)
[01:09:38.814] Build status: STARTED
[01:09:38.814] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:32:0) after 4m40s
[01:14:18.852] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:32:0)
[01:14:18.881] Build status: STARTED
[01:14:18.881] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:33:0) after 6m53s
[01:21:12.257] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:33:0)
[01:21:12.283] Build status: STARTED
[01:21:12.283] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:34:0) after 5m4s
[01:26:16.757] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:34:0)
[01:26:16.782] Build status: STARTED
[01:26:16.782] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:35:0) after 1m19s
[01:27:35.862] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:35:0)
[01:27:35.887] Build status: STARTED
[01:27:35.887] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:36:0) after 5m24s
[01:33:00.141] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:36:0)
[01:33:00.185] Build status: STARTED
[01:33:00.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:37:0) after 4m12s
[01:37:12.209] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:37:0)
[01:37:12.352] Build status: STARTED
[01:37:12.352] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:38:0) after 3m41s
[01:40:53.643] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:38:0)
[01:40:53.665] Build status: STARTED
[01:40:53.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:39:0) after 1m0s
[01:41:53.685] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:39:0)
[01:41:53.705] Build status: STARTED
[01:41:53.706] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:40:0) after 7m59s
[01:49:52.976] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:40:0)
[01:49:53.032] Build status: STARTED
[01:49:53.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:41:0) after 8m14s
[01:58:07.542] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:41:0)
[01:58:07.577] Build status: STARTED
[01:58:07.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:42:0) after 3m28s
[02:01:35.864] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:42:0)
[02:01:35.884] Build status: STARTED
[02:01:35.884] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:43:0) after 4m59s
[02:06:34.891] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:43:0)
[02:06:34.911] Build status: STARTED
[02:06:34.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:44:0) after 3m58s
[02:10:33.133] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:44:0)
[02:10:33.214] Build status: STARTED
[02:10:33.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:45:0) after 9m27s
[02:20:00.237] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:45:0)
[02:20:00.259] Build status: STARTED
[02:20:00.259] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:46:0) after 6m0s
[02:26:00.235] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:46:0)
[02:26:00.270] Build status: STARTED
[02:26:00.270] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:47:0) after 6m11s
[02:32:11.418] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:47:0)
[02:32:11.439] Build status: STARTED
[02:32:11.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:48:0) after 7m11s
[02:39:22.471] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:48:0)
[02:39:22.527] Build status: STARTED
[02:39:22.527] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:49:0) after 4m36s
[02:43:58.584] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:49:0)
[02:43:58.626] Build status: STARTED
[02:43:58.626] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:50:0) after 3m4s
[02:47:02.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:50:0)
[02:47:02.829] Build status: STARTED
[02:47:02.829] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:51:0) after 6m41s
[02:53:44.098] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:51:0)
[02:53:44.116] Build status: STARTED
[02:53:44.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:52:0) after 7m39s
[03:01:23.272] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:52:0)
[03:01:23.296] Build status: STARTED
[03:01:23.296] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:53:0) after 1m59s
[03:03:22.332] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:53:0)
[03:03:22.372] Build status: STARTED
[03:03:22.372] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:54:0) after 2m37s
[03:05:59.401] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:54:0)
[03:05:59.424] Build status: STARTED
[03:05:59.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:55:0) after 8m43s
[03:14:42.555] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:55:0)
[03:14:42.583] Build status: STARTED
[03:14:42.583] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:56:0) after 4m21s
[03:19:03.659] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:56:0)
[03:19:03.682] Build status: STARTED
[03:19:03.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:57:0) after 5m52s
[03:24:55.780] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:57:0)
[03:24:55.812] Build status: STARTED
[03:24:55.812] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:58:0) after 1m27s
[03:26:22.873] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:58:0)
[03:26:22.906] Build status: STARTED
[03:26:22.906] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:59:0) after 7m8s
[03:33:30.928] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:59:0)
[03:33:30.949] Build status: STARTED
[03:33:30.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:60:0) after 3m10s
[03:36:41.490] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:60:0)
[03:36:41.525] Build status: STARTED
[03:36:41.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:61:0) after 1m34s
[03:38:15.568] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:61:0)
[03:38:15.589] Build status: STARTED
[03:38:15.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:62:0) after 8m3s
[03:46:18.628] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:62:0)
[03:46:18.648] Build status: STARTED
[03:46:18.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8973692325491527744:63:0) after 8m26s
[03:48:45.850] Received PubSub notification, asking Buildbucket for the build status
[03:48:45.876] Build:
{
"id": "8808262263510528001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 13576,
"createdBy": "project:chromium",
"createTime": "2022-07-18T22:32:09.327610456Z",
"startTime": "2022-07-18T22:34:36.527653Z",
"endTime": "2022-07-19T03:48:44.903173171Z",
"updateTime": "2022-07-19T03:48:44.903173171Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5a3f9deda7d79e7e5ecaa8a66730a87348a3e854",
"ref": "refs/heads/main"
}
}
}
[03:48:45.876] Invocation finished in 5h16m37.822317704s with status FAILED