[01:40:13.135] New invocation is queued and will start shortly
[01:40:14.657] Starting the invocation (attempt 1)
[01:40:14.694] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/27fd61ee916fbc74bbca4d47c108d192d5f2c224"
[01:40:14.694] Popped gitiles commit info from properties and tags
[01:40:14.694] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:40:14.695] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:40:14.695] Buildbucket request:
{
"requestId": "8922312012290980400",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "27fd61ee916fbc74bbca4d47c108d192d5f2c224",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8922312012290980400"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:40:15.302] Scheduled build:
{
"id": "8756881949908694001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 19369,
"createdBy": "project:chromium",
"createTime": "2024-02-06T01:40:14.778919369Z",
"updateTime": "2024-02-06T01:40:14.778919369Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "27fd61ee916fbc74bbca4d47c108d192d5f2c224",
"ref": "refs/heads/main"
}
}
}
[01:40:15.302] Task URL: https://cr-buildbucket.appspot.com/build/8756881949908694001
[01:40:15.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:2:0) after 9m52s
[01:40:16.205] Received PubSub notification, asking Buildbucket for the build status
[01:40:16.228] Build status: SCHEDULED
[01:41:29.598] Received PubSub notification, asking Buildbucket for the build status
[01:41:29.699] Build status: STARTED
[01:50:07.328] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:2:0)
[01:50:07.363] Build status: STARTED
[01:50:07.363] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:5:0) after 7m54s
[01:58:01.603] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:5:0)
[01:58:01.725] Build status: STARTED
[01:58:01.726] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:6:0) after 6m21s
[02:04:22.749] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:6:0)
[02:04:22.837] Build status: STARTED
[02:04:22.837] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:7:0) after 1m37s
[02:05:59.909] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:7:0)
[02:05:59.931] Build status: STARTED
[02:05:59.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:8:0) after 6m22s
[02:12:21.989] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:8:0)
[02:12:22.127] Build status: STARTED
[02:12:22.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:9:0) after 9m51s
[02:22:13.179] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:9:0)
[02:22:13.400] Build status: STARTED
[02:22:13.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:10:0) after 4m20s
[02:26:33.432] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:10:0)
[02:26:33.463] Build status: STARTED
[02:26:33.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:11:0) after 7m23s
[02:33:56.512] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:11:0)
[02:33:56.532] Build status: STARTED
[02:33:56.532] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:12:0) after 7m17s
[02:41:14.028] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:12:0)
[02:41:14.051] Build status: STARTED
[02:41:14.051] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:13:0) after 3m22s
[02:44:36.082] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:13:0)
[02:44:36.253] Build status: STARTED
[02:44:36.253] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:14:0) after 2m48s
[02:47:24.344] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:14:0)
[02:47:24.381] Build status: STARTED
[02:47:24.381] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:15:0) after 9m12s
[02:56:36.421] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:15:0)
[02:56:36.495] Build status: STARTED
[02:56:36.495] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:16:0) after 8m24s
[03:05:00.545] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:16:0)
[03:05:00.571] Build status: STARTED
[03:05:00.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:17:0) after 3m12s
[03:08:12.675] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:17:0)
[03:08:12.771] Build status: STARTED
[03:08:12.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:18:0) after 8m43s
[03:16:55.801] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:18:0)
[03:16:55.827] Build status: STARTED
[03:16:55.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:19:0) after 2m0s
[03:18:55.927] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:19:0)
[03:18:55.945] Build status: STARTED
[03:18:55.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:20:0) after 8m47s
[03:27:43.076] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:20:0)
[03:27:43.095] Build status: STARTED
[03:27:43.095] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:21:0) after 5m57s
[03:33:40.188] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:21:0)
[03:33:40.206] Build status: STARTED
[03:33:40.206] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:22:0) after 7m46s
[03:41:26.455] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:22:0)
[03:41:26.476] Build status: STARTED
[03:41:26.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:23:0) after 7m29s
[03:48:55.914] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:23:0)
[03:48:55.933] Build status: STARTED
[03:48:55.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:24:0) after 7m54s
[03:56:50.032] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:24:0)
[03:56:50.048] Build status: STARTED
[03:56:50.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:25:0) after 6m52s
[04:03:42.070] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:25:0)
[04:03:42.157] Build status: STARTED
[04:03:42.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:26:0) after 6m42s
[04:10:24.430] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:26:0)
[04:10:24.453] Build status: STARTED
[04:10:24.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:27:0) after 8m4s
[04:18:28.569] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:27:0)
[04:18:28.597] Build status: STARTED
[04:18:28.597] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:28:0) after 5m40s
[04:24:08.716] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:28:0)
[04:24:08.768] Build status: STARTED
[04:24:08.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:29:0) after 3m30s
[04:27:38.839] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:29:0)
[04:27:38.856] Build status: STARTED
[04:27:38.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:30:0) after 7m22s
[04:35:01.028] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:30:0)
[04:35:01.051] Build status: STARTED
[04:35:01.051] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:31:0) after 4m17s
[04:39:18.400] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:31:0)
[04:39:18.421] Build status: STARTED
[04:39:18.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:32:0) after 6m33s
[04:45:51.564] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:32:0)
[04:45:51.583] Build status: STARTED
[04:45:51.583] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:33:0) after 6m5s
[04:51:56.604] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:33:0)
[04:51:56.624] Build status: STARTED
[04:51:56.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:34:0) after 2m43s
[04:54:39.745] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:34:0)
[04:54:39.879] Build status: STARTED
[04:54:39.879] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:35:0) after 3m23s
[04:58:03.219] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:35:0)
[04:58:03.244] Build status: STARTED
[04:58:03.245] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:36:0) after 1m23s
[04:59:26.274] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:36:0)
[04:59:26.292] Build status: STARTED
[04:59:26.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:37:0) after 5m23s
[05:04:49.414] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:37:0)
[05:04:49.434] Build status: STARTED
[05:04:49.434] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:38:0) after 6m48s
[05:11:37.560] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:38:0)
[05:11:37.579] Build status: STARTED
[05:11:37.579] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:39:0) after 1m44s
[05:13:21.641] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:39:0)
[05:13:21.662] Build status: STARTED
[05:13:21.662] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:40:0) after 4m9s
[05:17:30.831] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:40:0)
[05:17:30.854] Build status: STARTED
[05:17:30.854] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:41:0) after 4m46s
[05:22:16.889] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:41:0)
[05:22:16.921] Build status: STARTED
[05:22:16.921] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:42:0) after 2m39s
[05:24:56.099] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:42:0)
[05:24:56.131] Build status: STARTED
[05:24:56.131] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:43:0) after 6m2s
[05:30:58.241] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:43:0)
[05:30:58.260] Build status: STARTED
[05:30:58.260] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:44:0) after 5m47s
[05:36:45.376] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:44:0)
[05:36:45.399] Build status: STARTED
[05:36:45.399] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:45:0) after 7m18s
[05:44:03.717] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:45:0)
[05:44:03.744] Build status: STARTED
[05:44:03.744] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:46:0) after 1m43s
[05:45:46.778] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:46:0)
[05:45:46.796] Build status: STARTED
[05:45:46.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:47:0) after 2m59s
[05:48:45.920] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:47:0)
[05:48:45.939] Build status: STARTED
[05:48:45.939] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:48:0) after 2m54s
[05:51:40.031] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:48:0)
[05:51:40.058] Build status: STARTED
[05:51:40.058] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:49:0) after 1m39s
[05:53:19.091] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:49:0)
[05:53:19.116] Build status: STARTED
[05:53:19.116] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:50:0) after 1m15s
[05:54:34.814] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:50:0)
[05:54:34.842] Build status: STARTED
[05:54:34.842] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:51:0) after 4m9s
[05:58:44.054] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:51:0)
[05:58:44.073] Build status: STARTED
[05:58:44.073] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:52:0) after 7m52s
[06:06:36.907] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:52:0)
[06:06:36.927] Build status: STARTED
[06:06:36.927] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:53:0) after 5m11s
[06:11:47.982] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:53:0)
[06:11:48.010] Build status: STARTED
[06:11:48.010] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:54:0) after 7m21s
[06:19:09.138] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:54:0)
[06:19:09.167] Build status: STARTED
[06:19:09.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:55:0) after 9m32s
[06:28:41.364] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:55:0)
[06:28:41.384] Build status: STARTED
[06:28:41.384] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:56:0) after 5m37s
[06:34:18.489] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:56:0)
[06:34:18.511] Build status: STARTED
[06:34:18.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:57:0) after 8m18s
[06:42:36.557] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:57:0)
[06:42:36.584] Build status: STARTED
[06:42:36.584] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:58:0) after 5m37s
[06:48:13.665] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:58:0)
[06:48:13.682] Build status: STARTED
[06:48:13.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:59:0) after 4m52s
[06:53:05.763] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:59:0)
[06:53:05.781] Build status: STARTED
[06:53:05.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:60:0) after 9m30s
[07:02:35.804] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:60:0)
[07:02:35.822] Build status: STARTED
[07:02:35.822] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:61:0) after 3m24s
[07:05:59.940] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:61:0)
[07:06:00.000] Build status: STARTED
[07:06:00.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:62:0) after 5m10s
[07:11:10.043] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:62:0)
[07:11:10.071] Build status: STARTED
[07:11:10.071] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:63:0) after 8m31s
[07:19:41.476] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:63:0)
[07:19:41.493] Build status: STARTED
[07:19:41.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:64:0) after 6m44s
[07:26:25.522] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:64:0)
[07:26:25.574] Build status: STARTED
[07:26:25.574] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:65:0) after 3m49s
[07:30:15.284] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:65:0)
[07:30:15.311] Build status: STARTED
[07:30:15.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:66:0) after 8m57s
[07:39:12.335] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:66:0)
[07:39:12.350] Build status: STARTED
[07:39:12.350] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:8922312012290980400:67:0) after 6m57s
[07:41:39.280] Received PubSub notification, asking Buildbucket for the build status
[07:41:39.311] Build:
{
"id": "8756881949908694001",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 19369,
"createdBy": "project:chromium",
"createTime": "2024-02-06T01:40:14.778919369Z",
"startTime": "2024-02-06T01:41:29.052014Z",
"endTime": "2024-02-06T07:41:38.807508Z",
"updateTime": "2024-02-06T07:41:38.807508Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "27fd61ee916fbc74bbca4d47c108d192d5f2c224",
"ref": "refs/heads/main"
}
}
}
[07:41:39.311] Invocation finished in 6h1m26.1880297s with status FAILED