[19:47:50.320] New invocation is queued and will start shortly
[19:47:51.390] Starting the invocation (attempt 1)
[19:47:51.423] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/5803ec0918b4928305c315b98c4d9a6a4de92bf5"
[19:47:51.423] Popped gitiles commit info from properties and tags
[19:47:51.423] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:47:51.423] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:47:51.423] Buildbucket request:
{
"requestId": "9009941448651082624",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5803ec0918b4928305c315b98c4d9a6a4de92bf5",
"ref": "refs/heads/master"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "9009941448651082624"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxTSan"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:47:51.974] Scheduled build:
{
"id": "8844511386749025152",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 10003,
"createdBy": "project:chromium",
"createTime": "2021-06-13T19:47:51.502137Z",
"updateTime": "2021-06-13T19:47:51.899123Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5803ec0918b4928305c315b98c4d9a6a4de92bf5",
"ref": "refs/heads/master"
}
}
}
[19:47:51.974] Task URL: https://cr-buildbucket.appspot.com/build/8844511386749025152
[19:47:51.974] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:2:0) after 8m0s
[19:48:09.429] Received PubSub notification, asking Buildbucket for the build status
[19:48:09.454] Build status: STARTED
[19:55:51.989] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:2:0)
[19:55:52.019] Build status: STARTED
[19:55:52.019] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:4:0) after 6m26s
[20:02:18.035] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:4:0)
[20:02:18.066] Build status: STARTED
[20:02:18.066] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:5:0) after 7m5s
[20:09:23.459] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:5:0)
[20:09:23.503] Build status: STARTED
[20:09:23.503] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:6:0) after 9m41s
[20:19:04.731] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:6:0)
[20:19:04.756] Build status: STARTED
[20:19:04.756] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:7:0) after 7m51s
[20:26:55.771] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:7:0)
[20:26:55.797] Build status: STARTED
[20:26:55.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:8:0) after 6m41s
[20:33:36.972] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:8:0)
[20:33:36.994] Build status: STARTED
[20:33:36.994] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:9:0) after 8m25s
[20:42:01.988] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:9:0)
[20:42:02.015] Build status: STARTED
[20:42:02.016] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:10:0) after 4m16s
[20:46:18.285] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:10:0)
[20:46:18.411] Build status: STARTED
[20:46:18.412] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:11:0) after 7m5s
[20:53:23.933] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:11:0)
[20:53:23.960] Build status: STARTED
[20:53:23.960] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:12:0) after 3m30s
[20:56:54.048] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:12:0)
[20:56:54.079] Build status: STARTED
[20:56:54.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:13:0) after 3m30s
[21:00:24.413] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:13:0)
[21:00:24.450] Build status: STARTED
[21:00:24.450] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:14:0) after 8m20s
[21:08:44.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:14:0)
[21:08:44.830] Build status: STARTED
[21:08:44.830] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:15:0) after 3m5s
[21:11:49.847] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:15:0)
[21:11:49.877] Build status: STARTED
[21:11:49.877] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:16:0) after 4m13s
[21:16:02.996] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:16:0)
[21:16:03.024] Build status: STARTED
[21:16:03.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:17:0) after 9m1s
[21:25:04.533] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:17:0)
[21:25:04.560] Build status: STARTED
[21:25:04.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:18:0) after 6m46s
[21:31:50.786] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:18:0)
[21:31:50.815] Build status: STARTED
[21:31:50.815] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:19:0) after 4m57s
[21:36:48.094] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:19:0)
[21:36:48.122] Build status: STARTED
[21:36:48.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:20:0) after 4m52s
[21:41:40.859] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:20:0)
[21:41:40.890] Build status: STARTED
[21:41:40.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:21:0) after 9m56s
[21:51:37.343] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:21:0)
[21:51:37.375] Build status: STARTED
[21:51:37.375] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:22:0) after 1m9s
[21:52:46.548] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:22:0)
[21:52:46.574] Build status: STARTED
[21:52:46.574] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:23:0) after 9m12s
[22:01:58.913] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:23:0)
[22:01:58.942] Build status: STARTED
[22:01:58.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:24:0) after 3m32s
[22:05:31.493] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:24:0)
[22:05:31.524] Build status: STARTED
[22:05:31.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:25:0) after 6m45s
[22:12:16.542] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:25:0)
[22:12:16.573] Build status: STARTED
[22:12:16.573] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:26:0) after 5m59s
[22:18:15.714] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:26:0)
[22:18:15.798] Build status: STARTED
[22:18:15.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:27:0) after 6m22s
[22:24:37.906] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:27:0)
[22:24:37.931] Build status: STARTED
[22:24:37.931] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:28:0) after 3m55s
[22:28:32.946] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:28:0)
[22:28:32.974] Build status: STARTED
[22:28:32.974] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:29:0) after 6m45s
[22:35:17.999] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:29:0)
[22:35:18.029] Build status: STARTED
[22:35:18.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:30:0) after 1m16s
[22:36:34.788] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:30:0)
[22:36:34.819] Build status: STARTED
[22:36:34.819] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:31:0) after 8m39s
[22:45:13.849] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:31:0)
[22:45:13.876] Build status: STARTED
[22:45:13.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:32:0) after 8m19s
[22:53:32.984] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:32:0)
[22:53:33.060] Build status: STARTED
[22:53:33.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:33:0) after 7m44s
[23:01:17.448] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:33:0)
[23:01:17.481] Build status: STARTED
[23:01:17.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:34:0) after 6m52s
[23:08:10.257] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:34:0)
[23:08:10.285] Build status: STARTED
[23:08:10.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:35:0) after 6m11s
[23:14:21.814] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:35:0)
[23:14:21.838] Build status: STARTED
[23:14:21.838] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:36:0) after 5m42s
[23:20:03.881] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:36:0)
[23:20:03.909] Build status: STARTED
[23:20:03.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:37:0) after 3m22s
[23:23:25.997] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:37:0)
[23:23:26.023] Build status: STARTED
[23:23:26.023] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:38:0) after 6m0s
[23:29:26.189] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:38:0)
[23:29:26.226] Build status: STARTED
[23:29:26.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:39:0) after 8m13s
[23:37:39.361] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:39:0)
[23:37:39.392] Build status: STARTED
[23:37:39.392] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:40:0) after 6m55s
[23:44:34.496] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:40:0)
[23:44:34.524] Build status: STARTED
[23:44:34.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:41:0) after 4m59s
[23:49:33.636] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:41:0)
[23:49:33.665] Build status: STARTED
[23:49:33.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:42:0) after 9m30s
[23:59:03.884] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:42:0)
[23:59:03.912] Build status: STARTED
[23:59:03.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:43:0) after 2m12s
[00:01:15.934] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:43:0)
[00:01:15.967] Build status: STARTED
[00:01:15.967] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:44:0) after 6m22s
[00:07:38.069] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:44:0)
[00:07:38.099] Build status: STARTED
[00:07:38.099] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:45:0) after 5m6s
[00:12:44.115] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:45:0)
[00:12:44.150] Build status: STARTED
[00:12:44.150] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:46:0) after 6m40s
[00:19:24.167] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:46:0)
[00:19:24.195] Build status: STARTED
[00:19:24.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:47:0) after 4m34s
[00:23:58.336] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:47:0)
[00:23:58.433] Build status: STARTED
[00:23:58.433] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:48:0) after 2m50s
[00:26:48.731] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:48:0)
[00:26:48.762] Build status: STARTED
[00:26:48.762] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:49:0) after 7m16s
[00:34:04.886] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:49:0)
[00:34:04.913] Build status: STARTED
[00:34:04.913] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:50:0) after 3m2s
[00:37:06.960] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:50:0)
[00:37:06.988] Build status: STARTED
[00:37:06.988] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:51:0) after 2m49s
[00:39:56.007] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:51:0)
[00:39:56.037] Build status: STARTED
[00:39:56.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:52:0) after 4m25s
[00:44:21.627] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:52:0)
[00:44:21.667] Build status: STARTED
[00:44:21.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:53:0) after 7m5s
[00:51:26.998] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:53:0)
[00:51:27.025] Build status: STARTED
[00:51:27.025] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:54:0) after 1m48s
[00:53:15.058] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:54:0)
[00:53:15.086] Build status: STARTED
[00:53:15.086] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:55:0) after 5m7s
[00:58:22.462] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:55:0)
[00:58:22.489] Build status: STARTED
[00:58:22.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:56:0) after 5m8s
[01:03:30.699] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:56:0)
[01:03:30.739] Build status: STARTED
[01:03:30.739] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:57:0) after 2m13s
[01:05:44.268] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:57:0)
[01:05:44.345] Build status: STARTED
[01:05:44.345] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:58:0) after 9m23s
[01:15:07.367] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:58:0)
[01:15:07.392] Build status: STARTED
[01:15:07.392] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:59:0) after 6m49s
[01:21:56.413] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:59:0)
[01:21:56.457] Build status: STARTED
[01:21:56.457] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:60:0) after 8m19s
[01:30:15.854] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:60:0)
[01:30:15.884] Build status: STARTED
[01:30:15.884] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:61:0) after 8m35s
[01:38:50.906] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:61:0)
[01:38:50.934] Build status: STARTED
[01:38:50.934] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxTSan:9009941448651082624:62:0) after 8m55s
[01:40:06.251] Received PubSub notification, asking Buildbucket for the build status
[01:40:06.277] Build:
{
"id": "8844511386749025152",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxTSan"
},
"number": 10003,
"createdBy": "project:chromium",
"createTime": "2021-06-13T19:47:51.502137Z",
"startTime": "2021-06-13T19:48:07.332254Z",
"endTime": "2021-06-14T01:40:05.710912934Z",
"updateTime": "2021-06-14T01:40:05.711228740Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5803ec0918b4928305c315b98c4d9a6a4de92bf5",
"ref": "refs/heads/master"
}
}
}
[01:40:06.277] Invocation finished in 5h52m15.972776496s with status FAILED