[14:51:43.658] New invocation is queued and will start shortly
[14:51:44.976] Starting the invocation (attempt 1)
[14:51:45.016] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[14:51:45.016] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[14:51:45.016] Buildbucket request:
{
"bucket": "luci.chromium.ci",
"client_operation_id": "9030344395788160080",
"parameters_json": "{\"builder_name\":\"ToTiOSDevice\",\"properties\":{\"branch\":\"refs/heads/master\",\"repository\":\"https://chromium.googlesource.com/chromium/src\",\"revision\":\"e66cafeaecb1380bfab166de05a27ded93aa8558\"}}",
"pubsub_callback": {
"auth_token": "...",
"topic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
},
"tags": [
"builder:ToTiOSDevice",
"scheduler_invocation_id:9030344395788160080",
"scheduler_job_id:chromium/ToTiOSDevice",
"user_agent:luci-scheduler",
"buildset:commit/git/e66cafeaecb1380bfab166de05a27ded93aa8558",
"buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e66cafeaecb1380bfab166de05a27ded93aa8558",
"gitiles_ref:refs/heads/master"
]
}
[14:51:45.828] Buildbucket response:
{
"build": {
"bucket": "luci.chromium.ci",
"canary_preference": "PROD",
"created_by": "project:chromium",
"created_ts": "1604155905096663",
"id": "8864914333616179904",
"parameters_json": "{\"builder_name\": \"ToTiOSDevice\", \"properties\": {\"branch\": \"refs/heads/master\", \"repository\": \"https://chromium.googlesource.com/chromium/src\", \"revision\": \"e66cafeaecb1380bfab166de05a27ded93aa8558\"}}",
"project": "chromium",
"result_details_json": "{\"properties\": {}}",
"service_account": "chromium-ci-builder@chops-service-accounts.iam.gserviceaccount.com",
"status": "SCHEDULED",
"status_changed_ts": "1604155905703236",
"tags": [
"build_address:luci.chromium.ci/ToTiOSDevice/20444",
"builder:ToTiOSDevice",
"buildset:commit/git/e66cafeaecb1380bfab166de05a27ded93aa8558",
"buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e66cafeaecb1380bfab166de05a27ded93aa8558",
"gitiles_ref:refs/heads/master",
"scheduler_invocation_id:9030344395788160080",
"scheduler_job_id:chromium/ToTiOSDevice",
"swarming_hostname:chromium-swarm.appspot.com",
"swarming_tag:log_location:logdog://logs.chromium.org/chromium/buildbucket/cr-buildbucket.appspot.com/8864914333616179904/+/annotations",
"swarming_tag:luci_project:chromium",
"swarming_tag:recipe_name:chromium",
"swarming_tag:recipe_package:infra/recipe_bundles/chromium.googlesource.com/chromium/tools/build",
"swarming_task_id:",
"user_agent:luci-scheduler"
],
"updated_ts": "1604155905703349",
"url": "https://ci.chromium.org/b/8864914333616179904",
"utcnow_ts": "1604155905820234"
}
}
[14:51:45.828] Task URL: https://ci.chromium.org/b/8864914333616179904
[14:51:45.828] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:2:0) after 1m0s
[14:52:45.809] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:2:0)
[14:52:45.809] Timer tick, asking Buildbucket for the build status
[14:52:45.915] Build 8864914333616179904: status "SCHEDULED", result "", failure_reason "", cancelation_reason ""
[14:52:45.915] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:3:0) after 1m0s
[14:53:45.909] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:3:0)
[14:53:45.909] Timer tick, asking Buildbucket for the build status
[14:53:46.009] Build 8864914333616179904: status "SCHEDULED", result "", failure_reason "", cancelation_reason ""
[14:53:46.009] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:4:0) after 1m0s
[14:54:46.084] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:4:0)
[14:54:46.084] Timer tick, asking Buildbucket for the build status
[14:54:46.223] Build 8864914333616179904: status "SCHEDULED", result "", failure_reason "", cancelation_reason ""
[14:54:46.223] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:5:0) after 1m0s
[14:55:06.493] Received PubSub notification, asking Buildbucket for the build status
[14:55:06.572] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:55:46.289] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:5:0)
[14:55:46.289] Timer tick, asking Buildbucket for the build status
[14:55:46.472] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:55:46.472] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:7:0) after 1m0s
[14:56:46.612] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:7:0)
[14:56:46.612] Timer tick, asking Buildbucket for the build status
[14:56:46.729] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:56:46.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:8:0) after 1m0s
[14:57:46.741] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:8:0)
[14:57:46.741] Timer tick, asking Buildbucket for the build status
[14:57:46.868] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:57:46.868] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:9:0) after 1m0s
[14:58:46.916] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:9:0)
[14:58:46.916] Timer tick, asking Buildbucket for the build status
[14:58:47.092] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:58:47.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:10:0) after 1m0s
[14:59:47.131] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:10:0)
[14:59:47.131] Timer tick, asking Buildbucket for the build status
[14:59:49.319] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[14:59:49.319] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:11:0) after 1m0s
[15:00:49.357] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:11:0)
[15:00:49.357] Timer tick, asking Buildbucket for the build status
[15:00:49.425] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:00:49.425] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:12:0) after 1m0s
[15:01:49.455] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:12:0)
[15:01:49.455] Timer tick, asking Buildbucket for the build status
[15:01:49.600] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:01:49.600] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:13:0) after 1m0s
[15:02:49.628] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:13:0)
[15:02:49.628] Timer tick, asking Buildbucket for the build status
[15:02:49.773] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:02:49.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:14:0) after 1m0s
[15:03:49.888] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:14:0)
[15:03:49.888] Timer tick, asking Buildbucket for the build status
[15:03:50.059] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:03:50.059] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:15:0) after 1m0s
[15:04:50.092] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:15:0)
[15:04:50.092] Timer tick, asking Buildbucket for the build status
[15:04:50.389] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:04:50.389] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:16:0) after 1m0s
[15:05:50.505] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:16:0)
[15:05:50.505] Timer tick, asking Buildbucket for the build status
[15:05:50.589] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:05:50.589] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:17:0) after 1m0s
[15:06:50.662] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:17:0)
[15:06:50.662] Timer tick, asking Buildbucket for the build status
[15:06:50.944] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:06:50.944] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:18:0) after 1m0s
[15:07:51.028] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:18:0)
[15:07:51.028] Timer tick, asking Buildbucket for the build status
[15:07:51.240] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:07:51.240] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:19:0) after 1m0s
[15:08:51.286] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:19:0)
[15:08:51.286] Timer tick, asking Buildbucket for the build status
[15:08:51.370] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:08:51.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:20:0) after 1m0s
[15:09:51.407] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:20:0)
[15:09:51.407] Timer tick, asking Buildbucket for the build status
[15:09:52.456] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:09:52.456] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:21:0) after 1m0s
[15:10:52.509] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:21:0)
[15:10:52.509] Timer tick, asking Buildbucket for the build status
[15:10:52.881] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:10:52.881] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:22:0) after 1m0s
[15:11:52.866] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:22:0)
[15:11:52.866] Timer tick, asking Buildbucket for the build status
[15:11:53.078] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:11:53.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:23:0) after 1m0s
[15:12:53.142] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:23:0)
[15:12:53.142] Timer tick, asking Buildbucket for the build status
[15:12:53.212] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:12:53.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:24:0) after 1m0s
[15:13:53.234] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:24:0)
[15:13:53.234] Timer tick, asking Buildbucket for the build status
[15:13:53.307] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:13:53.307] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:25:0) after 1m0s
[15:14:53.358] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:25:0)
[15:14:53.358] Timer tick, asking Buildbucket for the build status
[15:14:53.494] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:14:53.494] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:26:0) after 1m0s
[15:15:53.553] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:26:0)
[15:15:53.553] Timer tick, asking Buildbucket for the build status
[15:15:53.622] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:15:53.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:27:0) after 1m0s
[15:16:53.645] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:27:0)
[15:16:53.645] Timer tick, asking Buildbucket for the build status
[15:16:53.774] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:16:53.774] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:28:0) after 1m0s
[15:17:53.817] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:28:0)
[15:17:53.817] Timer tick, asking Buildbucket for the build status
[15:17:53.895] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:17:53.895] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:29:0) after 1m0s
[15:18:53.954] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:29:0)
[15:18:53.954] Timer tick, asking Buildbucket for the build status
[15:18:54.027] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:18:54.027] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:30:0) after 1m0s
[15:19:54.100] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:30:0)
[15:19:54.100] Timer tick, asking Buildbucket for the build status
[15:19:54.204] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:19:54.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:31:0) after 1m0s
[15:20:54.246] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:31:0)
[15:20:54.246] Timer tick, asking Buildbucket for the build status
[15:20:54.318] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:20:54.318] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:32:0) after 1m0s
[15:21:54.389] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:32:0)
[15:21:54.389] Timer tick, asking Buildbucket for the build status
[15:21:54.536] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:21:54.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:33:0) after 1m0s
[15:22:54.588] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:33:0)
[15:22:54.588] Timer tick, asking Buildbucket for the build status
[15:22:54.649] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:22:54.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:34:0) after 1m0s
[15:23:55.002] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:34:0)
[15:23:55.002] Timer tick, asking Buildbucket for the build status
[15:23:55.302] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:23:55.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:35:0) after 1m0s
[15:25:06.521] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:35:0)
[15:25:06.521] Timer tick, asking Buildbucket for the build status
[15:25:06.690] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:25:06.690] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:36:0) after 1m0s
[15:26:07.325] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:36:0)
[15:26:07.325] Timer tick, asking Buildbucket for the build status
[15:26:07.404] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:26:07.404] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:37:0) after 1m0s
[15:27:07.537] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:37:0)
[15:27:07.537] Timer tick, asking Buildbucket for the build status
[15:27:07.752] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:27:07.752] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:38:0) after 1m0s
[15:28:08.057] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:38:0)
[15:28:08.058] Timer tick, asking Buildbucket for the build status
[15:28:08.191] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:28:08.191] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:39:0) after 1m0s
[15:29:08.506] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:39:0)
[15:29:08.506] Timer tick, asking Buildbucket for the build status
[15:29:08.575] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:29:08.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:40:0) after 1m0s
[15:30:08.761] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:40:0)
[15:30:08.761] Timer tick, asking Buildbucket for the build status
[15:30:08.821] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:30:08.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:41:0) after 1m0s
[15:31:09.632] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:41:0)
[15:31:09.632] Timer tick, asking Buildbucket for the build status
[15:31:10.462] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:31:10.462] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:42:0) after 1m0s
[15:32:10.623] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:42:0)
[15:32:10.623] Timer tick, asking Buildbucket for the build status
[15:32:10.790] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:32:10.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:43:0) after 1m0s
[15:33:10.813] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:43:0)
[15:33:10.813] Timer tick, asking Buildbucket for the build status
[15:33:10.891] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:33:10.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:44:0) after 1m0s
[15:34:10.974] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:44:0)
[15:34:10.974] Timer tick, asking Buildbucket for the build status
[15:34:11.090] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:34:11.090] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:45:0) after 1m0s
[15:35:11.234] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:45:0)
[15:35:11.234] Timer tick, asking Buildbucket for the build status
[15:35:11.309] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:35:11.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:46:0) after 1m0s
[15:36:11.380] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:46:0)
[15:36:11.380] Timer tick, asking Buildbucket for the build status
[15:36:11.497] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:36:11.497] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:47:0) after 1m0s
[15:37:11.601] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:47:0)
[15:37:11.601] Timer tick, asking Buildbucket for the build status
[15:37:11.719] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:37:11.719] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:48:0) after 1m0s
[15:38:11.956] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:48:0)
[15:38:11.956] Timer tick, asking Buildbucket for the build status
[15:38:12.190] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:38:12.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:49:0) after 1m0s
[15:39:12.232] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:49:0)
[15:39:12.232] Timer tick, asking Buildbucket for the build status
[15:39:12.299] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:39:12.299] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:50:0) after 1m0s
[15:40:12.353] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:50:0)
[15:40:12.353] Timer tick, asking Buildbucket for the build status
[15:40:12.705] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:40:12.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:51:0) after 1m0s
[15:41:12.831] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:51:0)
[15:41:12.831] Timer tick, asking Buildbucket for the build status
[15:41:12.905] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:41:12.905] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:52:0) after 1m0s
[15:42:12.934] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:52:0)
[15:42:12.934] Timer tick, asking Buildbucket for the build status
[15:42:13.083] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:42:13.083] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:53:0) after 1m0s
[15:43:13.158] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:53:0)
[15:43:13.158] Timer tick, asking Buildbucket for the build status
[15:43:13.227] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:43:13.227] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:54:0) after 1m0s
[15:44:13.252] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:54:0)
[15:44:13.252] Timer tick, asking Buildbucket for the build status
[15:44:13.365] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:44:13.365] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:55:0) after 1m0s
[15:45:13.651] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:55:0)
[15:45:13.651] Timer tick, asking Buildbucket for the build status
[15:45:13.866] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:45:13.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:56:0) after 1m0s
[15:46:14.065] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:56:0)
[15:46:14.065] Timer tick, asking Buildbucket for the build status
[15:46:14.519] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:46:14.519] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:57:0) after 1m0s
[15:47:14.510] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:57:0)
[15:47:14.510] Timer tick, asking Buildbucket for the build status
[15:47:14.778] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:47:14.778] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:58:0) after 1m0s
[15:48:14.813] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:58:0)
[15:48:14.813] Timer tick, asking Buildbucket for the build status
[15:48:14.884] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:48:14.884] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:59:0) after 1m0s
[15:49:15.126] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:59:0)
[15:49:15.126] Timer tick, asking Buildbucket for the build status
[15:49:15.348] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:49:15.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:60:0) after 1m0s
[15:50:15.451] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:60:0)
[15:50:15.451] Timer tick, asking Buildbucket for the build status
[15:50:15.584] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:50:15.584] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:61:0) after 1m0s
[15:51:15.651] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:61:0)
[15:51:15.651] Timer tick, asking Buildbucket for the build status
[15:51:15.792] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:51:15.792] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:62:0) after 1m0s
[15:52:15.814] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:62:0)
[15:52:15.814] Timer tick, asking Buildbucket for the build status
[15:52:15.935] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:52:15.935] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:63:0) after 1m0s
[15:53:15.991] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:63:0)
[15:53:15.991] Timer tick, asking Buildbucket for the build status
[15:53:16.204] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:53:16.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:64:0) after 1m0s
[15:54:16.244] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:64:0)
[15:54:16.244] Timer tick, asking Buildbucket for the build status
[15:54:16.470] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:54:16.470] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:65:0) after 1m0s
[15:55:16.665] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:65:0)
[15:55:16.665] Timer tick, asking Buildbucket for the build status
[15:55:16.924] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:55:16.924] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:66:0) after 1m0s
[15:56:17.047] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:66:0)
[15:56:17.047] Timer tick, asking Buildbucket for the build status
[15:56:17.217] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:56:17.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:67:0) after 1m0s
[15:57:17.238] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:67:0)
[15:57:17.238] Timer tick, asking Buildbucket for the build status
[15:57:17.353] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:57:17.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:68:0) after 1m0s
[15:58:17.410] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:68:0)
[15:58:17.410] Timer tick, asking Buildbucket for the build status
[15:58:17.504] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:58:17.504] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:69:0) after 1m0s
[15:59:17.611] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:69:0)
[15:59:17.611] Timer tick, asking Buildbucket for the build status
[15:59:18.178] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[15:59:18.178] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:70:0) after 1m0s
[16:00:18.367] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:70:0)
[16:00:18.367] Timer tick, asking Buildbucket for the build status
[16:00:19.083] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:00:19.083] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:71:0) after 1m0s
[16:01:19.327] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:71:0)
[16:01:19.327] Timer tick, asking Buildbucket for the build status
[16:01:19.464] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:01:19.464] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:72:0) after 1m0s
[16:02:19.535] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:72:0)
[16:02:19.535] Timer tick, asking Buildbucket for the build status
[16:02:19.805] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:02:19.805] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:73:0) after 1m0s
[16:03:19.855] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:73:0)
[16:03:19.855] Timer tick, asking Buildbucket for the build status
[16:03:19.936] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:03:19.936] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:74:0) after 1m0s
[16:04:19.991] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:74:0)
[16:04:19.991] Timer tick, asking Buildbucket for the build status
[16:04:20.168] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:04:20.168] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:75:0) after 1m0s
[16:05:20.327] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:75:0)
[16:05:20.327] Timer tick, asking Buildbucket for the build status
[16:05:20.490] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:05:20.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:76:0) after 1m0s
[16:06:20.545] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:76:0)
[16:06:20.545] Timer tick, asking Buildbucket for the build status
[16:06:20.734] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:06:20.734] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:77:0) after 1m0s
[16:07:20.782] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:77:0)
[16:07:20.782] Timer tick, asking Buildbucket for the build status
[16:07:20.949] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:07:20.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:78:0) after 1m0s
[16:08:21.094] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:78:0)
[16:08:21.094] Timer tick, asking Buildbucket for the build status
[16:08:21.172] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:08:21.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:79:0) after 1m0s
[16:09:21.236] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:79:0)
[16:09:21.236] Timer tick, asking Buildbucket for the build status
[16:09:21.304] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:09:21.304] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:80:0) after 1m0s
[16:10:21.371] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:80:0)
[16:10:21.371] Timer tick, asking Buildbucket for the build status
[16:10:21.431] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:10:21.431] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:81:0) after 1m0s
[16:11:21.499] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:81:0)
[16:11:21.499] Timer tick, asking Buildbucket for the build status
[16:11:21.785] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:11:21.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:82:0) after 1m0s
[16:12:21.856] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:82:0)
[16:12:21.856] Timer tick, asking Buildbucket for the build status
[16:12:21.961] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:12:21.961] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:83:0) after 1m0s
[16:13:22.053] Handling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:83:0)
[16:13:22.053] Timer tick, asking Buildbucket for the build status
[16:13:22.134] Build 8864914333616179904: status "STARTED", result "", failure_reason "", cancelation_reason ""
[16:13:22.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTiOSDevice:9030344395788160080:84:0) after 1m0s
[16:13:55.478] Received PubSub notification, asking Buildbucket for the build status
[16:13:55.554] Build 8864914333616179904: status "COMPLETED", result "FAILURE", failure_reason "BUILD_FAILURE", cancelation_reason ""
[16:13:55.554] Buildbucket build:
{
"build": {
"bucket": "luci.chromium.ci",
"canary_preference": "PROD",
"completed_ts": "1604160834986821",
"created_by": "project:chromium",
"created_ts": "1604155905096663",
"failure_reason": "BUILD_FAILURE",
"id": "8864914333616179904",
"parameters_json": "{\"builder_name\": \"ToTiOSDevice\", \"properties\": {\"branch\": \"refs/heads/master\", \"repository\": \"https://chromium.googlesource.com/chromium/src\", \"revision\": \"e66cafeaecb1380bfab166de05a27ded93aa8558\"}}",
"project": "chromium",
"result": "FAILURE",
"result_details_json": "{\"error\": {\"message\": \"#### Step _compile_ failed. Error logs are shown below:\\n```\\n[6952/21508] OBJCXX obj/ios/web/js_messaging/js_messaging/web_frame_impl.o\\nFAILED: obj/ios/web/js_messaging/js_messaging/web_frame_impl.o\\n../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/ios/web/js_messaging/js_messag...(too long)\\nAssertion failed: (CurOffs \u003c= CurLayObj-\u003efirst), function getObjCEncodingForStructureImpl, file /opt/s/w/ir/cache/builder/src/third_party/llvm/clang/lib/AST/ASTContext.cpp, line 7718.\\nPLEASE submit a bug report to https://crbug.com and run tools/clang/scripts/process_crashreports.py (only works inside Google) which will upload a report and include the crash backtrace, preprocessed source, and associated run script.\\nStack dump:\\n```\\n##### ...The message was too long...\\n#### More information in raw_io.output[failure_summary]\"}, \"properties\": {\"$build/goma\": {\"use_luci_auth\": true}, \"$recipe_engine/isolated\": {\"server\": \"https://isolateserver.appspot.com\"}, \"$recipe_engine/milo/blamelist_pins\": [{\"host\": \"chromium.googlesource.com\", \"id\": \"e66cafeaecb1380bfab166de05a27ded93aa8558\", \"project\": \"chromium/src\"}], \"$recipe_engine/path\": {\"cache_dir\": \"/opt/s/w/ir/cache\", \"temp_dir\": \"/opt/s/w/ir/tmp/rt\"}, \"$recipe_engine/runtime\": {\"is_experimental\": false, \"is_luci\": true}, \"bot_id\": \"build716-m9\", \"branch\": \"refs/heads/master\", \"builder_group\": \"chromium.clang\", \"buildername\": \"ToTiOSDevice\", \"buildnumber\": 20444, \"clang_revision\": \"llvmorg-12-init-5627-gf086e85e-2\", \"got_angle_revision\": \"42ad027982404039e6c9a921b2f6354f55b65a2e\", \"got_dawn_revision\": \"0a374dce665328391ab3b68fd9e667b018dd9dcd\", \"got_nacl_revision\": \"f9e6f257d1a0934dbeed3c5c8f83b651dc4f55e1\", \"got_revision\": \"e66cafeaecb1380bfab166de05a27ded93aa8558\", \"got_revision_cp\": \"refs/heads/master@{#822963}\", \"got_swarming_client_revision\": \"d46ea7635f2911208268170512cb611412488fd8\", \"got_swiftshader_revision\": \"e0de282e678b72dac25b81f8d25305231aec01f8\", \"got_v8_revision\": \"69c856dc816ff1dd65fcc536095b44399a7434b9\", \"got_v8_revision_cp\": \"refs/heads/8.8.214@{#1}\", \"got_webrtc_revision\": \"ee24eb66d23ffea5ef729dc340e1fc4381c72a4a\", \"got_webrtc_revision_cp\": \"refs/heads/master@{#32507}\", \"path_config\": \"generic\", \"perf_dashboard_machine_group\": \"ChromiumClang\", \"recipe\": \"chromium\", \"repository\": \"https://chromium.googlesource.com/chromium/src\", \"revision\": \"e66cafeaecb1380bfab166de05a27ded93aa8558\", \"xcode_build_version\": \"12a7209\"}, \"swarming\": {\"bot_dimensions\": {\"builder\": [\"ToTiOS\", \"ToTiOSDevice\"], \"caches\": [\"builder_ae5debd901c74ee0906916cb62621bceb2a8b657b42c2e2e9e89a0f6a2e5e53b_v2\", \"git\", \"goma_v2\", \"vpython\", \"xcode_ios_12a7209\"], \"cores\": [\"12\"], \"cpu\": [\"x86\", \"x86-64\", \"x86-64-i7-8700B\"], \"gce\": [\"0\"], \"gpu\": [\"8086\", \"8086:3e9b\"], \"hidpi\": [\"0\"], \"id\": [\"build716-m9\"], \"mac_model\": [\"Macmini8,1\"], \"machine_type\": [\"n1-standard-8\"], \"os\": [\"Mac\", \"Mac-10.15\", \"Mac-10.15.6\", \"Mac-10.15.6-19G2021\"], \"pool\": [\"luci.chromium.ci\"], \"python\": [\"2.7.16\"], \"server_version\": [\"5614-d862eaa\"], \"ssd\": [\"1\"], \"zone\": [\"us\", \"us-atl\", \"us-atl-golo\", \"us-atl-golo-m9\"]}}, \"ui\": {\"info\": \"#### Step _compile_ failed. Error logs are shown below:\\n```\\n[6952/21508] OBJCXX obj/ios/web/js_messaging/js_messaging/web_frame_impl.o\\nFAILED: obj/ios/web/js_messaging/js_messaging/web_frame_impl.o\\n../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/ios/web/js_messaging/js_messag...(too long)\\nAssertion failed: (CurOffs \u003c= CurLayObj-\u003efirst), function getObjCEncodingForStructureImpl, file /opt/s/w/ir/cache/builder/src/third_party/llvm/clang/lib/AST/ASTContext.cpp, line 7718.\\nPLEASE submit a bug report to https://crbug.com and run tools/clang/scripts/process_crashreports.py (only works inside Google) which will upload a report and include the crash backtrace, preprocessed source, and associated run script.\\nStack dump:\\n```\\n##### ...The message was too long...\\n#### More information in raw_io.output[failure_summary]\"}}",
"service_account": "chromium-ci-builder@chops-service-accounts.iam.gserviceaccount.com",
"started_ts": "1604156105692737",
"status": "COMPLETED",
"status_changed_ts": "1604156106126094",
"tags": [
"build_address:luci.chromium.ci/ToTiOSDevice/20444",
"builder:ToTiOSDevice",
"buildset:commit/git/e66cafeaecb1380bfab166de05a27ded93aa8558",
"buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e66cafeaecb1380bfab166de05a27ded93aa8558",
"gitiles_ref:refs/heads/master",
"scheduler_invocation_id:9030344395788160080",
"scheduler_job_id:chromium/ToTiOSDevice",
"swarming_hostname:chromium-swarm.appspot.com",
"swarming_tag:log_location:logdog://logs.chromium.org/chromium/buildbucket/cr-buildbucket.appspot.com/8864914333616179904/+/annotations",
"swarming_tag:luci_project:chromium",
"swarming_tag:os:Mac",
"swarming_tag:recipe_name:chromium",
"swarming_tag:recipe_package:infra/recipe_bundles/chromium.googlesource.com/chromium/tools/build",
"swarming_task_id:4f97f6220d6ff310",
"user_agent:luci-scheduler"
],
"updated_ts": "1604160835059179",
"url": "https://ci.chromium.org/b/8864914333616179904",
"utcnow_ts": "1604160835545732"
}
}
[16:13:55.554] Invocation finished in 1h22m11.912894392s with status FAILED