[19:30:44.805] New invocation is queued and will start shortly
[19:30:45.933] Starting the invocation (attempt 1)
[19:30:45.964] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/e2a09433a8f644959d261c2f186f3e685dba66bc"
[19:30:45.964] Popped gitiles commit info from properties and tags
[19:30:45.964] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:30:45.964] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:30:45.964] Buildbucket request:
{
"requestId": "8851216638836854528",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e2a09433a8f644959d261c2f186f3e685dba66bc",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851216638836854528"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:30:46.531] Scheduled build:
{
"id": "8685786576916074449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2373,
"createdBy": "project:chromium",
"createTime": "2026-03-31T19:30:46.016283492Z",
"updateTime": "2026-03-31T19:30:46.016283492Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e2a09433a8f644959d261c2f186f3e685dba66bc",
"ref": "refs/heads/main"
}
}
}
[19:30:46.531] Task URL: https://cr-buildbucket.appspot.com/build/8685786576916074449
[19:30:46.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:2:0) after 4m29s
[19:30:47.443] Received PubSub notification, asking Buildbucket for the build status
[19:30:47.460] Build status: SCHEDULED
[19:31:29.584] Received PubSub notification, asking Buildbucket for the build status
[19:31:29.617] Build status: STARTED
[19:35:15.739] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:2:0)
[19:35:15.761] Build status: STARTED
[19:35:15.761] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:5:0) after 6m17s
[19:41:32.782] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:5:0)
[19:41:32.847] Build status: STARTED
[19:41:32.847] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:6:0) after 7m45s
[19:49:17.872] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:6:0)
[19:49:17.893] Build status: STARTED
[19:49:17.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:7:0) after 7m8s
[19:56:25.974] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:7:0)
[19:56:26.024] Build status: STARTED
[19:56:26.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:8:0) after 2m39s
[19:59:05.048] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:8:0)
[19:59:05.067] Build status: STARTED
[19:59:05.067] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:9:0) after 3m34s
[20:02:39.134] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:9:0)
[20:02:39.200] Build status: STARTED
[20:02:39.200] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:10:0) after 4m43s
[20:07:22.272] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:10:0)
[20:07:22.297] Build status: STARTED
[20:07:22.297] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:11:0) after 7m36s
[20:14:58.340] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:11:0)
[20:14:58.360] Build status: STARTED
[20:14:58.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:12:0) after 6m12s
[20:21:10.622] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:12:0)
[20:21:10.644] Build status: STARTED
[20:21:10.645] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:13:0) after 4m40s
[20:25:50.686] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:13:0)
[20:25:50.761] Build status: STARTED
[20:25:50.761] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:14:0) after 5m18s
[20:31:08.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:14:0)
[20:31:08.813] Build status: STARTED
[20:31:08.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:15:0) after 1m16s
[20:32:24.834] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:15:0)
[20:32:24.896] Build status: STARTED
[20:32:24.896] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:16:0) after 8m19s
[20:40:44.020] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:16:0)
[20:40:44.046] Build status: STARTED
[20:40:44.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:17:0) after 6m30s
[20:47:14.138] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:17:0)
[20:47:14.160] Build status: STARTED
[20:47:14.160] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:18:0) after 5m34s
[20:52:48.184] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:18:0)
[20:52:48.199] Build status: STARTED
[20:52:48.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:19:0) after 4m24s
[20:57:12.246] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:19:0)
[20:57:12.264] Build status: STARTED
[20:57:12.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:20:0) after 1m15s
[20:58:27.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:20:0)
[20:58:27.325] Build status: STARTED
[20:58:27.325] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:21:0) after 8m33s
[21:07:00.422] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:21:0)
[21:07:00.442] Build status: STARTED
[21:07:00.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:22:0) after 2m43s
[21:09:43.525] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:22:0)
[21:09:43.545] Build status: STARTED
[21:09:43.545] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:23:0) after 1m59s
[21:11:42.632] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:23:0)
[21:11:42.654] Build status: STARTED
[21:11:42.654] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:24:0) after 6m1s
[21:17:43.687] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:24:0)
[21:17:43.701] Build status: STARTED
[21:17:43.701] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:25:0) after 4m44s
[21:22:27.726] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:25:0)
[21:22:27.758] Build status: STARTED
[21:22:27.758] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:26:0) after 3m24s
[21:25:51.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:26:0)
[21:25:51.776] Build status: STARTED
[21:25:51.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:27:0) after 2m9s
[21:28:00.856] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:27:0)
[21:28:00.876] Build status: STARTED
[21:28:00.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:28:0) after 9m3s
[21:37:03.893] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:28:0)
[21:37:03.918] Build status: STARTED
[21:37:03.918] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:29:0) after 2m44s
[21:39:47.986] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:29:0)
[21:39:48.001] Build status: STARTED
[21:39:48.001] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:30:0) after 1m48s
[21:41:36.075] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:30:0)
[21:41:36.125] Build status: STARTED
[21:41:36.125] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:31:0) after 1m4s
[21:42:40.220] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:31:0)
[21:42:40.278] Build status: STARTED
[21:42:40.278] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:32:0) after 6m19s
[21:48:59.438] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:32:0)
[21:48:59.500] Build status: STARTED
[21:48:59.500] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:33:0) after 8m35s
[21:57:34.515] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:33:0)
[21:57:34.594] Build status: STARTED
[21:57:34.594] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:34:0) after 1m9s
[21:58:43.641] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:34:0)
[21:58:43.659] Build status: STARTED
[21:58:43.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:35:0) after 1m49s
[22:00:32.763] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:35:0)
[22:00:32.814] Build status: STARTED
[22:00:32.814] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:36:0) after 1m32s
[22:02:04.887] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:36:0)
[22:02:04.965] Build status: STARTED
[22:02:04.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:37:0) after 6m15s
[22:08:20.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:37:0)
[22:08:20.052] Build status: STARTED
[22:08:20.052] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:38:0) after 9m59s
[22:18:19.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:38:0)
[22:18:19.145] Build status: STARTED
[22:18:19.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:39:0) after 9m26s
[22:27:45.140] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:39:0)
[22:27:45.162] Build status: STARTED
[22:27:45.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:40:0) after 5m10s
[22:32:55.388] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:40:0)
[22:32:55.476] Build status: STARTED
[22:32:55.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:41:0) after 6m35s
[22:39:30.485] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:41:0)
[22:39:30.507] Build status: STARTED
[22:39:30.507] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:42:0) after 3m44s
[22:43:14.551] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:42:0)
[22:43:14.565] Build status: STARTED
[22:43:14.565] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:43:0) after 3m34s
[22:46:48.568] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:43:0)
[22:46:48.624] Build status: STARTED
[22:46:48.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:44:0) after 5m46s
[22:52:34.630] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:44:0)
[22:52:34.653] Build status: STARTED
[22:52:34.653] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:45:0) after 9m54s
[23:02:28.700] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:45:0)
[23:02:28.720] Build status: STARTED
[23:02:28.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:46:0) after 1m21s
[23:03:49.795] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:46:0)
[23:03:49.824] Build status: STARTED
[23:03:49.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:47:0) after 1m34s
[23:05:23.890] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:47:0)
[23:05:23.909] Build status: STARTED
[23:05:23.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:48:0) after 4m57s
[23:10:20.930] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:48:0)
[23:10:20.979] Build status: STARTED
[23:10:20.979] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:49:0) after 2m32s
[23:12:52.968] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:49:0)
[23:12:52.985] Build status: STARTED
[23:12:52.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:50:0) after 3m36s
[23:16:29.058] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:50:0)
[23:16:29.078] Build status: STARTED
[23:16:29.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:51:0) after 1m16s
[23:17:45.097] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:51:0)
[23:17:45.129] Build status: STARTED
[23:17:45.130] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:52:0) after 3m53s
[23:21:38.151] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:52:0)
[23:21:38.186] Build status: STARTED
[23:21:38.186] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:53:0) after 1m56s
[23:23:34.229] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:53:0)
[23:23:34.248] Build status: STARTED
[23:23:34.248] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:54:0) after 1m45s
[23:25:19.370] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:54:0)
[23:25:19.401] Build status: STARTED
[23:25:19.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:55:0) after 9m3s
[23:34:22.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:55:0)
[23:34:22.672] Build status: STARTED
[23:34:22.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:56:0) after 4m3s
[23:38:25.696] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:56:0)
[23:38:25.746] Build status: STARTED
[23:38:25.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:57:0) after 9m3s
[23:47:28.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:57:0)
[23:47:28.808] Build status: STARTED
[23:47:28.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:58:0) after 5m39s
[23:53:07.827] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:58:0)
[23:53:07.869] Build status: STARTED
[23:53:07.869] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:59:0) after 6m49s
[23:59:56.846] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:59:0)
[23:59:56.879] Build status: STARTED
[23:59:56.879] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:60:0) after 9m55s
[00:09:51.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:60:0)
[00:09:51.912] Build status: STARTED
[00:09:51.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:61:0) after 5m19s
[00:15:11.044] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:61:0)
[00:15:11.112] Build status: STARTED
[00:15:11.112] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851216638836854528:62:0) after 3m55s
[00:15:49.665] Received PubSub notification, asking Buildbucket for the build status
[00:15:49.686] Build:
{
"id": "8685786576916074449",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2373,
"createdBy": "project:chromium",
"createTime": "2026-03-31T19:30:46.016283492Z",
"startTime": "2026-03-31T19:31:29.326220323Z",
"endTime": "2026-04-01T00:15:49.414153443Z",
"updateTime": "2026-04-01T00:15:49.414153443Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "e2a09433a8f644959d261c2f186f3e685dba66bc",
"ref": "refs/heads/main"
}
}
}
[00:15:49.687] Invocation finished in 4h45m4.885611956s with status FAILED