[21:46:02.099] New invocation is queued and will start shortly
[21:46:03.605] Starting the invocation (attempt 1)
[21:46:03.635] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/a74e233110e33fef97dfa5cd1e78f4c74786d029"
[21:46:03.635] Popped gitiles commit info from properties and tags
[21:46:03.635] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:46:03.635] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:46:03.636] Buildbucket request:
{
"requestId": "8850573948482979968",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a74e233110e33fef97dfa5cd1e78f4c74786d029",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850573948482979968"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:46:04.215] Scheduled build:
{
"id": "8685143886142548065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2409,
"createdBy": "project:chromium",
"createTime": "2026-04-07T21:46:03.701048390Z",
"updateTime": "2026-04-07T21:46:03.701048390Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a74e233110e33fef97dfa5cd1e78f4c74786d029",
"ref": "refs/heads/main"
}
},
"canary": true
}
[21:46:04.215] Task URL: https://cr-buildbucket.appspot.com/build/8685143886142548065
[21:46:04.215] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:2:0) after 8m24s
[21:46:05.084] Received PubSub notification, asking Buildbucket for the build status
[21:46:05.106] Build status: SCHEDULED
[21:46:43.582] Received PubSub notification, asking Buildbucket for the build status
[21:46:43.612] Build status: STARTED
[21:54:28.389] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:2:0)
[21:54:28.531] Build status: STARTED
[21:54:28.531] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:5:0) after 8m44s
[22:03:12.548] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:5:0)
[22:03:12.564] Build status: STARTED
[22:03:12.564] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:6:0) after 9m12s
[22:12:24.588] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:6:0)
[22:12:24.615] Build status: STARTED
[22:12:24.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:7:0) after 7m36s
[22:20:00.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:7:0)
[22:20:00.706] Build status: STARTED
[22:20:00.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:8:0) after 3m39s
[22:23:39.774] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:8:0)
[22:23:39.791] Build status: STARTED
[22:23:39.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:9:0) after 3m25s
[22:27:04.826] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:9:0)
[22:27:04.851] Build status: STARTED
[22:27:04.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:10:0) after 4m10s
[22:31:15.060] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:10:0)
[22:31:15.121] Build status: STARTED
[22:31:15.121] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:11:0) after 2m48s
[22:34:03.109] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:11:0)
[22:34:03.123] Build status: STARTED
[22:34:03.123] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:12:0) after 5m9s
[22:39:12.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:12:0)
[22:39:12.161] Build status: STARTED
[22:39:12.161] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:13:0) after 3m15s
[22:42:27.290] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:13:0)
[22:42:27.313] Build status: STARTED
[22:42:27.313] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:14:0) after 7m45s
[22:50:12.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:14:0)
[22:50:12.426] Build status: STARTED
[22:50:12.426] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:15:0) after 3m51s
[22:54:03.513] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:15:0)
[22:54:03.616] Build status: STARTED
[22:54:03.616] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:16:0) after 9m54s
[23:03:57.790] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:16:0)
[23:03:57.808] Build status: STARTED
[23:03:57.808] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:17:0) after 1m3s
[23:05:00.882] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:17:0)
[23:05:00.906] Build status: STARTED
[23:05:00.906] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:18:0) after 9m8s
[23:14:08.966] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:18:0)
[23:14:08.980] Build status: STARTED
[23:14:08.981] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:19:0) after 2m11s
[23:16:19.981] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:19:0)
[23:16:19.996] Build status: STARTED
[23:16:19.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:20:0) after 4m26s
[23:20:46.166] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:20:0)
[23:20:46.190] Build status: STARTED
[23:20:46.191] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:21:0) after 8m29s
[23:29:15.215] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:21:0)
[23:29:15.232] Build status: STARTED
[23:29:15.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:22:0) after 4m54s
[23:34:09.325] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:22:0)
[23:34:09.343] Build status: STARTED
[23:34:09.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:23:0) after 4m13s
[23:38:22.418] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:23:0)
[23:38:22.435] Build status: STARTED
[23:38:22.435] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:24:0) after 1m8s
[23:39:30.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:24:0)
[23:39:30.530] Build status: STARTED
[23:39:30.530] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:25:0) after 8m24s
[23:47:54.553] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:25:0)
[23:47:54.575] Build status: STARTED
[23:47:54.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:26:0) after 2m7s
[23:50:01.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:26:0)
[23:50:01.624] Build status: STARTED
[23:50:01.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:27:0) after 4m14s
[23:54:15.615] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:27:0)
[23:54:15.645] Build status: STARTED
[23:54:15.645] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:28:0) after 2m5s
[23:56:20.689] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:28:0)
[23:56:20.707] Build status: STARTED
[23:56:20.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:29:0) after 1m44s
[23:58:04.779] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:29:0)
[23:58:04.894] Build status: STARTED
[23:58:04.894] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:30:0) after 3m14s
[00:01:18.916] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:30:0)
[00:01:18.933] Build status: STARTED
[00:01:18.933] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:31:0) after 7m51s
[00:09:10.029] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:31:0)
[00:09:10.047] Build status: STARTED
[00:09:10.047] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:32:0) after 4m36s
[00:13:46.173] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:32:0)
[00:13:46.219] Build status: STARTED
[00:13:46.219] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:33:0) after 5m51s
[00:19:37.329] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:33:0)
[00:19:37.348] Build status: STARTED
[00:19:37.348] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:34:0) after 3m27s
[00:23:04.643] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:34:0)
[00:23:04.668] Build status: STARTED
[00:23:04.668] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:35:0) after 2m40s
[00:25:44.683] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:35:0)
[00:25:44.726] Build status: STARTED
[00:25:44.726] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:36:0) after 2m50s
[00:28:34.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:36:0)
[00:28:34.749] Build status: STARTED
[00:28:34.749] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:37:0) after 4m7s
[00:32:41.870] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:37:0)
[00:32:41.894] Build status: STARTED
[00:32:41.894] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:38:0) after 6m27s
[00:39:09.003] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:38:0)
[00:39:09.022] Build status: STARTED
[00:39:09.022] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:39:0) after 6m42s
[00:45:51.050] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:39:0)
[00:45:51.071] Build status: STARTED
[00:45:51.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:40:0) after 7m32s
[00:53:23.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:40:0)
[00:53:23.255] Build status: STARTED
[00:53:23.255] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:41:0) after 2m41s
[00:56:04.355] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:41:0)
[00:56:04.429] Build status: STARTED
[00:56:04.429] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:42:0) after 7m19s
[01:03:23.492] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:42:0)
[01:03:23.508] Build status: STARTED
[01:03:23.508] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:43:0) after 2m52s
[01:06:15.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:43:0)
[01:06:15.666] Build status: STARTED
[01:06:15.667] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:44:0) after 6m4s
[01:12:19.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:44:0)
[01:12:19.681] Build status: STARTED
[01:12:19.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:45:0) after 2m10s
[01:14:29.751] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:45:0)
[01:14:29.765] Build status: STARTED
[01:14:29.766] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:46:0) after 7m39s
[01:22:08.831] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:46:0)
[01:22:08.926] Build status: STARTED
[01:22:08.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:47:0) after 4m58s
[01:27:06.952] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:47:0)
[01:27:06.972] Build status: STARTED
[01:27:06.972] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:48:0) after 5m38s
[01:32:44.995] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:48:0)
[01:32:45.052] Build status: STARTED
[01:32:45.052] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:49:0) after 8m54s
[01:41:39.061] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:49:0)
[01:41:39.134] Build status: STARTED
[01:41:39.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:50:0) after 3m12s
[01:44:51.162] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:50:0)
[01:44:51.181] Build status: STARTED
[01:44:51.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:51:0) after 5m50s
[01:50:41.211] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:51:0)
[01:50:41.225] Build status: STARTED
[01:50:41.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:52:0) after 3m9s
[01:53:50.337] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:52:0)
[01:53:50.424] Build status: STARTED
[01:53:50.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:53:0) after 8m6s
[02:01:56.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:53:0)
[02:01:56.570] Build status: STARTED
[02:01:56.570] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:54:0) after 7m24s
[02:09:20.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:54:0)
[02:09:20.577] Build status: STARTED
[02:09:20.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:55:0) after 1m12s
[02:10:32.743] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:55:0)
[02:10:32.824] Build status: STARTED
[02:10:32.824] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8850573948482979968:56:0) after 9m28s
[02:18:34.746] Received PubSub notification, asking Buildbucket for the build status
[02:18:34.768] Build:
{
"id": "8685143886142548065",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2409,
"createdBy": "project:chromium",
"createTime": "2026-04-07T21:46:03.701048390Z",
"startTime": "2026-04-07T21:46:43.224864070Z",
"endTime": "2026-04-08T02:18:34.436912202Z",
"updateTime": "2026-04-08T02:18:34.436912202Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a74e233110e33fef97dfa5cd1e78f4c74786d029",
"ref": "refs/heads/main"
}
},
"canary": true
}
[02:18:34.768] Invocation finished in 4h32m32.68292869s with status FAILED