[21:33:27.677] New invocation is queued and will start shortly
[21:33:28.966] Starting the invocation (attempt 1)
[21:33:29.000] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/f1e84d4f02cd7ad2df1ee23499e3788d6fc2d1dc"
[21:33:29.000] Popped gitiles commit info from properties and tags
[21:33:29.000] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[21:33:29.000] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[21:33:29.000] Buildbucket request:
{
"requestId": "8851571306179479456",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f1e84d4f02cd7ad2df1ee23499e3788d6fc2d1dc",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851571306179479456"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[21:33:29.648] Scheduled build:
{
"id": "8686141244054604129",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2353,
"createdBy": "project:chromium",
"createTime": "2026-03-27T21:33:29.076830690Z",
"updateTime": "2026-03-27T21:33:29.076830690Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f1e84d4f02cd7ad2df1ee23499e3788d6fc2d1dc",
"ref": "refs/heads/main"
}
}
}
[21:33:29.648] Task URL: https://cr-buildbucket.appspot.com/build/8686141244054604129
[21:33:29.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:2:0) after 9m38s
[21:33:29.880] Received PubSub notification, asking Buildbucket for the build status
[21:33:29.923] Build status: SCHEDULED
[21:34:14.633] Received PubSub notification, asking Buildbucket for the build status
[21:34:14.647] Build status: STARTED
[21:43:07.667] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:2:0)
[21:43:07.698] Build status: STARTED
[21:43:07.698] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:5:0) after 6m59s
[21:50:06.766] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:5:0)
[21:50:06.831] Build status: STARTED
[21:50:06.831] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:6:0) after 8m30s
[21:58:36.853] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:6:0)
[21:58:36.892] Build status: STARTED
[21:58:36.892] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:7:0) after 3m49s
[22:02:25.927] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:7:0)
[22:02:26.031] Build status: STARTED
[22:02:26.031] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:8:0) after 7m36s
[22:10:02.098] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:8:0)
[22:10:02.113] Build status: STARTED
[22:10:02.113] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:9:0) after 1m48s
[22:11:50.239] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:9:0)
[22:11:50.257] Build status: STARTED
[22:11:50.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:10:0) after 3m16s
[22:15:06.278] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:10:0)
[22:15:06.303] Build status: STARTED
[22:15:06.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:11:0) after 5m44s
[22:20:50.351] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:11:0)
[22:20:50.371] Build status: STARTED
[22:20:50.371] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:12:0) after 8m46s
[22:29:36.393] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:12:0)
[22:29:36.444] Build status: STARTED
[22:29:36.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:13:0) after 7m55s
[22:37:31.499] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:13:0)
[22:37:31.516] Build status: STARTED
[22:37:31.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:14:0) after 9m30s
[22:47:01.584] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:14:0)
[22:47:01.599] Build status: STARTED
[22:47:01.599] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:15:0) after 6m16s
[22:53:17.621] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:15:0)
[22:53:17.715] Build status: STARTED
[22:53:17.715] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:16:0) after 7m45s
[23:01:02.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:16:0)
[23:01:02.978] Build status: STARTED
[23:01:02.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:17:0) after 2m53s
[23:03:56.133] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:17:0)
[23:03:56.154] Build status: STARTED
[23:03:56.154] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:18:0) after 2m5s
[23:06:01.244] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:18:0)
[23:06:01.270] Build status: STARTED
[23:06:01.271] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:19:0) after 9m57s
[23:15:58.345] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:19:0)
[23:15:58.370] Build status: STARTED
[23:15:58.370] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:20:0) after 5m24s
[23:21:22.439] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:20:0)
[23:21:22.454] Build status: STARTED
[23:21:22.454] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:21:0) after 9m9s
[23:30:31.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:21:0)
[23:30:31.547] Build status: STARTED
[23:30:31.547] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:22:0) after 5m13s
[23:35:44.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:22:0)
[23:35:44.706] Build status: STARTED
[23:35:44.706] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:23:0) after 3m24s
[23:39:08.840] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:23:0)
[23:39:09.620] Build status: STARTED
[23:39:09.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:24:0) after 8m18s
[23:47:27.602] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:24:0)
[23:47:27.625] Build status: STARTED
[23:47:27.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:25:0) after 6m58s
[23:54:25.668] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:25:0)
[23:54:25.785] Build status: STARTED
[23:54:25.785] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:26:0) after 9m29s
[00:03:54.806] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:26:0)
[00:03:54.844] Build status: STARTED
[00:03:54.844] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:27:0) after 7m21s
[00:11:15.869] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:27:0)
[00:11:15.938] Build status: STARTED
[00:11:15.938] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:28:0) after 9m17s
[00:20:32.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:28:0)
[00:20:32.985] Build status: STARTED
[00:20:32.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:29:0) after 5m56s
[00:26:29.057] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:29:0)
[00:26:29.088] Build status: STARTED
[00:26:29.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:30:0) after 9m22s
[00:35:51.109] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:30:0)
[00:35:51.134] Build status: STARTED
[00:35:51.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:31:0) after 9m25s
[00:45:16.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:31:0)
[00:45:16.359] Build status: STARTED
[00:45:16.359] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:32:0) after 3m54s
[00:49:10.408] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:32:0)
[00:49:10.423] Build status: STARTED
[00:49:10.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:33:0) after 1m57s
[00:51:07.471] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:33:0)
[00:51:07.493] Build status: STARTED
[00:51:07.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:34:0) after 5m53s
[00:57:00.585] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:34:0)
[00:57:00.609] Build status: STARTED
[00:57:00.609] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:35:0) after 6m27s
[01:03:27.728] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:35:0)
[01:03:27.744] Build status: STARTED
[01:03:27.744] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:36:0) after 2m14s
[01:05:41.768] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:36:0)
[01:05:41.826] Build status: STARTED
[01:05:41.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:37:0) after 7m58s
[01:13:39.952] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:37:0)
[01:13:40.002] Build status: STARTED
[01:13:40.002] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:38:0) after 6m6s
[01:19:45.983] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:38:0)
[01:19:46.002] Build status: STARTED
[01:19:46.002] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:39:0) after 2m36s
[01:22:22.040] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:39:0)
[01:22:22.068] Build status: STARTED
[01:22:22.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:40:0) after 1m32s
[01:23:54.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:40:0)
[01:23:54.106] Build status: STARTED
[01:23:54.106] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:41:0) after 6m37s
[01:30:31.178] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:41:0)
[01:30:31.260] Build status: STARTED
[01:30:31.260] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:42:0) after 3m0s
[01:33:31.374] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:42:0)
[01:33:31.391] Build status: STARTED
[01:33:31.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:43:0) after 9m58s
[01:43:29.457] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:43:0)
[01:43:29.482] Build status: STARTED
[01:43:29.482] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:44:0) after 9m23s
[01:52:52.543] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:44:0)
[01:52:52.560] Build status: STARTED
[01:52:52.560] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:45:0) after 2m42s
[01:55:34.588] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:45:0)
[01:55:34.606] Build status: STARTED
[01:55:34.606] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:46:0) after 9m41s
[02:05:15.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:46:0)
[02:05:15.647] Build status: STARTED
[02:05:15.647] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:47:0) after 4m20s
[02:09:35.667] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:47:0)
[02:09:35.705] Build status: STARTED
[02:09:35.705] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:48:0) after 5m33s
[02:15:08.741] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:48:0)
[02:15:08.770] Build status: STARTED
[02:15:08.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64:8851571306179479456:49:0) after 2m40s
[02:15:26.769] Received PubSub notification, asking Buildbucket for the build status
[02:15:26.791] Build:
{
"id": "8686141244054604129",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64"
},
"number": 2353,
"createdBy": "project:chromium",
"createTime": "2026-03-27T21:33:29.076830690Z",
"startTime": "2026-03-27T21:34:14.448215926Z",
"endTime": "2026-03-28T02:15:25.942012201Z",
"updateTime": "2026-03-28T02:15:25.942012201Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "f1e84d4f02cd7ad2df1ee23499e3788d6fc2d1dc",
"ref": "refs/heads/main"
}
}
}
[02:15:26.791] Invocation finished in 4h41m59.125900396s with status FAILED