[19:53:30.670] New invocation is queued and will start shortly
[19:53:31.830] Starting the invocation (attempt 1)
[19:53:31.855] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/9207d9568b8f3eaab68540bf32b952e9f1dd9478"
[19:53:31.855] Popped gitiles commit info from properties and tags
[19:53:31.855] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:53:31.856] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:53:31.856] Buildbucket request:
{
"requestId": "8850943415732611488",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9207d9568b8f3eaab68540bf32b952e9f1dd9478",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8850943415732611488"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:53:32.550] Scheduled build:
{
"id": "8685513353761769393",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2082,
"createdBy": "project:chromium",
"createTime": "2026-04-03T19:53:31.921788466Z",
"updateTime": "2026-04-03T19:53:31.921788466Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9207d9568b8f3eaab68540bf32b952e9f1dd9478",
"ref": "refs/heads/main"
}
}
}
[19:53:32.550] Task URL: https://cr-buildbucket.appspot.com/build/8685513353761769393
[19:53:32.550] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:2:0) after 2m27s
[19:53:33.278] Received PubSub notification, asking Buildbucket for the build status
[19:53:33.295] Build status: SCHEDULED
[19:54:14.194] Received PubSub notification, asking Buildbucket for the build status
[19:54:14.273] Build status: STARTED
[19:55:59.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:2:0)
[19:55:59.595] Build status: STARTED
[19:55:59.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:5:0) after 6m49s
[20:02:48.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:5:0)
[20:02:48.745] Build status: STARTED
[20:02:48.746] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:6:0) after 3m25s
[20:06:13.763] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:6:0)
[20:06:13.804] Build status: STARTED
[20:06:13.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:7:0) after 7m8s
[20:13:21.878] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:7:0)
[20:13:21.926] Build status: STARTED
[20:13:21.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:8:0) after 8m55s
[20:22:16.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:8:0)
[20:22:16.983] Build status: STARTED
[20:22:16.983] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:9:0) after 4m49s
[20:27:06.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:9:0)
[20:27:06.126] Build status: STARTED
[20:27:06.126] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:10:0) after 9m9s
[20:36:15.281] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:10:0)
[20:36:15.309] Build status: STARTED
[20:36:15.309] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:11:0) after 9m29s
[20:45:44.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:11:0)
[20:45:44.328] Build status: STARTED
[20:45:44.328] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:12:0) after 3m48s
[20:49:32.345] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:12:0)
[20:49:32.369] Build status: STARTED
[20:49:32.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:13:0) after 1m24s
[20:50:56.446] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:13:0)
[20:50:56.466] Build status: STARTED
[20:50:56.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:14:0) after 7m2s
[20:57:58.559] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:14:0)
[20:57:58.576] Build status: STARTED
[20:57:58.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:15:0) after 7m51s
[21:05:49.570] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:15:0)
[21:05:49.587] Build status: STARTED
[21:05:49.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:16:0) after 7m52s
[21:13:41.590] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:16:0)
[21:13:41.613] Build status: STARTED
[21:13:41.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:17:0) after 8m27s
[21:22:08.614] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:17:0)
[21:22:08.671] Build status: STARTED
[21:22:08.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:18:0) after 3m9s
[21:25:17.785] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:18:0)
[21:25:17.802] Build status: STARTED
[21:25:17.802] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:19:0) after 2m0s
[21:27:17.819] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:19:0)
[21:27:17.870] Build status: STARTED
[21:27:17.870] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:20:0) after 6m36s
[21:33:53.936] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:20:0)
[21:33:53.951] Build status: STARTED
[21:33:53.951] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:21:0) after 8m47s
[21:42:40.982] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:21:0)
[21:42:41.006] Build status: STARTED
[21:42:41.007] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:22:0) after 3m44s
[21:46:25.119] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:22:0)
[21:46:25.138] Build status: STARTED
[21:46:25.138] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:23:0) after 4m11s
[21:50:36.190] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:23:0)
[21:50:36.204] Build status: STARTED
[21:50:36.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:24:0) after 6m16s
[21:56:52.224] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:24:0)
[21:56:52.291] Build status: STARTED
[21:56:52.291] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:25:0) after 2m12s
[21:59:04.394] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:25:0)
[21:59:04.424] Build status: STARTED
[21:59:04.424] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:26:0) after 8m29s
[22:07:33.600] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:26:0)
[22:07:33.689] Build status: STARTED
[22:07:33.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:27:0) after 8m48s
[22:16:21.755] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:27:0)
[22:16:21.809] Build status: STARTED
[22:16:21.809] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:28:0) after 2m57s
[22:19:18.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:28:0)
[22:19:18.814] Build status: STARTED
[22:19:18.814] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:29:0) after 9m14s
[22:28:32.899] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:29:0)
[22:28:32.917] Build status: STARTED
[22:28:32.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:30:0) after 4m13s
[22:32:45.933] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:30:0)
[22:32:45.959] Build status: STARTED
[22:32:45.959] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:31:0) after 3m21s
[22:36:07.025] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:31:0)
[22:36:07.046] Build status: STARTED
[22:36:07.046] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:32:0) after 7m36s
[22:43:43.140] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:32:0)
[22:43:43.167] Build status: STARTED
[22:43:43.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:33:0) after 1m59s
[22:45:42.282] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:33:0)
[22:45:42.303] Build status: STARTED
[22:45:42.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:34:0) after 9m13s
[22:54:55.396] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:34:0)
[22:54:55.416] Build status: STARTED
[22:54:55.416] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:35:0) after 6m41s
[23:01:36.458] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:35:0)
[23:01:36.475] Build status: STARTED
[23:01:36.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:36:0) after 7m29s
[23:09:05.555] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:36:0)
[23:09:05.624] Build status: STARTED
[23:09:05.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:37:0) after 1m14s
[23:10:19.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:37:0)
[23:10:19.639] Build status: STARTED
[23:10:19.639] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:38:0) after 4m24s
[23:14:43.736] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:38:0)
[23:14:43.751] Build status: STARTED
[23:14:43.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:39:0) after 5m31s
[23:20:14.769] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:39:0)
[23:20:14.798] Build status: STARTED
[23:20:14.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:40:0) after 5m23s
[23:25:37.795] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:40:0)
[23:25:37.811] Build status: STARTED
[23:25:37.811] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:41:0) after 8m17s
[23:33:54.971] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:41:0)
[23:33:54.997] Build status: STARTED
[23:33:54.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:42:0) after 4m3s
[23:37:58.094] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:42:0)
[23:37:58.122] Build status: STARTED
[23:37:58.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:43:0) after 7m10s
[23:45:08.182] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:43:0)
[23:45:08.205] Build status: STARTED
[23:45:08.205] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:44:0) after 2m40s
[23:47:48.227] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:44:0)
[23:47:48.244] Build status: STARTED
[23:47:48.244] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:45:0) after 3m36s
[23:51:24.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:45:0)
[23:51:24.273] Build status: STARTED
[23:51:24.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:46:0) after 9m50s
[00:01:14.391] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:46:0)
[00:01:14.410] Build status: STARTED
[00:01:14.410] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:47:0) after 6m42s
[00:07:56.429] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:47:0)
[00:07:56.446] Build status: STARTED
[00:07:56.446] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:48:0) after 7m44s
[00:15:40.466] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:48:0)
[00:15:40.482] Build status: STARTED
[00:15:40.482] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:49:0) after 8m26s
[00:24:06.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:49:0)
[00:24:06.554] Build status: STARTED
[00:24:06.554] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:50:0) after 9m47s
[00:33:53.578] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:50:0)
[00:33:53.594] Build status: STARTED
[00:33:53.594] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:51:0) after 3m40s
[00:37:33.797] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:51:0)
[00:37:34.264] Build status: STARTED
[00:37:34.264] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:52:0) after 4m18s
[00:41:52.336] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:52:0)
[00:41:52.353] Build status: STARTED
[00:41:52.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:53:0) after 3m38s
[00:45:30.372] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:53:0)
[00:45:30.400] Build status: STARTED
[00:45:30.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:54:0) after 5m23s
[00:50:53.420] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:54:0)
[00:50:53.436] Build status: STARTED
[00:50:53.436] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:55:0) after 2m42s
[00:53:35.466] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:55:0)
[00:53:35.665] Build status: STARTED
[00:53:35.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:56:0) after 2m51s
[00:56:26.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:56:0)
[00:56:26.676] Build status: STARTED
[00:56:26.676] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:57:0) after 9m14s
[01:05:40.695] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:57:0)
[01:05:40.720] Build status: STARTED
[01:05:40.720] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:58:0) after 3m7s
[01:08:47.834] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:58:0)
[01:08:47.851] Build status: STARTED
[01:08:47.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:59:0) after 5m47s
[01:14:34.925] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:59:0)
[01:14:34.950] Build status: STARTED
[01:14:34.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:60:0) after 4m36s
[01:19:10.973] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:60:0)
[01:19:10.998] Build status: STARTED
[01:19:10.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:61:0) after 8m34s
[01:27:44.984] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:61:0)
[01:27:44.998] Build status: STARTED
[01:27:44.998] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8850943415732611488:62:0) after 8m32s
[01:32:24.710] Received PubSub notification, asking Buildbucket for the build status
[01:32:24.730] Build:
{
"id": "8685513353761769393",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2082,
"createdBy": "project:chromium",
"createTime": "2026-04-03T19:53:31.921788466Z",
"startTime": "2026-04-03T19:54:13.897186492Z",
"endTime": "2026-04-04T01:32:24.365838901Z",
"updateTime": "2026-04-04T01:32:24.365838901Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "9207d9568b8f3eaab68540bf32b952e9f1dd9478",
"ref": "refs/heads/main"
}
}
}
[01:32:24.730] Invocation finished in 5h38m54.071617149s with status FAILED