[23:58:24.127] New invocation is queued and will start shortly
[23:58:25.162] Starting the invocation (attempt 1)
[23:58:25.190] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/7512f9a47d4c8ea70290807dbfa50aac3a8a3f3e"
[23:58:25.190] Popped gitiles commit info from properties and tags
[23:58:25.190] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[23:58:25.193] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[23:58:25.194] Buildbucket request:
{
"requestId": "8851743381220943568",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7512f9a47d4c8ea70290807dbfa50aac3a8a3f3e",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851743381220943568"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[23:58:25.921] Scheduled build:
{
"id": "8686313319340326465",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2005,
"createdBy": "project:chromium",
"createTime": "2026-03-25T23:58:25.295556189Z",
"updateTime": "2026-03-25T23:58:25.295556189Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7512f9a47d4c8ea70290807dbfa50aac3a8a3f3e",
"ref": "refs/heads/main"
}
}
}
[23:58:25.921] Task URL: https://cr-buildbucket.appspot.com/build/8686313319340326465
[23:58:25.922] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:2:0) after 3m16s
[23:58:26.809] Received PubSub notification, asking Buildbucket for the build status
[23:58:26.831] Build status: SCHEDULED
[23:59:15.626] Received PubSub notification, asking Buildbucket for the build status
[23:59:15.673] Build status: STARTED
[00:01:41.968] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:2:0)
[00:01:42.032] Build status: STARTED
[00:01:42.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:5:0) after 4m6s
[00:05:48.060] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:5:0)
[00:05:48.087] Build status: STARTED
[00:05:48.108] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:6:0) after 3m24s
[00:09:12.130] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:6:0)
[00:09:12.181] Build status: STARTED
[00:09:12.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:7:0) after 9m55s
[00:19:07.205] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:7:0)
[00:19:07.297] Build status: STARTED
[00:19:07.297] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:8:0) after 4m0s
[00:23:07.320] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:8:0)
[00:23:07.342] Build status: STARTED
[00:23:07.342] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:9:0) after 1m51s
[00:24:58.535] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:9:0)
[00:24:58.563] Build status: STARTED
[00:24:58.563] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:10:0) after 4m8s
[00:29:06.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:10:0)
[00:29:06.645] Build status: STARTED
[00:29:06.645] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:11:0) after 4m20s
[00:33:26.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:11:0)
[00:33:26.755] Build status: STARTED
[00:33:26.755] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:12:0) after 2m48s
[00:36:14.875] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:12:0)
[00:36:14.891] Build status: STARTED
[00:36:14.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:13:0) after 5m31s
[00:41:45.986] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:13:0)
[00:41:46.035] Build status: STARTED
[00:41:46.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:14:0) after 5m21s
[00:47:07.061] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:14:0)
[00:47:07.086] Build status: STARTED
[00:47:07.086] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:15:0) after 6m35s
[00:53:42.207] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:15:0)
[00:53:42.295] Build status: STARTED
[00:53:42.295] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:16:0) after 2m24s
[00:56:06.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:16:0)
[00:56:06.338] Build status: STARTED
[00:56:06.338] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:17:0) after 1m47s
[00:57:53.396] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:17:0)
[00:57:53.411] Build status: STARTED
[00:57:53.411] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:18:0) after 8m0s
[01:05:53.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:18:0)
[01:05:53.579] Build status: STARTED
[01:05:53.579] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:19:0) after 3m27s
[01:09:20.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:19:0)
[01:09:20.707] Build status: STARTED
[01:09:20.707] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:20:0) after 3m22s
[01:12:42.738] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:20:0)
[01:12:42.779] Build status: STARTED
[01:12:42.779] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:21:0) after 3m30s
[01:16:12.978] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:21:0)
[01:16:12.995] Build status: STARTED
[01:16:12.995] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:22:0) after 5m17s
[01:21:30.027] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:22:0)
[01:21:30.045] Build status: STARTED
[01:21:30.045] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:23:0) after 8m28s
[01:29:58.050] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:23:0)
[01:29:58.068] Build status: STARTED
[01:29:58.068] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:24:0) after 6m48s
[01:36:46.155] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:24:0)
[01:36:46.231] Build status: STARTED
[01:36:46.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:25:0) after 8m40s
[01:45:26.462] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:25:0)
[01:45:26.478] Build status: STARTED
[01:45:26.478] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:26:0) after 1m31s
[01:46:57.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:26:0)
[01:46:57.569] Build status: STARTED
[01:46:57.569] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:27:0) after 2m49s
[01:49:46.645] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:27:0)
[01:49:46.659] Build status: STARTED
[01:49:46.659] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:28:0) after 9m50s
[01:59:36.740] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:28:0)
[01:59:36.909] Build status: STARTED
[01:59:36.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:29:0) after 6m51s
[02:06:27.931] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:29:0)
[02:06:27.948] Build status: STARTED
[02:06:27.948] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:30:0) after 2m25s
[02:08:52.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:30:0)
[02:08:52.958] Build status: STARTED
[02:08:52.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:31:0) after 8m41s
[02:17:34.026] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:31:0)
[02:17:34.093] Build status: STARTED
[02:17:34.093] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:32:0) after 2m42s
[02:20:16.132] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:32:0)
[02:20:16.165] Build status: STARTED
[02:20:16.165] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:33:0) after 2m23s
[02:22:39.404] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:33:0)
[02:22:39.486] Build status: STARTED
[02:22:39.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:34:0) after 9m53s
[02:32:32.529] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:34:0)
[02:32:32.571] Build status: STARTED
[02:32:32.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:35:0) after 2m40s
[02:35:12.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:35:0)
[02:35:12.566] Build status: STARTED
[02:35:12.566] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:36:0) after 7m27s
[02:42:39.661] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:36:0)
[02:42:39.679] Build status: STARTED
[02:42:39.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:37:0) after 9m43s
[02:52:22.699] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:37:0)
[02:52:22.726] Build status: STARTED
[02:52:22.726] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:38:0) after 8m47s
[03:01:09.742] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:38:0)
[03:01:09.804] Build status: STARTED
[03:01:09.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:39:0) after 8m16s
[03:09:25.828] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:39:0)
[03:09:25.899] Build status: STARTED
[03:09:25.899] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:40:0) after 1m44s
[03:11:09.975] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:40:0)
[03:11:09.988] Build status: STARTED
[03:11:09.988] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:41:0) after 2m18s
[03:13:28.015] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:41:0)
[03:13:28.034] Build status: STARTED
[03:13:28.035] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:42:0) after 5m5s
[03:18:33.072] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:42:0)
[03:18:33.153] Build status: STARTED
[03:18:33.153] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:43:0) after 9m43s
[03:28:16.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:43:0)
[03:28:16.243] Build status: STARTED
[03:28:16.243] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:44:0) after 9m6s
[03:37:22.334] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:44:0)
[03:37:22.350] Build status: STARTED
[03:37:22.350] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:45:0) after 8m28s
[03:45:50.414] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:45:0)
[03:45:50.444] Build status: STARTED
[03:45:50.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:46:0) after 5m39s
[03:51:29.499] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:46:0)
[03:51:29.524] Build status: STARTED
[03:51:29.524] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:47:0) after 8m18s
[03:59:47.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:47:0)
[03:59:47.575] Build status: STARTED
[03:59:47.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:48:0) after 8m52s
[04:08:39.633] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:48:0)
[04:08:39.673] Build status: STARTED
[04:08:39.673] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:49:0) after 4m29s
[04:13:08.744] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:49:0)
[04:13:08.761] Build status: STARTED
[04:13:08.761] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:50:0) after 5m19s
[04:18:27.786] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:50:0)
[04:18:27.800] Build status: STARTED
[04:18:27.800] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:51:0) after 7m47s
[04:26:14.827] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:51:0)
[04:26:14.929] Build status: STARTED
[04:26:14.929] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:52:0) after 1m39s
[04:27:54.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:52:0)
[04:27:54.173] Build status: STARTED
[04:27:54.173] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:53:0) after 2m50s
[04:30:44.264] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:53:0)
[04:30:44.287] Build status: STARTED
[04:30:44.287] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:54:0) after 2m29s
[04:33:13.658] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:54:0)
[04:33:13.674] Build status: STARTED
[04:33:13.675] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:55:0) after 9m51s
[04:43:04.810] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:55:0)
[04:43:04.895] Build status: STARTED
[04:43:04.895] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:56:0) after 8m33s
[04:51:37.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:56:0)
[04:51:37.994] Build status: STARTED
[04:51:37.994] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:57:0) after 6m3s
[04:57:41.048] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:57:0)
[04:57:41.064] Build status: STARTED
[04:57:41.065] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:58:0) after 1m21s
[04:59:02.088] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:58:0)
[04:59:02.108] Build status: STARTED
[04:59:02.108] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:59:0) after 1m6s
[05:00:08.178] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:59:0)
[05:00:08.195] Build status: STARTED
[05:00:08.195] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:60:0) after 7m20s
[05:07:28.216] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:60:0)
[05:07:28.272] Build status: STARTED
[05:07:28.272] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:61:0) after 9m15s
[05:16:43.293] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:61:0)
[05:16:43.312] Build status: STARTED
[05:16:43.312] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:62:0) after 6m58s
[05:23:41.389] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:62:0)
[05:23:41.480] Build status: STARTED
[05:23:41.481] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:63:0) after 1m39s
[05:25:20.541] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:63:0)
[05:25:20.557] Build status: STARTED
[05:25:20.557] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:64:0) after 3m5s
[05:28:25.615] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:64:0)
[05:28:25.638] Build status: STARTED
[05:28:25.638] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:65:0) after 4m52s
[05:33:17.665] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:65:0)
[05:33:17.683] Build status: STARTED
[05:33:17.683] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851743381220943568:66:0) after 4m27s
[05:35:13.329] Received PubSub notification, asking Buildbucket for the build status
[05:35:13.347] Build:
{
"id": "8686313319340326465",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2005,
"createdBy": "project:chromium",
"createTime": "2026-03-25T23:58:25.295556189Z",
"startTime": "2026-03-25T23:59:15.170199120Z",
"endTime": "2026-03-26T05:35:13.067206954Z",
"updateTime": "2026-03-26T05:35:13.067206954Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "7512f9a47d4c8ea70290807dbfa50aac3a8a3f3e",
"ref": "refs/heads/main"
}
}
}
[05:35:13.347] Invocation finished in 5h36m49.228198612s with status FAILED