[01:21:33.256] New invocation is queued and will start shortly
[01:21:35.216] Starting the invocation (attempt 1)
[01:21:35.248] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/d6fb2ac71ba993553eecb43bb5d1161ff1364fcf"
[01:21:35.248] Popped gitiles commit info from properties and tags
[01:21:35.248] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:21:35.249] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:21:35.249] Buildbucket request:
{
"requestId": "8851194567942626080",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d6fb2ac71ba993553eecb43bb5d1161ff1364fcf",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8851194567942626080"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacArm64PGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:21:35.920] Scheduled build:
{
"id": "8685764505139205025",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2072,
"createdBy": "project:chromium",
"createTime": "2026-04-01T01:21:35.303125429Z",
"updateTime": "2026-04-01T01:21:35.303125429Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d6fb2ac71ba993553eecb43bb5d1161ff1364fcf",
"ref": "refs/heads/main"
}
}
}
[01:21:35.920] Task URL: https://cr-buildbucket.appspot.com/build/8685764505139205025
[01:21:35.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:2:0) after 6m3s
[01:21:36.818] Received PubSub notification, asking Buildbucket for the build status
[01:21:36.837] Build status: SCHEDULED
[01:22:25.339] Received PubSub notification, asking Buildbucket for the build status
[01:22:25.364] Build status: STARTED
[01:27:38.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:2:0)
[01:27:38.956] Build status: STARTED
[01:27:38.956] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:5:0) after 4m5s
[01:31:43.980] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:5:0)
[01:31:43.997] Build status: STARTED
[01:31:43.997] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:6:0) after 8m18s
[01:40:02.007] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:6:0)
[01:40:02.026] Build status: STARTED
[01:40:02.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:7:0) after 4m4s
[01:44:06.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:7:0)
[01:44:06.127] Build status: STARTED
[01:44:06.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:8:0) after 5m51s
[01:49:57.227] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:8:0)
[01:49:57.257] Build status: STARTED
[01:49:57.257] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:9:0) after 2m12s
[01:52:09.334] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:9:0)
[01:52:09.354] Build status: STARTED
[01:52:09.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:10:0) after 8m12s
[02:00:21.476] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:10:0)
[02:00:21.493] Build status: STARTED
[02:00:21.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:11:0) after 1m33s
[02:01:54.511] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:11:0)
[02:01:54.561] Build status: STARTED
[02:01:54.561] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:12:0) after 4m37s
[02:06:31.626] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:12:0)
[02:06:31.664] Build status: STARTED
[02:06:31.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:13:0) after 5m8s
[02:11:39.707] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:13:0)
[02:11:39.734] Build status: STARTED
[02:11:39.734] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:14:0) after 8m0s
[02:19:39.801] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:14:0)
[02:19:39.819] Build status: STARTED
[02:19:39.819] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:15:0) after 9m51s
[02:29:30.842] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:15:0)
[02:29:30.862] Build status: STARTED
[02:29:30.862] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:16:0) after 7m22s
[02:36:52.882] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:16:0)
[02:36:52.902] Build status: STARTED
[02:36:52.902] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:17:0) after 2m19s
[02:39:11.924] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:17:0)
[02:39:11.944] Build status: STARTED
[02:39:11.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:18:0) after 2m51s
[02:42:03.009] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:18:0)
[02:42:03.026] Build status: STARTED
[02:42:03.026] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:19:0) after 7m9s
[02:49:12.087] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:19:0)
[02:49:12.165] Build status: STARTED
[02:49:12.166] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:20:0) after 5m5s
[02:54:17.197] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:20:0)
[02:54:17.249] Build status: STARTED
[02:54:17.249] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:21:0) after 9m46s
[03:04:03.310] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:21:0)
[03:04:03.354] Build status: STARTED
[03:04:03.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:22:0) after 5m50s
[03:09:53.415] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:22:0)
[03:09:53.430] Build status: STARTED
[03:09:53.430] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:23:0) after 9m56s
[03:19:49.475] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:23:0)
[03:19:49.493] Build status: STARTED
[03:19:49.493] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:24:0) after 9m3s
[03:28:52.564] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:24:0)
[03:28:52.581] Build status: STARTED
[03:28:52.581] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:25:0) after 9m1s
[03:37:53.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:25:0)
[03:37:53.644] Build status: STARTED
[03:37:53.644] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:26:0) after 7m39s
[03:45:32.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:26:0)
[03:45:32.751] Build status: STARTED
[03:45:32.751] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:27:0) after 6m3s
[03:51:35.806] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:27:0)
[03:51:35.825] Build status: STARTED
[03:51:35.825] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:28:0) after 5m56s
[03:57:31.846] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:28:0)
[03:57:31.864] Build status: STARTED
[03:57:31.864] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:29:0) after 5m41s
[04:03:12.998] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:29:0)
[04:03:13.036] Build status: STARTED
[04:03:13.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:30:0) after 6m27s
[04:09:40.059] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:30:0)
[04:09:40.078] Build status: STARTED
[04:09:40.078] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:31:0) after 4m1s
[04:13:41.127] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:31:0)
[04:13:41.153] Build status: STARTED
[04:13:41.153] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:32:0) after 9m33s
[04:23:14.176] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:32:0)
[04:23:14.192] Build status: STARTED
[04:23:14.192] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:33:0) after 9m56s
[04:33:10.216] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:33:0)
[04:33:10.231] Build status: STARTED
[04:33:10.231] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:34:0) after 9m57s
[04:43:07.255] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:34:0)
[04:43:07.273] Build status: STARTED
[04:43:07.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:35:0) after 8m34s
[04:51:41.294] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:35:0)
[04:51:41.311] Build status: STARTED
[04:51:41.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:36:0) after 5m47s
[04:57:28.431] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:36:0)
[04:57:28.448] Build status: STARTED
[04:57:28.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:37:0) after 3m14s
[05:00:42.473] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:37:0)
[05:00:42.528] Build status: STARTED
[05:00:42.529] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:38:0) after 9m52s
[05:10:34.717] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:38:0)
[05:10:34.737] Build status: STARTED
[05:10:34.737] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:39:0) after 8m52s
[05:19:26.817] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:39:0)
[05:19:26.840] Build status: STARTED
[05:19:26.840] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:40:0) after 4m32s
[05:23:58.956] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:40:0)
[05:23:58.974] Build status: STARTED
[05:23:58.974] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:41:0) after 6m29s
[05:30:27.994] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:41:0)
[05:30:28.029] Build status: STARTED
[05:30:28.029] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:42:0) after 9m14s
[05:39:42.101] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:42:0)
[05:39:42.120] Build status: STARTED
[05:39:42.120] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:43:0) after 9m45s
[05:49:27.245] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:43:0)
[05:49:27.269] Build status: STARTED
[05:49:27.270] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:44:0) after 1m29s
[05:50:56.327] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:44:0)
[05:50:56.347] Build status: STARTED
[05:50:56.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:45:0) after 6m53s
[05:57:49.381] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:45:0)
[05:57:49.402] Build status: STARTED
[05:57:49.402] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:46:0) after 9m16s
[06:07:05.425] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:46:0)
[06:07:05.444] Build status: STARTED
[06:07:05.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:47:0) after 3m42s
[06:10:47.506] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:47:0)
[06:10:47.603] Build status: STARTED
[06:10:47.603] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:48:0) after 4m52s
[06:15:39.654] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:48:0)
[06:15:39.671] Build status: STARTED
[06:15:39.671] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:49:0) after 7m20s
[06:22:59.695] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:49:0)
[06:22:59.712] Build status: STARTED
[06:22:59.712] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:50:0) after 7m53s
[06:30:52.735] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:50:0)
[06:30:52.755] Build status: STARTED
[06:30:52.759] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:51:0) after 2m20s
[06:33:12.791] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:51:0)
[06:33:12.813] Build status: STARTED
[06:33:12.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:52:0) after 8m32s
[06:41:44.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:52:0)
[06:41:44.805] Build status: STARTED
[06:41:44.805] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:53:0) after 3m49s
[06:45:33.827] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:53:0)
[06:45:33.851] Build status: STARTED
[06:45:33.851] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:54:0) after 9m56s
[06:55:30.062] Handling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:54:0)
[06:55:30.122] Build status: STARTED
[06:55:30.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacArm64PGO:8851194567942626080:55:0) after 5m8s
[06:59:06.366] Received PubSub notification, asking Buildbucket for the build status
[06:59:06.381] Build:
{
"id": "8685764505139205025",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacArm64PGO"
},
"number": 2072,
"createdBy": "project:chromium",
"createTime": "2026-04-01T01:21:35.303125429Z",
"startTime": "2026-04-01T01:22:25.156605090Z",
"endTime": "2026-04-01T06:58:56.737025285Z",
"updateTime": "2026-04-01T06:58:56.737025285Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "d6fb2ac71ba993553eecb43bb5d1161ff1364fcf",
"ref": "refs/heads/main"
}
}
}
[06:59:06.381] Invocation finished in 5h37m33.135501791s with status FAILED