[12:56:26.698] New invocation is queued and will start shortly
[12:56:28.288] Starting the invocation (attempt 1)
[12:56:28.318] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/5bf589c03b6134d8f5ee406628106f46a2586af8"
[12:56:28.318] Popped gitiles commit info from properties and tags
[12:56:28.318] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[12:56:28.318] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[12:56:28.319] Buildbucket request:
{
"requestId": "8842815928285923664",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5bf589c03b6134d8f5ee406628106f46a2586af8",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842815928285923664"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[12:56:28.850] Scheduled build:
{
"id": "8677385865853769793",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3571,
"createdBy": "project:chromium",
"createTime": "2026-07-02T12:56:28.396128339Z",
"updateTime": "2026-07-02T12:56:28.396128339Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5bf589c03b6134d8f5ee406628106f46a2586af8",
"ref": "refs/heads/main"
}
}
}
[12:56:28.850] Task URL: https://cr-buildbucket.appspot.com/build/8677385865853769793
[12:56:28.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:2:0) after 5m23s
[12:56:29.795] Received PubSub notification, asking Buildbucket for the build status
[12:56:29.811] Build status: SCHEDULED
[12:56:32.306] Received PubSub notification, asking Buildbucket for the build status
[12:56:32.326] Build status: STARTED
[13:01:51.867] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:2:0)
[13:01:51.885] Build status: STARTED
[13:01:51.885] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:5:0) after 4m35s
[13:06:27.045] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:5:0)
[13:06:27.169] Build status: STARTED
[13:06:27.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:6:0) after 9m36s
[13:16:03.283] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:6:0)
[13:16:03.307] Build status: STARTED
[13:16:03.308] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:7:0) after 1m8s
[13:17:11.326] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:7:0)
[13:17:11.343] Build status: STARTED
[13:17:11.343] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:8:0) after 1m43s
[13:18:54.351] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:8:0)
[13:18:54.373] Build status: STARTED
[13:18:54.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:9:0) after 7m10s
[13:26:04.395] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:9:0)
[13:26:04.419] Build status: STARTED
[13:26:04.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:10:0) after 9m51s
[13:35:55.487] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:10:0)
[13:35:55.508] Build status: STARTED
[13:35:55.508] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:11:0) after 5m17s
[13:41:12.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:11:0)
[13:41:12.496] Build status: STARTED
[13:41:12.496] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:12:0) after 6m24s
[13:47:36.563] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:12:0)
[13:47:36.585] Build status: STARTED
[13:47:36.585] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:13:0) after 6m54s
[13:54:30.652] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:13:0)
[13:54:30.674] Build status: STARTED
[13:54:30.675] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:14:0) after 4m14s
[13:58:44.695] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:14:0)
[13:58:44.716] Build status: STARTED
[13:58:44.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:15:0) after 4m35s
[14:03:19.779] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:15:0)
[14:03:19.797] Build status: STARTED
[14:03:19.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:16:0) after 7m59s
[14:11:18.864] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:16:0)
[14:11:18.885] Build status: STARTED
[14:11:18.885] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:17:0) after 5m13s
[14:16:31.903] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:17:0)
[14:16:31.923] Build status: STARTED
[14:16:31.923] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:18:0) after 8m13s
[14:24:45.104] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:18:0)
[14:24:45.127] Build status: STARTED
[14:24:45.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:19:0) after 5m2s
[14:29:47.191] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:19:0)
[14:29:47.335] Build status: STARTED
[14:29:47.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:20:0) after 6m6s
[14:35:53.353] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:20:0)
[14:35:53.373] Build status: STARTED
[14:35:53.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:21:0) after 6m51s
[14:42:44.429] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:21:0)
[14:42:44.625] Build status: STARTED
[14:42:44.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:22:0) after 5m40s
[14:48:24.643] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:22:0)
[14:48:24.663] Build status: STARTED
[14:48:24.663] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:23:0) after 4m6s
[14:52:30.694] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:23:0)
[14:52:30.716] Build status: STARTED
[14:52:30.716] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:24:0) after 5m37s
[14:58:07.742] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:24:0)
[14:58:07.764] Build status: STARTED
[14:58:07.764] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:25:0) after 4m37s
[15:02:44.860] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:25:0)
[15:02:44.920] Build status: STARTED
[15:02:44.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:26:0) after 2m56s
[15:05:40.893] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:26:0)
[15:05:40.920] Build status: STARTED
[15:05:40.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:27:0) after 6m11s
[15:11:52.024] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:27:0)
[15:11:52.127] Build status: STARTED
[15:11:52.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:28:0) after 4m48s
[15:16:40.225] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:28:0)
[15:16:40.246] Build status: STARTED
[15:16:40.246] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:29:0) after 6m58s
[15:23:38.362] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:29:0)
[15:23:38.380] Build status: STARTED
[15:23:38.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:30:0) after 6m26s
[15:30:04.561] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:30:0)
[15:30:04.587] Build status: STARTED
[15:30:04.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:31:0) after 8m17s
[15:38:21.611] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:31:0)
[15:38:21.630] Build status: STARTED
[15:38:21.630] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:32:0) after 8m4s
[15:46:25.926] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:32:0)
[15:46:25.961] Build status: STARTED
[15:46:25.961] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:33:0) after 5m12s
[15:51:38.040] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:33:0)
[15:51:38.061] Build status: STARTED
[15:51:38.061] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:34:0) after 3m59s
[15:55:37.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:34:0)
[15:55:37.530] Build status: STARTED
[15:55:37.530] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:35:0) after 6m51s
[16:02:28.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:35:0)
[16:02:28.576] Build status: STARTED
[16:02:28.576] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:36:0) after 5m35s
[16:08:03.805] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:36:0)
[16:08:03.857] Build status: STARTED
[16:08:03.857] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:37:0) after 3m22s
[16:11:25.881] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:37:0)
[16:11:25.976] Build status: STARTED
[16:11:25.976] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:38:0) after 8m21s
[16:19:47.083] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:38:0)
[16:19:47.102] Build status: STARTED
[16:19:47.102] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:39:0) after 6m8s
[16:25:55.078] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:39:0)
[16:25:55.097] Build status: STARTED
[16:25:55.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:40:0) after 1m17s
[16:27:12.115] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:40:0)
[16:27:12.142] Build status: STARTED
[16:27:12.142] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:41:0) after 2m39s
[16:29:51.157] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:41:0)
[16:29:51.182] Build status: STARTED
[16:29:51.182] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:42:0) after 9m26s
[16:39:17.373] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:42:0)
[16:39:17.394] Build status: STARTED
[16:39:17.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:43:0) after 6m5s
[16:45:22.504] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:43:0)
[16:45:22.526] Build status: STARTED
[16:45:22.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:44:0) after 3m13s
[16:48:35.545] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:44:0)
[16:48:35.571] Build status: STARTED
[16:48:35.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:45:0) after 5m45s
[16:54:20.590] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:45:0)
[16:54:20.607] Build status: STARTED
[16:54:20.607] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:46:0) after 9m50s
[17:04:10.575] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:46:0)
[17:04:10.595] Build status: STARTED
[17:04:10.595] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:47:0) after 7m48s
[17:11:58.662] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:47:0)
[17:11:58.682] Build status: STARTED
[17:11:58.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:48:0) after 7m23s
[17:19:21.734] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:48:0)
[17:19:21.911] Build status: STARTED
[17:19:21.911] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:49:0) after 5m9s
[17:24:30.931] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:49:0)
[17:24:30.985] Build status: STARTED
[17:24:30.985] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:50:0) after 6m54s
[17:31:25.003] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:50:0)
[17:31:25.024] Build status: STARTED
[17:31:25.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:51:0) after 8m21s
[17:39:46.132] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:51:0)
[17:39:46.155] Build status: STARTED
[17:39:46.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:52:0) after 4m25s
[17:44:11.173] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:52:0)
[17:44:11.190] Build status: STARTED
[17:44:11.190] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:53:0) after 2m54s
[17:47:05.206] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:53:0)
[17:47:05.224] Build status: STARTED
[17:47:05.224] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:54:0) after 6m16s
[17:53:21.240] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:54:0)
[17:53:21.261] Build status: STARTED
[17:53:21.261] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:55:0) after 9m14s
[18:02:35.237] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:55:0)
[18:02:35.253] Build status: STARTED
[18:02:35.253] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:56:0) after 7m24s
[18:09:59.254] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:56:0)
[18:09:59.272] Build status: STARTED
[18:09:59.273] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:57:0) after 9m41s
[18:19:40.504] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:57:0)
[18:19:40.525] Build status: STARTED
[18:19:40.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:58:0) after 7m57s
[18:27:37.575] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:58:0)
[18:27:37.630] Build status: STARTED
[18:27:37.630] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:59:0) after 9m42s
[18:37:19.654] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:59:0)
[18:37:19.677] Build status: STARTED
[18:37:19.677] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:60:0) after 9m19s
[18:46:38.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:60:0)
[18:46:38.794] Build status: STARTED
[18:46:38.794] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:61:0) after 2m18s
[18:48:56.949] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:61:0)
[18:48:56.966] Build status: STARTED
[18:48:56.966] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:62:0) after 7m34s
[18:56:31.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:62:0)
[18:56:31.037] Build status: STARTED
[18:56:31.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:63:0) after 5m38s
[19:02:09.075] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:63:0)
[19:02:09.097] Build status: STARTED
[19:02:09.097] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842815928285923664:64:0) after 6m1s
[19:02:53.388] Received PubSub notification, asking Buildbucket for the build status
[19:02:53.409] Build:
{
"id": "8677385865853769793",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3571,
"createdBy": "project:chromium",
"createTime": "2026-07-02T12:56:28.396128339Z",
"startTime": "2026-07-02T12:56:32.129735904Z",
"endTime": "2026-07-02T19:02:52.733985732Z",
"updateTime": "2026-07-02T19:02:52.733985732Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5bf589c03b6134d8f5ee406628106f46a2586af8",
"ref": "refs/heads/main"
}
}
}
[19:02:53.409] Invocation finished in 6h6m26.716686636s with status FAILED