[07:15:00.222] New invocation is queued and will start shortly
[07:15:01.300] Starting the invocation (attempt 1)
[07:15:01.325] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/5fc534825914eb3f1229c390db86f71ae66bbae8"
[07:15:01.325] Popped gitiles commit info from properties and tags
[07:15:01.325] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[07:15:01.325] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[07:15:01.325] Buildbucket request:
{
"requestId": "8842928006879360064",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5fc534825914eb3f1229c390db86f71ae66bbae8",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842928006879360064"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[07:15:01.863] Scheduled build:
{
"id": "8677497944993719057",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3565,
"createdBy": "project:chromium",
"createTime": "2026-07-01T07:15:01.399403465Z",
"updateTime": "2026-07-01T07:15:01.399403465Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5fc534825914eb3f1229c390db86f71ae66bbae8",
"ref": "refs/heads/main"
}
}
}
[07:15:01.863] Task URL: https://cr-buildbucket.appspot.com/build/8677497944993719057
[07:15:01.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:2:0) after 6m4s
[07:15:02.942] Received PubSub notification, asking Buildbucket for the build status
[07:15:03.004] Build status: SCHEDULED
[07:15:05.451] Received PubSub notification, asking Buildbucket for the build status
[07:15:05.500] Build status: STARTED
[07:21:05.896] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:2:0)
[07:21:05.926] Build status: STARTED
[07:21:05.926] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:5:0) after 1m9s
[07:22:14.950] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:5:0)
[07:22:14.981] Build status: STARTED
[07:22:14.981] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:6:0) after 9m21s
[07:31:36.014] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:6:0)
[07:31:36.032] Build status: STARTED
[07:31:36.032] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:7:0) after 7m55s
[07:39:31.050] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:7:0)
[07:39:31.079] Build status: STARTED
[07:39:31.079] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:8:0) after 2m51s
[07:42:22.129] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:8:0)
[07:42:22.155] Build status: STARTED
[07:42:22.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:9:0) after 7m5s
[07:49:27.214] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:9:0)
[07:49:27.233] Build status: STARTED
[07:49:27.233] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:10:0) after 2m11s
[07:51:38.253] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:10:0)
[07:51:38.282] Build status: STARTED
[07:51:38.283] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:11:0) after 2m5s
[07:53:43.324] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:11:0)
[07:53:43.347] Build status: STARTED
[07:53:43.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:12:0) after 1m31s
[07:55:14.406] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:12:0)
[07:55:14.429] Build status: STARTED
[07:55:14.429] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:13:0) after 7m38s
[08:02:52.457] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:13:0)
[08:02:52.483] Build status: STARTED
[08:02:52.483] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:14:0) after 3m40s
[08:06:32.521] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:14:0)
[08:06:32.541] Build status: STARTED
[08:06:32.541] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:15:0) after 1m38s
[08:08:10.577] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:15:0)
[08:08:10.597] Build status: STARTED
[08:08:10.598] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:16:0) after 6m15s
[08:14:25.622] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:16:0)
[08:14:25.649] Build status: STARTED
[08:14:25.649] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:17:0) after 6m56s
[08:21:21.680] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:17:0)
[08:21:21.714] Build status: STARTED
[08:21:21.714] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:18:0) after 6m9s
[08:27:30.754] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:18:0)
[08:27:30.786] Build status: STARTED
[08:27:30.787] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:19:0) after 5m27s
[08:32:57.793] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:19:0)
[08:32:57.813] Build status: STARTED
[08:32:57.813] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:20:0) after 5m17s
[08:38:14.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:20:0)
[08:38:15.137] Build status: STARTED
[08:38:15.137] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:21:0) after 3m53s
[08:42:08.282] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:21:0)
[08:42:08.298] Build status: STARTED
[08:42:08.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:22:0) after 6m1s
[08:48:09.317] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:22:0)
[08:48:09.338] Build status: STARTED
[08:48:09.338] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:23:0) after 6m42s
[08:54:51.361] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:23:0)
[08:54:51.429] Build status: STARTED
[08:54:51.429] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:24:0) after 4m26s
[08:59:17.479] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:24:0)
[08:59:17.499] Build status: STARTED
[08:59:17.499] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:25:0) after 9m20s
[09:08:37.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:25:0)
[09:08:37.596] Build status: STARTED
[09:08:37.596] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:26:0) after 7m4s
[09:15:41.594] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:26:0)
[09:15:41.629] Build status: STARTED
[09:15:41.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:27:0) after 1m2s
[09:16:43.644] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:27:0)
[09:16:43.669] Build status: STARTED
[09:16:43.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:28:0) after 5m2s
[09:21:45.661] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:28:0)
[09:21:45.706] Build status: STARTED
[09:21:45.706] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:29:0) after 1m42s
[09:23:27.828] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:29:0)
[09:23:27.909] Build status: STARTED
[09:23:27.909] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:30:0) after 8m23s
[09:31:50.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:30:0)
[09:31:50.965] Build status: STARTED
[09:31:50.965] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:31:0) after 3m22s
[09:35:12.973] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:31:0)
[09:35:12.999] Build status: STARTED
[09:35:12.999] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:32:0) after 8m44s
[09:43:57.020] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:32:0)
[09:43:57.037] Build status: STARTED
[09:43:57.037] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:33:0) after 9m7s
[09:53:04.052] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:33:0)
[09:53:04.069] Build status: STARTED
[09:53:04.069] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:34:0) after 8m34s
[10:01:38.093] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:34:0)
[10:01:38.108] Build status: STARTED
[10:01:38.108] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:35:0) after 4m41s
[10:06:19.159] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:35:0)
[10:06:19.181] Build status: STARTED
[10:06:19.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:36:0) after 6m17s
[10:12:36.318] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:36:0)
[10:12:36.364] Build status: STARTED
[10:12:36.364] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:37:0) after 6m10s
[10:18:46.414] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:37:0)
[10:18:46.432] Build status: STARTED
[10:18:46.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:38:0) after 4m50s
[10:23:36.456] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:38:0)
[10:23:36.491] Build status: STARTED
[10:23:36.492] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:39:0) after 6m20s
[10:29:56.559] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:39:0)
[10:29:56.581] Build status: STARTED
[10:29:56.581] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:40:0) after 5m44s
[10:35:40.611] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:40:0)
[10:35:40.629] Build status: STARTED
[10:35:40.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:41:0) after 6m12s
[10:41:52.646] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:41:0)
[10:41:52.664] Build status: STARTED
[10:41:52.664] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:42:0) after 5m19s
[10:47:11.733] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:42:0)
[10:47:11.752] Build status: STARTED
[10:47:11.752] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:43:0) after 9m4s
[10:56:15.772] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:43:0)
[10:56:15.791] Build status: STARTED
[10:56:15.791] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:44:0) after 8m1s
[11:04:16.818] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:44:0)
[11:04:16.879] Build status: STARTED
[11:04:16.879] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:45:0) after 4m13s
[11:08:29.907] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:45:0)
[11:08:29.925] Build status: STARTED
[11:08:29.925] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:46:0) after 5m22s
[11:13:51.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:46:0)
[11:13:51.970] Build status: STARTED
[11:13:51.970] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:47:0) after 3m0s
[11:16:51.988] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:47:0)
[11:16:52.008] Build status: STARTED
[11:16:52.008] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:48:0) after 5m11s
[11:22:03.126] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:48:0)
[11:22:03.157] Build status: STARTED
[11:22:03.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:49:0) after 4m22s
[11:26:25.202] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:49:0)
[11:26:25.226] Build status: STARTED
[11:26:25.226] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:50:0) after 6m48s
[11:35:47.864] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:50:0)
[11:35:47.890] Build status: STARTED
[11:35:47.890] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:51:0) after 4m7s
[11:39:54.897] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:51:0)
[11:39:55.015] Build status: STARTED
[11:39:55.015] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:52:0) after 3m19s
[11:43:14.035] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:52:0)
[11:43:14.049] Build status: STARTED
[11:43:14.049] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:53:0) after 6m16s
[11:49:30.071] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:53:0)
[11:49:30.088] Build status: STARTED
[11:49:30.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:54:0) after 3m22s
[11:52:52.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:54:0)
[11:52:52.205] Build status: STARTED
[11:52:52.205] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:55:0) after 3m2s
[11:55:54.235] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:55:0)
[11:55:54.271] Build status: STARTED
[11:55:54.271] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:56:0) after 1m34s
[11:57:28.291] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:56:0)
[11:57:28.307] Build status: STARTED
[11:57:28.307] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:57:0) after 3m44s
[12:01:12.330] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:57:0)
[12:01:12.350] Build status: STARTED
[12:01:12.350] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:58:0) after 6m48s
[12:08:00.375] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:58:0)
[12:08:00.404] Build status: STARTED
[12:08:00.404] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842928006879360064:59:0) after 3m17s
[12:10:56.962] Received PubSub notification, asking Buildbucket for the build status
[12:10:57.021] Build:
{
"id": "8677497944993719057",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3565,
"createdBy": "project:chromium",
"createTime": "2026-07-01T07:15:01.399403465Z",
"startTime": "2026-07-01T07:15:05.286944713Z",
"endTime": "2026-07-01T12:10:56.699587744Z",
"updateTime": "2026-07-01T12:10:56.699587744Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "5fc534825914eb3f1229c390db86f71ae66bbae8",
"ref": "refs/heads/main"
}
}
}
[12:10:57.021] Invocation finished in 4h55m56.804085666s with status FAILED