[05:13:30.542] New invocation is queued and will start shortly
[05:13:31.869] Starting the invocation (attempt 1)
[05:13:31.893] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/ad9b403011b1891d0435c997cf49c4986814d8e4"
[05:13:31.893] Popped gitiles commit info from properties and tags
[05:13:31.893] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[05:13:31.893] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[05:13:31.893] Buildbucket request:
{
"requestId": "8863048177208681968",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ad9b403011b1891d0435c997cf49c4986814d8e4",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8863048177208681968"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[05:13:32.453] Scheduled build:
{
"id": "8697618115083281537",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 261,
"createdBy": "project:chromium",
"createTime": "2025-11-21T05:13:31.943539283Z",
"updateTime": "2025-11-21T05:13:31.943539283Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ad9b403011b1891d0435c997cf49c4986814d8e4",
"ref": "refs/heads/main"
}
}
}
[05:13:32.453] Task URL: https://cr-buildbucket.appspot.com/build/8697618115083281537
[05:13:32.453] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:2:0) after 8m42s
[05:13:33.493] Received PubSub notification, asking Buildbucket for the build status
[05:13:33.515] Build status: SCHEDULED
[05:13:43.941] Received PubSub notification, asking Buildbucket for the build status
[05:13:43.968] Build status: STARTED
[05:22:14.496] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:2:0)
[05:22:14.511] Build status: STARTED
[05:22:14.511] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:5:0) after 9m24s
[05:31:38.677] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:5:0)
[05:31:38.702] Build status: STARTED
[05:31:38.702] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:6:0) after 8m44s
[05:40:22.770] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:6:0)
[05:40:22.796] Build status: STARTED
[05:40:22.796] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:7:0) after 8m7s
[05:48:29.812] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:7:0)
[05:48:29.829] Build status: STARTED
[05:48:29.829] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:8:0) after 5m57s
[05:54:26.882] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:8:0)
[05:54:26.912] Build status: STARTED
[05:54:26.912] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:9:0) after 4m51s
[05:59:18.016] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:9:0)
[05:59:18.098] Build status: STARTED
[05:59:18.098] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:10:0) after 2m6s
[06:01:24.198] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:10:0)
[06:01:24.218] Build status: STARTED
[06:01:24.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:11:0) after 5m57s
[06:07:21.279] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:11:0)
[06:07:21.310] Build status: STARTED
[06:07:21.310] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:12:0) after 7m15s
[06:14:36.336] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:12:0)
[06:14:36.353] Build status: STARTED
[06:14:36.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:13:0) after 3m47s
[06:18:23.383] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:13:0)
[06:18:23.396] Build status: STARTED
[06:18:23.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:14:0) after 1m30s
[06:19:53.449] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:14:0)
[06:19:53.466] Build status: STARTED
[06:19:53.466] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:15:0) after 6m50s
[06:26:43.586] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:15:0)
[06:26:43.602] Build status: STARTED
[06:26:43.602] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:16:0) after 7m41s
[06:34:24.713] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:16:0)
[06:34:24.729] Build status: STARTED
[06:34:24.729] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:17:0) after 2m55s
[06:37:20.109] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:17:0)
[06:37:20.124] Build status: STARTED
[06:37:20.125] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:18:0) after 1m39s
[06:38:59.251] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:18:0)
[06:38:59.267] Build status: STARTED
[06:38:59.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:19:0) after 9m53s
[06:48:52.288] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:19:0)
[06:48:52.327] Build status: STARTED
[06:48:52.327] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:20:0) after 2m28s
[06:51:20.340] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:20:0)
[06:51:20.396] Build status: STARTED
[06:51:20.396] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:21:0) after 2m39s
[06:53:59.514] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:21:0)
[06:53:59.530] Build status: STARTED
[06:53:59.530] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:22:0) after 1m13s
[06:55:12.664] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:22:0)
[06:55:12.678] Build status: STARTED
[06:55:12.679] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:23:0) after 8m49s
[07:04:01.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:23:0)
[07:04:01.893] Build status: STARTED
[07:04:01.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:24:0) after 2m43s
[07:06:44.913] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:24:0)
[07:06:44.946] Build status: STARTED
[07:06:44.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:25:0) after 1m26s
[07:08:11.064] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:25:0)
[07:08:11.092] Build status: STARTED
[07:08:11.092] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:26:0) after 9m7s
[07:17:18.507] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:26:0)
[07:17:18.564] Build status: STARTED
[07:17:18.564] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:27:0) after 7m19s
[07:24:37.579] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:27:0)
[07:24:37.607] Build status: STARTED
[07:24:37.607] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:28:0) after 7m59s
[07:32:36.637] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:28:0)
[07:32:36.657] Build status: STARTED
[07:32:36.657] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:29:0) after 2m37s
[07:35:13.674] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:29:0)
[07:35:13.689] Build status: STARTED
[07:35:13.689] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:30:0) after 3m47s
[07:39:00.718] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:30:0)
[07:39:00.736] Build status: STARTED
[07:39:00.736] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:31:0) after 6m37s
[07:45:37.756] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:31:0)
[07:45:37.773] Build status: STARTED
[07:45:37.773] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:32:0) after 3m4s
[07:48:41.787] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:32:0)
[07:48:41.804] Build status: STARTED
[07:48:41.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:33:0) after 6m59s
[07:55:41.195] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:33:0)
[07:55:41.210] Build status: STARTED
[07:55:41.210] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:34:0) after 4m14s
[07:59:55.427] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:34:0)
[07:59:55.443] Build status: STARTED
[07:59:55.443] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:35:0) after 7m17s
[08:07:12.491] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:35:0)
[08:07:12.502] Build status: STARTED
[08:07:12.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:36:0) after 4m4s
[08:11:16.525] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:36:0)
[08:11:16.561] Build status: STARTED
[08:11:16.561] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:37:0) after 5m33s
[08:16:49.629] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:37:0)
[08:16:49.648] Build status: STARTED
[08:16:49.648] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:38:0) after 6m32s
[08:23:21.683] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:38:0)
[08:23:21.765] Build status: STARTED
[08:23:21.765] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:39:0) after 3m13s
[08:26:34.783] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:39:0)
[08:26:34.856] Build status: STARTED
[08:26:34.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:40:0) after 4m34s
[08:31:09.002] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:40:0)
[08:31:09.016] Build status: STARTED
[08:31:09.016] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:41:0) after 1m45s
[08:32:54.155] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:41:0)
[08:32:54.181] Build status: STARTED
[08:32:54.181] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:42:0) after 9m25s
[08:42:19.200] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:42:0)
[08:42:19.217] Build status: STARTED
[08:42:19.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:43:0) after 9m14s
[08:51:33.196] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:43:0)
[08:51:33.216] Build status: STARTED
[08:51:33.216] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:44:0) after 9m56s
[09:01:29.382] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:44:0)
[09:01:29.400] Build status: STARTED
[09:01:29.400] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:45:0) after 9m27s
[09:10:56.544] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:45:0)
[09:10:56.559] Build status: STARTED
[09:10:56.559] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:46:0) after 9m16s
[09:20:12.627] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:46:0)
[09:20:12.642] Build status: STARTED
[09:20:12.642] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:47:0) after 3m55s
[09:24:07.768] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:47:0)
[09:24:07.781] Build status: STARTED
[09:24:07.781] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:48:0) after 4m16s
[09:28:23.799] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:48:0)
[09:28:23.816] Build status: STARTED
[09:28:23.816] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:49:0) after 8m9s
[09:36:32.835] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:49:0)
[09:36:32.852] Build status: STARTED
[09:36:32.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:50:0) after 4m21s
[09:40:53.883] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:50:0)
[09:40:53.901] Build status: STARTED
[09:40:53.901] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:51:0) after 8m29s
[09:49:22.970] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:51:0)
[09:49:22.984] Build status: STARTED
[09:49:22.984] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:52:0) after 2m0s
[09:51:23.041] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:52:0)
[09:51:23.064] Build status: STARTED
[09:51:23.064] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:53:0) after 6m9s
[09:57:32.083] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:53:0)
[09:57:32.110] Build status: STARTED
[09:57:32.110] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:54:0) after 1m3s
[09:58:35.133] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:54:0)
[09:58:35.154] Build status: STARTED
[09:58:35.154] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:55:0) after 5m17s
[10:03:52.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:55:0)
[10:03:52.169] Build status: STARTED
[10:03:52.169] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:56:0) after 1m13s
[10:05:05.218] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:56:0)
[10:05:05.232] Build status: STARTED
[10:05:05.232] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:57:0) after 2m32s
[10:07:37.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:57:0)
[10:07:37.286] Build status: STARTED
[10:07:37.286] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:58:0) after 2m8s
[10:09:45.362] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:58:0)
[10:09:45.376] Build status: STARTED
[10:09:45.376] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:59:0) after 2m24s
[10:12:09.394] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:59:0)
[10:12:09.409] Build status: STARTED
[10:12:09.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:60:0) after 2m22s
[10:14:31.434] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:60:0)
[10:14:31.486] Build status: STARTED
[10:14:31.486] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:61:0) after 1m42s
[10:16:13.503] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:61:0)
[10:16:13.571] Build status: STARTED
[10:16:13.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:62:0) after 6m33s
[10:22:46.603] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:62:0)
[10:22:46.674] Build status: STARTED
[10:22:46.674] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:63:0) after 7m29s
[10:30:15.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:63:0)
[10:30:15.703] Build status: STARTED
[10:30:15.703] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8863048177208681968:64:0) after 3m6s
[10:33:01.791] Received PubSub notification, asking Buildbucket for the build status
[10:33:01.805] Build:
{
"id": "8697618115083281537",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 261,
"createdBy": "project:chromium",
"createTime": "2025-11-21T05:13:31.943539283Z",
"startTime": "2025-11-21T05:13:43.458965044Z",
"endTime": "2025-11-21T10:32:34.204527478Z",
"updateTime": "2025-11-21T10:32:34.204527478Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "ad9b403011b1891d0435c997cf49c4986814d8e4",
"ref": "refs/heads/main"
}
}
}
[10:33:01.805] Invocation finished in 5h19m31.273193347s with status FAILED