[10:31:21.009] New invocation is queued and will start shortly
[10:31:22.036] Starting the invocation (attempt 1)
[10:31:22.062] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/11382d95d14ce6a9676ae5079b1e2e358cedb905"
[10:31:22.062] Popped gitiles commit info from properties and tags
[10:31:22.062] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[10:31:22.063] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[10:31:22.063] Buildbucket request:
{
"requestId": "8846901920350347120",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "11382d95d14ce6a9676ae5079b1e2e358cedb905",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8846901920350347120"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[10:31:22.600] Scheduled build:
{
"id": "8681471858494682929",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2688,
"createdBy": "project:chromium",
"createTime": "2026-05-18T10:31:22.157957699Z",
"updateTime": "2026-05-18T10:31:22.157957699Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "11382d95d14ce6a9676ae5079b1e2e358cedb905",
"ref": "refs/heads/main"
}
}
}
[10:31:22.600] Task URL: https://cr-buildbucket.appspot.com/build/8681471858494682929
[10:31:22.600] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:2:0) after 9m20s
[10:32:21.789] Received PubSub notification, asking Buildbucket for the build status
[10:32:21.816] Build status: SCHEDULED
[10:33:46.284] Received PubSub notification, asking Buildbucket for the build status
[10:33:46.321] Build status: STARTED
[10:40:42.894] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:2:0)
[10:40:42.915] Build status: STARTED
[10:40:42.915] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:5:0) after 9m52s
[10:50:35.029] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:5:0)
[10:50:35.049] Build status: STARTED
[10:50:35.049] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:6:0) after 5m51s
[10:56:26.066] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:6:0)
[10:56:26.087] Build status: STARTED
[10:56:26.087] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:7:0) after 4m5s
[11:00:31.308] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:7:0)
[11:00:31.325] Build status: STARTED
[11:00:31.325] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:8:0) after 9m45s
[11:10:16.387] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:8:0)
[11:10:16.471] Build status: STARTED
[11:10:16.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:9:0) after 2m23s
[11:12:39.510] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:9:0)
[11:12:39.529] Build status: STARTED
[11:12:39.529] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:10:0) after 8m48s
[11:21:27.666] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:10:0)
[11:21:27.804] Build status: STARTED
[11:21:27.804] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:11:0) after 5m44s
[11:27:12.008] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:11:0)
[11:27:12.024] Build status: STARTED
[11:27:12.024] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:12:0) after 1m43s
[11:28:55.193] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:12:0)
[11:28:55.220] Build status: STARTED
[11:28:55.220] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:13:0) after 3m45s
[11:32:40.489] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:13:0)
[11:32:40.506] Build status: STARTED
[11:32:40.506] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:14:0) after 8m14s
[11:40:54.526] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:14:0)
[11:40:54.542] Build status: STARTED
[11:40:54.542] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:15:0) after 1m6s
[11:42:00.809] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:15:0)
[11:42:00.834] Build status: STARTED
[11:42:00.834] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:16:0) after 2m12s
[11:44:13.020] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:16:0)
[11:44:13.048] Build status: STARTED
[11:44:13.048] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:17:0) after 5m53s
[11:50:06.088] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:17:0)
[11:50:06.104] Build status: STARTED
[11:50:06.104] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:18:0) after 8m0s
[11:58:06.151] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:18:0)
[11:58:06.184] Build status: STARTED
[11:58:06.184] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:19:0) after 1m18s
[11:59:24.404] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:19:0)
[11:59:24.427] Build status: STARTED
[11:59:24.427] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:20:0) after 9m44s
[12:09:08.589] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:20:0)
[12:09:08.613] Build status: STARTED
[12:09:08.613] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:21:0) after 3m53s
[12:13:02.039] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:21:0)
[12:13:02.082] Build status: STARTED
[12:13:02.082] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:22:0) after 9m27s
[12:22:29.096] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:22:0)
[12:22:29.119] Build status: STARTED
[12:22:29.119] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:23:0) after 9m4s
[12:31:33.231] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:23:0)
[12:31:33.262] Build status: STARTED
[12:31:33.262] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:24:0) after 6m37s
[12:38:10.276] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:24:0)
[12:38:10.297] Build status: STARTED
[12:38:10.297] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:25:0) after 7m56s
[12:46:06.330] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:25:0)
[12:46:06.347] Build status: STARTED
[12:46:06.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:26:0) after 3m15s
[12:49:21.361] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:26:0)
[12:49:21.380] Build status: STARTED
[12:49:21.380] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:27:0) after 2m51s
[12:52:12.486] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:27:0)
[12:52:12.504] Build status: STARTED
[12:52:12.504] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:28:0) after 6m49s
[12:59:01.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:28:0)
[12:59:01.575] Build status: STARTED
[12:59:01.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:29:0) after 1m56s
[13:00:57.552] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:29:0)
[13:00:57.574] Build status: STARTED
[13:00:57.574] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:30:0) after 3m14s
[13:04:11.595] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:30:0)
[13:04:11.631] Build status: STARTED
[13:04:11.631] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:31:0) after 4m51s
[13:09:02.671] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:31:0)
[13:09:02.687] Build status: STARTED
[13:09:02.687] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:32:0) after 8m7s
[13:17:09.701] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:32:0)
[13:17:09.717] Build status: STARTED
[13:17:09.717] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:33:0) after 2m3s
[13:19:12.808] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:33:0)
[13:19:12.829] Build status: STARTED
[13:19:12.829] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:34:0) after 6m56s
[13:26:08.841] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:34:0)
[13:26:08.858] Build status: STARTED
[13:26:08.858] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:35:0) after 2m46s
[13:28:54.886] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:35:0)
[13:28:54.902] Build status: STARTED
[13:28:54.902] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:36:0) after 9m44s
[13:38:39.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:36:0)
[13:38:39.214] Build status: STARTED
[13:38:39.215] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:37:0) after 6m6s
[13:44:45.275] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:37:0)
[13:44:45.358] Build status: STARTED
[13:44:45.358] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:38:0) after 5m13s
[13:49:58.479] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:38:0)
[13:49:58.499] Build status: STARTED
[13:49:58.499] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:39:0) after 5m41s
[13:55:39.556] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:39:0)
[13:55:39.571] Build status: STARTED
[13:55:39.571] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:40:0) after 5m2s
[14:00:41.614] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:40:0)
[14:00:41.630] Build status: STARTED
[14:00:41.630] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:41:0) after 5m40s
[14:06:21.647] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:41:0)
[14:06:21.673] Build status: STARTED
[14:06:21.673] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:42:0) after 1m33s
[14:07:54.688] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:42:0)
[14:07:54.721] Build status: STARTED
[14:07:54.721] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:43:0) after 3m38s
[14:11:32.738] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:43:0)
[14:11:32.765] Build status: STARTED
[14:11:32.765] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:44:0) after 5m2s
[14:16:34.777] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:44:0)
[14:16:34.798] Build status: STARTED
[14:16:34.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:45:0) after 1m19s
[14:17:53.810] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:45:0)
[14:17:53.837] Build status: STARTED
[14:17:53.838] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:46:0) after 8m22s
[14:26:15.853] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:46:0)
[14:26:15.876] Build status: STARTED
[14:26:15.876] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:47:0) after 5m55s
[14:32:10.943] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:47:0)
[14:32:10.958] Build status: STARTED
[14:32:10.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:48:0) after 4m3s
[14:36:14.024] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:48:0)
[14:36:14.060] Build status: STARTED
[14:36:14.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:49:0) after 2m31s
[14:38:45.110] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:49:0)
[14:38:45.127] Build status: STARTED
[14:38:45.127] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:50:0) after 6m9s
[14:44:54.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:50:0)
[14:44:54.199] Build status: STARTED
[14:44:54.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:51:0) after 1m45s
[14:46:39.248] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:51:0)
[14:46:40.476] Build status: STARTED
[14:46:40.476] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:52:0) after 5m3s
[14:51:43.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:52:0)
[14:51:43.502] Build status: STARTED
[14:51:43.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:53:0) after 9m1s
[15:00:44.554] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:53:0)
[15:00:44.587] Build status: STARTED
[15:00:44.587] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:54:0) after 4m10s
[15:04:54.640] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:54:0)
[15:04:54.665] Build status: STARTED
[15:04:54.665] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:55:0) after 1m58s
[15:06:52.690] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:55:0)
[15:06:52.762] Build status: STARTED
[15:06:52.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:56:0) after 4m15s
[15:11:07.780] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:56:0)
[15:11:07.807] Build status: STARTED
[15:11:07.807] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:57:0) after 6m20s
[15:17:28.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:57:0)
[15:17:28.036] Build status: STARTED
[15:17:28.036] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:58:0) after 9m6s
[15:26:34.102] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:58:0)
[15:26:34.121] Build status: STARTED
[15:26:34.121] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:59:0) after 5m24s
[15:31:58.141] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:59:0)
[15:31:58.157] Build status: STARTED
[15:31:58.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:60:0) after 3m13s
[15:35:11.207] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:60:0)
[15:35:11.278] Build status: STARTED
[15:35:11.278] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:61:0) after 4m16s
[15:39:27.292] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:61:0)
[15:39:27.320] Build status: STARTED
[15:39:27.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:62:0) after 4m53s
[15:44:20.340] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:62:0)
[15:44:20.361] Build status: STARTED
[15:44:20.361] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:63:0) after 1m56s
[15:46:16.377] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:63:0)
[15:46:16.403] Build status: STARTED
[15:46:16.403] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8846901920350347120:64:0) after 1m9s
[15:47:10.871] Received PubSub notification, asking Buildbucket for the build status
[15:47:10.894] Build:
{
"id": "8681471858494682929",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 2688,
"createdBy": "project:chromium",
"createTime": "2026-05-18T10:31:22.157957699Z",
"startTime": "2026-05-18T10:33:07.045888009Z",
"endTime": "2026-05-18T15:47:00.279643099Z",
"updateTime": "2026-05-18T15:47:00.279643099Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "11382d95d14ce6a9676ae5079b1e2e358cedb905",
"ref": "refs/heads/main"
}
}
}
[15:47:10.894] Invocation finished in 5h15m49.889878463s with status FAILED