[22:06:28.839] New invocation is queued and will start shortly
[22:06:30.063] Starting the invocation (attempt 1)
[22:06:30.085] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/3bc706a74cf7e6bcc8abd3a1789364ba54701f79"
[22:06:30.085] Popped gitiles commit info from properties and tags
[22:06:30.085] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[22:06:30.085] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[22:06:30.085] Buildbucket request:
{
"requestId": "8842871920000319584",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3bc706a74cf7e6bcc8abd3a1789364ba54701f79",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8842871920000319584"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[22:06:30.594] Scheduled build:
{
"id": "8677441857958944465",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3568,
"createdBy": "project:chromium",
"createTime": "2026-07-01T22:06:30.164073643Z",
"updateTime": "2026-07-01T22:06:30.164073643Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3bc706a74cf7e6bcc8abd3a1789364ba54701f79",
"ref": "refs/heads/main"
}
}
}
[22:06:30.594] Task URL: https://cr-buildbucket.appspot.com/build/8677441857958944465
[22:06:30.594] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:2:0) after 2m3s
[22:06:31.540] Received PubSub notification, asking Buildbucket for the build status
[22:06:31.558] Build status: SCHEDULED
[22:06:34.624] Received PubSub notification, asking Buildbucket for the build status
[22:06:34.643] Build status: STARTED
[22:08:33.928] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:2:0)
[22:08:33.950] Build status: STARTED
[22:08:33.950] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:5:0) after 4m19s
[22:12:52.994] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:5:0)
[22:12:53.060] Build status: STARTED
[22:12:53.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:6:0) after 5m34s
[22:18:27.079] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:6:0)
[22:18:27.096] Build status: STARTED
[22:18:27.096] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:7:0) after 9m32s
[22:27:59.144] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:7:0)
[22:27:59.178] Build status: STARTED
[22:27:59.178] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:8:0) after 4m53s
[22:32:52.232] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:8:0)
[22:32:52.279] Build status: STARTED
[22:32:52.279] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:9:0) after 9m10s
[22:42:02.297] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:9:0)
[22:42:02.316] Build status: STARTED
[22:42:02.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:10:0) after 9m51s
[22:51:53.420] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:10:0)
[22:51:53.439] Build status: STARTED
[22:51:53.439] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:11:0) after 7m35s
[22:59:28.443] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:11:0)
[22:59:28.461] Build status: STARTED
[22:59:28.461] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:12:0) after 8m29s
[23:07:57.607] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:12:0)
[23:07:57.629] Build status: STARTED
[23:07:57.629] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:13:0) after 2m6s
[23:10:03.727] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:13:0)
[23:10:03.749] Build status: STARTED
[23:10:03.749] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:14:0) after 9m24s
[23:19:27.851] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:14:0)
[23:19:27.978] Build status: STARTED
[23:19:27.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:15:0) after 3m39s
[23:23:07.144] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:15:0)
[23:23:07.162] Build status: STARTED
[23:23:07.162] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:16:0) after 2m34s
[23:25:41.182] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:16:0)
[23:25:41.208] Build status: STARTED
[23:25:41.208] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:17:0) after 6m44s
[23:32:25.228] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:17:0)
[23:32:25.250] Build status: STARTED
[23:32:25.250] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:18:0) after 4m25s
[23:36:50.267] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:18:0)
[23:36:50.324] Build status: STARTED
[23:36:50.324] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:19:0) after 3m0s
[23:39:50.353] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:19:0)
[23:39:50.489] Build status: STARTED
[23:39:50.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:20:0) after 8m5s
[23:47:55.772] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:20:0)
[23:47:55.788] Build status: STARTED
[23:47:55.788] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:21:0) after 4m40s
[23:52:35.860] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:21:0)
[23:52:35.879] Build status: STARTED
[23:52:35.879] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:22:0) after 7m29s
[00:00:04.929] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:22:0)
[00:00:04.977] Build status: STARTED
[00:00:04.978] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:23:0) after 4m56s
[00:05:01.035] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:23:0)
[00:05:01.066] Build status: STARTED
[00:05:01.066] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:24:0) after 7m51s
[00:12:52.124] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:24:0)
[00:12:52.145] Build status: STARTED
[00:12:52.145] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:25:0) after 1m9s
[00:14:01.165] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:25:0)
[00:14:01.209] Build status: STARTED
[00:14:01.209] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:26:0) after 6m16s
[00:20:17.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:26:0)
[00:20:17.442] Build status: STARTED
[00:20:17.442] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:27:0) after 6m3s
[00:26:20.500] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:27:0)
[00:26:20.516] Build status: STARTED
[00:26:20.516] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:28:0) after 7m0s
[00:33:20.540] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:28:0)
[00:33:20.562] Build status: STARTED
[00:33:20.562] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:29:0) after 9m25s
[00:42:45.600] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:29:0)
[00:42:45.622] Build status: STARTED
[00:42:45.622] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:30:0) after 8m19s
[00:51:04.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:30:0)
[00:51:04.798] Build status: STARTED
[00:51:04.798] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:31:0) after 1m9s
[00:52:13.833] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:31:0)
[00:52:13.862] Build status: STARTED
[00:52:13.862] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:32:0) after 8m47s
[01:01:00.880] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:32:0)
[01:01:00.904] Build status: STARTED
[01:01:00.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:33:0) after 1m25s
[01:02:25.924] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:33:0)
[01:02:25.941] Build status: STARTED
[01:02:25.941] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:34:0) after 7m37s
[01:10:03.075] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:34:0)
[01:10:03.088] Build status: STARTED
[01:10:03.088] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:35:0) after 8m5s
[01:18:08.316] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:35:0)
[01:18:08.359] Build status: STARTED
[01:18:08.360] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:36:0) after 4m7s
[01:22:15.428] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:36:0)
[01:22:15.448] Build status: STARTED
[01:22:15.448] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:37:0) after 2m49s
[01:25:04.456] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:37:0)
[01:25:04.475] Build status: STARTED
[01:25:04.475] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:38:0) after 6m39s
[01:31:43.486] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:38:0)
[01:31:43.505] Build status: STARTED
[01:31:43.505] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:39:0) after 3m35s
[01:35:18.623] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:39:0)
[01:35:18.643] Build status: STARTED
[01:35:18.643] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:40:0) after 3m55s
[01:39:13.689] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:40:0)
[01:39:13.713] Build status: STARTED
[01:39:13.713] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:41:0) after 1m27s
[01:40:40.773] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:41:0)
[01:40:40.790] Build status: STARTED
[01:40:40.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:42:0) after 7m9s
[01:47:49.834] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:42:0)
[01:47:49.850] Build status: STARTED
[01:47:49.850] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:43:0) after 2m26s
[01:50:15.921] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:43:0)
[01:50:15.945] Build status: STARTED
[01:50:15.946] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:44:0) after 5m31s
[01:55:46.996] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:44:0)
[01:55:47.020] Build status: STARTED
[01:55:47.020] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:45:0) after 6m23s
[02:02:10.006] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:45:0)
[02:02:10.030] Build status: STARTED
[02:02:10.030] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:46:0) after 3m26s
[02:05:36.013] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:46:0)
[02:05:36.060] Build status: STARTED
[02:05:36.060] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:47:0) after 1m58s
[02:07:34.094] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:47:0)
[02:07:34.109] Build status: STARTED
[02:07:34.109] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:48:0) after 2m0s
[02:09:34.183] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:48:0)
[02:09:34.198] Build status: STARTED
[02:09:34.198] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:49:0) after 3m21s
[02:12:55.216] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:49:0)
[02:12:55.267] Build status: STARTED
[02:12:55.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:50:0) after 7m39s
[02:20:34.291] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:50:0)
[02:20:34.310] Build status: STARTED
[02:20:34.310] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:51:0) after 6m44s
[02:27:18.414] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:51:0)
[02:27:18.432] Build status: STARTED
[02:27:18.432] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:52:0) after 8m31s
[02:35:49.491] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:52:0)
[02:35:49.526] Build status: STARTED
[02:35:49.526] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:53:0) after 8m6s
[02:43:55.564] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:53:0)
[02:43:55.582] Build status: STARTED
[02:43:55.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:54:0) after 7m45s
[02:51:40.708] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:54:0)
[02:51:40.735] Build status: STARTED
[02:51:40.736] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:55:0) after 8m45s
[03:00:25.804] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:55:0)
[03:00:25.839] Build status: STARTED
[03:00:25.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8842871920000319584:56:0) after 4m28s
[03:04:09.232] Received PubSub notification, asking Buildbucket for the build status
[03:04:09.251] Build:
{
"id": "8677441857958944465",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3568,
"createdBy": "project:chromium",
"createTime": "2026-07-01T22:06:30.164073643Z",
"startTime": "2026-07-01T22:06:34.405904661Z",
"endTime": "2026-07-02T03:04:08.951593931Z",
"updateTime": "2026-07-02T03:04:08.951593931Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "3bc706a74cf7e6bcc8abd3a1789364ba54701f79",
"ref": "refs/heads/main"
}
}
}
[03:04:09.251] Invocation finished in 4h57m40.418087049s with status FAILED