[00:16:02.505] New invocation is queued and will start shortly
[00:16:03.586] Starting the invocation (attempt 1)
[00:16:03.611] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/99e485acd20846a5b8a561b69e5ab7d0e9994ce3"
[00:16:03.611] Popped gitiles commit info from properties and tags
[00:16:03.611] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[00:16:03.612] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[00:16:03.612] Buildbucket request:
{
"requestId": "8841504814225258640",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "99e485acd20846a5b8a561b69e5ab7d0e9994ce3",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8841504814225258640"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[00:16:04.285] Scheduled build:
{
"id": "8676074752328976993",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3798,
"createdBy": "project:chromium",
"createTime": "2026-07-17T00:16:03.691976713Z",
"updateTime": "2026-07-17T00:16:03.691976713Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "99e485acd20846a5b8a561b69e5ab7d0e9994ce3",
"ref": "refs/heads/main"
}
},
"canary": true
}
[00:16:04.285] Task URL: https://cr-buildbucket.appspot.com/build/8676074752328976993
[00:16:04.285] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:2:0) after 2m19s
[00:16:04.752] Received PubSub notification, asking Buildbucket for the build status
[00:16:04.794] Build status: SCHEDULED
[00:16:08.371] Received PubSub notification, asking Buildbucket for the build status
[00:16:08.397] Build status: STARTED
[00:18:23.332] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:2:0)
[00:18:23.362] Build status: STARTED
[00:18:23.362] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:5:0) after 4m11s
[00:22:34.384] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:5:0)
[00:22:34.446] Build status: STARTED
[00:22:34.447] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:6:0) after 5m6s
[00:27:40.562] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:6:0)
[00:27:40.584] Build status: STARTED
[00:27:40.584] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:7:0) after 3m8s
[00:30:48.612] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:7:0)
[00:30:48.646] Build status: STARTED
[00:30:48.646] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:8:0) after 2m20s
[00:33:08.667] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:8:0)
[00:33:08.696] Build status: STARTED
[00:33:08.696] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:9:0) after 8m25s
[00:41:33.741] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:9:0)
[00:41:33.772] Build status: STARTED
[00:41:33.772] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:10:0) after 9m59s
[00:51:32.962] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:10:0)
[00:51:32.984] Build status: STARTED
[00:51:32.984] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:11:0) after 6m4s
[00:57:37.019] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:11:0)
[00:57:37.040] Build status: STARTED
[00:57:37.040] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:12:0) after 7m25s
[01:05:02.258] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:12:0)
[01:05:02.298] Build status: STARTED
[01:05:02.298] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:13:0) after 3m48s
[01:08:50.338] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:13:0)
[01:08:50.354] Build status: STARTED
[01:08:50.354] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:14:0) after 2m42s
[01:11:32.367] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:14:0)
[01:11:32.390] Build status: STARTED
[01:11:32.390] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:15:0) after 6m10s
[01:17:42.413] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:15:0)
[01:17:42.431] Build status: STARTED
[01:17:42.431] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:16:0) after 2m7s
[01:19:49.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:16:0)
[01:19:49.534] Build status: STARTED
[01:19:49.534] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:17:0) after 9m51s
[01:29:40.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:17:0)
[01:29:40.558] Build status: STARTED
[01:29:40.559] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:18:0) after 6m45s
[01:36:25.682] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:18:0)
[01:36:25.699] Build status: STARTED
[01:36:25.699] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:19:0) after 4m18s
[01:40:43.979] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:19:0)
[01:40:43.996] Build status: STARTED
[01:40:43.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:20:0) after 4m50s
[01:45:34.111] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:20:0)
[01:45:34.147] Build status: STARTED
[01:45:34.147] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:21:0) after 1m45s
[01:47:19.209] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:21:0)
[01:47:19.228] Build status: STARTED
[01:47:19.228] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:22:0) after 1m40s
[01:48:59.327] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:22:0)
[01:48:59.346] Build status: STARTED
[01:48:59.346] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:23:0) after 1m17s
[01:50:16.369] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:23:0)
[01:50:16.385] Build status: STARTED
[01:50:16.385] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:24:0) after 7m24s
[01:57:40.434] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:24:0)
[01:57:40.451] Build status: STARTED
[01:57:40.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:25:0) after 1m5s
[01:58:45.592] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:25:0)
[01:58:45.611] Build status: STARTED
[01:58:45.611] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:26:0) after 9m2s
[02:07:47.665] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:26:0)
[02:07:47.680] Build status: STARTED
[02:07:47.680] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:27:0) after 3m42s
[02:11:29.752] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:27:0)
[02:11:29.776] Build status: STARTED
[02:11:29.776] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:28:0) after 2m28s
[02:13:57.821] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:28:0)
[02:13:57.878] Build status: STARTED
[02:13:57.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:29:0) after 9m55s
[02:23:52.878] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:29:0)
[02:23:52.898] Build status: STARTED
[02:23:52.898] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:30:0) after 3m9s
[02:27:01.960] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:30:0)
[02:27:01.986] Build status: STARTED
[02:27:01.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:31:0) after 3m23s
[02:30:25.175] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:31:0)
[02:30:25.207] Build status: STARTED
[02:30:25.207] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:32:0) after 9m58s
[02:40:23.279] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:32:0)
[02:40:23.316] Build status: STARTED
[02:40:23.316] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:33:0) after 5m12s
[02:45:35.374] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:33:0)
[02:45:35.398] Build status: STARTED
[02:45:35.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:34:0) after 6m22s
[02:51:57.488] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:34:0)
[02:51:57.510] Build status: STARTED
[02:51:57.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:35:0) after 8m17s
[03:00:14.631] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:35:0)
[03:00:14.651] Build status: STARTED
[03:00:14.651] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:36:0) after 7m6s
[03:07:20.679] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:36:0)
[03:07:20.702] Build status: STARTED
[03:07:20.702] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:37:0) after 5m4s
[03:12:24.812] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:37:0)
[03:12:24.836] Build status: STARTED
[03:12:24.836] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:38:0) after 9m41s
[03:22:05.862] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:38:0)
[03:22:05.882] Build status: STARTED
[03:22:05.882] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:39:0) after 9m10s
[03:31:15.931] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:39:0)
[03:31:15.984] Build status: STARTED
[03:31:15.984] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:40:0) after 6m13s
[03:37:29.037] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:40:0)
[03:37:29.061] Build status: STARTED
[03:37:29.061] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:41:0) after 1m51s
[03:39:20.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:41:0)
[03:39:20.217] Build status: STARTED
[03:39:20.217] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:42:0) after 6m18s
[03:45:38.243] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:42:0)
[03:45:38.269] Build status: STARTED
[03:45:38.269] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:43:0) after 2m54s
[03:48:32.292] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:43:0)
[03:48:32.330] Build status: STARTED
[03:48:32.330] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:44:0) after 2m30s
[03:51:02.457] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:44:0)
[03:51:02.474] Build status: STARTED
[03:51:02.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:45:0) after 6m20s
[03:57:22.549] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:45:0)
[03:57:22.590] Build status: STARTED
[03:57:22.590] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:46:0) after 7m58s
[04:05:20.616] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:46:0)
[04:05:20.638] Build status: STARTED
[04:05:20.638] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:47:0) after 7m23s
[04:12:43.718] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:47:0)
[04:12:43.771] Build status: STARTED
[04:12:43.771] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:48:0) after 7m5s
[04:19:48.848] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:48:0)
[04:19:48.882] Build status: STARTED
[04:19:48.882] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:49:0) after 6m2s
[04:25:50.893] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:49:0)
[04:25:50.909] Build status: STARTED
[04:25:50.910] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:50:0) after 6m42s
[04:32:32.933] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:50:0)
[04:32:32.953] Build status: STARTED
[04:32:32.953] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:51:0) after 2m40s
[04:35:13.067] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:51:0)
[04:35:13.084] Build status: STARTED
[04:35:13.084] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:52:0) after 5m41s
[04:40:54.094] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:52:0)
[04:40:54.203] Build status: STARTED
[04:40:54.203] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:53:0) after 7m10s
[04:48:04.227] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:53:0)
[04:48:04.244] Build status: STARTED
[04:48:04.244] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:54:0) after 8m17s
[04:56:21.267] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:54:0)
[04:56:21.364] Build status: STARTED
[04:56:21.364] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:55:0) after 3m52s
[05:00:13.388] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:55:0)
[05:00:13.409] Build status: STARTED
[05:00:13.409] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:56:0) after 9m18s
[05:09:31.454] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:56:0)
[05:09:31.471] Build status: STARTED
[05:09:31.471] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:57:0) after 2m2s
[05:11:33.492] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:57:0)
[05:11:33.680] Build status: STARTED
[05:11:33.680] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:58:0) after 2m47s
[05:14:20.703] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:58:0)
[05:14:20.724] Build status: STARTED
[05:14:20.725] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8841504814225258640:59:0) after 9m29s
[05:18:45.496] Received PubSub notification, asking Buildbucket for the build status
[05:18:45.519] Build:
{
"id": "8676074752328976993",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 3798,
"createdBy": "project:chromium",
"createTime": "2026-07-17T00:16:03.691976713Z",
"startTime": "2026-07-17T00:16:07.978835243Z",
"endTime": "2026-07-17T05:18:45.167196091Z",
"updateTime": "2026-07-17T05:18:45.167196091Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "99e485acd20846a5b8a561b69e5ab7d0e9994ce3",
"ref": "refs/heads/main"
}
},
"canary": true
}
[05:18:45.519] Invocation finished in 5h2m43.020272985s with status FAILED