[01:04:34.800] New invocation is queued and will start shortly
[01:04:35.881] Starting the invocation (attempt 1)
[01:04:35.918] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/64653cbd69e2f6c663a35cea06d524bbba939d57"
[01:04:35.918] Popped gitiles commit info from properties and tags
[01:04:35.918] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[01:04:35.918] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[01:04:35.918] Buildbucket request:
{
"requestId": "8861976674871946752",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "64653cbd69e2f6c663a35cea06d524bbba939d57",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8861976674871946752"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[01:04:36.419] Scheduled build:
{
"id": "8696546612963215889",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 524,
"createdBy": "project:chromium",
"createTime": "2025-12-03T01:04:35.995497550Z",
"updateTime": "2025-12-03T01:04:35.995497550Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "64653cbd69e2f6c663a35cea06d524bbba939d57",
"ref": "refs/heads/main"
}
}
}
[01:04:36.419] Task URL: https://cr-buildbucket.appspot.com/build/8696546612963215889
[01:04:36.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:2:0) after 3m9s
[01:04:37.343] Received PubSub notification, asking Buildbucket for the build status
[01:04:37.460] Build status: SCHEDULED
[01:04:41.656] Received PubSub notification, asking Buildbucket for the build status
[01:04:41.740] Build status: STARTED
[01:07:45.431] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:2:0)
[01:07:45.451] Build status: STARTED
[01:07:45.451] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:5:0) after 7m33s
[01:15:18.466] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:5:0)
[01:15:18.538] Build status: STARTED
[01:15:18.538] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:6:0) after 1m28s
[01:16:46.584] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:6:0)
[01:16:46.605] Build status: STARTED
[01:16:46.605] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:7:0) after 4m5s
[01:20:51.603] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:7:0)
[01:20:51.617] Build status: STARTED
[01:20:51.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:8:0) after 4m25s
[01:25:16.750] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:8:0)
[01:25:16.790] Build status: STARTED
[01:25:16.790] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:9:0) after 8m56s
[01:34:12.845] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:9:0)
[01:34:12.859] Build status: STARTED
[01:34:12.859] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:10:0) after 2m33s
[01:36:45.874] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:10:0)
[01:36:45.897] Build status: STARTED
[01:36:45.897] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:11:0) after 3m21s
[01:40:06.910] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:11:0)
[01:40:06.929] Build status: STARTED
[01:40:06.929] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:12:0) after 2m33s
[01:42:39.954] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:12:0)
[01:42:39.971] Build status: STARTED
[01:42:39.971] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:13:0) after 3m21s
[01:46:00.991] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:13:0)
[01:46:01.043] Build status: STARTED
[01:46:01.043] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:14:0) after 6m44s
[01:52:45.236] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:14:0)
[01:52:45.276] Build status: STARTED
[01:52:45.277] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:15:0) after 4m57s
[01:57:42.287] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:15:0)
[01:57:42.302] Build status: STARTED
[01:57:42.302] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:16:0) after 3m17s
[02:00:59.287] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:16:0)
[02:00:59.303] Build status: STARTED
[02:00:59.303] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:17:0) after 5m35s
[02:06:34.314] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:17:0)
[02:06:34.329] Build status: STARTED
[02:06:34.329] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:18:0) after 9m56s
[02:16:30.342] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:18:0)
[02:16:30.365] Build status: STARTED
[02:16:30.365] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:19:0) after 4m11s
[02:20:41.378] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:19:0)
[02:20:41.394] Build status: STARTED
[02:20:41.394] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:20:0) after 1m58s
[02:22:39.448] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:20:0)
[02:22:39.463] Build status: STARTED
[02:22:39.463] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:21:0) after 7m25s
[02:30:04.481] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:21:0)
[02:30:04.496] Build status: STARTED
[02:30:04.496] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:22:0) after 1m14s
[02:31:18.513] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:22:0)
[02:31:18.529] Build status: STARTED
[02:31:18.529] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:23:0) after 8m27s
[02:39:45.617] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:23:0)
[02:39:45.640] Build status: STARTED
[02:39:45.640] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:24:0) after 4m55s
[02:44:40.655] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:24:0)
[02:44:40.669] Build status: STARTED
[02:44:40.669] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:25:0) after 6m21s
[02:51:01.781] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:25:0)
[02:51:01.800] Build status: STARTED
[02:51:01.800] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:26:0) after 7m5s
[02:58:06.926] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:26:0)
[02:58:06.942] Build status: STARTED
[02:58:06.942] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:27:0) after 5m25s
[03:03:32.267] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:27:0)
[03:03:32.283] Build status: STARTED
[03:03:32.283] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:28:0) after 4m19s
[03:07:51.355] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:28:0)
[03:07:51.368] Build status: STARTED
[03:07:51.368] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:29:0) after 9m8s
[03:16:59.380] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:29:0)
[03:16:59.398] Build status: STARTED
[03:16:59.398] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:30:0) after 3m46s
[03:20:45.463] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:30:0)
[03:20:45.482] Build status: STARTED
[03:20:45.482] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:31:0) after 5m21s
[03:26:06.538] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:31:0)
[03:26:06.553] Build status: STARTED
[03:26:06.553] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:32:0) after 9m55s
[03:36:01.619] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:32:0)
[03:36:01.636] Build status: STARTED
[03:36:01.636] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:33:0) after 6m41s
[03:42:42.692] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:33:0)
[03:42:42.706] Build status: STARTED
[03:42:42.706] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:34:0) after 8m3s
[03:50:45.823] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:34:0)
[03:50:45.839] Build status: STARTED
[03:50:45.839] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:35:0) after 7m34s
[03:58:19.864] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:35:0)
[03:58:19.878] Build status: STARTED
[03:58:19.878] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:36:0) after 7m4s
[04:05:23.940] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:36:0)
[04:05:23.958] Build status: STARTED
[04:05:23.958] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:37:0) after 2m56s
[04:08:19.951] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:37:0)
[04:08:20.006] Build status: STARTED
[04:08:20.006] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:38:0) after 4m59s
[04:13:19.022] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:38:0)
[04:13:19.139] Build status: STARTED
[04:13:19.139] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:39:0) after 1m46s
[04:15:05.157] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:39:0)
[04:15:05.175] Build status: STARTED
[04:15:05.175] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:40:0) after 2m38s
[04:17:43.295] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:40:0)
[04:17:43.307] Build status: STARTED
[04:17:43.307] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:41:0) after 8m34s
[04:26:17.458] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:41:0)
[04:26:17.474] Build status: STARTED
[04:26:17.474] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:42:0) after 4m16s
[04:30:33.568] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:42:0)
[04:30:33.582] Build status: STARTED
[04:30:33.582] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:43:0) after 5m47s
[04:36:20.597] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:43:0)
[04:36:20.623] Build status: STARTED
[04:36:20.624] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:44:0) after 4m22s
[04:40:42.668] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:44:0)
[04:40:42.682] Build status: STARTED
[04:40:42.682] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:45:0) after 3m21s
[04:44:03.749] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:45:0)
[04:44:03.777] Build status: STARTED
[04:44:03.777] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:46:0) after 9m49s
[04:53:52.844] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:46:0)
[04:53:52.867] Build status: STARTED
[04:53:52.867] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:47:0) after 7m47s
[05:01:39.885] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:47:0)
[05:01:39.901] Build status: STARTED
[05:01:39.901] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:48:0) after 8m36s
[05:10:15.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:48:0)
[05:10:15.957] Build status: STARTED
[05:10:15.957] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:49:0) after 5m3s
[05:15:18.972] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:49:0)
[05:15:18.990] Build status: STARTED
[05:15:18.990] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:50:0) after 1m32s
[05:16:51.051] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:50:0)
[05:16:51.067] Build status: STARTED
[05:16:51.067] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:51:0) after 3m40s
[05:20:31.134] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:51:0)
[05:20:31.154] Build status: STARTED
[05:20:31.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:52:0) after 1m36s
[05:22:07.169] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:52:0)
[05:22:07.184] Build status: STARTED
[05:22:07.184] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:53:0) after 5m45s
[05:27:52.153] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:53:0)
[05:27:52.173] Build status: STARTED
[05:27:52.173] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:54:0) after 8m30s
[05:36:22.194] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:54:0)
[05:36:22.212] Build status: STARTED
[05:36:22.212] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:55:0) after 6m54s
[05:43:16.178] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:55:0)
[05:43:16.193] Build status: STARTED
[05:43:16.193] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:56:0) after 1m30s
[05:44:46.185] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:56:0)
[05:44:46.199] Build status: STARTED
[05:44:46.199] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:57:0) after 6m37s
[05:51:23.304] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:57:0)
[05:51:23.320] Build status: STARTED
[05:51:23.320] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:58:0) after 1m45s
[05:53:08.390] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:58:0)
[05:53:08.412] Build status: STARTED
[05:53:08.412] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8861976674871946752:59:0) after 9m44s
[05:59:20.702] Received PubSub notification, asking Buildbucket for the build status
[05:59:20.752] Build:
{
"id": "8696546612963215889",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 524,
"createdBy": "project:chromium",
"createTime": "2025-12-03T01:04:35.995497550Z",
"startTime": "2025-12-03T01:04:41.502627005Z",
"endTime": "2025-12-03T05:59:20.281189869Z",
"updateTime": "2025-12-03T05:59:20.281189869Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "64653cbd69e2f6c663a35cea06d524bbba939d57",
"ref": "refs/heads/main"
}
}
}
[05:59:20.752] Invocation finished in 4h54m45.961038439s with status FAILED