[19:43:41.524] New invocation is queued and will start shortly
[19:43:42.651] Starting the invocation (attempt 1)
[19:43:42.679] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/66fda3582cc632f2fd307901eab8d61d1b4992f1"
[19:43:42.679] Popped gitiles commit info from properties and tags
[19:43:42.679] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[19:43:42.679] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[19:43:42.679] Buildbucket request:
{
"requestId": "8860003730135197328",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "66fda3582cc632f2fd307901eab8d61d1b4992f1",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8860003730135197328"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTMacPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[19:43:43.199] Scheduled build:
{
"id": "8694573668192651873",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 781,
"createdBy": "project:chromium",
"createTime": "2025-12-24T19:43:42.750179401Z",
"updateTime": "2025-12-24T19:43:42.750179401Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "66fda3582cc632f2fd307901eab8d61d1b4992f1",
"ref": "refs/heads/main"
}
},
"canary": true
}
[19:43:43.200] Task URL: https://cr-buildbucket.appspot.com/build/8694573668192651873
[19:43:43.200] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:2:0) after 2m58s
[19:43:52.648] Received PubSub notification, asking Buildbucket for the build status
[19:43:52.666] Build status: SCHEDULED
[19:45:05.662] Received PubSub notification, asking Buildbucket for the build status
[19:45:05.673] Build status: STARTED
[19:46:41.212] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:2:0)
[19:46:41.267] Build status: STARTED
[19:46:41.267] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:5:0) after 1m9s
[19:47:50.319] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:5:0)
[19:47:50.332] Build status: STARTED
[19:47:50.332] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:6:0) after 6m18s
[19:54:08.348] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:6:0)
[19:54:08.369] Build status: STARTED
[19:54:08.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:7:0) after 4m28s
[19:58:36.443] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:7:0)
[19:58:36.458] Build status: STARTED
[19:58:36.458] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:8:0) after 6m15s
[20:04:51.518] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:8:0)
[20:04:51.532] Build status: STARTED
[20:04:51.533] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:9:0) after 8m56s
[20:13:47.677] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:9:0)
[20:13:47.691] Build status: STARTED
[20:13:47.691] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:10:0) after 2m0s
[20:15:47.789] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:10:0)
[20:15:47.820] Build status: STARTED
[20:15:47.820] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:11:0) after 4m23s
[20:20:10.941] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:11:0)
[20:20:11.000] Build status: STARTED
[20:20:11.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:12:0) after 6m22s
[20:26:33.015] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:12:0)
[20:26:33.033] Build status: STARTED
[20:26:33.033] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:13:0) after 4m34s
[20:31:07.149] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:13:0)
[20:31:07.164] Build status: STARTED
[20:31:07.164] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:14:0) after 7m32s
[20:38:39.300] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:14:0)
[20:38:39.321] Build status: STARTED
[20:38:39.321] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:15:0) after 6m41s
[20:45:20.403] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:15:0)
[20:45:20.425] Build status: STARTED
[20:45:20.425] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:16:0) after 8m14s
[20:53:34.473] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:16:0)
[20:53:34.490] Build status: STARTED
[20:53:34.490] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:17:0) after 3m25s
[20:56:59.605] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:17:0)
[20:56:59.619] Build status: STARTED
[20:56:59.620] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:18:0) after 5m20s
[21:02:19.901] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:18:0)
[21:02:19.920] Build status: STARTED
[21:02:19.920] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:19:0) after 3m25s
[21:05:44.934] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:19:0)
[21:05:44.949] Build status: STARTED
[21:05:44.949] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:20:0) after 3m13s
[21:08:58.064] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:20:0)
[21:08:58.085] Build status: STARTED
[21:08:58.085] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:21:0) after 8m42s
[21:17:40.204] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:21:0)
[21:17:40.218] Build status: STARTED
[21:17:40.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:22:0) after 5m12s
[21:22:52.315] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:22:0)
[21:22:52.335] Build status: STARTED
[21:22:52.335] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:23:0) after 8m37s
[21:31:29.352] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:23:0)
[21:31:29.373] Build status: STARTED
[21:31:29.373] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:24:0) after 8m8s
[21:39:37.388] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:24:0)
[21:39:37.421] Build status: STARTED
[21:39:37.421] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:25:0) after 3m28s
[21:43:05.474] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:25:0)
[21:43:05.556] Build status: STARTED
[21:43:05.556] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:26:0) after 4m9s
[21:47:14.570] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:26:0)
[21:47:14.593] Build status: STARTED
[21:47:14.593] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:27:0) after 6m14s
[21:53:28.610] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:27:0)
[21:53:28.639] Build status: STARTED
[21:53:28.639] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:28:0) after 1m36s
[21:55:04.705] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:28:0)
[21:55:04.718] Build status: STARTED
[21:55:04.718] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:29:0) after 1m13s
[21:56:17.861] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:29:0)
[21:56:17.872] Build status: STARTED
[21:56:17.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:30:0) after 5m2s
[22:01:19.990] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:30:0)
[22:01:20.003] Build status: STARTED
[22:01:20.003] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:31:0) after 1m28s
[22:02:48.016] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:31:0)
[22:02:48.043] Build status: STARTED
[22:02:48.043] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:32:0) after 5m46s
[22:08:34.057] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:32:0)
[22:08:34.072] Build status: STARTED
[22:08:34.072] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:33:0) after 4m46s
[22:13:20.142] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:33:0)
[22:13:20.157] Build status: STARTED
[22:13:20.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:34:0) after 2m1s
[22:15:21.296] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:34:0)
[22:15:21.374] Build status: STARTED
[22:15:21.374] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:35:0) after 5m53s
[22:21:14.390] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:35:0)
[22:21:14.406] Build status: STARTED
[22:21:14.406] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:36:0) after 8m1s
[22:29:15.519] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:36:0)
[22:29:15.536] Build status: STARTED
[22:29:15.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:37:0) after 7m42s
[22:36:57.554] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:37:0)
[22:36:57.608] Build status: STARTED
[22:36:57.608] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:38:0) after 5m40s
[22:42:37.966] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:38:0)
[22:42:37.986] Build status: STARTED
[22:42:37.986] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:39:0) after 6m26s
[22:49:04.172] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:39:0)
[22:49:04.185] Build status: STARTED
[22:49:04.185] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:40:0) after 6m56s
[22:56:00.200] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:40:0)
[22:56:00.214] Build status: STARTED
[22:56:00.214] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:41:0) after 6m33s
[23:02:33.236] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:41:0)
[23:02:33.275] Build status: STARTED
[23:02:33.275] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:42:0) after 2m9s
[23:04:42.337] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:42:0)
[23:04:42.353] Build status: STARTED
[23:04:42.353] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:43:0) after 9m29s
[23:14:11.471] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:43:0)
[23:14:11.489] Build status: STARTED
[23:14:11.489] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:44:0) after 5m58s
[23:20:09.509] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:44:0)
[23:20:09.525] Build status: STARTED
[23:20:09.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:45:0) after 2m46s
[23:22:55.582] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:45:0)
[23:22:55.617] Build status: STARTED
[23:22:55.617] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:46:0) after 5m16s
[23:28:11.630] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:46:0)
[23:28:11.644] Build status: STARTED
[23:28:11.644] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:47:0) after 2m27s
[23:30:38.657] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:47:0)
[23:30:38.730] Build status: STARTED
[23:30:38.730] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:48:0) after 8m1s
[23:38:40.134] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:48:0)
[23:38:40.151] Build status: STARTED
[23:38:40.152] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:49:0) after 9m49s
[23:48:29.180] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:49:0)
[23:48:29.193] Build status: STARTED
[23:48:29.193] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:50:0) after 1m18s
[23:49:47.220] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:50:0)
[23:49:47.237] Build status: STARTED
[23:49:47.238] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:51:0) after 3m51s
[23:53:38.253] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:51:0)
[23:53:38.265] Build status: STARTED
[23:53:38.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:52:0) after 5m14s
[23:58:52.363] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:52:0)
[23:58:52.375] Build status: STARTED
[23:58:52.375] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:53:0) after 7m49s
[00:06:41.463] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:53:0)
[00:06:41.482] Build status: STARTED
[00:06:41.482] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:54:0) after 6m29s
[00:13:10.842] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:54:0)
[00:13:10.863] Build status: STARTED
[00:13:10.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:55:0) after 9m37s
[00:22:47.875] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:55:0)
[00:22:47.893] Build status: STARTED
[00:22:47.893] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:56:0) after 2m25s
[00:25:12.949] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:56:0)
[00:25:13.115] Build status: STARTED
[00:25:13.115] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:57:0) after 8m15s
[00:33:28.139] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:57:0)
[00:33:28.155] Build status: STARTED
[00:33:28.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:58:0) after 8m1s
[00:41:29.147] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:58:0)
[00:41:29.167] Build status: STARTED
[00:41:29.167] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:59:0) after 9m57s
[00:51:26.382] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:59:0)
[00:51:26.417] Build status: STARTED
[00:51:26.417] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:60:0) after 2m38s
[00:54:04.675] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:60:0)
[00:54:04.690] Build status: STARTED
[00:54:04.690] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:61:0) after 9m22s
[01:03:26.783] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:61:0)
[01:03:26.801] Build status: STARTED
[01:03:26.801] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:62:0) after 6m30s
[01:09:56.822] Handling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:62:0)
[01:09:56.856] Build status: STARTED
[01:09:56.856] Scheduling timer "check-buildbucket-build-status" (chromium/ToTMacPGO:8860003730135197328:63:0) after 8m4s
[01:17:34.808] Received PubSub notification, asking Buildbucket for the build status
[01:17:34.877] Build:
{
"id": "8694573668192651873",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTMacPGO"
},
"number": 781,
"createdBy": "project:chromium",
"createTime": "2025-12-24T19:43:42.750179401Z",
"startTime": "2025-12-24T19:44:35.751854302Z",
"endTime": "2025-12-25T01:17:34.476984945Z",
"updateTime": "2025-12-25T01:17:34.476984945Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "66fda3582cc632f2fd307901eab8d61d1b4992f1",
"ref": "refs/heads/main"
}
},
"canary": true
}
[01:17:34.877] Invocation finished in 5h33m53.362813953s with status FAILED