[10:15:47.247] New invocation is queued and will start shortly
[10:15:50.327] Starting the invocation (attempt 1)
[10:15:50.350] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/a79d2b14ab87ae331bf8d13e28887c35879ce5a8"
[10:15:50.350] Popped gitiles commit info from properties and tags
[10:15:50.350] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[10:15:50.351] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[10:15:50.351] Buildbucket request:
{
"requestId": "8857502744547367584",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a79d2b14ab87ae331bf8d13e28887c35879ce5a8",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8857502744547367584"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[10:15:50.945] Scheduled build:
{
"id": "8692072680570954113",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 19404,
"createdBy": "project:chromium",
"createTime": "2026-01-21T10:15:50.411052554Z",
"updateTime": "2026-01-21T10:15:50.411052554Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a79d2b14ab87ae331bf8d13e28887c35879ce5a8",
"ref": "refs/heads/main"
}
},
"canary": true
}
[10:15:50.945] Task URL: https://cr-buildbucket.appspot.com/build/8692072680570954113
[10:15:50.945] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:2:0) after 8m6s
[10:15:51.826] Received PubSub notification, asking Buildbucket for the build status
[10:15:51.847] Build status: SCHEDULED
[10:16:46.062] Received PubSub notification, asking Buildbucket for the build status
[10:16:46.077] Build status: STARTED
[10:23:56.976] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:2:0)
[10:23:57.000] Build status: STARTED
[10:23:57.000] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:5:0) after 3m55s
[10:27:52.024] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:5:0)
[10:27:52.136] Build status: STARTED
[10:27:52.136] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:6:0) after 7m33s
[10:35:25.168] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:6:0)
[10:35:25.222] Build status: STARTED
[10:35:25.222] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:7:0) after 5m34s
[10:40:59.243] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:7:0)
[10:40:59.265] Build status: STARTED
[10:40:59.265] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:8:0) after 5m28s
[10:46:27.300] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:8:0)
[10:46:27.454] Build status: STARTED
[10:46:27.454] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:9:0) after 4m5s
[10:50:32.488] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:9:0)
[10:50:32.502] Build status: STARTED
[10:50:32.502] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:10:0) after 2m16s
[10:52:48.567] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:10:0)
[10:52:48.591] Build status: STARTED
[10:52:48.591] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:11:0) after 6m35s
[10:59:23.609] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:11:0)
[10:59:23.635] Build status: STARTED
[10:59:23.635] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:12:0) after 7m19s
[11:06:42.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:12:0)
[11:06:42.891] Build status: STARTED
[11:06:42.891] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:13:0) after 4m18s
[11:11:00.905] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:13:0)
[11:11:00.937] Build status: STARTED
[11:11:00.937] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:14:0) after 2m46s
[11:13:46.957] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:14:0)
[11:13:46.973] Build status: STARTED
[11:13:46.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:15:0) after 5m5s
[11:18:51.994] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:15:0)
[11:18:52.011] Build status: STARTED
[11:18:52.011] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:16:0) after 7m46s
[11:26:38.164] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:16:0)
[11:26:38.206] Build status: STARTED
[11:26:38.206] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:17:0) after 7m27s
[11:34:05.231] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:17:0)
[11:34:05.247] Build status: STARTED
[11:34:05.247] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:18:0) after 2m34s
[11:36:39.314] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:18:0)
[11:36:39.333] Build status: STARTED
[11:36:39.333] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:19:0) after 6m29s
[11:43:08.372] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:19:0)
[11:43:08.386] Build status: STARTED
[11:43:08.387] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:20:0) after 7m43s
[11:50:51.381] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:20:0)
[11:50:51.395] Build status: STARTED
[11:50:51.395] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:21:0) after 6m55s
[11:57:46.495] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:21:0)
[11:57:46.510] Build status: STARTED
[11:57:46.510] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:22:0) after 3m28s
[12:01:14.608] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:22:0)
[12:01:14.625] Build status: STARTED
[12:01:14.625] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:23:0) after 4m31s
[12:05:45.802] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:23:0)
[12:05:45.817] Build status: STARTED
[12:05:45.817] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:24:0) after 7m44s
[12:13:29.836] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:24:0)
[12:13:29.852] Build status: STARTED
[12:13:29.852] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:25:0) after 1m17s
[12:14:46.930] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:25:0)
[12:14:46.979] Build status: STARTED
[12:14:46.980] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:26:0) after 1m18s
[12:16:05.001] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:26:0)
[12:16:05.021] Build status: STARTED
[12:16:05.021] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:27:0) after 4m52s
[12:20:57.111] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:27:0)
[12:20:57.130] Build status: STARTED
[12:20:57.130] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:28:0) after 2m21s
[12:23:18.157] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:28:0)
[12:23:18.172] Build status: STARTED
[12:23:18.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:29:0) after 8m25s
[12:31:43.202] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:29:0)
[12:31:43.218] Build status: STARTED
[12:31:43.218] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:30:0) after 5m59s
[12:37:42.245] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:30:0)
[12:37:42.258] Build status: STARTED
[12:37:42.258] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:31:0) after 7m59s
[12:45:41.316] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:31:0)
[12:45:41.351] Build status: STARTED
[12:45:41.352] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:32:0) after 9m54s
[12:55:35.450] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:32:0)
[12:55:35.467] Build status: STARTED
[12:55:35.467] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:33:0) after 8m40s
[13:04:15.539] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:33:0)
[13:04:15.555] Build status: STARTED
[13:04:15.555] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:34:0) after 5m53s
[13:10:08.577] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:34:0)
[13:10:08.604] Build status: STARTED
[13:10:08.604] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:35:0) after 5m3s
[13:15:11.621] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:35:0)
[13:15:11.637] Build status: STARTED
[13:15:11.637] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:36:0) after 1m59s
[13:17:10.671] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:36:0)
[13:17:10.688] Build status: STARTED
[13:17:10.688] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:37:0) after 8m16s
[13:25:26.707] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:37:0)
[13:25:26.722] Build status: STARTED
[13:25:26.722] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:38:0) after 3m42s
[13:29:08.740] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:38:0)
[13:29:08.752] Build status: STARTED
[13:29:08.753] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:39:0) after 9m3s
[13:38:11.777] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:39:0)
[13:38:11.826] Build status: STARTED
[13:38:11.826] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:40:0) after 3m48s
[13:41:59.932] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:40:0)
[13:41:59.955] Build status: STARTED
[13:41:59.955] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:41:0) after 7m1s
[13:49:01.008] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:41:0)
[13:49:01.160] Build status: STARTED
[13:49:01.160] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:42:0) after 8m13s
[13:57:14.175] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:42:0)
[13:57:14.293] Build status: STARTED
[13:57:14.293] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:43:0) after 7m55s
[14:05:09.514] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:43:0)
[14:05:09.536] Build status: STARTED
[14:05:09.536] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:44:0) after 4m17s
[14:09:26.562] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:44:0)
[14:09:26.580] Build status: STARTED
[14:09:26.580] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:45:0) after 8m12s
[14:17:38.570] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:45:0)
[14:17:38.614] Build status: STARTED
[14:17:38.615] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:46:0) after 6m3s
[14:23:41.638] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:46:0)
[14:23:41.654] Build status: STARTED
[14:23:41.654] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:47:0) after 2m58s
[14:26:39.722] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:47:0)
[14:26:39.738] Build status: STARTED
[14:26:39.738] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:48:0) after 4m58s
[14:31:37.806] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:48:0)
[14:31:37.821] Build status: STARTED
[14:31:37.821] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:49:0) after 6m5s
[14:37:42.862] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:49:0)
[14:37:42.903] Build status: STARTED
[14:37:42.903] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:50:0) after 6m20s
[14:44:02.979] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:50:0)
[14:44:03.013] Build status: STARTED
[14:44:03.013] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:51:0) after 7m16s
[14:51:19.030] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:51:0)
[14:51:19.044] Build status: STARTED
[14:51:19.045] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:52:0) after 4m28s
[14:55:47.065] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:52:0)
[14:55:47.089] Build status: STARTED
[14:55:47.089] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:53:0) after 8m31s
[15:04:18.157] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:53:0)
[15:04:18.172] Build status: STARTED
[15:04:18.172] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:54:0) after 4m19s
[15:08:37.194] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:54:0)
[15:08:37.268] Build status: STARTED
[15:08:37.268] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:55:0) after 5m27s
[15:14:04.325] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:55:0)
[15:14:04.369] Build status: STARTED
[15:14:04.369] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:56:0) after 4m14s
[15:18:18.410] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:56:0)
[15:18:18.444] Build status: STARTED
[15:18:18.444] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:57:0) after 1m41s
[15:19:59.457] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:57:0)
[15:19:59.472] Build status: STARTED
[15:19:59.472] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:58:0) after 3m9s
[15:23:08.492] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:58:0)
[15:23:08.509] Build status: STARTED
[15:23:08.509] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8857502744547367584:59:0) after 1m48s
[15:23:37.981] Received PubSub notification, asking Buildbucket for the build status
[15:23:37.995] Build:
{
"id": "8692072680570954113",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 19404,
"createdBy": "project:chromium",
"createTime": "2026-01-21T10:15:50.411052554Z",
"startTime": "2026-01-21T10:16:45.774512761Z",
"endTime": "2026-01-21T15:23:37.683323204Z",
"updateTime": "2026-01-21T15:23:37.683323204Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "a79d2b14ab87ae331bf8d13e28887c35879ce5a8",
"ref": "refs/heads/main"
}
},
"canary": true
}
[15:23:37.995] Invocation finished in 5h7m50.760292444s with status FAILED