[22:38:30.721] New invocation is queued and will start shortly
[22:38:32.993] Starting the invocation (attempt 1)
[22:38:33.024] Popped buildset tag "buildset:commit/gitiles/chromium.googlesource.com/chromium/src/+/c58add96a24cee7771e0f4f7d5a047f042eae1be"
[22:38:33.024] Popped gitiles commit info from properties and tags
[22:38:33.024] Preparing PubSub topic for "https://cr-buildbucket.appspot.com"
[22:38:33.026] PubSub topic is "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
[22:38:33.026] Buildbucket request:
{
"requestId": "8867149891798538976",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"properties": {
},
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c58add96a24cee7771e0f4f7d5a047f042eae1be",
"ref": "refs/heads/main"
},
"tags": [
{
"key": "scheduler_invocation_id",
"value": "8867149891798538976"
},
{
"key": "scheduler_job_id",
"value": "chromium/ToTLinuxPGO"
},
{
"key": "user_agent",
"value": "luci-scheduler"
}
],
"notify": {
"pubsubTopic": "projects/luci-scheduler/topics/scheduler.buildbucket.cr-buildbucket~appspot.gserviceaccount.com"
}
}
[22:38:33.769] Scheduled build:
{
"id": "8701719828611905889",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 18115,
"createdBy": "project:chromium",
"createTime": "2025-10-06T22:38:33.097993904Z",
"updateTime": "2025-10-06T22:38:33.097993904Z",
"status": "SCHEDULED",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c58add96a24cee7771e0f4f7d5a047f042eae1be",
"ref": "refs/heads/main"
}
}
}
[22:38:33.769] Task URL: https://cr-buildbucket.appspot.com/build/8701719828611905889
[22:38:33.769] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:2:0) after 7m20s
[22:38:34.706] Received PubSub notification, asking Buildbucket for the build status
[22:38:34.787] Build status: SCHEDULED
[22:39:25.236] Received PubSub notification, asking Buildbucket for the build status
[22:39:25.256] Build status: STARTED
[22:45:53.816] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:2:0)
[22:45:53.863] Build status: STARTED
[22:45:53.863] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:5:0) after 2m7s
[22:48:00.882] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:5:0)
[22:48:00.895] Build status: STARTED
[22:48:00.895] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:6:0) after 8m47s
[22:56:47.927] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:6:0)
[22:56:47.973] Build status: STARTED
[22:56:47.973] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:7:0) after 8m51s
[23:05:39.088] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:7:0)
[23:05:39.107] Build status: STARTED
[23:05:39.107] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:8:0) after 4m38s
[23:10:17.134] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:8:0)
[23:10:17.149] Build status: STARTED
[23:10:17.149] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:9:0) after 9m30s
[23:19:47.248] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:9:0)
[23:19:47.268] Build status: STARTED
[23:19:47.268] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:10:0) after 6m36s
[23:26:23.403] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:10:0)
[23:26:23.704] Build status: STARTED
[23:26:23.704] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:11:0) after 4m20s
[23:30:43.706] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:11:0)
[23:30:43.723] Build status: STARTED
[23:30:43.723] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:12:0) after 4m29s
[23:35:12.743] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:12:0)
[23:35:12.763] Build status: STARTED
[23:35:12.763] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:13:0) after 8m57s
[23:44:09.898] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:13:0)
[23:44:09.917] Build status: STARTED
[23:44:09.917] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:14:0) after 8m40s
[23:52:49.989] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:14:0)
[23:52:50.009] Build status: STARTED
[23:52:50.009] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:15:0) after 3m23s
[23:56:13.032] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:15:0)
[23:56:13.046] Build status: STARTED
[23:56:13.047] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:16:0) after 4m29s
[00:00:42.064] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:16:0)
[00:00:42.134] Build status: STARTED
[00:00:42.134] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:17:0) after 4m47s
[00:05:29.271] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:17:0)
[00:05:29.292] Build status: STARTED
[00:05:29.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:18:0) after 5m51s
[00:11:20.309] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:18:0)
[00:11:20.654] Build status: STARTED
[00:11:20.654] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:19:0) after 5m33s
[00:16:53.724] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:19:0)
[00:16:53.742] Build status: STARTED
[00:16:53.742] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:20:0) after 8m20s
[00:25:13.778] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:20:0)
[00:25:13.797] Build status: STARTED
[00:25:13.797] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:21:0) after 8m12s
[00:33:25.855] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:21:0)
[00:33:25.872] Build status: STARTED
[00:33:25.872] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:22:0) after 7m48s
[00:41:14.085] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:22:0)
[00:41:14.100] Build status: STARTED
[00:41:14.100] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:23:0) after 2m37s
[00:43:51.140] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:23:0)
[00:43:51.162] Build status: STARTED
[00:43:51.163] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:24:0) after 1m39s
[00:45:30.138] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:24:0)
[00:45:30.178] Build status: STARTED
[00:45:30.178] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:25:0) after 4m12s
[00:49:42.181] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:25:0)
[00:49:42.204] Build status: STARTED
[00:49:42.204] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:26:0) after 9m49s
[00:59:31.279] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:26:0)
[00:59:31.300] Build status: STARTED
[00:59:31.300] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:27:0) after 6m41s
[01:06:12.292] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:27:0)
[01:06:12.310] Build status: STARTED
[01:06:12.311] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:28:0) after 5m35s
[01:11:47.332] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:28:0)
[01:11:47.347] Build status: STARTED
[01:11:47.347] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:29:0) after 7m41s
[01:19:28.374] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:29:0)
[01:19:28.391] Build status: STARTED
[01:19:28.391] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:30:0) after 3m31s
[01:22:59.434] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:30:0)
[01:22:59.488] Build status: STARTED
[01:22:59.488] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:31:0) after 1m15s
[01:24:14.696] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:31:0)
[01:24:14.714] Build status: STARTED
[01:24:14.714] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:32:0) after 8m44s
[01:32:58.848] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:32:0)
[01:32:58.866] Build status: STARTED
[01:32:58.866] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:33:0) after 2m47s
[01:35:45.890] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:33:0)
[01:35:45.904] Build status: STARTED
[01:35:45.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:34:0) after 3m58s
[01:39:43.926] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:34:0)
[01:39:43.954] Build status: STARTED
[01:39:43.954] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:35:0) after 2m41s
[01:42:25.001] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:35:0)
[01:42:25.018] Build status: STARTED
[01:42:25.018] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:36:0) after 9m34s
[01:51:59.139] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:36:0)
[01:51:59.155] Build status: STARTED
[01:51:59.155] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:37:0) after 5m51s
[01:57:50.266] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:37:0)
[01:57:50.288] Build status: STARTED
[01:57:50.288] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:38:0) after 9m16s
[02:07:06.311] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:38:0)
[02:07:06.326] Build status: STARTED
[02:07:06.326] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:39:0) after 2m30s
[02:09:36.344] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:39:0)
[02:09:36.364] Build status: STARTED
[02:09:36.364] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:40:0) after 4m12s
[02:13:48.385] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:40:0)
[02:13:48.399] Build status: STARTED
[02:13:48.399] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:41:0) after 7m38s
[02:21:26.559] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:41:0)
[02:21:26.575] Build status: STARTED
[02:21:26.575] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:42:0) after 5m41s
[02:27:07.654] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:42:0)
[02:27:07.672] Build status: STARTED
[02:27:07.672] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:43:0) after 3m49s
[02:30:56.697] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:43:0)
[02:30:56.718] Build status: STARTED
[02:30:56.718] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:44:0) after 8m58s
[02:39:54.853] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:44:0)
[02:39:55.010] Build status: STARTED
[02:39:55.010] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:45:0) after 2m26s
[02:42:21.065] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:45:0)
[02:42:21.122] Build status: STARTED
[02:42:21.122] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:46:0) after 5m44s
[02:48:05.144] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:46:0)
[02:48:05.157] Build status: STARTED
[02:48:05.157] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:47:0) after 9m20s
[02:57:25.276] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:47:0)
[02:57:25.292] Build status: STARTED
[02:57:25.292] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:48:0) after 6m2s
[03:03:27.374] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:48:0)
[03:03:27.419] Build status: STARTED
[03:03:27.419] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:49:0) after 5m36s
[03:09:03.508] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:49:0)
[03:09:03.525] Build status: STARTED
[03:09:03.525] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:50:0) after 3m41s
[03:12:44.546] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:50:0)
[03:12:44.577] Build status: STARTED
[03:12:44.577] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:51:0) after 7m52s
[03:20:36.612] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:51:0)
[03:20:36.632] Build status: STARTED
[03:20:36.632] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:52:0) after 1m46s
[03:22:22.663] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:52:0)
[03:22:22.678] Build status: STARTED
[03:22:22.678] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:53:0) after 3m44s
[03:26:06.803] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:53:0)
[03:26:06.827] Build status: STARTED
[03:26:06.827] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:54:0) after 3m33s
[03:29:39.887] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:54:0)
[03:29:39.904] Build status: STARTED
[03:29:39.904] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:55:0) after 3m38s
[03:33:17.977] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:55:0)
[03:33:17.996] Build status: STARTED
[03:33:17.996] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:56:0) after 6m39s
[03:39:57.002] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:56:0)
[03:39:57.030] Build status: STARTED
[03:39:57.031] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:57:0) after 2m24s
[03:42:21.078] Handling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:57:0)
[03:42:21.096] Build status: STARTED
[03:42:21.096] Scheduling timer "check-buildbucket-build-status" (chromium/ToTLinuxPGO:8867149891798538976:58:0) after 5m21s
[03:43:23.624] Received PubSub notification, asking Buildbucket for the build status
[03:43:23.643] Build:
{
"id": "8701719828611905889",
"builder": {
"project": "chromium",
"bucket": "ci",
"builder": "ToTLinuxPGO"
},
"number": 18115,
"createdBy": "project:chromium",
"createTime": "2025-10-06T22:38:33.097993904Z",
"startTime": "2025-10-06T22:39:25.019990092Z",
"endTime": "2025-10-07T03:42:53.779490118Z",
"updateTime": "2025-10-07T03:42:53.779490118Z",
"status": "FAILURE",
"input": {
"gitilesCommit": {
"host": "chromium.googlesource.com",
"project": "chromium/src",
"id": "c58add96a24cee7771e0f4f7d5a047f042eae1be",
"ref": "refs/heads/main"
}
}
}
[03:43:23.643] Invocation finished in 5h4m52.967799732s with status FAILED