Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows VMs sometimes fail to startup and recover #1845

Closed
meteorcloudy opened this issue Jan 8, 2024 · 4 comments
Closed

Windows VMs sometimes fail to startup and recover #1845

meteorcloudy opened this issue Jan 8, 2024 · 4 comments

Comments

@meteorcloudy
Copy link
Member

During a normal boot up of a Windows VM:

Serial port 1 output for bk-windows-13l5
�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01HCSM BBS Table full.
BdsDxe: loading Boot0001 "UEFI Google PersistentDisk " from PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)
BdsDxe: starting Boot0001 "UEFI Google PersistentDisk " from PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)

UEFI: Attempting to start image.
Description: UEFI Google PersistentDisk 
FilePath: PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)
OptionNumber: 1.

�[2J�[01;01H�[=3h�[2J�[01;01H�[1m�[37m�[40m�[2J�[01;01H2024/01/08 10:15:39 GCEGuestAgent: GCE Agent Started (version 20231004.02)
2024/01/08 10:15:39 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:40 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:41 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:42 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:43 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:44 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:45 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:46 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 11:15:47 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:48 GCEInstanceSetup: Enable google_osconfig_agent during the specialize configuration pass.
2024/01/08 11:15:48 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:15:48 GCEInstanceSetup: Starting sysprep specialize phase.
2024/01/08 10:15:49 GCEInstanceSetup: All networks set to DHCP.
2024/01/08 10:15:49 GCEInstanceSetup: VirtIO network adapter detected.
2024/01/08 10:15:49 GCEInstanceSetup: Running 'netsh' with arguments 'interface ipv4 set interface Ethernet mtu=1460'
2024/01/08 10:15:49 GCEInstanceSetup: --> Ok.
2024/01/08 10:15:49 GCEInstanceSetup: MTU set to 1460.
2024/01/08 10:15:49 GCEInstanceSetup: Running 'route' with arguments '/p add 169.254.169.254 mask 255.255.255.255 0.0.0.0 if 6 metric 1'
2024/01/08 10:15:49 GCEInstanceSetup: Getting hostname from metadata server.
2024/01/08 10:15:49 GCEInstanceSetup: Renamed from bk-testing-windows-image-1698927213 to bk-windows-13l5.
2024/01/08 10:15:49 GCEInstanceSetup: Configuring WinRM...
2024/01/08 11:15:49 GCEGuestAgent: Starting the scheduler to run jobs
2024/01/08 11:15:49 GCEGuestAgent: start: []
2024/01/08 11:15:49 GCEGuestAgent: added: [now 2024-01-08 11:15:49.8798113 +0100 CET entry 1 next 2024-01-09 11:15:49 +0100 CET]
2024/01/08 10:15:50 GCEInstanceSetup: --> written C:\temp\cert\cert.p12
2024/01/08 10:15:50 GCEInstanceSetup: Waiting for WinRM to be running...
2024/01/08 10:15:50 GCEInstanceSetup: Setup of WinRM complete.
2024/01/08 10:15:51 GCEMetadataScripts: Starting specialize scripts (version 20231004.02).
2024/01/08 10:15:51 GCEMetadataScripts: No specialize scripts to run.
2024/01/08 10:15:51 GCEInstanceSetup: Finished with sysprep specialize phase, restarting...
2024/01/08 11:15:51 GCEGuestAgent: Error watching metadata: context canceled
2024/01/08 11:15:51 GCEGuestAgent: GCE Agent Stopped
�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01HCSM BBS Table full.
BdsDxe: loading Boot0003 "Windows Boot Manager" from HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: starting Boot0003 "Windows Boot Manager" from HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi

UEFI: Attempting to start image.
Description: Windows Boot Manager
FilePath: HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi
OptionNumber: 3.

�[2J�[01;01H�[=3h�[2J�[01;01H�[1m�[37m�[40m�[2J�[01;01H2024/01/08 09:16:09 GCEGuestAgent: GCE Agent Started (version 20231004.02)
2024/01/08 09:16:09 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:10 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:11 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:12 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:13 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:13 GCEInstanceSetup: Enable google_osconfig_agent during the specialize configuration pass.
2024/01/08 09:16:13 GCEInstanceSetup: WinRM certificate details: Subject: CN=bk-windows-13l5, Thumbprint: 510FF642B86D223C18BDA5DB3E7A37B343EED927
2024/01/08 09:16:13 GCEInstanceSetup: RDP certificate details: Subject: CN=bk-windows-13l5, Thumbprint: D882959B4E19D8AF9884FD9C4154D96E100678EF
2024-01-08T09:16:14.3741Z OSConfigAgent Info: OSConfig Agent (version 20231003.00.0+win@1) started.
2024/01/08 09:16:14 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:14 GCEInstanceSetup: Checking instance license activation status.
2024/01/08 09:16:15 GCEInstanceSetup: bk-windows-13l5 needs to be activated by a KMS Server.
2024/01/08 09:16:15 GCEInstanceSetup: Key Management Service machine name set to kms.windows.googlecloud.com successfully.
2024/01/08 09:16:15 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:15 GCEInstanceSetup: Installed product key WMDGN-G9PQG-XVVXX-R3X43-63DFG successfully.
2024/01/08 09:16:15 GCEInstanceSetup: Could not contact activation server. Will retry activation later.
2024/01/08 09:16:15 GCEInstanceSetup: Running 'schtasks' with arguments '/change /tn GCEStartup /enable'
2024/01/08 09:16:16 GCEInstanceSetup: --> SUCCESS: The parameters of scheduled task "GCEStartup" have been changed.
2024/01/08 09:16:16 GCEInstanceSetup: Running 'schtasks' with arguments '/run /tn GCEStartup'
2024/01/08 09:16:16 GCEInstanceSetup: --> SUCCESS: Attempted to run the scheduled task "GCEStartup".
2024/01/08 09:16:16 GCEInstanceSetup: ------------------------------------------------------------
2024/01/08 09:16:16 GCEInstanceSetup: Instance setup finished. bk-windows-13l5 is ready to use.
2024/01/08 09:16:16 GCEInstanceSetup: ------------------------------------------------------------
2024/01/08 09:16:16 GCEMetadataScripts: Starting startup scripts (version 20231004.02).
2024/01/08 09:16:16 GCEMetadataScripts: Found windows-startup-script-ps1 in metadata.
2024/01/08 09:16:16 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1: Loading support for ZIP files...
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1: Creating build folder on PD-SSD...
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1:
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1:
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1:     Directory: C:\
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1:
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1:
2024/01/08 09:16:16 GCEMetadataScripts: windows-startup-script-ps1: Mode                LastWriteTime         Length Name
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: ----                -------------         ------ ----
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: d-----         1/8/2024   9:16 AM                b
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Setting environment variables...
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Creating Buildkite agent environment hook...
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Creating Buildkite agent pre-exit hook...
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Enabling SECreateSymbolicLinkPrivilege permission...
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1:
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: The task has completed successfully.
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: See log %windir%\security\logs\scesrv.log for detail info.
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Completed 6 percent (0/15) 	Process Privilege Rights area        
Completed 13 percent (1/15) 	Process Privilege Rights area        
Completed 20 percent (2/15) 	Process Privilege Rights area        
Completed 100 percent (15/15) 	Process Privilege Rights area
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: The task has completed successfully.
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: See log %windir%\security\logs\scesrv.log for detail info.
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Getting Buildkite Agent token from GCS...
2024/01/08 09:16:17 GCEMetadataScripts: windows-startup-script-ps1: Failed to download token: Exception calling "DownloadFile" with "2" argument(s): "The remote name could not be resolved: 'storage.googleapis.com'"
2024/01/08 09:16:18 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:19 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:16:20 GCEGuestAgent: Starting the scheduler to run jobs
2024/01/08 09:16:20 GCEGuestAgent: start: []
2024/01/08 09:16:20 GCEGuestAgent: added: [now 2024-01-08 09:16:20.7628544 +0000 GMT entry 1 next 2024-01-09 09:16:20 +0000 GMT]
2024/01/08 09:16:27 GCEMetadataScripts: windows-startup-script-ps1: Decrypting Buildkite Agent token using KMS...
2024/01/08 09:16:29 GCEMetadataScripts: windows-startup-script-ps1: Configuring Buildkite Agent...
2024/01/08 09:16:29 GCEMetadataScripts: windows-startup-script-ps1: Starting Buildkite agent...
�[33m2024-01-08 09:16:30 WARN  �[0m �[0mThe git-mirrors experiment has been promoted to a stable feature in agent version v3.47.0. You can safely remove the `--experiment git-mirrors` flag to silence this message and continue using the feature�[0m

�[38;5;48m   _           _ _     _ _    _ _                                _
  | |         (_) |   | | |  (_) |                              | |
  | |__  _   _ _| | __| | | ___| |_ ___    __ _  __ _  ___ _ __ | |_
  | '_ \| | | | | |/ _` | |/ / | __/ _ \  / _` |/ _` |/ _ \ '_ \| __|
  | |_) | |_| | | | (_| |   <| | ||  __/ | (_| | (_| |  __/ | | | |_
  |_.__/ \__,_|_|_|\__,_|_|\_\_|\__\___|  \__,_|\__, |\___|_| |_|\__|
                                                 __/ |
 https://buildkite.com/agent                    |___/
�[0m
�[1;36m2024-01-08 09:16:30 NOTICE�[0m �[0mStarting buildkite-agent v3.58.0 with PID: 4220�[0m
�[1;36m2024-01-08 09:16:30 NOTICE�[0m �[0mThe agent source code can be found here: https://github.com/buildkite/agent�[0m
�[1;36m2024-01-08 09:16:30 NOTICE�[0m �[0mFor questions and support, email us at: [email protected]�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mConfiguration loaded�[0m �[1;38;5;251mpath=�[0m�[0mc:\buildkite\buildkite-agent.cfg�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mRunning builds within a pseudoterminal (PTY) has been disabled�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mAgents will disconnect after a job run has completed�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mRegistering agent with Buildkite...�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mSuccessfully registered agent "bk-windows-13l5" with tags [queue=windows, kind=worker, os=windows]�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mStarting 1 Agent(s)�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[0mYou can press Ctrl-C to stop the agents�[0m
�[1;36m2024-01-08 09:16:30 NOTICE�[0m �[0mStarting HTTP health check server on 0.0.0.0:8080�[0m
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[38;5;243mbk-windows-13l5�[0m �[0mConnecting to Buildkite...�[0m 
�[38;5;48m2024-01-08 09:16:30 INFO  �[0m �[38;5;243mbk-windows-13l5�[0m �[0mWaiting for work...�[0m

However, sometimes the VM fails to boot up with log:

Serial port 1 output for bk-windows-d1qp
�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01HCSM BBS Table full.
BdsDxe: loading Boot0001 "UEFI Google PersistentDisk " from PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)
BdsDxe: starting Boot0001 "UEFI Google PersistentDisk " from PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)

UEFI: Attempting to start image.
Description: UEFI Google PersistentDisk 
FilePath: PciRoot(0x0)/Pci(0x3,0x0)/Scsi(0x1,0x0)
OptionNumber: 1.

�[2J�[01;01H�[=3h�[2J�[01;01H�[1m�[37m�[40m�[2J�[01;01H2024/01/08 10:20:27 GCEGuestAgent: GCE Agent Started (version 20231004.02)
2024/01/08 10:20:27 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:28 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:29 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:30 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:31 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:32 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:33 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:34 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 11:20:35 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:35 GCEInstanceSetup: Enable google_osconfig_agent during the specialize configuration pass.
2024/01/08 10:20:36 GCEInstanceSetup: Starting sysprep specialize phase.
2024/01/08 11:20:36 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:20:36 GCEInstanceSetup: All networks set to DHCP.
2024/01/08 10:20:36 GCEInstanceSetup: VirtIO network adapter detected.
2024/01/08 10:20:36 GCEInstanceSetup: Running 'netsh' with arguments 'interface ipv4 set interface Ethernet mtu=1460'
2024/01/08 10:20:36 GCEInstanceSetup: --> Ok.
2024/01/08 10:20:36 GCEInstanceSetup: MTU set to 1460.
2024/01/08 10:20:36 GCEInstanceSetup: Running 'route' with arguments '/p add 169.254.169.254 mask 255.255.255.255 0.0.0.0 if 6 metric 1'
2024/01/08 10:20:37 GCEInstanceSetup: Getting hostname from metadata server.
2024/01/08 10:20:37 GCEInstanceSetup: Renamed from bk-testing-windows-image-1698927213 to bk-windows-d1qp.
2024/01/08 10:20:37 GCEInstanceSetup: Configuring WinRM...
2024/01/08 10:20:37 GCEInstanceSetup: Running 'C:\Program Files\Google\Compute Engine\tools\certgen.exe' with arguments '-outDir C:\temp\cert -hostname bk-windows-d1qp'
2024/01/08 11:20:37 GCEGuestAgent: Starting the scheduler to run jobs
2024/01/08 11:20:37 GCEGuestAgent: start: []
2024/01/08 11:20:37 GCEGuestAgent: Scheduling job: telemetryJobID
2024/01/08 11:20:37 GCEGuestAgent: Scheduling job "telemetryJobID" to run at 24.000000 hr interval
2024/01/08 11:20:37 GCEGuestAgent: Successfully scheduled job telemetryJobID
2024/01/08 10:20:37 GCEInstanceSetup: Waiting for WinRM to be running...
2024/01/08 10:20:38 GCEInstanceSetup: Setup of WinRM complete.
2024/01/08 10:20:38 GCEMetadataScripts: Starting specialize scripts (version 20231004.02).
2024/01/08 10:20:38 GCEMetadataScripts: No specialize scripts to run.
2024/01/08 10:20:38 GCEInstanceSetup: Finished with sysprep specialize phase, restarting...
2024/01/08 11:20:38 GCEGuestAgent: Error watching metadata: context canceled
2024/01/08 11:20:38 GCEGuestAgent: GCE Agent Stopped
�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01HCSM BBS Table full.
BdsDxe: loading Boot0003 "Windows Boot Manager" from HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: starting Boot0003 "Windows Boot Manager" from HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi

UEFI: Attempting to start image.
Description: Windows Boot Manager
FilePath: HD(2,GPT,0360DC16-603D-4640-A954-EEFDCD1337B6,0x8000,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi
OptionNumber: 3.

�[2J�[01;01H�[=3h�[2J�[01;01H�[1m�[37m�[40m�[2J�[01;01H2024/01/08 09:20:57 GCEGuestAgent: GCE Agent Started (version 20231004.02)
2024/01/08 09:20:57 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:20:58 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:20:59 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:21:00 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:21:00 GCEInstanceSetup: Enable google_osconfig_agent during the specialize configuration pass.
2024/01/08 09:21:01 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 09:21:01 GCEInstanceSetup: WinRM certificate details: Subject: CN=bk-windows-d1qp, Thumbprint: 54F5F5027C9E34CDD3EEFBD93957639CF901A545
2024/01/08 09:21:01 GCEInstanceSetup: RDP certificate details: Subject: CN=bk-windows-d1qp, Thumbprint: 160EBEE14E4A7C397B01CE66C0A1C8F3E224C409
2024/01/08 09:21:01 GCEInstanceSetup: Checking instance license activation status.
2024-01-08T09:21:02.1221Z OSConfigAgent Info: OSConfig Agent (version 20231003.00.0+win@1) started.
2024/01/08 09:21:02 GCEGuestAgent: Could not set default route to metadata: could not find default route

And those VMs will linger around and eventually all Windows VMs will be disconnected from Bazel CI.
@meteorcloudy
Copy link
Member Author

On some machines, we are also seeing:

2024/01/08 10:29:22 GCEMetadataScripts: windows-startup-script-ps1: Failed to download token: Exception calling "DownloadFile" with "2" argument(s): "The remote name could not be resolved: 'storage.googleapis.com'"
2024/01/08 10:29:23 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:24 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:25 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:26 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:27 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:28 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:29 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:30 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:31 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:31 GCEMetadataScripts: windows-startup-script-ps1: Failed to download token: Exception calling "DownloadFile" with "2" argument(s): "The remote name could not be resolved: 'storage.googleapis.com'"
2024/01/08 10:29:32 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:33 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:34 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:35 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:36 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:37 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:38 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:39 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:40 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:41 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:41 GCEMetadataScripts: windows-startup-script-ps1: Failed to download token: Exception calling "DownloadFile" with "2" argument(s): "The remote name could not be resolved: 'storage.googleapis.com'"
2024/01/08 10:29:42 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:43 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:44 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:45 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:47 GCEGuestAgent: Could not set default route to metadata: could not find default route
2024/01/08 10:29:48 GCEGuestAgent: Could not set default route to metadata: could not find default route

@meteorcloudy
Copy link
Member Author

Relevant code:

continuous-integration/buildkite/startup-windows-pdssd.ps1

Lines 99 to 101 in d17d97b

$buildkite_agent_token_url = "https://storage.googleapis.com/bazel-untrusted-encrypted-secrets/buildkite-untrusted-agent-token.enc"
$project = "bazel-untrusted"
$key = "buildkite-untrusted-agent-token"

@meteorcloudy
Copy link
Member Author

We should probably not do a while true loop to allow the VM to fail and be recreated:

continuous-integration/buildkite/startup-windows-pdssd.ps1

Lines 104 to 114 in d17d97b

Write-Host "Getting Buildkite Agent token from GCS..."
while ($true) {
try {
(New-Object Net.WebClient).DownloadFile($buildkite_agent_token_url, $buildkite_agent_token_file)
break
} catch {
$msg = $_.Exception.Message
Write-Host "Failed to download token: $msg"
Start-Sleep -Seconds 10
}
}

meteorcloudy added a commit that referenced this issue Jan 10, 2024
@meteorcloudy
Set an upper limit for token download retries
8bff27c 
Addresses #1845
@meteorcloudy meteorcloudy mentioned this issue Jan 10, 2024
Merged
meteorcloudy added a commit that referenced this issue Jan 11, 2024
@meteorcloudy
Set an upper limit for token download retries (#1848)
ba56013 
Addresses
#1845
@meteorcloudy
Copy link
Member Author

The problem is worked around by #1848

@meteorcloudy meteorcloudy mentioned this issue Feb 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@meteorcloudy