cdk deploy intermittently fails with ENOMEM

[bkchy]

Hi,

Recently, a number of people in our development team have been seeing cdk deploy fail with the following error:

[sdk error] CloudFormation.DescribeChangeSet({"StackName":"*******","ChangeSetName":"cdk-deploy-change-set"}) -> Error: connect ENOMEM *******:443 - Local (0.0.0.0:0)

If we enable debug logging for Node.js we see that the connection attempt to cloudformation.us-east-1.amazonaws.com is terminating almost immediately:

NET 31088: connect: attempting to connect to *******:443 (addressType: 4)
NET 31088: destroy
NET 31088: close
NET 31088: close handle

Interestingly, prior connections to CloudFormation to same address have succeeded in the same execution of the deploy command and the error does not always occur in the same spot. It seems every now and then polling the stack status during the deploy causes this error. We have tried a number things without any success:

• tried setting --concurrency 1
• changed between wired and wireless connections.
• changed between home and office network
• tried with or without VPN
• changed TCP keep-alive settings at Mac OS level
• disabled keep-alive entirely by editing the bundled index.js of CDK
• although we thought ENOMEM (no memory) error was a bit misleading we tried to change the Node.js heap settings like --max_old_space_size=... --max_semi_space_size=...
• tried different versions of Node.js 20,x and 22.x.
• upgraded CDK to latest (2.174.1).

Has anyone else run into an issue like this? We have been using AWS CDK on Node.js for almost 6 years now and this is the first time we are seeing this. All our developers are using Macbook with M1 or M2 chip.

Any suggestion for resolution or further diagnostics will be much appreciated.

[advdv]

This started happing to us as well, the error message looks like @bkchy metions, but also sometimes we get an "extended" error. This happens intermittently, we're on MacOS, Node v22.13, CDK is version: 2.175.1 (build afe6e87):

Error occurred while monitoring stack: Error: connect ENOMEM 3.78.206.123:443 - Local (0.0.0.0:0)
    at internalConnect (node:net:1115:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1520:9)
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:132:8) {
  errno: -12,
  code: 'ENOMEM',
  syscall: 'connect',
  address: '3.78.206.123',
  port: 443,
  '$metadata': [Object]
}

[bkchy]

We just discovered that disabling realtime internet scanning feature of our Sophos Endpoint software (endpoint security solution) seems to make this problem go away. We will monitor it for next couple of days to see if it's sticky but if you have any network filter/proxy enabled in your Mac (Settings -> Network -> VPN & Filters -> Filters & Proxies) then try disabling them to see if the situation improves.

[advdv]

Right, this could indeed be our AdGuard proxy messing things up. I'll see if things improve, thank you