Hazelcast eureka client is not able to register after Eureka Server restart

[mkraszew]

Hi,

Currently we use Eureka Server with Spring Boot 2.5.5 - org.springframework.cloud:spring-cloud-starter-netflix-eureka-server:jar:3.0.4 and we plan to upgrade it to Spring Boot 3.1.6 - org.springframework.cloud:spring-cloud-starter-netflix-eureka-server:jar:4.0.3.

Unfortunately, there is an issue with newer version of Eureka Server. After restarting Eureka Server cache's eureka client is not able to register itself but our application's eureka client is able to do that. Looks like an issue related with cache eureka client.

In the logs of eureka server we can see some warnings:

2024-01-22T15:45:48.487+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.s.a.dao.DaoAuthenticationProvider : Authenticated user
2024-01-22T15:45:48.487+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.s.w.a.www.BasicAuthenticationFilter : Set SecurityContextHolder to UsernamePasswordAuthenticationToken [Principal=org.springframework.security.core.userdetails.User [Username=admin, Password=[PROTECTED], Enabled=true, AccountNonExpired=true, credentialsNonExpired=true, AccountNonLocked=true, Granted Authorities=[ROLE_ADMIN, ROLE_USER]], Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=null], Granted Authorities=[ROLE_ADMIN, ROLE_USER]]
2024-01-22T15:45:48.488+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Secured PUT /eureka/apps/USER-CACHE/f0899c53-47a8-4a2b-bab4-99855fe3c55c?status=UP&lastDirtyTimestamp=1705934237735
2024-01-22T15:45:48.489+01:00 WARN 33229 --- [nio-8761-exec-9] c.n.e.registry.AbstractInstanceRegistry : DS: Registry: lease doesn't exist, registering resource: USER-CACHE - f0899c53-47a8-4a2b-bab4-99855fe3c55c
2024-01-22T15:45:48.489+01:00 WARN 33229 --- [nio-8761-exec-9] c.n.eureka.resources.InstanceResource : Not Found (Renew): USER-CACHE - f0899c53-47a8-4a2b-bab4-99855fe3c55c
2024-01-22T15:45:48.489+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Securing PUT /error?status=UP&lastDirtyTimestamp=1705934237735
2024-01-22T15:45:48.490+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Secured PUT /error?status=UP&lastDirtyTimestamp=1705934237735

In the logs of our application we have an error related to cache:

2024-01-22T15:51:48.563+01:00 ERROR 32183 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_USER-CACHE/f0899c53-47a8-4a2b-bab4-99855fe3c55c - was unable to send heartbeat!
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:837)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1401)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

Bellow our cache configuration:

@Bean
    public CacheManager cacheManager(HazelcastInstance hazelcastInstance) {
        log.debug("Starting HazelcastCacheManager");
        return new com.hazelcast.spring.cache.HazelcastCacheManager(hazelcastInstance);
    }

 @Bean
    public HazelcastInstance hazelcastInstance() {
        log.debug("Configuring Hazelcast");
        HazelcastInstance hazelCastInstance = Hazelcast.getHazelcastInstanceByName(appName);
        if (hazelCastInstance != null) {
            log.debug("Hazelcast already initialized");
            return hazelCastInstance;
        }
        Config config = new Config();
        config.setInstanceName(appName);
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        config
            .getNetworkConfig()
            .getJoin()
            .getEurekaConfig()
            .setEnabled(true)
            .setProperty("self-registration", "true")
            .setProperty("namespace", "hazelcast")
            .setProperty("use-metadata-for-host-and-port", "false")
            .setProperty("skip-eureka-registration-verification", "true")
            .setProperty("use-classpath-eureka-client-props", "false")
            .setProperty("shouldUseDns", "false")
            .setProperty("name", appName + "-cache")
            .setProperty("serviceUrl.default", eurekaServiceUrl);
        config.setManagementCenterConfig(new ManagementCenterConfig());
       // config.addMapConfig(initializeDefaultMapConfig(jHipsterProperties));
       // config.addMapConfig(initializeDomainMapConfig(jHipsterProperties));
        return Hazelcast.newHazelcastInstance(config);
    }