You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have deployed 3scale apimanager with 3scale operator 2.12. When these are huge number of connections, OpenShift would trigger Alert CoreDNSErrorsHigh .
There was also huge number of error logs, known as SERVFAIL, with CoreDNS, like below:
[INFO] 10.20.10.100:42782 - 766 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000114929s
[INFO] 10.20.10.100:32785 - 54091 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000045799s
[INFO] 10.20.10.100:38736 - 38959 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000495619s
[INFO] 10.20.10.100:53221 - 55209 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000111221s
[INFO] 10.20.10.100:43456 - 55627 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000087746s
[INFO] 10.20.10.100:53734 - 57461 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000093702s
[INFO] 10.20.10.100:56550 - 34310 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000131092s
[INFO] 10.20.10.100:40476 - 52329 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000077204s
[INFO] 10.20.10.100:48084 - 49744 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000089607s
[INFO] 10.20.10.100:43484 - 50657 "A IN backend-listener. udp 34 false 512" SERVFAIL qr,aa,rd,ra 34 0.000053069s
[user@localhost ~]$ oc image info registry.redhat.io/3scale-amp2/apicast-gateway-rhel8@sha256:e6970547e76e9671a11e45ebe55c1ccf8d3833e22ce232adb2fe963b0f0263cb
Name: registry.redhat.io/3scale-amp2/apicast-gateway-rhel8@sha256:e6970547e76e9671a11e45ebe55c1ccf8d3833e22ce232adb2fe963b0f0263cb
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created: 147d ago
Image Size: 114.5MB in 3 layers
Layers: 79.43MB sha256:1b3417e31a5e0e64f861e121d4efed3152e75aaa85026cd784cd0070e063daa3
1.796kB sha256:809fe483e88523e7021d76b001a552856f216430023bdc0aeff8fce8df385535
35.03MB sha256:949460b6bc6cc93018087e3896c839f232ec361c13b612f5e5dab922f060d3bf
OS: linux
Arch: amd64
Entrypoint: container-entrypoint
Command: scripts/run
Working Dir: /opt/app-root
User: 1001
Exposes Ports: 8080/tcp
Environment: PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=oci
AUTO_UPDATE_INTERVAL=0
HOME=/opt/app-root/src
BUILD_PATH=/remote-source/app
LD_LIBRARY_PATH=:/opt/app-root/lib
LUA_CPATH=./?.so;/usr/lib64/lua/5.1/?.so;/usr/lib64/lua/5.1/loadall.so;/usr/local/lib64/lua/5.1/?.so
LUA_PATH=/usr/lib64/lua/5.1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/*/?.lua;
Labels: architecture=x86_64
build-date=2022-09-13T21:46:03.758871
com.redhat.build-host=cpt-1005.osbs.prod.upshift.rdu2.redhat.com
com.redhat.component=3scale-amp-apicast-gateway-container
com.redhat.license_terms=https://www.redhat.com/agreements
description=APIcast is not a standalone API gateway therefore it needs connection to the 3scale API management platform. The container includes OpenResty and uses LuaRocks to install dependencies (rocks are installed in the application folder).
distribution-scope=public
io.k8s.description=APIcast is not a standalone API gateway therefore it needs connection to the 3scale API management platform. The container includes OpenResty and uses LuaRocks to install dependencies (rocks are installed in the application folder).
io.k8s.display-name=3scale API gateway (APIcast)
io.openshift.expose-services=8080:apicast
io.openshift.tags=integration, nginx, lua, openresty, api, gateway, 3scale, rhamp
[email protected]
name=3scale-amp2/apicast-gateway-rhel8
release=74
summary=3scale's API gateway (APIcast) is an OpenResty application which consists of two parts: Nginx configuration and Lua files.
url=https://access.redhat.com/containers/#/registry.access.redhat.com/3scale-amp2/apicast-gateway-rhel8/images/1.21.0-74
vcs-ref=3e532f5dc893e3ed1d3d374d3f266e9704b358b0
vcs-type=git
vendor=Red Hat, Inc.
version=1.21.0
Steps to Reproduce
Change Environment APICAST_LOG_LEVEL to debug
Watch out the logs of API Cast, there would show DNS resolve procedure, like below:
Query backend-listener.3scale.svc.cluster.local. Found 10.10.10.100 and write cache to backend-listener.3scale.svc.cluster.local., not backend-listener
Thanks for looking at this and for the great description and detailed doc.
I looked at it and indeed it seems an issue.
We are currently working on other tasks with higher priorities. But I have created a Jira issue for this issue. https://issues.redhat.com/browse/THREESCALE-9301 so it can be added to the backlog for future sprints
Issue Description
I have deployed 3scale apimanager with 3scale operator 2.12. When these are huge number of connections, OpenShift would trigger Alert
CoreDNSErrorsHigh
.There was also huge number of error logs, known as
SERVFAIL
, with CoreDNS, like below:Version
OpenResty Version
API Cast Gateway Image Version
Steps to Reproduce
APICAST_LOG_LEVEL
todebug
First time to request:
Second time to request:
Current Result
Current DNS resolve produce for API Cast, like below:
First Time to query
backend-listener
backend-listener
backend-listener.
, and gotSERVFAIL
forbackend-listener.
backend-listener.3scale.svc.cluster.local.
Found 10.10.10.100 and write cache tobackend-listener.3scale.svc.cluster.local.
, notbackend-listener
Second Time to query
backend-listener
backend-listener
. I think cache shold be hit at this step.backend-listener.
, still GotSERVFAIL
forbackend-listener.
backend-listener.3scale.svc.cluster.local.
, cache hits forbackend-listener.3scale.svc.cluster.local.
Expected Result
Second Time to query
backend-listener
, the cache should be hit at first step to avoid querying CoreDNS.The text was updated successfully, but these errors were encountered: