Skip to content
This repository has been archived by the owner on Oct 6, 2023. It is now read-only.

Broker Error Log: feeder: error occured while processing client #131

Open
itsul opened this issue Sep 28, 2017 · 7 comments
Open

Broker Error Log: feeder: error occured while processing client #131

itsul opened this issue Sep 28, 2017 · 7 comments
Assignees
@bouda1

Comments

@itsul
Copy link

itsul commented Sep 28, 2017 •
edited
Loading

Hi,

one of our customer has sometimes problems with a outside poller connection. Sometimes the status icon from the display counter is in a red/error state. While the problem was present the vpn was working fine and a test telnet connection to bbdo port 5669 was successfully. Only the broker itself was disconnected. We investigated much time in troubleshooting and increased the log. The monitoring system is the only system with problems, other applications are working well via VPN include important applications like SAP ( user will report immediately if an error occured).

Display Poller Broker Logsettings:
grafik

Centreon Versions:
centreon-broker-cbmod-3.0.8-1.el6.x86_64
centreon-broker-storage-3.0.8-1.el6.x86_64
centreon-syslog-frontend-1.5.4-1.el6.noarch
centreon-engine-1.7.2-3.el6.x86_64
centreon-connector-1.1.2-1.el6.x86_64
centreon-common-2.8.12-7.el6.noarch
centreon-bam-server-3.5.5-1.el6.noarch
centreon-bi-server-3.1.5-1.el6.noarch
centreon-broker-3.0.8-1.el6.x86_64
centreon-connector-ssh-1.1.2-1.el6.x86_64
centreon-perl-libs-2.8.12-7.el6.noarch
centreon-poller-centreon-engine-2.8.12-7.el6.noarch
centreon-pp-manager-2.2.0-4.el6.noarch
centreon-2.8.12-7.el6.noarch
centreon-engine-daemon-1.7.2-3.el6.x86_64
centreon-connector-perl-1.1.2-1.el6.x86_64
centreon-release-3.4-4.el6.noarch
centreon-base-config-centreon-engine-2.8.12-7.el6.noarch
centreon-syslog-server-1.2.5-6.el6.noarch
centreon-engine-extcommands-1.7.2-3.el6.x86_64
centreon-broker-cbd-3.0.8-1.el6.x86_64
centreon-clib-1.4.2-1.el6.x86_64
centreon-broker-core-3.0.8-1.el6.x86_64
centreon-trap-2.8.12-7.el6.noarch
centreon-web-2.8.12-7.el6.noarch
centreon-license-manager-1.0.1-3.el6.noarch

OS: CentOS 6.9

Additional environment details :Hardware maschine with a central and a display poller. Display poller is located in the US and connected via MPLS from Telekom.

Broker log from display poller with a human readable dateformat:
[Thu Sep 28 06:23:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13866': TCP peer '1.2.3.4:37084' is disconnected
[Thu Sep 28 06:28:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13867': TCP peer '1.2.3.4:38130' is disconnected
[Thu Sep 28 06:33:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13868': TCP peer '1.2.3.4:39174' is disconnected
[Thu Sep 28 06:38:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13869': TCP peer '1.2.3.4:40220' is disconnected
[Thu Sep 28 06:43:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13870': TCP peer '1.2.3.4:41272' is disconnected
[Thu Sep 28 06:48:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13871': TCP peer '1.2.3.4:42326' is disconnected
[Thu Sep 28 06:53:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13872': TCP peer '1.2.3.4:43376' is disconnected
[Thu Sep 28 06:58:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13873': TCP peer '1.2.3.4:44426' is disconnected
[Thu Sep 28 07:03:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13874': TCP peer '1.2.3.4:45478' is disconnected
[Thu Sep 28 07:08:10 2017] error: feeder: error occured while processing client 'central-broker-master-input-13875': TCP peer '1.2.3.4:46524' is disconnected
[Thu Sep 28 07:13:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13876': TCP peer '1.2.3.4:47568' is disconnected
[Thu Sep 28 07:18:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13877': TCP peer '1.2.3.4:48616' is disconnected
[Thu Sep 28 07:23:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13878': TCP peer '1.2.3.4:49668' is disconnected
[Thu Sep 28 07:28:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13879': TCP peer '1.2.3.4:50714' is disconnected
[Thu Sep 28 07:33:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13880': TCP peer '1.2.3.4:51766' is disconnected
[Thu Sep 28 07:38:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13881': TCP peer '1.2.3.4:52812' is disconnected
[Thu Sep 28 07:43:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13882': TCP peer '1.2.3.4:53864' is disconnected
[Thu Sep 28 07:48:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13883': TCP peer '1.2.3.4:54916' is disconnected
[Thu Sep 28 07:53:09 2017] error: feeder: error occured while processing client 'central-broker-master-input-13884': TCP peer '1.2.3.4:55966' is disconnected

The log reports every few minutes problems with the connection but the status of the display counter is ok. The problem is rare and sporadic, log is constant.

grafik

Regards
@bouda1
Copy link
Contributor

bouda1 commented Sep 29, 2017

Hi,

During a such situation, did you try to keep telnet running long enough ? Is it not interrupted after several minutes ?

Another question, on your poller output you can configure the retry interval to reduce the time between too connections. Did you try to change it with a small value, 5 seconds or so ?

Regards.

@bouda1 bouda1 self-assigned this Oct 3, 2017
@itsul
Copy link
Author

itsul commented Oct 4, 2017

Hi,

the last time we changed the value to 60. Since this moment, there are only 2 log entries since today night. Much better than before. Now we changed the value to 5 as you suggested.

Let`s wait a couple of days, what happen in the logs.

Regards

@itsul
Copy link
Author

itsul commented Oct 20, 2017 •
edited
Loading

Unfortunately the log will be filled every few seconds:
[Fri Oct 20 12:48:20 2017] error: feeder: error occured while processing client 'central-broker-master-input-4694': TCP peer '10.1.1.1:53796' is disconnected
[Fri Oct 20 12:53:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4695': TCP peer '10.1.1.1:54894' is disconnected
[Fri Oct 20 12:58:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4696': TCP peer '10.1.1.1:55998' is disconnected
[Fri Oct 20 13:03:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4697': TCP peer '10.1.1.1:57098' is disconnected
[Fri Oct 20 13:08:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4698': TCP peer '10.1.1.1:58194' is disconnected
[Fri Oct 20 13:13:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4699': TCP peer '10.1.1.1:59290' is disconnected
[Fri Oct 20 13:18:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4700': TCP peer '10.1.1.1:60386' is disconnected
[Fri Oct 20 13:23:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4701': TCP peer '10.1.1.1:33250' is disconnected
[Fri Oct 20 13:28:19 2017] error: feeder: error occured while processing client 'central-broker-master-input-4702': TCP peer '10.1.1.1:34356' is disconnected
[Fri Oct 20 13:33:20 2017] error: feeder: error occured while processing client 'central-broker-master-input-4703': TCP peer '10.1.1.1:35458' is disconnected
[Fri Oct 20 13:38:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4704': TCP peer '10.1.1.1:36564' is disconnected
[Fri Oct 20 13:43:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4705': TCP peer '10.1.1.1:37662' is disconnected
[Fri Oct 20 13:48:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4706': TCP peer '10.1.1.1:38758' is disconnected
[Fri Oct 20 13:53:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4707': TCP peer '10.1.1.1:39852' is disconnected
[Fri Oct 20 13:58:22 2017] error: feeder: error occured while processing client 'central-broker-master-input-4708': TCP peer '10.1.1.1:40950' is disconnected
[Fri Oct 20 14:03:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4709': TCP peer '10.1.1.1:42050' is disconnected
[Fri Oct 20 14:08:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4710': TCP peer '10.1.1.1:43150' is disconnected
[Fri Oct 20 14:13:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4711': TCP peer '10.1.1.1:44244' is disconnected
[Fri Oct 20 14:18:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4712': TCP peer '10.1.1.1:45344' is disconnected
[Fri Oct 20 14:23:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4713': TCP peer '10.1.1.1:46438' is disconnected
[Fri Oct 20 14:28:21 2017] error: feeder: error occured while processing client 'central-broker-master-input-4714': TCP peer '10.1.1.1:47534' is disconnected
[Fri Oct 20 14:33:22 2017] error: feeder: error occured while processing client 'central-broker-master-input-4715': TCP peer '10.1.1.1:48630' is disconnected

@alvaro-octal
Copy link

alvaro-octal commented Dec 24, 2017 •
edited
Loading

It seems to be similar, my poller shows as not running and none of its host appear at the monitoring tab:

OS: CentOS 7.4 (AArch64)
centreon-engine: 1.8.1
centreon-broker: 3.0.12

Every time I restart the poller's cbd it shows this at central cbd: /var/log/centreon-broker/central-broker-master.log (note that poller has id 2 not 1)

[1514117240] info:    storage: committing transaction
[1514117240] info:    SQL: committing transaction
[1514117240] info:    SQL: processing poller status event (id: 1, last alive: 1514117240)
[1514117240] info:    SQL: committing transaction
[1514117240] info:    storage: committing transaction
[1514117241] info:    storage: committing transaction
[1514117241] info:    SQL: committing transaction
[1514117241] info:    SQL: processing poller status event (id: 1, last alive: 1514117241)
[1514117241] info:    SQL: committing transaction
[1514117241] info:    storage: committing transaction
[1514117241] error:   feeder: error occured while processing client 'central-broker-master-input-15': TCP peer '1.2.3.4:37692' is disconnected
[1514117241] info:    feeder: thread of client 'central-broker-master-input-15' will exit
[1514117242] info:    storage: committing transaction
[1514117242] info:    SQL: committing transaction
[1514117243] info:    storage: committing transaction
[1514117243] info:    SQL: committing transaction
[1514117244] info:    storage: committing transaction
[1514117244] info:    SQL: committing transaction
[1514117244] info:    TCP: new client connected
[1514117244] info:    file: opening new file '/var/lib/centreon-broker//central-broker-master.queue.central-broker-master-input-16'
[1514117244] info:    file: end of file '/var/lib/centreon-broker//central-broker-master.queue.central-broker-master-input-16' reached, erasing file
[1514117244] info:    multiplexing: 'central-broker-master-input-16' start with 0 in queue and the queue file is disable
[1514117244] info:    feeder: thread of client 'central-broker-master-input-16' is starting
[1514117244] info:    BBDO: peer is using protocol version 2.0.0, we're using version 2.0.0
[1514117244] info:    BBDO: we have extensions 'compression' and peer has ''
[1514117244] info:    SQL: committing transaction
[1514117244] info:    storage: committing transaction

@lpinsivy
Copy link
Contributor

Hi,

For your Centreon Broker output on distant poller:

  • Do you enable TLS encryption? (Auto / No / Yes)
  • If yes, do you use certificate?
  • Do you enable negociation? (No/ yes)
  • Do you enable "One peer retention"? (No / Yes)
  • Do you use compression (zlib)? (Auto / No / Yes)
  • Do you enable compression level and which level? (No / Yes)
  • Do you enable compression buffer size and which buffer size? (No / Yes)
  • Do you use IP address or FQDN to connect to Centreon central server?
  • Do you define failover output?
  • If yes what type of output?
  • Do you define buffering timeout?

Regards,

@itsul
Copy link
Author

itsul commented Jan 22, 2018

Hi,

find the requested information below:

image

image

Regards

@bouda1
Copy link
Contributor

bouda1 commented Feb 7, 2018

The error tag is too much in those logs. The connection is lost, broker or cbd see that, the connection is established again and data are sent.
Maybe we could just write a medium log information and an error could arise when the new connection fails.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bouda1 bouda1

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lpinsivy @alvaro-octal @bouda1 @itsul