Zabbix 4.x

README.md

@@ -1,19 +1,70 @@
-Zabbix Database Cleanup
-=======================
+# Zabbix SQL repo
 
-This repo contains a few useful queries to cleanup old orphaned data in a Zabbix database. While Zabbix does Housekeeping, it does not clear orphaned data very well. Especially if you come from a long upgrade-line (1.x to 1.6 to 1.8 to 2.x), chances are your database is riddled with orphaned data that is consuming disk space.
+This repo contains a few useful queries for Zabbix database, mostly to cleanup old and/or orphaned data.
 
+- [Important notes](#important-notes)
+- [Usage](#usage)
+  * [Orphaned data](#orphaned-data)
+  * [Old data](#old-data)
+  * [Unused data](#unused-data)
+  * [Stop email flood](#stop-email-flood)
+  * [LLD triggers](#lld-triggers)
 
-Usage: check how many orphaned rows you have
-============================================
+### Important notes
 
-These queries are safe to run, but if you have a large database please note that these can take a while (read: a few hours is normal).
+* If you have a large database please note that these can take a while (a few hours is normal).
+* Use the queries on your own risk. Take **backups** first. The queries were (mostly) tested against Zabbix 1.8-3.0.
+* Some scripts are Mysql or Postgresql specific, they're named *.my.sql and *.pg.sql, respectively. Some are also Zabbix version specific. Filenames are self-explaining.
+* Patches are welcome.
 
-    user@host-$ mysql zabbix < checkscript.sql
+### Usage
 
-Usage: delete the orphaned rows
-===============================
+#### Orphaned data
 
-Use the queries above on your won risk. Take *backups* first. Once you have back-ups, go ahead and be brave.
+Orphaned data is history which belongs to deleted hosts and similar.
 
-     user@host-$ mysql zabbix < cleanup.sql
+    mysql zabbix < check-orphaned-data.sql
+    psql -A -R ' : ' -P 'footer=off' zabbix < check-orphaned-data.zbx2x.sql
+
+    mysql zabbix < delete-orphaned-data.sql
+    psql -A -R ' : ' -P 'footer=off' zabbix < delete-orphaned-data.zbx2x.sql
+
+#### Old data
+
+This set of queries allows you to delete all data older than a specified period. Default is 1 week for history, 3 months for trends - edit sql at your own discretion.
+
+    mysql zabbix < check-old-data.my.sql
+    psql -A -R ' : ' -P 'footer=off' zabbix < check-old-data.pg.sql
+
+    mysql zabbix < delete-old-data.my.sql
+    psql -A -R ' : ' -P 'footer=off' zabbix < delete-old-data.pg.sql
+
+**NOTE**: due to the way the databases work (Mysql in particular), running these scripts won't reduce Zabbix db size if it's already bloated. You will have to dump and reload the db after that. What the scripts do is keep its size more or less constant if you run them regularly.
+
+#### Unused data
+
+This deletes all history for disabled items. May come in handy when you disable a significant number of items and no longer need the collected data.
+
+    mysql zabbix < check-unused-data.sql
+    psql -A -R ' : '  -P 'footer=off' zabbix < check-unused-data.sql
+
+    mysql zabbix < delete-unused-data.sql
+    psql -A -R ' : '  -P 'footer=off' zabbix < delete-unused-data.sql
+
+#### Stop email flood
+
+(Use stop-and-delete-email-alerts.sql if you're not interested in alert history)
+
+    sudo service zabbix-server stop
+    psql zabbix < stop-email-alerts.sql
+    sudo service zabbix-server start
+
+#### LLD triggers
+
+Zabbix will create automatically create "Mounted Filesystem Discovery" triggers, which you can't disable or delete from the web interface. These queries will allow you to delete them all at once (disk space and inodes). [Inspired by](https://www.zabbix.com/forum/showthread.php?t=44028).
+
+    mysql zabbix < check-lld-triggers.my.sql
+    psql -A -R ' : '  -P 'footer=off' zabbix < check-lld-triggers.pg.sql
+
+    mysql zabbix < delete-lld-triggers.my.sql
+    psql -A -R ' : '  -P 'footer=off' zabbix < delete-lld-triggers.pg.sql

check-lld-triggers.my.sql

@@ -0,0 +1,13 @@
+SELECT count(triggerid)
+AS diskspace
+FROM triggers
+WHERE description LIKE 'Free disk space is less than 20|%%'
+ESCAPE '|'
+AND NOT description LIKE '%{#FSNAME}%';
+
+SELECT count(triggerid)
+AS inodes
+FROM triggers
+WHERE description LIKE 'Free disk space is less than 20|%%'
+ESCAPE '|'
+AND NOT description LIKE '%{#FSNAME}%';

check-lld-triggers.pg.sql

@@ -0,0 +1,3 @@
+SELECT count(triggerid)
+FROM triggers
+WHERE description ~* 'disk space.*less.*20%' AND NOT description  ~* '{#';

check-old-data.my.sql

@@ -0,0 +1,16 @@
+-- intervals in days
+SET @history_interval = 7;
+SET @trends_interval = 90;
+
+SELECT count(alertid) AS alerts FROM alerts WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(acknowledgeid) AS acknowledges FROM acknowledges WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(eventid) AS events FROM events WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+
+SELECT count(itemid) AS history FROM history WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(itemid) AS history_uint FROM history_uint WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(itemid) AS history_str FROM history_str WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(itemid) AS history_text FROM history_text WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+SELECT count(itemid) AS history_log FROM history_log WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@history_interval * 24 * 60 * 60);
+
+SELECT count(itemid) AS trends FROM trends WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@trends_interval * 24 * 60 * 60);
+SELECT count(itemid) AS trends_uint FROM trends_uint WHERE (UNIX_TIMESTAMP(NOW()) - clock) > (@trends_interval * 24 * 60 * 60);

check-old-data.pg.sql

@@ -0,0 +1,18 @@
+-- intevals in days
+\set history_interval 7
+\set trends_interval 90
+
+SELECT count(alertid) AS alerts FROM alerts WHERE age(to_timestamp(alerts.clock)) > (:history_interval * interval '1 day');
+
+SELECT count(acknowledgeid) AS acknowledges FROM acknowledges WHERE age(to_timestamp(acknowledges.clock)) > (:history_interval * interval '1 day');
+
+SELECT count(eventid) AS events FROM events WHERE age(to_timestamp(events.clock)) > (:history_interval * interval '1 day');
+
+SELECT count(itemid) AS history FROM history WHERE age(to_timestamp(history.clock)) > (:history_interval * interval '1 day');
+SELECT count(itemid) AS history_uint FROM history_uint WHERE age(to_timestamp(history_uint.clock)) > (:history_interval * interval '1 day') ;
+SELECT count(itemid) AS history_str FROM history_str WHERE age(to_timestamp(history_str.clock)) > (:history_interval * interval '1 day') ;
+SELECT count(itemid) AS history_text FROM history_text WHERE age(to_timestamp(history_text.clock)) > (:history_interval * interval '1 day') ;
+SELECT count(itemid) AS history_log FROM history_log WHERE age(to_timestamp(history_log.clock)) > (:history_interval * interval '1 day') ;
+
+SELECT count(itemid) AS trends FROM trends WHERE age(to_timestamp(trends.clock)) > (:trends_interval * interval '1 day');
+SELECT count(itemid) AS trends_uint FROM trends_uint WHERE age(to_timestamp(trends_uint.clock)) > (:trends_interval * interval '1 day') ;

check-orphaned-data.zbx18.sql

@@ -0,0 +1,84 @@
+-- Count the amount of rows that would be deleted for all orphaned acknowledge entries
+SELECT COUNT(*) AS acknowedges_missing_user FROM acknowledges WHERE NOT userid IN (SELECT userid FROM users);
+SELECT COUNT(*) AS acknowedges_missing_event FROM acknowledges WHERE NOT eventid IN (SELECT eventid FROM events);
+
+-- Count the amount of rows that would be deleted for orphaned alerts entries
+SELECT COUNT(*) AS alerts_missing_action FROM alerts WHERE NOT actionid IN (SELECT actionid FROM actions);
+SELECT COUNT(*) AS alerts_missing_event FROM alerts WHERE NOT eventid IN (SELECT eventid FROM events);
+SELECT COUNT(*) AS alerts_missing_user FROM alerts WHERE NOT userid IN (SELECT userid FROM users);
+SELECT COUNT(*) AS alerts_missing_mediatype FROM alerts WHERE NOT mediatypeid IN (SELECT mediatypeid FROM media_type);
+
+-- Count the amount of rows that would be deleted for orphaned application entries that no longer map back to a host
+SELECT COUNT(*) AS applications_missing_host FROM applications WHERE NOT hostid IN (SELECT hostid FROM hosts);
+
+-- Count the amount of rows that would be deleted for orphaned auditlog details (such as logins)
+-- SELECT COUNT(*) AS auditlog_missing_details FROM auditlog_details WHERE NOT auditid IN (SELECT auditid FROM auditlog); -- This takes exponential time, disabled
+SELECT COUNT(*) AS auditlog_missing_user FROM auditlog WHERE NOT userid IN (SELECT userid FROM users);
+
+-- Count the amount of rows that would be deleted for orphaned conditions
+SELECT COUNT(*) AS conditions_missing_acction FROM conditions WHERE NOT actionid IN (SELECT actionid FROM actions);
+
+-- Count the amount of rows that would be deleted for orphaned functions
+SELECT COUNT(*) AS functions_missing_item FROM functions WHERE NOT itemid IN (SELECT itemid FROM items);
+SELECT COUNT(*) AS functions_missing_trigger FROM functions WHERE NOT triggerid IN (SELECT triggerid FROM triggers);
+
+-- Count the amount of rows that would be deleted for orphaned graph items
+SELECT COUNT(*) AS graph_items_missing_graphs FROM graphs_items WHERE NOT graphid IN (SELECT graphid FROM graphs);
+SELECT COUNT(*) AS graph_items_missing_items FROM graphs_items WHERE NOT itemid IN (SELECT itemid FROM items);
+
+-- Count the amount of records in the history/trends table for items that no longer exist
+SELECT COUNT(itemid) AS history_missing_item FROM history WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS history_uint_missing_item FROM history_uint WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS history_log_missing_item FROM history_log WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS history_str_missing_item FROM history_str WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS history_sync_missing_item FROM history_sync WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS history_text_missing_item FROM history_text WHERE itemid NOT IN (SELECT itemid FROM items);
+
+-- Count the amount of rows that would be deleted for orphaned host_profiles
+SELECT COUNT(*) AS host_profiles_missing_hosts FROM hosts_profiles WHERE NOT hostid IN (SELECT hostid FROM hosts);
+SELECT COUNT(*) AS host_profiles_external_missing_hosts FROM hosts_profiles_ext WHERE NOT hostid IN (SELECT hostid FROM hosts);
+
+-- Count the amount of rows that would be deleted for orphaned host macro's
+SELECT COUNT(*) AS host_macros_missing_hosts FROM hostmacro WHERE NOT hostid IN (SELECT hostid FROM hosts);
+
+-- Count the amount of rows that would be deleted for orphaned item data
+SELECT COUNT(itemid) AS items_missing_hosts FROM items WHERE hostid NOT IN (SELECT hostid FROM hosts);
+SELECT COUNT(*) AS item_apps_missing_apps FROM items_applications WHERE applicationid NOT IN (SELECT applicationid FROM applications);
+SELECT COUNT(*) AS item_apps_missing_items FROM items_applications WHERE itemid NOT IN (SELECT itemid FROM items);
+
+-- Count the amount of rows that would be deleted for orphaned HTTP check data
+SELECT COUNT(*) AS http_step_missing_tests FROM httpstep WHERE NOT httptestid IN (SELECT httptestid FROM httptest);
+SELECT COUNT(*) AS http_step_items_missing_steps FROM httpstepitem WHERE NOT httpstepid IN (SELECT httpstepid FROM httpstep);
+SELECT COUNT(*) AS http_step_items_missing_items FROM httpstepitem WHERE NOT itemid IN (SELECT itemid FROM items);
+SELECT COUNT(*) AS http_test_missing_apps FROM httptest WHERE applicationid NOT IN (SELECT applicationid FROM applications);
+
+-- Count the amount of rows that would be deleted for orphaned maintenance data
+SELECT COUNT(*) AS maintenance_group_missing_maintenances FROM maintenances_groups WHERE maintenanceid NOT IN (SELECT maintenanceid FROM maintenances);
+SELECT COUNT(*) AS maintenance_group_missing_groups FROM maintenances_groups WHERE groupid NOT IN (SELECT groupid FROM groups);
+SELECT COUNT(*) AS maintenances_hosts_missing_maintenances FROM maintenances_hosts WHERE maintenanceid NOT IN (SELECT maintenanceid FROM maintenances);
+SELECT COUNT(*) AS maintenances_hosts_missing_hosts FROM maintenances_hosts WHERE hostid NOT IN (SELECT hostid FROM hosts);
+SELECT COUNT(*) AS maintenances_windows_missing_maintenances FROM maintenances_windows WHERE maintenanceid NOT IN (SELECT maintenanceid FROM maintenances);
+SELECT COUNT(*) AS maintenances_windows_missing_windows FROM maintenances_windows WHERE timeperiodid NOT IN (SELECT timeperiodid FROM timeperiods);
+
+-- Count the amount of rows that would be deleted for orphaned mappings
+SELECT COUNT(*) AS mappings_missing_valuemap FROM mappings WHERE NOT valuemapid IN (SELECT valuemapid FROM valuemaps);
+
+-- Count the amount of rows that would be deleted for orphaned media / user items
+SELECT COUNT(*) AS media_missing_user FROM media WHERE NOT userid IN (SELECT userid FROM users);
+SELECT COUNT(*) AS media_missing_type FROM media WHERE NOT mediatypeid IN (SELECT mediatypeid FROM media_type);
+SELECT COUNT(*) AS rights_missing_user FROM rights WHERE NOT groupid IN (SELECT usrgrpid FROM usrgrp);
+SELECT COUNT(*) AS rights_missing_group FROM rights WHERE NOT id IN (SELECT groupid FROM groups);
+
+-- Count the amount of rows that would be deleted for orphaned screens
+SELECT COUNT(*) AS screens_items_missing_screen FROM screens_items WHERE screenid NOT IN (SELECT screenid FROM screens);
+
+SELECT COUNT(*) AS sessions_missing_user FROM sessions WHERE NOT userid IN (SELECT userid FROM users);
+
+SELECT COUNT(itemid) AS trends_missing_item FROM trends WHERE itemid NOT IN (SELECT itemid FROM items);
+SELECT COUNT(itemid) AS trends_uint_missing_item FROM trends_uint WHERE itemid NOT IN (SELECT itemid FROM items);
+
+-- Count the amount of rows that would be deleted for orphaned events & triggers
+SELECT COUNT(*) AS trigger_down_dependency_missing_trigger FROM trigger_depends WHERE triggerid_down NOT IN (SELECT triggerid FROM triggers);
+SELECT COUNT(*) AS trigger_up_dependency_missing_trigger FROM trigger_depends WHERE triggerid_up NOT IN (SELECT triggerid FROM triggers);
+
+SELECT COUNT(*) AS user_history_missing_user FROM user_history WHERE NOT userid IN (SELECT userid FROM users);