Zabbix 7.

0 Performance
Tuning Best Practices

Kārlis Saliņš
Technical Support Engineer
New things in Zabbix 7.0

New processes
Adjusted health monitoring
Performance improvements
Proxy load balancing

2
New processes

Asynchronous processes
▪ SNMPAgentPoller
▪ AgentPoller
▪ HTTPAgentPoller
▪ Discoverer*

MaxConcurrentChecksPerPoller
▪ Maximum 1000 threads per worker
▪ OS configuration required

*Discoverer process is changed, however, network discovery is still performed

3
#Edit service file Increasing file
descriptor amount
systemctl edit zabbix-server(or proxy)
#Add these lines, numbers can be different
[Service]
LimitNOFILE=100000
TasksMax=32768
#Reload systemctl daemon Modification of service files is
required
systemctl daemon-reload
▪ Zabbix server
#Restart the service
▪ Zabbix proxy
systemctl restart zabbix-server(or proxy)

4
SMNPAgentPoller

Allows asynchronous monitoring of SNMP devices

Most default templates have been converted to use
SNMPAgentPoller
Custom templates need to be adjusted
Works only with certain OID syntax
▪ walk[oid,oid,oid…]
▪ get[oid]

5
Converting a custom template
Instead of discovery rule, a new walk item needs to be created

6
Converting a custom template
Discovery rule needs to be converted to dependent discovery rule

7
Converting a custom template
Discovery rule needs to be converted to dependent discovery rule

8
Converting a custom template
Use preprocessing to convert walk output to JSON on LLD rule

9
Converting a custom template
Convert item prototypes to dependent items which will take values from walk item

10
Converting a custom template
Use preprocessing to extract specific values from the master item on the item prototype

11
Converting a custom template
Use preprocessing to limit LLD rule execution time

12
Proxy memory buffer

Available proxy memory buffer methods (ProxyBufferMode):

«disk»
▪ All data gets stored in DB
▪ Default for old environments after upgrade

«memory»
▪ All data gets stored in memory (RAM)
▪ No protection against data loss

«hybrid»
▪ Recommended
▪ Uses memory in most cases
▪ Data loss protection using DB
▪ Default for new installations

13
Proxy load balancing

Proxy groups managed by Zabbix server

Auto rebalancing of hosts
In case of issues, host gets automatically
assigned to a working proxy

14
 Proxy LLD by Zabbix server
Zabbix server health template:
Does LLD of proxies that are
connected to the server
Creates basic items and triggers
Shows various statistics from
proxies

*Remember to retrieve the latest health templates after

upgrading to a new major Zabbix version!

15
Configuration updates

Configuration updates are now incremental

Zabbix server reloads configuration every 10s
Zabbix proxy reloads configuration every 10s
Active agent reloads configuration every 5s
Version/Component Zabbix server Zabbix proxy Active agent
6.0 1 minute 1 hour 2 minutes
7.0 10 seconds 10 seconds 5 seconds

16
Performance improvements

Faster frontend permission checks

Immediate maintenance
Faster trigger execution
Logging of SNMP v3 duplicate engine ID
Configuration file validation
Item timeouts

17
Zabbix internal process tuning

Workers

Caches

Managers

18
Tuning workers

Most worker usage needs to be between 40-60%

Zabbix server and proxy have almost the same set of workers

Alerter Escalator HTTP poller

History syncer History poller Timer

Text Text Text

LLD worker Java poller Poller
19
History syncers

Writes data into the database

Calculates triggers
1 History syncer can deal with ~1000 NVPS

20
History syncers

Causes:
Data cannot be written in the DB fast enough
Lots of triggers to be calculated
Most often DB related

Fixes:
Tune and check DB performance
Check triggers
Improve hardware
Increase history syncer amount (1 history syncer = 1000 NVPS!)
21
LLD workers

Performs low-level discovery

High impact on DB performance
Only on Zabbix server
Usually, best not to increase above the default amount

22
LLD workers

Causes:
Frequent execution of low-level discovery rules
Most often DB related
Fixes:
Tune and check DB performance
Increase update interval for low-level discovery rules
Increase “Discard unchanged with heartbeat” period for dependent low-level discovery
rules
Increase LLD workers (Every LLD worker causes huge load on DB!)
23
Tuning caches

Most cache usage should be between 40% and 60%

Zabbix server and proxy have almost the same set of caches

Value cache Configuration cache Trend cache

24
Value cache

Stores values for easier accessibility

Is used for trigger calculation
Is used for calculated items
Almost limitless
Never should be full!

25
Value cache

Causes:
Lots of triggers
Lots of functions that use forecasting
Lots of calculated items
Fixes:
Increase ValueCacheSize (Make sure there is enough RAM on the server!)
Adjust forecast/trigger periods
26
Configuration cache

Stores configuration
Almost limitless
Never should be full, if it gets full, Zabbix server/proxy
crashes!

27
Configuration cache

Causes:
Lots of hosts, items, triggers
Fixes:
Increase CacheSize (Make sure there is enough RAM on the server!)

28
History and history index cache

History cache
Stores values after they are preprocessed

History syncer takes the values from this cache and writes them to the
database

2 GB limit

Never should be full and should be as empty as possible!

History index cache

Indexes history cache

¼ size of history cache

29
History and history index cache

Causes:
Data cannot be written in the DB fast enough
Most of the time issues arise together with history syncer
Fixes:
Tune and check DB performance
Increase HistoryCacheSize and HistoryIndexCacheSize(Make sure there is enough RAM
on the server - this only applies if your DB is big)

30
Tuning other processes and caches

Most process worker and cache usage should be maintained between

40% and 60%

If the default settings are being used, it's acceptable for the usage to be
lower

There is no “best” configuration, every Zabbix environment needs to be

tuned individually

31
Tuning managers

Manager processes cannot be tuned!

High manager usage typically happens when worker usage grows

Needs to be checked what is causing the issues – diaginfo, reading log

files, etc.

32
Queue

Values that have not yet arrived

Fixable
Many different reasons why it appears

33
How to

Use built-in health templates

Dashboards of server/proxy performance

Monitor OS metrics of server/proxy using Zabbix agent

Remember to update the templates when upgrading to next major

release

34
Set up proxy monitoring

1. Set up the proxy

2. Create a host on the frontend
3. Set up the host to be monitored by itself
4. Link template «Zabbix proxy health»
5. Link Zabbix agent template (optional)

35
Health monitoring

36
Tuning frontend

Frontend can become slow and unresponsive over time

Lots of improvements in Zabbix 7.0

Issues can be caused by web server or the DB

37
Tuning frontend

Enable debug mode!

Debug can be done for the whole page or individual widgets

Web server issue Database issue

38
Tuning frontend

Web server configuration can be changed

/etc/php-fpm.d/zabbix.conf

39
Tuning databases
MySQL tunable parameters
▪ innodb_flush_log_at_trx_commit = 0
▪ innodb_flush_method = O_DIRECT
▪ optimizer_switch=index_condition_pushdown=off
▪ innodb_buffer_pool_size=(75-80% of RAM if
standalone DB or 60% if shared with the proxy)
PostgreSQL tunable parameters
▪ Use PGTune ( https://pgtune.leopard.in.ua/ )

40
Tuning triggers

Huge amount of trigger changes in a short time period creates lots of events

Lots of events = load on DB

Regularly observe Reports > Top 100 triggers or dashboard widget

41
Tuning triggers

Use proper functions

Adjust time intervals

Alert fatigue

LLD overrides allow to not discover triggers based on regexp

Recovery expression

Limit nodata() usage

42
Tuning templates

Default templates are good, however, require attention

Switch old SNMP templates to use asynchronous checks

Think about update interval

43
Thank you!

Kārlis Saliņš
Technical Support Engineer