[Check_mk (english)] Regarding High IOPS using Check Mk

Dhawal Doshy dhawal.doshy at gmail.com
Tue Feb 10 17:14:33 CET 2015


Joddy, you may benefit by moving out the RRD function to a different
server altogether.

Also, using RRDcached helps.

A slightly dangerous option but increases performance is to move the
rrd journal to a tmpfs (reboots *will* cause performance data loss
unless you flush to disk before a reboot).

Last, see if you can use SSDs, though do your research on caveats of
an RRD + SSD solution.



On Tue, Feb 10, 2015 at 12:30 AM, Andreas Döhler
<andreas.doehler at gmail.com> wrote:
> Hi Joddy,
>
> that's a hard question whats the best way :)
> With your active check amount i would do a setup with more then one host.
> Something like one monitoring server per server room or any other
> distribution. You need to get load from your monitoring machine.
>
> One important thing i would do is separate the check machine from the web
> frontend. At my bigger installations i have the web frontend as a virtual
> machine with OMD and the only checks going to the distributed check hosts to
> check if they are alive.
>
> I think that's all i recommend at the moment. (beside the test of CMC
> performance)
>
> For version 1.2.6 it would also be an option to evaluate the rule based
> notification to reduce the notification handling of your check hosts.
>
> Best regards
> Andreas
>
>
> Joddy Street <peeyushraj8 at gmail.com> schrieb am Thu Feb 05 2015 at 09:21:50:
>
>> Thanks for the answer Andreas,
>>
>> How would you suggest improvement in the current setup in terms of
>> hardware ?
>>
>>
>> On Tue, Feb 3, 2015 at 12:16 AM, Andreas Döhler
>> <andreas.doehler at gmail.com> wrote:
>>>
>>> Inside the benchmark the numbers are generated with simulated output. It
>>> is no real active check against an agent.
>>>
>>> Two questions.
>>>
>>> - all your sites running on the same machine? Is this needed?
>>
>> We have a single Physical Machine for polling the hosts. So yes.
>>>
>>> - If yes to the first question - with the amount of your active checks
>>> and hosts you need to do more then 350 checks per second. I think this is
>>> not possible with your hardware.
>>
>> We do have an option to upgrade the Hardware, how would you suggest to
>> modify the hardware ?
>>>
>>>
>>> I think the limit of your server will be something around 32k active
>>> checks with 5 minute interval.
>>>
>>> For your system it would also be interesting to test the CMC core from
>>> the subscription version as it is very fast at check execution. But it
>>> cannot help with long check run time of your active checks.
>>>
>>> Best regards
>>> Andreas
>>>
>>> Joddy Street <peeyushraj8 at gmail.com> schrieb am Mon Feb 02 2015 at
>>> 14:12:50:
>>>>
>>>> Hi,
>>>>
>>>> I was looking at :
>>>> https://mathias-kettner.de/checkmk_checkmk_benchmarks.html
>>>>
>>>> Our base server is now defined as :
>>>>
>>>> 32 Core CPU
>>>> Hyper-threading enabled
>>>> 256 GB RAM
>>>> connected to normal Disk with RPM 10K RPM (150 TPS)
>>>> Disk Space : 500 GB
>>>>
>>>> We have installed : OMD Version 1.10 (CMK Version 1.2.2 p3)
>>>>
>>>> We have 8 OMD Instances, running as Slave
>>>>
>>>> We have 10K Hosts, per host 30 Services : 10,000 / 3,00,000
>>>>
>>>> Of which : 1,00,000 are Active Checks and 2,00,000 Passive Checks
>>>>
>>>> Running per 5 minutes.
>>>>
>>>> Looking at :
>>>>
>>>> top : sys cpu usage is : 90 %
>>>> top : memory usage is : 90 %
>>>> iostats : avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>>                               10.61        0.00      89.39    0.00
>>>> 0.00       0.00
>>>>
>>>> Now during :
>>>>
>>>> sync and activate changes : it takes 6 hours ( random killing of sync
>>>> processes)
>>>> monitoring of Passive Check : shows very old executed data : check time
>>>> difference is greater than 1 hour
>>>> monitoring of Active Checks : shows very old executed data : check time
>>>> difference is greater than 1 hour
>>>>
>>>>
>>>> How should we be able to improve this performance ? We don't really have
>>>> option of visualizing the OS.
>>>>
>>>> Please suggest.
>>>>
>>>> _______________________________________________
>>>> checkmk-en mailing list
>>>> checkmk-en at lists.mathias-kettner.de
>>>> http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en
>>
>>
>>
>>
>> --
>> < J >
>
>
> _______________________________________________
> checkmk-en mailing list
> checkmk-en at lists.mathias-kettner.de
> http://lists.mathias-kettner.de/mailman/listinfo/checkmk-en
>


More information about the checkmk-en mailing list