How to correctly monitor Linux "real" memory usage?

Hi!

This is exactly the issue we have as well.

PRTG SSH sensor monitors the free physical memory. But that isn't really a very meaningful number with the way Linux handles memory. The buffers and cache usage can be more or less ignored, as they will be immediately discarded if any processes need to allocate the memory used by them. Therefore it would make more sense to have a sensor taking this into account.

We built a sensor using the sensor factory feature which sums up the effective free memory as we call it, but it is a pain to do so since you need to create a couple of SNMP sensors, one SSH sensor and then the factory built sensor to calculate the real number.

Any comments/ideas to this?

Jesper

Im having the same issue as well... im using the snmp linux memory sensor and its experiencing the same issues described above... is there a way to fix the sensor? Its not very useful as is = (

Same issue here. Any update on this?

Christoph

Hi,
For this you can use our 'SSH Script Advanced'-Sensor.
As an example you can use a example-script:
Please create a file on your linux-box in the path /var/prtg/scriptsxml/<script_name>.sh with the following content

#!/bin/bash
meminfo="/usr/bin/free"

xmlresult=`cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<prtg>
EOF
`

if [ -f $meminfo ]; then
  result=`free -b | grep 'Mem\|Swap'`
  while read line; do
    if [[ $line == Mem* ]]; then
      total=`echo $line | awk '{print $2}'`
      used=`echo $line | awk '{print $3}'`
      free=`echo $line | awk '{print $4}'`
      shared=`echo $line | awk '{print $5}'`
      buffers=`echo $line | awk '{print $6}'`
      cache=`echo $line | awk '{print $7}'`
    else
      swtotal=`echo $line | awk '{print $2}'`
      swused=`echo $line | awk '{print $3}'`
      swfree=`echo $line | awk '{print $4}'`
    fi
  done <<< "$result"

  totalusedperc=`echo $used $buffers $cache $total | \
    awk '{printf("%.3f", 100-($1-$2-$3)*100/$4)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$totalusedperc</value>
  </result>
EOF
`
  totalfreebytes=`echo $used $buffers $cache | \
    awk '{printf("%i", ($1-$2-$3))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$totalfreebytes</value>
  </result>
EOF
`


  swapusedperc=`echo $swtotal $swused | \
    awk '{printf("%.3f", ($2/$1)*100)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$swapusedperc</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swused</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swfree</value>
  </result>
EOF
`


  xmlresult+=`cat <<EOF

  <text>OK</text>
</prtg>
EOF
`

else
  xmlresult+=`cat <<EOF

  <error>1</error>
  <text>This sensor is not supported by your system, missing $proc</text>
</prtg>
EOF
`
fi

echo "$xmlresult"

exit 0

and make it executable ('chmod +x'). After that you can add it in PRTG as a new 'SSH Script Advanced'-Sensor and select the sensor from the 'Script'-dropdown.

The sensor provides 7 Channels:
Physical Free (in KByte, MByte, GByte or TByte, depending on your device-setting 'Channel Unit Configuration)
Pyhsical Used Percent (in %)
Swap Free (units like Physical Free)
Swap Used (units like Physical Free)
Swap Used Percent (in %)
Total Free (units like Physical Free)
Total Used Percent (in %)

Kind Regards

Thanks a lot for this script!

Thanks to Jim for letting us know that there was a bug in the script.

Please use the updated script instead:

#!/bin/bash
meminfo="/usr/bin/free"

xmlresult=`cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<prtg>
EOF
`

if [ -f $meminfo ]; then
  result=`free -b | grep 'Mem\|Swap'`
  while read line; do
    if [[ $line == Mem* ]]; then
      total=`echo $line | awk '{print $2}'`
      used=`echo $line | awk '{print $3}'`
      free=`echo $line | awk '{print $4}'`
      shared=`echo $line | awk '{print $5}'`
      buffers=`echo $line | awk '{print $6}'`
      cache=`echo $line | awk '{print $7}'`
    else
      swtotal=`echo $line | awk '{print $2}'`
      swused=`echo $line | awk '{print $3}'`
      swfree=`echo $line | awk '{print $4}'`
    fi
  done <<< "$result"

  physicalusedperc=`echo $used $buffers $cache $total | \
    awk '{printf("%.3f", ($1-$2-$3)*100/$4)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$physicalusedperc</value>
  </result>
EOF
`
  physicalfreebytes=`echo $used $buffers $cache $total | \
    awk '{printf("%i", $4-($1-$2-$3))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$physicalfreebytes</value>
  </result>
EOF
`


  swapusedperc=`echo $swtotal $swused | \
    awk '{printf("%.3f", ($2/$1)*100)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$swapusedperc</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swused</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swfree</value>
  </result>
EOF
`
  totalusedperc=`echo $used $buffers $cache $total $swused $swtotal | \
    awk '{printf("%.3f", ($1-$2-$3+$5)*100/($4+$6))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$totalusedperc</value>
  </result>
EOF
`
  totalfreebytes=`echo $used $buffers $cache $total $swused $swtotal | \
    awk '{printf("%i", ($4+$6)-($1-$2-$3+$5))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$totalfreebytes</value>
  </result>
EOF
`


  xmlresult+=`cat <<EOF

  <text>OK</text>
</prtg>
EOF
`

else
  xmlresult+=`cat <<EOF

  <error>1</error>
  <text>This sensor is not supported by your system, missing $proc</text>
</prtg>
EOF
`
fi

echo "$xmlresult"

exit 0

Note: If you're using CentOS 7 and above please please refer to the following.

Another small bug: if you have more than 2 GB memory free, this script will allways report 2.048 MB. In "physicalfreebytes" and "totalfreebytes" your have to replace

printf("%i".....

with

printf("%.0f".....

Thanks camaoag Works like a charm,

Do you plane to fix in the the software itself?

No, this is a different statistic than what PRTG collects for memory and at this time, the request for real memory usage has only come up a few time which is why we published the above custom script.

As a general comment, in response to Greg Campion's comment - I see that same answer to other, seemingly minor, sensor improvement requests within the PRTG forums (jmx support comes to mind).

While the SSH script will work, i imagine anyone who monitors a linux box wants to know how much memory is ACTUALLY utilized vs what is used for cache.

For context, most linux boxes will use 99% of it's memory most of the time, which means the default PRTG linux memory sensor is useless.

It makes me sad. PRTG is a decent product, but it could be awesome.

I will bring this up again in the next dev meeting and see what comes of it. I'll post here when I know more.

" While the SSH script will work, i imagine anyone who monitors a linux box wants to know how much memory is ACTUALLY utilized vs what is used for cache.

For context, most linux boxes will use 99% of it's memory most of the time, which means the default PRTG linux memory sensor is useless. "

---------------------

Yes I agree with this. We really need a working solution (SNMP), it should not require custom scripts and custom sensors - it should just work out the box using SNMP. I'd prefer not to use SSH collection or have two collection protocols running (SNMP+SSH).

Unfortunately I'm not finding the output from the SNMP disk free sensors particularly useful either.

After discussing it once again with the developers, this cannot be changed since the SNMP Value that we are querying for is technically correct, it's just that the cached and buffered memory shows in that SNMP value as being used. The above work arounds for this problem or turning off the cache would be the only way to use the SNMP sensor and get the value that you are looking for.

OK thanks. I've looked at the output using an snmp walk. I just wonder if PTRG could do an internal calculation though to provide it (using the numbers coming back from SNMP).

It would be possible to do this but it would only work for distros that worked with this memory usage system. Other distributions don't use the cache etc in this way and those happen to be in the majority of the systems that are monitored by PRTG.

In free version 3.3.10 the line :

-/+ buffers/cache:

dissapeared, and the line Mem is now the line that add the cache to the free memory. So the script don't show the correct value anymore

This script seems not to work in CentOS 7. The "Physical Used Percent" reports negative numbers. Is there any updates?

Thanks

Hi,

you are right, in CentOS from version 7 and up the 'free' command returns different values compared to other / older distributions / version.

If you are using CentOS 7 and above please modify the script to

#!/bin/bash
meminfo="/usr/bin/free"

xmlresult=`cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<prtg>
EOF
`

if [ -f $meminfo ]; then
  result=`free -b | grep 'Mem\|Swap'`
  while read line; do
    if [[ $line == Mem* ]]; then
      total=`echo $line | awk '{print $2}'`
      used=`echo $line | awk '{print $3}'`
      free=`echo $line | awk '{print $4}'`
      shared=`echo $line | awk '{print $5}'`
      buffers=`echo $line | awk '{print $6}'`
      available=`echo $line | awk '{print $7}'`
    else
      swtotal=`echo $line | awk '{print $2}'`
      swused=`echo $line | awk '{print $3}'`
      swfree=`echo $line | awk '{print $4}'`
    fi
  done <<< "$result"

  physicalusedperc=`echo $used $buffers $total | \
    awk '{printf("%.3f", ($1-$2)*100/$3)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$physicalusedperc</value>
  </result>
EOF
`
  physicalfreebytes=`echo $used $buffers $total | \
    awk '{printf("%i", $3-($1-$2))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$physicalfreebytes</value>
  </result>
EOF
`


  swapusedperc=`echo $swtotal $swused | \
    awk '{printf("%.3f", ($2/$1)*100)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$swapusedperc</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swused</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swfree</value>
  </result>
EOF
`
  totalusedperc=`echo $used $buffers $total $swused $swtotal | \
    awk '{printf("%.3f", ($1-$2+$4)*100/($3+$5))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$totalusedperc</value>
  </result>
EOF
`
  totalfreebytes=`echo $used $buffers $total $swused $swtotal | \
    awk '{printf("%i", ($3+$5)-($1-$2+$4))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Total Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$totalfreebytes</value>
  </result>
EOF
`


  xmlresult+=`cat <<EOF

  <text>OK</text>
</prtg>
EOF
`

else
  xmlresult+=`cat <<EOF

  <error>1</error>
  <text>This sensor is not supported by your system, missing $proc</text>
</prtg>
EOF
`
fi

echo "$xmlresult"

exit 0

The script above (listed as for Centos 7) is giving me negative values on CentOS Linux release 7.3.1611.

<?xml version="1.0" encoding='UTF-8'?> <prtg> <result> <channel>Physical Used Percent</channel> <float>1</float> <unit>Percent</unit> <value>-71.703</value> </result> <result> <channel>Physical Free</channel> <float>0</float> <unit>BytesMemory</unit> <value>13580980224</value> </result> <result> <channel>Swap Used Percent</channel> <float>1</float> <unit>Percent</unit> <value>0.320</value> </result> <result> <channel>Swap Used</channel> <float>0</float> <unit>BytesMemory</unit> <value>17182720</value> </result> <result> <channel>Swap Free</channel> <float>0</float> <unit>BytesMemory</unit> <value>5351522304</value> </result> <result> <channel>Total Used Percent</channel> <float>1</float> <unit>Percent</unit> <value>-42.583</value> </result> <result> <channel>Total Free</channel> <float>0</float> <unit>BytesMemory</unit> <value>18932502528</value> </result> <text>OK</text> </prtg>

I added the SSH Script Advanced and am also getting negative values. I tried both the scripts. The Linux machine is an Ubuntu Server 16.04.2 LTS

Any thoughts on this?

I am also wondering why we can't just pull the "Available" memory and show that:

root@trk-kvm-01:~# free -h
              total        used        free      shared  buff/cache   available
Mem:            47G        6.5G        580M        447M         40G         39G
Swap:           47G         93M         47G

Hi Paul,

What about the SNMP Linux Meminfo Sensor - did you try that one already?

Kind regards,
Stephan Linke, Tech Support Team

The script for CentOS 7 still shows the percentage sensors as a negative number. Any thoughts on fixing?

Stephan, the SNMP Linux Meminfo Sensor doesn't process cache correctly to display a value that's of any use. Please read comments from Greg Campion [Paessler Support].

Thanks

Sorry, didn't see that one. I'll check the script using the latest Ubuntu / CentOS tomorrow and let you guys know where to add the corresponding fixes.

Kind regards,
Stephan Linke, Tech Support Team

Thanks Stephan,

I have 3 hosts all with the same script (I believe) and all producing different results. Two hosts are Ubuntu 16.04 LTS and one is 14.04 LTS

Is there anything I can offer from here that would help?

I am quite keen to see this working, the machines are KVM virtual hosts and accurate memory usage would be very helpful indeed.

regards

Stephan, you rock!

I installed it on my two xenial servers (16.04) and it works like a charm! I didn't have to do much at all. I didn't have the pip installed but once I did that and psutil it looks great! paul@trk-kvm-02:/var/prtg/scriptsxml# free -h total used free shared buff/cache available Mem: 54G 38G 585M 553M 16G 14G Swap: 55G 493M 55G

I am having some dependency issues in the perl on my 14.04 so I don't have a result for that yet but things are looking good.

I think the math is right. :-)

Thank you so much for that!!!

You're very welcome, glad it works :) Have a great weekend!

I rewrote this script for CentOS7 in bash (in case you don't want to install python or pip). Basically the problem was the ordering. I think my math works out, but feel free to beat me up on it. I also added alerting and min thresholds.

#!/bin/bash
meminfo="/usr/bin/free"

xmlresult=`cat <<EOF
<?xml version="1.0" encoding='UTF-8'?>
<prtg>
EOF
`

if [ -f $meminfo ]; then
  result=`free -b | grep 'Mem\|Swap'`
  while read line; do
    if [[ $line == Mem* ]]; then
      total=`echo $line | awk '{print $2}'`
      used=`echo $line | awk '{print $3}'`
      free=`echo $line | awk '{print $4}'`
      shared=`echo $line | awk '{print $5}'`
      buffers=`echo $line | awk '{print $6}'`
      available=`echo $line | awk '{print $7}'`
    else
      swtotal=`echo $line | awk '{print $2}'`
      swused=`echo $line | awk '{print $3}'`
      swfree=`echo $line | awk '{print $4}'`
    fi
  done <<< "$result"

  availableperc=`echo $available $total | \
    awk '{printf("%.3f",($1/$2)*100)}'`
  xmlresult+=`cat << EOF

  <result>
    <channel>Available Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$availableperc</value>
    <LimitMode>1</LimitMode>
    <LimitMinWarning>10</LimitMinWarning>
    <LimitMinError>5</LimitMinError>
  </result>
EOF
`
  availablebytes=$available
  xmlresult+=`cat << EOF

  <result>
    <channel>Available Bytes</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$availablebytes</value>
  </result>
EOF
`
physicalusedperc=`echo $free $total | \
    awk '{printf("%.3f", (100-(($1/$2)*100)))}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$physicalusedperc</value>
  </result>
EOF
`
  physicalfreebytes=$free
  xmlresult+=`cat <<EOF

  <result>
    <channel>Physical Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$physicalfreebytes</value>
  </result>
EOF
`
  swapusedperc=`echo $swtotal $swused | \
    awk '{printf("%.3f", ($2/$1)*100)}'`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used Percent</channel>
    <float>1</float>
    <unit>Percent</unit>
    <value>$swapusedperc</value>
    <limitMode>1</limitMode>
    <limitMinWarning>10</limitMinWarning>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Used</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swused</value>
  </result>
EOF
`
  xmlresult+=`cat <<EOF

  <result>
    <channel>Swap Free</channel>
    <float>0</float>
    <unit>BytesMemory</unit>
    <value>$swfree</value>
  </result>
EOF
`

  xmlresult+=`cat <<EOF

  <text>OK</text>
</prtg>
EOF
`

else
  xmlresult+=`cat <<EOF

  <error>1</error>
  <text>This sensor is not supported by your system, missing $proc</text>
</prtg>
EOF
`
fi

echo "$xmlresult"

exit 0

Thanks! :)

PRTG Scheduler | PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

Hi Stephan,

how can i use the python code? and what sensor which i use in PRTG?

Need your explanation.

Thank you for your help

This has to be used with a SSH Script (Advanced) Sensor. The script is copied to the target host, into /var/prtg/scriptsxml and then executed via the Sensor :)

PRTG Scheduler | PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

Just to add my 2 cents after opening a support case with PRTG.

Reading through the discussion above, while a technical reason was provided as to why the SNMP Memory sensor can't be changed (it's following the spec apparently), it was never mentioned why the "SSH Meminfo" sensor couldn't be fixed.

From my own research, it looks like "MemAvailable" is a fairly recent addition to /proc/meminfo. It was introduced in Linux Kernel 3.14 which was released early 2014. This obviously explains why MemAvailable wasn't used from the beginning, but that doesn't mean that "SSH Meminfo" can't be updated to use this new metric, with a fallback to "MemFree" for older kernels. You could even just add it as an additional channel as mentioned, which could read 0 for kernels that don't have the "MemAvailable" metric.

I think for these key metrics, PRTG really needs to provide better out-of-the-box solutions. Having to write a custom script which you then need to copy to every linux server seems kind of ridiculous for such a bog-standard sensor.

Thanks for your voice on this. We already have a ticket regarding a revamp of the Sensor and it got escalated to the corresponding people to have this implemented rather sooner than later. But no ETA yet, as always.

PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

Any new Information on the state of implementation 4 months later?

Thanks in advance

The SNMP Linux Meminfo has been updated to reflect the metrics better, the SSH derivate is however still open and I'm unable to tell when it will be adressed.