VMware Host Hardware Status (SOAP) Sensor returns warnings after Update to VMware 6.7

Hi Jacqueline,

I agree, the API should indeed return the exact same values as displayed in your vCenter. Unfortunately, this appears to be an issue with VMware 6.7, we have other customers reporting the same findings so we need to wait for a fix from VMware here.

As we cannot change the API output of the vCenter, you can enter the settings of the sensor and enter the warning string into the Known Warnings. You can get the string by enabling the sensor setting Write Result to Disk.

Kind regards,
Felix Saure, Tech Support Team

Hello Felix,

according to our observation this is not an issue directly related to vCenter. In our case the problem showed up after we updated the first esxi host to 6.7.0 Update 1. At this point of time the vCenter itself had been running on 6.7.0 Update 1 for about a week without any occurrences.

In my understanding this could be a change of VMware, which Paessler has to adopt the behavior of the appropriate sensor to.

But I understand that Paessler themselves are in close communication with VMware to get a short time fix resolving this issue.

Is this correct and is there any idea when this will happen?

Many thanks in advance.

Hi.

What is the fix for this? To wait for PRTG and vCenter to work it out? And then what? Update PRTG or vCenter? Will anyone update this thread when it is fixed? We are hesitant to update our PRTG in case if it breaks a service we use to read web/API calls.

Aidan

Hi there,

Updates of PRTG won't help here since the changes need to be deployed in the CIM extensions of VMware. PRTG can just work with the information provided by the API.

Kind regards,
Felix Saure, Tech Support Team

Hi,

We are currently "accepting" the warnings within the sensor under "Known Warnings" so we can separate the this warnings from real ones. That is not satisfactory, I know.

But how about a small change from prtg on this topic?

I think it is possible to check/filter the values without real value ("Unknown") and report them back into a separate channel (like Returning Unknown or something like that) not the warning channel?

In my opinion this would be a nice solution on part of prtg, even if it doesn’t fix the error.

I hope you consider my idea.

Jacqueline

Hi Jaqueline,

I get your point, unfortunately I cannot think of way to get this working with the current concept of PRTG just storing numerical value except for the sensor message. What would be possible (please don't see this as promise) is to store the number of filtered warnings in a channel. I'll forward this to our Product Owners for consideration.

Kind regards,
Felix Saure, Tech Support Team

Hi Felix, we have this issue as well in our vSphere 6.7 Update 1 environment. All of our vCenters and ESXi hosts are at the latest available patch, I hope you have an idea for this issue, since otherwise this sensor is useless.

What particular error do you get?

PRTG Scheduler | PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

That issue is definitely VMWare related as in "The hardware requires newer drivers". Did you check if any updates are available? There's not much we can do to fix this :/

PRTG Scheduler | PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

We already used the latest customized installation .iso from Dell, so there are no newer drivers available..

We do have some customers with that issue; you'll need to contact VMWare regarding that I'm afraid :( We're just returning what we get from the Managed Object Browser API endpoint here.

PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

Hi,

We have the sames issue with Dell ESXi host on 6.7 VMware version. Any update for this issue ?

Thank's in advance.

Best regards,

There's nothing we can do in this regard. I discussed this issue with our VMware administrators and I need to correct my previous reply.Some updates of the vCenter change the internal way how the vendor specific drivers (Dell, HP...) communicate with the vCenter via CIM. These specific drivers need to be adjusted after an update so that they do not throw a warning via the API calls.Based on our own experience, this takes about six to eight weeks until new packages get available after an update of the vCenter. I'm afraid that this is something that we cannot fix ourselves.

PRTGapi | Feature Requests | WMI Issues | SNMP Issues

Kind regards,
Stephan Linke, Tech Support Team

My solution to this was eventually to create a script to bypass this. I accomplished this with PowerCLI from VMware - the PowerShell extensions to communicate with VMware.

For details I would like to refer here - cause it is easier for me to maintain the script there - assuming it would need adjustments:

https://www.it-admins.com/prtg-and-vmware-6-7-vcenter-host-hardware-status/

PS: This is a user-script and not officially supported by PRTG - just thought it might help some others with the same issue...

Regards

Florian Rossmark

www.it-admins.com

@Florian Rossmark I'm confused as to what your script actually does. Does it actually get back the hardware statuses or does it just list the number of sensors that are in good/warning/error?

I don't really understand how the script works around the sensor to get the statuses of the hardware as shown here: https://www.paessler.com/manuals/prtg/vmware_virtual_machine_soap_sensor

@Paessler Support Any updates on this?

Aidan

Unfortunately, for now we do not have any news regarding this problem.

Aidan,

Your link points to the regular SOAP API for VM guests. This whole KB entry is about this: https://www.paessler.com/manuals/prtg/vmware_host_hardware_status_soap_sensor

Having said this - I simply created a script that reads each available HW information status from VMware (I can only read what VMware provides - if that's not enough - look in to remote access cards like iDRAC) and reply with the status of each sensor.

Now, if a sensor is in a warning or error state - this will be shown as additional sensor text giving you the chance to use your notification system and display / see what actually is wrong instead of just how many sensors in VMware have a warning/error state.

The original Paessler/PRTG sensor did something similar - actually even less cause it did not show you text based details nor count the sensors in unknown state..

So the script eventually mirrors the original Paessler/PRTG sensor while using the VMware SOAP API - install the PowerShell VMware module on the probe-system for all users:

install-module -name vmware.powercli -AllowClobber 

Hope this helps...

Regards

Florian

www.it-admins.com

We too experienced the problem, that warnings were shown for elements, whose state VMWare could not read. I tried the workaround Felix Saure recommended on 13 Nov. 2018. It worked and the sensors work as expected again. But it is only a workaround, so it still leaves us unsatisfied.

While putting together the string for the "Known Warnings" I saw, that the elements causing the warnings contain "Unknown" in their "healthState". So this information is transferred to PRTG. Is the behaviour of giving a warning for an unknown state wanted by VMWare? This seems reasonable to me, I am just curious.

Regards David

We indeed consider anything that is not OK within the Sensor as a Warning or Down state.

Since it is now March and this has not been fixed, can you at least just add an option to treat unknowns as healthy? That would solve the problem and require the user to opt-in to know that they won't get alerts for "legitimate" unknowns.

Hi there,

Our new version 20.1.57 has this option. With this you should be able to "easily" ignore those messages.

Kind regards,
Birk Guttmann, Tech Support Team

Thanks.

♥

had to exclude these warnings since 6.0 for our HPE blades. After every esxi update I had to edit every device to get it green again.

We have the sames issue with latest update of ESXi host on 6.7 VMware version and have uptodate PRTG Server. Ignoring the warnung message didn't work in our case. Since it is now almost December 2020 and this issue has still not been fixed, can you at least just add an option to treat unknowns as healthy? That would solve the problem and require the user to opt-in to know that they won't get alerts for "legitimate" unknowns.Any update for this issue ?

There is not much else for us we can do apart from preventing unknown issues from the Sensor state, as the issue is between the host hardware and VMware itself. I'm sorry, I know it's unsatisfactory, but we have no proper way to intervene here :(

Hello, we have the same issue. Setting "Do not show unknown states" doesn't work. Do we have to change more settings to remove the warnings for unknown states? Or has "Do not show unknown states" a bug?

Regards, Harald

Hi Harald,

we are not aware of any bug of the ""Do not show unknown states" setting. Can you post a screenshot of the "Warning/Error" message of the sensor.

Kind regards,
Matthias Kupfer - Team Tech Support

Since the update this doesnt appear to be working for us either..

We have put the error into "Know Errors" and set the "Handling Of Unknown States" to "Do not show unknows states" but this doesnt clear the error within the VMWare sensor.

Thanks

Adam

Hi Adam,

please let me know whether you have special characters in your string?

Same issue for me on ESXi 7.0U3 Entperise Plus. HPE DL380 Gen10 Servers. Do Not Show Unkown States is selected. I get the following message on 5 separate ESX Servers:

"Warning. No hardware status is available. Check if the query service is running on the server and if you have the vendor-specific CIM extensions installed.; No hardware status is available. Check if the query service is running on the server and if you have the vendor-specific CIM extensions installed."

I am not using WBEM, I'm using SOAP, but I have tried enabling CIM on a host anyway, just to rule out the possibility of that. I also tried adding the WBEM sensor after enabling CIM and it was unable to add that sensor type.

I have tried both the HPE Custom Image as well as the VMWare Image.

Any updates to this topic? After patching the Hosts to 7.0 Update 3c and vCenter 7.0.3.00300, I noticed the same behavior on ProLiant DL380 Gen10 Servers which where monitored with SOAP. No matter whether CIM is active or inactive, the warning is always issued.

Hi there,

please let me know whether you already checked via the "Result Handling" feature of the Sensor if these warnings are sent by the target device?

Hi, I'm getting similar things on our HPE ProLiant after upgrading it to 7 using HPE ESXi ISO. The warning message I get is "Warning. No hardware status is available. Check if the query service is running on the server and if you have the vendor-specific CIM extensions installed." I tried putting this into the Known Warnings but that isn't working. I only want to ignore this (server is a test bed, not production). Looks at the saved results I have:

0:16:58:44.0956758:
starting
######
0:16:58:44.0996768:
loading serviceContent from session-pool
######
0:16:58:44.1006770:
<RetrieveServiceContentResponse><returnval><rootFolder type="Folder">ha-folder-root</rootFolder><propertyCollector type="PropertyCollector">ha-property-collector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware ESXi</name><fullName>VMware ESXi 7.0.3 build-20328353</fullName><vendor>VMware, Inc.</vendor><version>7.0.3</version><patchLevel>0.55</patchLevel><build>20328353</build><localeVersion>INTL</localeVersion><localeBuild>000</localeBuild><osType>vmnix-x86</osType><productLineId>embeddedEsx</productLineId><apiType>HostAgent</apiType><apiVersion>7.0.3.0</apiVersion><licenseProductName>VMware ESX Server</licenseProductName><licenseProductVersion>7.0</licenseProductVersion></about><setting type="OptionManager">HostAgentSettings</setting><userDirectory type="UserDirectory">ha-user-directory</userDirectory><sessionManager type="SessionManager">ha-sessionmgr</sessionManager><authorizationManager type="AuthorizationManager">ha-authmgr</authorizationManager><serviceManager type="ServiceManager">ha-servicemanager</serviceManager><perfManager type="PerformanceManager">ha-perfmgr</perfManager><eventManager type="EventManager">ha-eventmgr</eventManager><taskManager type="TaskManager">ha-taskmgr</taskManager><accountManager type="HostLocalAccountManager">ha-localacctmgr</accountManager><diagnosticManager type="DiagnosticManager">ha-diagnosticmgr</diagnosticManager><licenseManager type="LicenseManager">ha-license-manager</licenseManager><searchIndex type="SearchIndex">ha-searchindex</searchIndex><fileManager type="FileManager">ha-nfc-file-manager</fileManager><datastoreNamespaceManager type="DatastoreNamespaceManager">ha-datastore-namespace-manager</datastoreNamespaceManager><virtualDiskManager type="VirtualDiskManager">ha-vdiskmanager</virtualDiskManager><ovfManager type="OvfManager">ha-ovf-manager</ovfManager><dvSwitchManager type="DistributedVirtualSwitchManager">ha-dvsmanager</dvSwitchManager><localizationManager type="LocalizationManager">ha-l10n-manager</localizationManager><storageResourceManager type="StorageResourceManager">ha-storage-resource-manager</storageResourceManager><guestOperationsManager type="GuestOperationsManager">ha-guest-operations-manager</guestOperationsManager><vStorageObjectManager type="HostVStorageObjectManager">ha-vstorage-object-manager</vStorageObjectManager><cryptoManager type="CryptoManagerHost">ha-crypto-manager</cryptoManager></returnval></RetrieveServiceContentResponse>
######
Starting WebServiceCall with parameters:

soapBody: <RetrieveProperties xmlns="urn:vim25"><_this type="PropertyCollector">ha-property-collector</_this><specSet><propSet><type>HostSystem</type><pathSet>summary.runtime.healthSystemRuntime</pathSet></propSet><objectSet><obj type="HostSystem">ha-host</obj></objectSet></specSet></RetrieveProperties>

0:16:58:44.1106799:
ResponseFilter: soapenv:Body
######
0:16:58:44.1106799:
Initializing WebRequest
######
0:16:58:44.1296839:
Opening RequestStream
######
0:16:58:44.1376856:
Initializing Stream-writer
######
0:16:58:44.1386861:
Writing Request to Stream: <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><RetrieveProperties xmlns="urn:vim25"><_this type="PropertyCollector">ha-property-collector</_this><specSet><propSet><type>HostSystem</type><pathSet>summary.runtime.healthSystemRuntime</pathSet></propSet><objectSet><obj type="HostSystem">ha-host</obj></objectSet></specSet></RetrieveProperties></soap:Body></soap:Envelope>
######
0:16:58:44.1386861:
WebRequest is: System.Net.HttpWebRequest
######
0:16:58:44.1396863:
Initializing WebResponse
######
0:16:58:44.6017962:
Got a WebResponse: System.Net.HttpWebResponse
######
0:16:58:44.6027963:
Reading Cookie
######
0:16:58:44.6027963:
Reading Response into Stream-reader
######
0:16:58:44.6037967:
Read to end
######
0:16:58:44.6107980:
WebService Response: <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<RetrievePropertiesResponse xmlns="urn:vim25"><returnval><obj type="HostSystem">ha-host</obj><propSet><name>summary.runtime.healthSystemRuntime</name><val xsi:type="HealthSystemRuntime"><systemHealthInfo><numericSensorInfo><name>Disk or Disk Bay 4 C1 P1I Bay 4</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.4.4</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 3 C1 P1I Bay 3</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.3.3</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 2 C1 P1I Bay 2</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.2.2</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 1 C1 P1I Bay 1</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.1.1</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo></systemHealthInfo><hardwareStatusInfo></hardwareStatusInfo></val></propSet></returnval></RetrievePropertiesResponse>
</soapenv:Body>
</soapenv:Envelope>
######
0:16:58:44.6117983:
Filter Payload
######
0:16:58:44.6137991:
Result: 
<RetrievePropertiesResponse><returnval><obj type="HostSystem">ha-host</obj><propSet><name>summary.runtime.healthSystemRuntime</name><val type="HealthSystemRuntime"><systemHealthInfo><numericSensorInfo><name>Disk or Disk Bay 4 C1 P1I Bay 4</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.4.4</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 3 C1 P1I Bay 3</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.3.3</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 2 C1 P1I Bay 2</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.2.2</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo><numericSensorInfo><name>Disk or Disk Bay 1 C1 P1I Bay 1</name><healthState><label>Green</label><summary>The sensor is operating under normal conditions</summary><key>Green</key></healthState><currentReading>1</currentReading><unitModifier>0</unitModifier><baseUnits>unspecified</baseUnits><rateUnits>none</rateUnits><sensorType>storage</sensorType><id>0.4.1.1</id><timeStamp>2022-11-18T05:30:14Z</timeStamp></numericSensorInfo></systemHealthInfo><hardwareStatusInfo></hardwareStatusInfo></val></propSet></returnval></RetrievePropertiesResponse>

######
0:16:58:44.6147990:
Loaded var into xmlDoc
######
0:16:58:44.6248014:
<?xml version='1.0' encoding='UTF-8' ?>
<prtg>
  <result>
    <channel>Normal States</channel>
    <unit>Count</unit>
    <float>0</float>
    <value>4</value>
  </result>
  <result>
    <channel>Warning States</channel>
    <unit>Count</unit>
    <float>0</float>
    <value>0</value>
    <warning>1</warning>
  </result>
  <result>
    <channel>Alert States</channel>
    <unit>Count</unit>
    <float>0</float>
    <value>0</value>
  </result>
  <result>
    <channel>Unknown States</channel>
    <unit>Count</unit>
    <float>0</float>
    <value>0</value>
  </result>
  <text>
    Warning. No hardware status is available. Check if the query service is running on the server and if you have the vendor-specific CIM extensions installed.
  </text>
</prtg>

######
0:16:58:44.6248014:
Duration: 0.5641339
######
Parameters:

user: root

password: *

server: X.X.X.X

option: -m

moid: ha-host

verbose: False

check: False

list: False

fips: False

logFolder: C:\ProgramData\Paessler\PRTG Network Monitor\Logs\sensors\

logFile: C:\ProgramData\Paessler\PRTG Network Monitor\Logs\sensors\Result of Sensor 4206.txt

debugname: Result of Sensor 4206.txt

cleanUpLog: False

exePath: C:\Program Files (x86)\PRTG Network Monitor\Sensor System

xmlns: urn:vim25

apiVersion: 7.0.3.0

type: health

exeName: C:\Program Files (x86)\PRTG Network Monitor\Sensor System\VMwareSensor

NetVersion: v4.0.30319

0:16:58:44.6338039:
Operating System Details

OS Version: 6.2.9200.0

OS Platform: Win32NT

OS SP: 

OS Version String: Microsoft Windows NT 6.2.9200.0

Hi there,

So you entered "Warning. No hardware status is available. Check if the query service is running on the server and if you have the vendor-specific CIM extensions installed." in the known warnings?
If this does not work I suspect the reason is the origin of this error is not the device it's PRTG which is not able to query.