PagerDuty HTTP notification

Hello,

we will put it onto the wishlist.

best regards.

Did some further testing on this.

The server I am sending to does not require Content-type set for JSON, so this should work as-is (but does not).

My notification has HTTP Action enabled with the following:

If I telnet to their server directly and post as below, it works.

----------------------Submit by telnet to events.pagerduty.com 80----------------------
POST /generic/2010-04-15/create_event.json HTTP/1.1
Host: events.pagerduty.com
Content-Length: 215

{ "service_key": "ServiceKeyDeleted", "incident_key": "prtg/test", "event_type": "trigger", "description": "Test event without placeholders", "details": {  "status": "status", "device": "device" } }

----------------------End----------------------

Can you shed any light on what else PRTG might be adding to the postdata? PRTG logs only show "HTTP/1.1 400 Bad Request" for a test notification. Any way I can get to the data being returned?

I think the problem is, that PRTG encodes the POSTDATA and that may break pagerduty's stuff. We will look into creating pagerduty support in one of the next versions.

Thanks for the update and for giving this some consideration.

Even a simple option to encode or not encode the postdata on an HTTP notification would be a convenient shorter-term solution.

It's also be helpful to have state change notification options for "Down Acknowledged" and "Paused" (this would let us do our acknowledgments in PRTG and then pass that notification out ).

Thanks again.

Would love to have this capability with PagerDuty, we ran into the encode issue too with Leftronic.

Hello,

Is there anything new on this subject?

I need to do HTTP Notification with PagerDuty and I have only HTTP/1.1 400 Bad Request logs when i try to test http notification on pagerduty.

Thanks.

Any update on "We will look into creating pagerduty support in one of the next versions." ?

I'm sorry, but due to a low number of requests it was postponed, but it is still on the list. Please bear with us.

I'm a little late to the party, and this isn't exactly a clean way of doing this, but the PagerDuty Nagios API will accept other content-types than JSON. You would just create a Nagios service within PagerDuty.

http://events.pagerduty.com/nagios/2010-04-15/create_event?pd_nagios_object=host&CONTACTPAGER=<pagerduty_api_key>&NOTIFICATIONTYPE=PROBLEM&HOSTNAME=<hostname>&HOSTSTATE=down

Cheers, Ryan

We are also very interested in using this feature, and would like to see it implemented as soon as possible.

-Groundspeak IT

Hi

I would like this too. I have had a bit of a stab at it and come up with the following powershell v3 script

You need to setup a generic api service in pagerduty and update the script with your personal key

Param(
  [string]$Sitename,
  [string]$Device,
  [string]$Name,
  [string]$Status,
  [string]$Down,  
  [string]$message
)
$Description = "$Device $Name $Status $Down"
	$Body = @{
	  "service_key"="<servicekey>";
      "event_type"="trigger";
      "description"=$Description;
      "client"=$Sitename;
      "client_url"="<url>";
      "details"="$Message"
	} | ConvertTo-Json
	Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body 
	#$Body | out-file ".\pager.txt"

Now this works flawlessly when running via powershell, when running it via the notification test, the logs come up with "Status sending EXE:Ok".

However I do not get anything at all.

Any ideas on how to debug this ?

Thanks

Ok, have modified the script a little bit

The trick seems to be that the PRTG notification setting does not enclose each passed variable in single quotes, so items with spaces in them break the script.

It now looks like (this is what works for us, and yes more can be added, but at present we don't need the extra info, we just need to know something has happened)

Param(
  [string]$probe,
  [string]$device,
  [string]$name,
  [string]$status,
  [string]$down,  
  [string]$message
)

$Description = "CRITICAL: $probe $device $name $status $down"
	$Body = @{
	  "service_key"="<apiservicekey>";
      "event_type"="trigger";
      "description"=$Description;
      "client"="$probe";
      "client_url"="<url>";
      "details"="$message"
	} | ConvertTo-Json
	Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body 
	#$Body | out-file ".\pager.txt"
	

In the prtg notification settings the variables under EXE action need to be specified as follows (in the parameter field)

'%probe' '%device' '%name' '%status' '%down' '%message'

Use at your on risk etc

Hopefully this helps someone Thanks

PRTG Guys, I don't understand why you don't deliver a simple integration to REST API's it doesn't require that much and they are all the same json based API's

please consider them.

Integration with PagerDuty, OpsGenie, and VictorOps native WebAPI calls would be an excellent feature. It feels like a majority of PRTG's competitors have exiting integration with these products. It would be a good sales feature, and also helpful to the pre-existing userbase.

Felipe, thank you very much for the feedback. I'll put this on the wish list!

If anyone is still looking for a resolution for this, I was able to overcome the headers. I configured a Nagios Service in PagerDuty, then I hit the nagios create_event. Below is how I was able to succesfully send a new trigger and resolve a trigger from PRTG

URL - Make sure it's HTTPS

https://events.pagerduty.com/nagios/2010-04-15/create_event

Postdata to trigger event

NOTIFICATIONTYPE=PROBLEM&pd_nagios_object=service&HOSTNAME=HOSTNAME&SERVICESTATE=DOWN&HOSTADDRESS=HOSTADDRESS&CONTACTPAGER=<YOUR API KEY>&pd_version=1.0&SERVICEDESC=TESTING

Postdata to resolve the event

NOTIFICATIONTYPE=RECOVERY&pd_nagios_object=service&HOSTNAME=HOSTNAME&SERVICESTATE=DOWN&HOSTADDRESS=HOSTADDRESS&CONTACTPAGER=<YOUR API KEY>&pd_version=1.0&SERVICEDESC=TESTING

Agreed. Bi-directional VictorOps, OpsGenie, and PagerDuty integration is something other solutions have and we are considering switching platforms for this support.

Dear response3

We collect user feedback to help us decide which features would help the most users. We continue to improve our notification emails, however integration with PagerDuty and similar tools is not something we are going for in the foreseeable future.

That does not mean it is off the table forever.

OpsGenie has instructions on how to integrate PRTG and OpsGenie: https://www.opsgenie.com/docs/integrations/prtg-integration

To expand (or update) on Rhyse posted way above:

I have 2 notifications, a "create pagerduty incident" and "resolve pagerduty incident"

Each of these has a executes a custom script.

The script for "create" is

Param(
  [string]$probe,
  [string]$device,
  [string]$name,
  [string]$status,
  [string]$down,  
  [string]$message
)

$Description = "CRITICAL: $probe $device $name $status $down"
	$Body = @{
	  "service_key"="<api key>";
      "event_type"="trigger";
      "incident_key"=$device;
      "description"=$Description;
      "client"="$probe";
      "client_url"="<url>";
      "details"="$message"
	} | ConvertTo-Json
	Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body 
	#$Body | out-file ".\pager.txt"

and for the "resolve":

Param(
  [string]$probe,
  [string]$device,
  [string]$name,
  [string]$status,
  [string]$down,  
  [string]$message
)

$Description = "CRITICAL: $probe $device $name $status $down"
	$Body = @{
	  "service_key"="<api key>";
      "event_type"="resolve";
      "incident_key"=$device;
      "description"=$Description;
      "client"="$probe";
      "client_url"="<url>";
      "details"="$message"
	} | ConvertTo-Json
	Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body 
	#$Body | out-file ".\pager.txt"

You will note the main difference is I am submitting the incident_key (using the deviceID). This means Pageryduty can resolve the incident if PRTG tells it.

Dear AndrewG,

As always, thank you for your contribution.

Best,
Sebastian

I grew tired of the issue of email and the not so great integration into pagerduty, difficult to read and sometimes dodgy on it's functionality. So I wrote a script to handle this and then had to update the script for their new API. You would need to create an API key with PagerDuty then switch the notification type to custom EXE in PRTG and just use a powershell script to call pagerduty. I am sure I left some things defaulted for my environment (I tried to remove all the defining stuff I just hard coded it since it doesn't change). This hopefully will help a few others get this rolling and I have suggested this to pagerduty as the preferred or alternative integration. If the sensor comes back down the script checks to see if alert exists and if it does then it skips saying it already exists unless it's paused or warning, or anything but down really, in which case it closes the ticket. I am sure there are optimizations that could be made it works right now.

Hope this helps.

#################  Start Here! #########################################
Param(
    [string]$Probe,
    [string]$Device,
    [string]$Name,
    [string]$Status,
    [string]$Down,
    [string]$DateTime,
    [string]$LinkDevice,
    [string]$Message
)

$Hour = Get-Date -format "HH:mm"
$text = "$Probe-$Device-$Name"
$Title = $text
$APIKey = "API_Key_20_character"
$servicekey = "service_key_32_characters_string"
$companyname = "subdomain"
$PagerdutyEventURL = "https://events.pagerduty.com/generic/2010-04-15/create_event.json"
$PagerdutyAPIURL = "https://api.pagerduty.com/incidents"

$pagerdutystatus1 = "triggered"
$pagerdutystatus2 = "acknowledged"
$Service = "SERVICE"
$data= "?statuses[]=$pagerdutystatus1`&statuses[]=$pagerdutystatus2`&time_zone=UTC&sort_by:desc&include[]=services:$Service"
$CALL0 = Invoke-WebRequest -Method GET -ContentType "application/vnd.pagerduty+json;version=2" $PagerdutyAPIURL/count$data -Headers @{"Authorization"=("Token token=" + $APIKey)}
$Response0 = $CALL0.content | ConvertFrom-JSON
$CALL1 = Invoke-WebRequest -Method GET -ContentType "application/vnd.pagerduty+json;version=2" $PagerdutyAPIURL$data -Headers @{"Authorization"=("Token token=" + $APIKey)}
$Response1 = $CALL1.content | ConvertFrom-JSON
write-host $Response0.triggered "Triggered and" $Response0.acknowledged "acknowledged"
write-host "Searching for incident_key or title matching ""$Title"""

function createincident 
{
	$Body= (
	 New-Object PSObject |
	 Add-Member -PassThru NoteProperty "service_key" "$servicekey" |
	 Add-Member -PassThru NoteProperty "event_type" "trigger" |
	 Add-Member -PassThru NoteProperty "description" "$Title" |
	 Add-Member -PassThru NoteProperty "incident_key" "$Title" |
	 Add-Member -PassThru NoteProperty "client" "PRTG" |
	 Add-Member -PassThru NoteProperty "client_url" "$LinkDevice" |
	 Add-Member -PassThru NoteProperty details (
	  New-Object PSObject |
	  Add-Member -PassThru NoteProperty "Message" "$Message" |
	  Add-Member -PassThru NoteProperty "Device" "$Device" |
	  Add-Member -PassThru NoteProperty "Name" "$Name" |
	  Add-Member -PassThru NoteProperty "Down" "$Down" |
	  Add-Member -PassThru NoteProperty "DateTime" "$DateTime" |
	  Add-Member -PassThru NoteProperty "Status" "$Status" |
	  Add-Member -PassThru NoteProperty "Probe" "$Probe"
	 ) |
	 Add-Member -PassThru NoteProperty contexts @(
	   (
	   New-Object PSObject |
	   Add-Member -PassThru NoteProperty -name "type" -value "link" |
	   Add-Member -PassThru NoteProperty -name "href" -value "$LinkDevice"
	   )
	   (
	   New-Object PSObject |
	   Add-Member -PassThru NoteProperty -name "type" -value "link" |
	   Add-Member -PassThru NoteProperty -name "href" -value "https://$companyname.pagerduty.com" |
	   Add-Member -PassThru NoteProperty -name "text" -value "View the incident on PagerDuty"
	   )
	 )
	) | ConvertTo-JSON -Compress

	$CALL2 = Invoke-WebRequest -Method POST -ContentType "application/json" -Body $Body -URI $PagerdutyEventURL
	$Response2 = $CALL2.content | ConvertFrom-JSON
	write-host $Response2.incident_key "created at $Hour"
}
If ($Response0.triggered -eq 0 -and $Response0.acknowledged -eq 0) {
	If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
		write-host "No open or acknowledged alerts! Whoo Hoo... D'oh!"
		createincident
		}
	else 
	{
		write-host "No incidents found resolving impossible"
	}
}
else
{
	If ($Response1.incidents.incident_key -contains $Title) {
		$Incidents = $Response1.incidents
		If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
			write-host "PagerDuty Ticket ""$Title"" already exists exiting"
		}
		else 
		{
		Foreach ($Incident in $Incidents | Where {$_.incident_key -eq $Title}) {
			write-host "******************** Start Of Incident Information ********************"
			write-host "id = " $Incident.id
			write-host "incident_number =" $Incident.incident_number
			write-host "incident_key =" $Incident.incident_key
			write-host "created_at =" $Incident.created_at
			write-host "status =" $Incident.status
			write-host "html_url =" $Incident.html_url
			write-host "urgency =" $Incident.urgency
			write-host "title =" $Incident.title
			write-host "description =" $Incident.description
			write-host "urgency =" $Incident.urgency
			write-host "********************* End Of Incident Information *********************"

				$Resolve = (
				New-Object PSObject |
				 Add-Member -PassThru NoteProperty "service_key" "$servicekey" |
				 Add-Member -PassThru NoteProperty "event_type" "resolve" |
				 Add-Member -PassThru NoteProperty "description" "$Title" |
				 Add-Member -PassThru NoteProperty "incident_key" "$Title" 
				) | ConvertTo-JSON -Compress
				
				write-host "PagerDuty Ticket $Title exists, resolving"
				$CallResolve = Invoke-WebRequest -Method POST -ContentType "application/json" -Body $Resolve -URI $PagerdutyEventURL
				$ResponseResolve = $CallResolve.content | ConvertFrom-JSON
				write-host "Ticket with incident_key"$ResponseResolve.incident_key"attempted ticket closing result:" $ResponseResolve.status
				exit
			} 
		}
	}
	else
	{
		If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
			write-host "PagerDuty Ticket ""$Title"" does not already exist"
			createincident
		}
		else
		{
		write-host "PagerDuty Ticket ""$Title"" does not exist and cannot be closed"
		}
	}
}
##################  End Here! ##########################################

Another vote for implementing bi-directional acks with PRTG and PagerDuty. The PD-PTRG integration with de-dedupe using a RegEx works OK, but it'd be so much better to only have to ack an incident in one place...

Another vote for implementing bi-directional alerts/acks with PagerDuty. It's been almost 10 years, any way this is coming any time soon?