PINGIDENTITY.

COM
TROUBLESHOOTING
Ping Identity Education
PINGFEDERATE LOGS

Copyright © 2015 Ping Identity Corp. All rights reserved.3

PINGFEDERATE LOGGING OPTIONS
§ We use Log4j 2
§ ArcSight
– Log into CEF
format
– Import into
ArcSight
§ LogRhythm
§ Database
– Oracle
– MS SQL Server
– MySQL

Copyright © 2015 Ping Identity Corp. All rights reserved.4

ADMIN LOG
§ Records events about PF Admin activity
• Password change/reset
• Account activation/deactivation
• Login attempts
• Data store create/modify/delete
• Certificate management actions
• Connection create/modify/delete, etc

Sample:
2013-08-19 18:20:48,390 DEBUG [AuditLogger] Administrator | UserAdmin,Admin,CryptoAdmin | Login
attempt | Login was successful

2013-08-19 18:21:29,109 DEBUG [AuditLogger] Administrator | UserAdmin,Admin,CryptoAdmin | Data store

created | LDAP-67771C232937987C915999B1E1D7120215F6689B

Copyright © 2015 Ping Identity Corp. All rights reserved.5

SERVER LOG
§ Records all PF runtime and administrative server activity
• Information is also sent to terminal or command window running the PingFederate server
• Main troubleshooting log used by Support and Client Services

• Send *entire* server.log to support (not just snippets)

Sample:
2013-08-20 16:53:45,635 INFO [com.pingidentity.appserver.jetty.SocketConnector] Not starting
listener class com.pingidentity.appserver.jetty.SocketConnector because port was set to -1

2013-08-05 16:45:34,788 tid:d381f3492 INFO

[org.sourceid.websso.servlet.IntegrationControllerServlet]
org.sourceid.websso.servlet.RenderPageException: Unable to resume processing because saved
state was not found for key: hIXsF0tcwpj9wSWN6ni0xD - rendering
state.not.found.error.page.template.html

§ Note: The word Exception is not necessarily a problem

Copyright © 2015 Ping Identity Corp. All rights reserved.6

TRANSACTION LOG
§ Configurable both globally and per-connection
§ 4 Modes:
• None
• Standard (timestamp; hostname:port; Connection ID; message type; etc.)
• Enhanced (Standard plus SAML_SUBJECT; binding; Signature Policy; etc.)
• Full (Enhanced plus all XML messages)

Sample:
2013-08-24 13:34:28,546 | 192.168.238.132:9041 | S | Sent Request | Connection
ID: idp:saml2 (IDP) | AuthnRequest | Target URL:
http://sm6.pinggcslab.sp.com:81/PFIsapiSample/Default.aspx?LOGON_USER=PFuser |
Endpoint: http://sm6.pinggcslab.idp.com:9030/idp/SSO.saml2

Copyright © 2015 Ping Identity Corp. All rights reserved.7

TRANSACTION LOGGING MODE –
CONNECTION

Copyright © 2015 Ping Identity Corp. All rights reserved.8

TRANSACTION LOGGING MODE –
GLOBALLY

Copyright © 2015 Ping Identity Corp. All rights reserved.9

REQUEST LOG
§ <date>.request.log records all HTTP requests for the given
date
§ Contains Jetty (web server) log messages
192.168.89.128 - [21/Sep/2013:14:21:33 +0000] "GET /IdpSample/ HTTP/1.1" 302 0
192.168.89.128 - [21/Sep/2013:14:21:38 +0000] "POST /pf-
ws/services/SSODirectoryService HTTP/1.0" 200 0
192.168.89.128 - [21/Sep/2013:14:21:35 +0000] "GET /IdpSample/MainPage HTTP/1.1"
200 3860

§ Errors caused by URL typos may be found here and not in

the server log

Copyright © 2015 Ping Identity Corp. All rights reserved.10

AUDIT LOG
§ Provided for security analysis and regulatory compliance purposes
§ Elements configured in log4j.xml file
§ Sample elements:
– Target SP app
– User attributes sent/received
– Type of transaction (e.g. SSO)
– Protocol (e.g. SAML 2.0)
– Status of transaction: success or failure
– etc

Copyright © 2015 Ping Identity Corp. All rights reserved.11

ADMIN API LOG
§ Actions performed using the administrative API
– Time the event occurred on the PingFederate server
– Administrator username performing the action
– Authentication method
– Client IP
– HTTP method
– REST endpoint
– HTTP status code

Copyright © 2015 Ping Identity Corp. All rights reserved.12

MANAGING LOG FILES
§ Default directory: <pf_install>/pingfederate/log
– Location can be changed in run.properties
› pf.log.dir property

§ Rollover:
– The transaction.log, the admin.log, the audit.log and the provisioner.log files roll
over at midnight each day.
› These files can become quite large, back up or remove older files on a routine basis.
– Other PingFederate log files roll over when they reach 10MB.
› The five most recent files are kept before overwriting the oldest.
– This number can be changed in the log4j.xml file

Copyright © 2015 Ping Identity Corp. All rights reserved.13

SUPPORT RESOURCES

Copyright © 2015 Ping Identity Corp. All rights reserved.14

ONLINE RESOURCES: SUPPORT CENTER
PORTAL
§ Main help portal:
– http://docs.pingidentity.com

§ Knowledge base:
– https://ping.force.com/Support/PingIdentityKnowledge
Home

Copyright © 2015 Ping Identity Corp. All rights reserved.15

ONLINE RESOURCES: VIDEO LIBRARY
https://docs.pingidentity.com/bundle/ping_sm_videoLibrary/

Copyright © 2015 Ping Identity Corp. All rights reserved.16

 ONLINE RESOURCES: Q&A COMMUNITY
https://ping.force.com/Support/Collaborate

Copyright © 2015 Ping Identity Corp. All rights reserved.17

USER GROUPS
https://ping.force.com/Support/PingIdentityUserGroup

Copyright © 2015 Ping Identity Corp. All rights reserved.18

TROUBLESHOOTING PHILOSOPHY 101

Copyright © 2015 Ping Identity Corp. All rights reserved.19

TROUBLESHOOTING METHODOLOGY
§ Think about the problem on a high level
§ User experience
– Don’t ask someone, ‘what do you think is the problem?’
– Instead ask the person to explain what they SEE
– Ask “What are the symptoms?”

§ Don’t make assumptions…start with a clean slate

§ Check the URL – which side is the error on?
§ Start with the symptoms, and work backwards
§ Devise a theory to explain the behavior

Copyright © 2015 Ping Identity Corp. All rights reserved.20

SOME NOTES…
§ Google!
– PingFederate is still a java application
› Search for exceptions in logs for classes that aren’t proprietary (jetty, jgroups…)
– Oracle has links to troubleshooting SSL with Java
§ PingFederate documentation has common issues and steps to resolve
– https://documentation.pingidentity.com/pingfederate/pf80/index.shtml#adminG
uide/concept/troubleshooting.html
§ Support Data Collector Tool
– https://community.pingidentity.com/PingIdentityArticle?id=kA340000000Gt7KC
AS

Copyright © 2015 Ping Identity Corp. All rights reserved.21

USEFUL NOTE – HEARTBEAT.PING
§ <pingfederate>/pf/heartbeat.ping
§ Customizable via heartbeat.page.template
§ Can be configured to include system information
such as memory and CPU usage

Copyright © 2015 Ping Identity Corp. All rights reserved.22

QUICK NOTE ON TROUBLESHOOTING IWA

§ Common issue – if the IWA or Kerberos adapter

isn’t working with Internet Explorer, check the IE
security settings:
– Verify that defaults are enabled
– Automatic logon only in the Intranet zone
– Enable Integrated Windows Authentication

Copyright © 2015 Ping Identity Corp. All rights reserved.23

TROUBLESHOOTING WITH LOGS

Copyright © 2015 Ping Identity Corp. All rights reserved.24

LOGS AND EVENTS FROM PINGFEDERATE
§ PingFederate log files
– Server.log is the main
source
› Search for PingFederate error
reference, if provided
– Request.log for possible
URL typos
– IWA-NTLM log for possible
failed authentication
Ways to consume this data
– View logs directly in a text
editor
– Use the logfilter script
– Logging consolidation tools
(Splunk)

Copyright © 2015 Ping Identity Corp. All rights reserved.25

LOG FILTER
§ Sorts through all the server logs in the log directory
§ logfilter.bat|sh
– In <pf_install>/pingfederate/bin

§ Returns lists of log entries based on

– Entity ID and Subject

– Tracking ID

– Session Cross-reference ID
– Not reference ID

http://documentation.pingidentity.com/display/PF610/Using+the+Server+Log+Filter

Copyright © 2015 Ping Identity Corp. All rights reserved.26

LOGGING CONSOLIDATION TOOLS
§ Consolidate your
PingFederate logs
for monitoring,
reporting and
analyzing (such as
Splunk)
– Generate
visualizations and
reports
– Real-time auditing &
analysis

Copyright © 2015 Ping Identity Corp. All rights reserved.27

SPLUNK APP FOR PINGFEDERATE
Real-time

splunk-audit.log

§ PingIdentity has created a Splunk application to process

PingFederate audit logs
– splunk-audit.log
§ PingFederate needs to be modified to write audit and server in
Splunk format
– Edit log4j.xml to include SecurityAudit2Splunk appender:
../pingfederate/server/default/conf/log4j.xml
› org.sourceid.websso.profiles.sp.SpAuditLogger
› org.sourceid.websso.profiles.Idp.IdpAuditLogger
Copyright © 2015 Ping Identity Corp. All rights reserved.28
SPLUNK APP FOR PINGFEDERATE
• Configure Splunk to
consume splunk-audit.log

§ Splunk for Pingfederate app will

read splunk-audit.log to create
real-time report
§ App can be downloaded from
Splunk apps store
§ App runs within Splunk server
SPLUNK APP FOR PINGFEDERATE (MAIN
PAGE)

Copyright © 2015 Ping Identity Corp. All rights reserved.30

 SPLUNK APP (DETAILED ANALYSIS)

Detailed Analysis

• Can perform fine grained search and sorting

• For example, your interested in SSO traffic from a specific subject/person
• Can save and print your result
SPLUNK APP (VISUAL CUSTOMIZATION)

§ Visualizations can be customized to personal needs

– Can select from existing report templates or create your own
– Reports can be displayed in a number of styles: pie, graph, etc
– Can add new panels to main page
§ Can Generate PDF
§ Can Print
Copyright © 2015 Ping Identity Corp. All rights reserved.32
NOTE: PINGFEDERATE MONITORING
OPTIONS
§ JMX
§ SNMP
– Get – Total
Transaction
Count
– Get – Total
Failed
Transaction
Count
– Trap – Server
Heartbeat

Copyright © 2015 Ping Identity Corp. All rights reserved.33

TROUBLESHOOTING WITH TRACES

Copyright © 2015 Ping Identity Corp. All rights reserved.34

HTTP HEADER TRACING TOOLS
§ Tools to help us answer questions about what’s going on during a SSO
transaction
– Where did this request come from?
– Where is it going to?
– Were there any cookies set? What are their names and domains?
§ Commonly used tools:
– LiveHTTPHeaders (Firefox add-on)
– SAML Tracer (Firefox add-on)
– Fiddler
– TamperData
– IEHTTPHeaders (Internet Explorer)

Copyright © 2015 Ping Identity Corp. All rights reserved.35

 LIVEHTTPHEADERS – EXAMPLE TRACE
https://idp.company.com:9031/idp/startSSO.ping?PartnerSpId=sp:partner:saml2&TargetResource=https://service.partner.com:903
1/SpSample&IdpAdapterId=IWAAUTH
GET SSO request
/idp/startSSO.ping?PartnerSpId=sp:partner:saml2&TargetResource=https://service.partner.com:9031/SpSample&IdpAdapterId=IWAA
UTH HTTP/1.1
Host: idp.company.com:9031
----------------------------------------------------------
https://idp.company.com:9031/idp/lYM3t/resumeSAML20/idp/startSSO.ping
GET /idp/lYM3t/resumeSAML20/idp/startSSO.ping HTTP/1.1 User authenticates and redirects back to resume path
Host: idp.company.com:9031
....
----------------------------------------------------------
https://sp.partner.com:9031/sp/ACS.saml2
POST /sp/ACS.saml2 HTTP/1.1
Host: sp.partner.com:9031
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.25) Gecko/20111212 Firefox/3.6.25 Redirects user to SP with
…………
Content-Type: application/x-www-form-urlencoded
a SAML assertion
Content-Length: 4442
SAMLResponse=PHNhbWxwOlJlc3BvbnNlIFZlcnNpb249IjIuMCIgSUQ9Im4zLTA4LTI2VDE1OjAzOjQ5LjY0MFoiIERlc3RpbmF0aW9uPSJ…….
----------------------------------------------------------
https://service.partner.com:9031/SpSample
GET /SpSample HTTP/1.1
Host: service.partner.com:9031
…………
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Redirects user to application
Keep-Alive: 115 with a Opentoken
Connection: keep-alive
Referer: https://idp.company.com:9031/idp/lYM3t/resumeSAML20/idp/startSSO.ping
Cookie: SPopentoken=T1RLAQKWYQH6n3s3bxpZtPjhNomEr158MBCjVn7SKhUAgk7oex8ZztAEAADAFX5E2eZ2ZzqXhr…..

Copyright © 2015 Ping Identity Corp. All rights reserved. 36

WIRESHARK

Copyright © 2015 Ping Identity Corp. All rights reserved.37

PROTOCOL ANALYSIS & PACKET TRACING
§ WireShark
– open-source network packet analysis tool

– Network troubleshooting and analysis

– Can view HTTP traffic as well as network-level activity

– Useful for server-to-server communications

– Often helpful troubleshooting IWA issues

§ Need to disable HTTPS to see PingFederate activity

– Or you need to obtain the private key of the server

§ Powerful tool, but perhaps ‘overkill’ for simple troubleshooting of PingFederate

– Helps to know TCP/IP at a protocol level

Copyright © 2015 Ping Identity Corp. All rights reserved.38

LAB: TROUBLESHOOTING
§ View sections of server.log
§ Fix broken PingFederate instances
– No answers provided!

Copyright © 2015 Ping Identity Corp. All rights reserved.39

PINGIDENTITY.COM