Troubleshoot steps when Integration Services are Hung

Troubleshoot steps when Integration Services are Hung

When you are unable to connect Integration Service from client, make sure Integration Service is available in Administrator Console

Login to one of the nodes in the IS grid and try pmcmd connect to Integration Service

pmcmd> connect  -sv   -d -u -p

Please share the output.

Please collect below information:-

1.Stack traces using pmstack on pmserver process

     pmstack -p

pmstack should be available in $INFA_HOME/tools/debugtools/pmstack directory. Refer to KB 13652 for more information.

2.List of open files using lsof on pmserver process

     lsof -p

You need to redirect the output to file as shown below:

     lsof -p 324564 > lsof_324564_1.txt

3.System calls using strace on pmserver process

     strace -ttTfo strace_pmserver.txt -p         (Linux)
     truss -faedEl -o truss_pmrepagent.txt -p     (AIX/Solaris)

You need to end strace manually. Let it run for two minutes and then end it.
Please make sure you run each of above commands minimum four times with an interval of two minutes.
Please note that you cannot run pmstack and strace on a process at the same time. You could run strace between pmstack intervals.

4.Core file using gcore on pmserver process on all the nodes in grid(only once)

     gcore -o pmserver_dump

Informatica support needs libscollector output from pmserver core file and not the actual core file. You could refer KB 144522 to run libscollector. This can be done at the end, once you are done collecting all the details.

You need to run all above commands on all the nodes in the grid.

5.Stack traces on associated Repository Service using pmstack

6.jstack on the node Java process. You need to collect this on all the nodes in the grid.

jstack -l > jstack1.txt

jstack should be available in $INFA_HOME/tools/debugtools/java/bin directory.

You need to run it four times with an interval of one minute.

7.netstat output on all the nodes in the grid

netstat -peano > netstat.out

8.Domain, Repository and Integration Service logs for the day separately.

9.Node logs(catalina.out, exceptions.log, node.log, ispLogs.log , node_jsf.log from $INFA_HOME\logs\ directory) from all the nodes