Troubleshoot steps when Integration Services are Hung
When you are unable to connect Integration Service from client, make sure Integration Service is available in Administrator Console
Login to one of the nodes in the IS grid and try pmcmd connect to Integration Service
pmcmd> connect -sv -d -u -p
Please share the output.
Please collect below information:-
1.Stack traces using pmstack on pmserver process
pmstack should be available in $INFA_HOME/tools/debugtools/pmstack directory. Refer to KB 13652 for more information.
2.List of open files using lsof on pmserver process
You need to redirect the output to file as shown below:
lsof -p 324564 > lsof_324564_1.txt
3.System calls using strace on pmserver process
strace -ttTfo strace_pmserver.txt -p (Linux)
truss -faedEl -o truss_pmrepagent.txt -p (AIX/Solaris)
You need to end strace manually. Let it run for two minutes and then end it.
Please make sure you run each of above commands minimum four times with an interval of two minutes.
Please note that you cannot run pmstack and strace on a process at the same time. You could run strace between pmstack intervals.
4.Core file using gcore on pmserver process on all the nodes in grid(only once)
gcore -o pmserver_dump
Informatica support needs libscollector output from pmserver core file and not the actual core file. You could refer KB 144522 to run libscollector. This can be done at the end, once you are done collecting all the details.
You need to run all above commands on all the nodes in the grid.
5.Stack traces on associated Repository Service using pmstack
6.jstack on the node Java process. You need to collect this on all the nodes in the grid.
jstack -l > jstack1.txt
jstack should be available in $INFA_HOME/tools/debugtools/java/bin directory.
You need to run it four times with an interval of one minute.
7.netstat output on all the nodes in the grid
netstat -peano > netstat.out
8.Domain, Repository and Integration Service logs for the day separately.
9.Node logs(catalina.out, exceptions.log, node.log, ispLogs.log , node_jsf.log from $INFA_HOME\logs\ directory) from all the nodes