Friday, November 21, 2014

ConfigEntine task "wp-create-ldap" failed with SOAP connector problem.

[timestamp] ssl.disable.url.hostname.verification.CWPKI0027I
[timestamp] Client code attempting to load security configuration
Could not access WebSphere profile using: username=wpsadmin password=PASSWORD_REMOVED portNumber=10025 ADMC0016E: The system cannot create a SOAP connector to connect to host at port 10025.
Caused by: java.lang.reflect.InvocationTargetException
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: PKIX path validation failed: The certificate expired at Fri Aug 19 05:01:01 EDT 2011; internal cause is: NotAfter: Tue Oct 01 05:01:01 EDT 2014;

Using wsadmin to connect the server with type SOAP, the same exceptions prevented the correct connection.


The default chained certificate has problems.

Resolving the problem

In WebSphere Application Server (WAS) version 7, a default chained certificate was introduced as a personal certificate. This certificate was created during profile set up and should be valid for a year. It should be renewed when its validity date expires.
When running ConfigEngine, it acts as a client that connects to the server through SOAP connector. It takes the parameters configured in /properties/ssl.client.props that points to the key store file containing the default personal certificate and residing at /etc/key.p12.
To renew the certificate, take the following steps
- Log in to the Integrated Solutions Console as the primary WAS administrative user;
- Navigate to Security > SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore> Personal certificates;
- Check the box besides "Default", and hit button "Renew". You should see the expiration date now is updated for another year.
- Restart servers.

Monday, November 17, 2014

How to uninstall IBM DB2 Content Manager Version 8.3 products if fix packs are also installed


When you run the product uninstall program, you see the following error:

"You must uninstall the most recently installed fix pack before running this uninstall."


Fix packs must be uninstalled from the system in reverse order before you can uninstall the product.


To uninstall the latest fix pack, run the command:

where VERSION is the version number of the fix pack.

Tip: To determine the current version installed for each IBM® DB2® Content Manager product, run the command IBMCMROOT/bin/cmlevel.

If multiple fix packs have already been applied, run the uninstallUpdate command for each fix pack starting with the latest fix pack and then continuing in reverse order. For example, if you have DB2 Content Manager fix pack 1, fix pack 3, and fix pack 6 installed, you should first uninstall fix pack 6, then fix pack 3, and then fix pack 1.

Once all fix packs have been uninstalled, run the uninstall program for the product you want to remove.
For DB2 Content Manager:
Run IBMCMROOT/_uninstCM/uninstall-cm.exe
For DB2 Information Integrator for Content:
Run IBMCMROOT/_uninstII4C/uninstall-ii4c.exe
For eClient:
Run IBMCMROOT/_uninstEC/uninstall-ec.exe

Wednesday, November 5, 2014

Troubleshooting WebSphere Application Server issues in Sametime Advanced 8

Where to look for errors (SystemOut, SystemErr, ffdc)
Logging/tracing usually found in /WebSphere/AppServer/profiles/profile_name/logs/. See Logging and tracing at the Info Center.

Server logs: 
Look in system error logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemErr.log
Look in system out logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemOut.log
ffdc logs: 
Look in ffdc logs, found in /WebSphere/AppServer/profiles/profile_name/logs/ffdc

How to look for CPU heap issues for WAS

To dump the javacore and/or heapdump on WebSphere:

(start the wsadmin console... you'll be prompted for login/pwd credentials...)
cd \ProgramFiles\IBM\WebSphere\AppServer\bin

(setup for the DUMPS... specifying application server to dump...)
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

(when ready to dump... execute the following to get a javacore file...)
wsadmin> $AdminControl invoke $jvm dumpThreads


(when ready to dump... execute the following to get a heapdump file...)
wsadmin> $AdminControl invoke $jvm generateHeapDump

javacore/heapdump file will be in the following directory:

How to monitor CPU for WebSphere Application Servver
See How to monitor CPU for EB.

How to turn on tracing

You can find on the following page Setting a diagnostic trace on a serverexternal link how to turn on tracing for the following:
  • How to turn on logging when looking for persistent chat errors
  • How to turn on logging when looking for vmm/ldap errors
  • How to turn on logging when looking for skilltap errors
How to turn on logging when looking for db errors:
Choose this detail level

How to turn on tracing for vmm issues
To look for issues with ldap we will need to enabled wmm tracing. In the admin console, turn on tracing for "*=all" at level FINEST. You should then see a trace.log file (where system.out and system.err live) with this tracing inside.

How to turn on Performance Monitoring Infrastructure PMI

Something to read before setting up:

This is a good page for reading on what to monitor when using PMI Monitoring overall system healthexternal link

Setting it up:
  1. In WebSphere ISC Console, go to Monitoring and Tuning - Performance Monitoring Infrastructure (PMI).
  2. Click server1.
  3. Enable either "Basic" monitoring or "Custom".

Basic monitoring should provide us with the 3 things we are interested in:
  • Number of DB Connections
  • Number of JMS Connections
  • Number of HTTP Sessions

If you are going to do "Custom" monitoring, enable the following statistics to get # of DB/JMS/HTTP connections/sessions: JDBC Connection Pools.CreateCount, JCA Connection Pools.CreateCount, Servlet Session Manager.LiveCount.

Once PMI is enabled you can look at the "Current Activity. To do this:
  1. In WebSphere ISC Console, go to Monitoring and Tuning - Performance Monitoring Infrastructure (PMI).
  2. Click Performance Viewer.
  3. Click Current Activity.
  4. Click server1.

From here on you can click on Summary Reports or Performance Modules which is more fine-grained. In Performance Modules, you can select the 3 statistics we are interested in. The picture below shows how to expand the trees and select the appropriate statistics.

Besides looking at Current Activity, you can View Logs. This option should be in the same location as Current Activity. Click View Logs and browse to the Server File where the PMI logs are saved. This location is usually: WAS\AppServer\profiles\AppSrv01\logs\tpv\

Adding WebSphere Application Servver as a Windows service

1. Modify IBM\was\AppServer\profiles\ST_Advanced_Profile\properties\soap.client.props file so you can stop Lotus Sametime Advanced with specifying a user name and passord. For example:

# SOAP Client Security Enablement
# - security enabled status  ( false[default], true  )

2. Configure WebSphere Application Servver to start as a service. User ID must have local security rights. Use the following syntax:

WASService.exe -add "service_name"
               -serverName server
               -profilePath server_profile_directory

For example:
D:\IBM\WAS\AppServer\bin\WASService -add "SametimeAdvanced" -serverName server1
   -profilePath "d:\ibm\was\AppServer\profiles\ST_Advanced_Profile"
   -startType automatic

3. Go to Click Start - Control Panel - Double-click Administrative Tools - Double-click Services. You should see IBM WebSphere Application Server V6.1 - node-name. which is the windows service you just created.

Note: To remove the service, type WASService.exe -remove"service_name" from WAS\bin\.

Source :

WebSphere Application Server - Quick How To

A quick list of "Good to Knows":
  • What is the default URL of the admin console: https://$hostname:10003/ibm/console/logon.jsp
  • What are the default portsHTTP: 8080, HTTPS: 443.
  • How to locate the logs: Logs can be found under$install_root/profiles/$profile_name/logs/$server_name. The default profile name is AppSrv01 and the default server name is server1. Example:/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1. SystemOut.log is the file containing everything that was logged to standard out. Logs can also be viewed from the admin console by navigating to Troubleshooting/Logging and Tracing/server_name/Runtime.
  • How to start/stop a server: If you're dealing with a "Network Deployment" type of installation (multiple application servers running under the control of the "deployment manager"), your can start/stop a server from the console (Server/Server Types/WebSphere application servers). Otherwise you have to do it from command line. Go to install_root/bin and run./ server_name, e.g., ./ server1 (this assumes that your installation has only one profile defined, otherwise you may need to "cd" to the profile_name/bindirectory). Make sure that you run all commands using the appropriate system account. To stop the server, run ./ server_name -username user_name -password password. user_name and password is the credentials of an admin account, typically the same one you use to login to the console.
  • How to deploy an application: In admin console, navigate to Applications/Application Types/WebSphere enterprise applications, click on "Install new application", select "Fast path", accept all the defaults except that on "step 2" make sure that you targeted correct servers (if you have multiple servers/clusters in your environment). Note that you can deploy a WAR file directly, you don't have to build an EAR. In this case, make sure that you set a context root on "step 4" screen of the wizard.
  • How to change context root of a Web application: Go to Applications/Application Types/WebSphere enterprise applications/application_name/Context Root For Web Modules in the console. Re-start the application after the change.
  • How to change the order of classloaders: If you're getting a ClassNotFoundException when you're starting the app, changing the order of classloaders is the first thing you may want to try. Go to Applications/Application Types/WebSphere enterprise applications/application_name/Manage Modules/module_name and make the appropriate selection in the "Class loader order" drop-down (this assumes you're doing it for a WAR module).
  • How to enable dynamic class reloading: If you need to frequently update your deployed application (e.g., you use a local WAS installation for development), enabling dynamic reloading could be a huge time saver. Go to your application in the console, "Class loading and update detection", set "Override class reloading settings ..." and set polling interval to 2 seconds. See this post for more details on how to configure your development environment to support class reloading.
  • How to find a host name and a port of the server: Go to Server/Server Types/WebSphere application servers. You'll find the host name in the Host Name column. To find a port, click on your server, and expand Ports. WC_defaulthost is the HTTP port and WC_defaulthost_secure is the HTTPS port.
  • How to kill a JVM: If the normal "stop" routine failed to stop the server in a reasonable amount of time, you may need to kill it. In a "Network Deployment" environment, simply navigate to the list of servers, select the server and click "Terminate". A node agent will kill the JVM for you. To achieve the same from command line (the only option if you're running standalone), cd toinstall_root/profiles/profile_name/logs/server_name, and kill the process ID contained in the file On Unix, you can simply do kill -9 `cat` (assumingserver1 is your server name). Use task manager or taskkill /PID on Windows.
  • How to browse JMS messages: Go to Buses/Your bus name/Destinations/Your destination/Queue points/Your queue point/Runtime/Messages.
  • Where to find configuration filesWAS has many configuration files, most of them are in XML/XMI format. The files are located under$install_root/profiles/$profile_name/config/cells/$cell_name.