Friday, November 21, 2014

ConfigEntine task "wp-create-ldap" failed with SOAP connector problem.

[timestamp] ssl.disable.url.hostname.verification.CWPKI0027I
[timestamp] Client code attempting to load security configuration
Could not access WebSphere profile using: username=wpsadmin password=PASSWORD_REMOVED portNumber=10025 ADMC0016E: The system cannot create a SOAP connector to connect to host at port 10025.
Caused by: java.lang.reflect.InvocationTargetException
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: PKIX path validation failed: The certificate expired at Fri Aug 19 05:01:01 EDT 2011; internal cause is: NotAfter: Tue Oct 01 05:01:01 EDT 2014;

Using wsadmin to connect the server with type SOAP, the same exceptions prevented the correct connection.


The default chained certificate has problems.

Resolving the problem

In WebSphere Application Server (WAS) version 7, a default chained certificate was introduced as a personal certificate. This certificate was created during profile set up and should be valid for a year. It should be renewed when its validity date expires.
When running ConfigEngine, it acts as a client that connects to the server through SOAP connector. It takes the parameters configured in /properties/ssl.client.props that points to the key store file containing the default personal certificate and residing at /etc/key.p12.
To renew the certificate, take the following steps
- Log in to the Integrated Solutions Console as the primary WAS administrative user;
- Navigate to Security > SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore> Personal certificates;
- Check the box besides "Default", and hit button "Renew". You should see the expiration date now is updated for another year.
- Restart servers.

Monday, November 17, 2014

How to uninstall IBM DB2 Content Manager Version 8.3 products if fix packs are also installed


When you run the product uninstall program, you see the following error:

"You must uninstall the most recently installed fix pack before running this uninstall."


Fix packs must be uninstalled from the system in reverse order before you can uninstall the product.


To uninstall the latest fix pack, run the command:

where VERSION is the version number of the fix pack.

Tip: To determine the current version installed for each IBM® DB2® Content Manager product, run the command IBMCMROOT/bin/cmlevel.

If multiple fix packs have already been applied, run the uninstallUpdate command for each fix pack starting with the latest fix pack and then continuing in reverse order. For example, if you have DB2 Content Manager fix pack 1, fix pack 3, and fix pack 6 installed, you should first uninstall fix pack 6, then fix pack 3, and then fix pack 1.

Once all fix packs have been uninstalled, run the uninstall program for the product you want to remove.
For DB2 Content Manager:
Run IBMCMROOT/_uninstCM/uninstall-cm.exe
For DB2 Information Integrator for Content:
Run IBMCMROOT/_uninstII4C/uninstall-ii4c.exe
For eClient:
Run IBMCMROOT/_uninstEC/uninstall-ec.exe

Wednesday, November 5, 2014

Troubleshooting WebSphere Application Server issues in Sametime Advanced 8

Where to look for errors (SystemOut, SystemErr, ffdc)
Logging/tracing usually found in /WebSphere/AppServer/profiles/profile_name/logs/. See Logging and tracing at the Info Center.

Server logs: 
Look in system error logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemErr.log
Look in system out logs, found in /WebSphere/AppServer/profiles/profile_name/logs/server1/SystemOut.log
ffdc logs: 
Look in ffdc logs, found in /WebSphere/AppServer/profiles/profile_name/logs/ffdc

How to look for CPU heap issues for WAS

To dump the javacore and/or heapdump on WebSphere:

(start the wsadmin console... you'll be prompted for login/pwd credentials...)
cd \ProgramFiles\IBM\WebSphere\AppServer\bin

(setup for the DUMPS... specifying application server to dump...)
wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

(when ready to dump... execute the following to get a javacore file...)
wsadmin> $AdminControl invoke $jvm dumpThreads


(when ready to dump... execute the following to get a heapdump file...)
wsadmin> $AdminControl invoke $jvm generateHeapDump

javacore/heapdump file will be in the following directory:

How to monitor CPU for WebSphere Application Servver
See How to monitor CPU for EB.

How to turn on tracing

You can find on the following page Setting a diagnostic trace on a serverexternal link how to turn on tracing for the following:
  • How to turn on logging when looking for persistent chat errors
  • How to turn on logging when looking for vmm/ldap errors
  • How to turn on logging when looking for skilltap errors
How to turn on logging when looking for db errors:
Choose this detail level

How to turn on tracing for vmm issues
To look for issues with ldap we will need to enabled wmm tracing. In the admin console, turn on tracing for "*=all" at level FINEST. You should then see a trace.log file (where system.out and system.err live) with this tracing inside.

How to turn on Performance Monitoring Infrastructure PMI

Something to read before setting up:

This is a good page for reading on what to monitor when using PMI Monitoring overall system healthexternal link

Setting it up:
  1. In WebSphere ISC Console, go to Monitoring and Tuning - Performance Monitoring Infrastructure (PMI).
  2. Click server1.
  3. Enable either "Basic" monitoring or "Custom".

Basic monitoring should provide us with the 3 things we are interested in:
  • Number of DB Connections
  • Number of JMS Connections
  • Number of HTTP Sessions

If you are going to do "Custom" monitoring, enable the following statistics to get # of DB/JMS/HTTP connections/sessions: JDBC Connection Pools.CreateCount, JCA Connection Pools.CreateCount, Servlet Session Manager.LiveCount.

Once PMI is enabled you can look at the "Current Activity. To do this:
  1. In WebSphere ISC Console, go to Monitoring and Tuning - Performance Monitoring Infrastructure (PMI).
  2. Click Performance Viewer.
  3. Click Current Activity.
  4. Click server1.

From here on you can click on Summary Reports or Performance Modules which is more fine-grained. In Performance Modules, you can select the 3 statistics we are interested in. The picture below shows how to expand the trees and select the appropriate statistics.

Besides looking at Current Activity, you can View Logs. This option should be in the same location as Current Activity. Click View Logs and browse to the Server File where the PMI logs are saved. This location is usually: WAS\AppServer\profiles\AppSrv01\logs\tpv\

Adding WebSphere Application Servver as a Windows service

1. Modify IBM\was\AppServer\profiles\ST_Advanced_Profile\properties\soap.client.props file so you can stop Lotus Sametime Advanced with specifying a user name and passord. For example:

# SOAP Client Security Enablement
# - security enabled status  ( false[default], true  )

2. Configure WebSphere Application Servver to start as a service. User ID must have local security rights. Use the following syntax:

WASService.exe -add "service_name"
               -serverName server
               -profilePath server_profile_directory

For example:
D:\IBM\WAS\AppServer\bin\WASService -add "SametimeAdvanced" -serverName server1
   -profilePath "d:\ibm\was\AppServer\profiles\ST_Advanced_Profile"
   -startType automatic

3. Go to Click Start - Control Panel - Double-click Administrative Tools - Double-click Services. You should see IBM WebSphere Application Server V6.1 - node-name. which is the windows service you just created.

Note: To remove the service, type WASService.exe -remove"service_name" from WAS\bin\.

Source :

WebSphere Application Server - Quick How To

A quick list of "Good to Knows":
  • What is the default URL of the admin console: https://$hostname:10003/ibm/console/logon.jsp
  • What are the default portsHTTP: 8080, HTTPS: 443.
  • How to locate the logs: Logs can be found under$install_root/profiles/$profile_name/logs/$server_name. The default profile name is AppSrv01 and the default server name is server1. Example:/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1. SystemOut.log is the file containing everything that was logged to standard out. Logs can also be viewed from the admin console by navigating to Troubleshooting/Logging and Tracing/server_name/Runtime.
  • How to start/stop a server: If you're dealing with a "Network Deployment" type of installation (multiple application servers running under the control of the "deployment manager"), your can start/stop a server from the console (Server/Server Types/WebSphere application servers). Otherwise you have to do it from command line. Go to install_root/bin and run./ server_name, e.g., ./ server1 (this assumes that your installation has only one profile defined, otherwise you may need to "cd" to the profile_name/bindirectory). Make sure that you run all commands using the appropriate system account. To stop the server, run ./ server_name -username user_name -password password. user_name and password is the credentials of an admin account, typically the same one you use to login to the console.
  • How to deploy an application: In admin console, navigate to Applications/Application Types/WebSphere enterprise applications, click on "Install new application", select "Fast path", accept all the defaults except that on "step 2" make sure that you targeted correct servers (if you have multiple servers/clusters in your environment). Note that you can deploy a WAR file directly, you don't have to build an EAR. In this case, make sure that you set a context root on "step 4" screen of the wizard.
  • How to change context root of a Web application: Go to Applications/Application Types/WebSphere enterprise applications/application_name/Context Root For Web Modules in the console. Re-start the application after the change.
  • How to change the order of classloaders: If you're getting a ClassNotFoundException when you're starting the app, changing the order of classloaders is the first thing you may want to try. Go to Applications/Application Types/WebSphere enterprise applications/application_name/Manage Modules/module_name and make the appropriate selection in the "Class loader order" drop-down (this assumes you're doing it for a WAR module).
  • How to enable dynamic class reloading: If you need to frequently update your deployed application (e.g., you use a local WAS installation for development), enabling dynamic reloading could be a huge time saver. Go to your application in the console, "Class loading and update detection", set "Override class reloading settings ..." and set polling interval to 2 seconds. See this post for more details on how to configure your development environment to support class reloading.
  • How to find a host name and a port of the server: Go to Server/Server Types/WebSphere application servers. You'll find the host name in the Host Name column. To find a port, click on your server, and expand Ports. WC_defaulthost is the HTTP port and WC_defaulthost_secure is the HTTPS port.
  • How to kill a JVM: If the normal "stop" routine failed to stop the server in a reasonable amount of time, you may need to kill it. In a "Network Deployment" environment, simply navigate to the list of servers, select the server and click "Terminate". A node agent will kill the JVM for you. To achieve the same from command line (the only option if you're running standalone), cd toinstall_root/profiles/profile_name/logs/server_name, and kill the process ID contained in the file On Unix, you can simply do kill -9 `cat` (assumingserver1 is your server name). Use task manager or taskkill /PID on Windows.
  • How to browse JMS messages: Go to Buses/Your bus name/Destinations/Your destination/Queue points/Your queue point/Runtime/Messages.
  • Where to find configuration filesWAS has many configuration files, most of them are in XML/XMI format. The files are located under$install_root/profiles/$profile_name/config/cells/$cell_name.

Sunday, September 14, 2014

Installation of CF03 on Portal 8.0 may fail with exception

When trying to install CF03 on Portal 8.0, Installation fails with below error:

opt/IBM/Rendering/PortalServer/installer/wp.config/config/includes/wp_cluster_cfg.xml:1436: Detected single app deployment time greater than 5 minutes.

Check the \wp_profile\ConfigEngine\log\ConfigTrace.log log file for the following error:-
/opt/IBM/Rendering/PortalServer/installer/wp.config/config/includes/wp_cluster_cfg.xml:1436: Detected single app deployment time greater than 5 minutes at at at  at

In addition to that you will see error like below:
[wplc-wait-for-sync-to-complete] Distribution of AJAX Proxy Configuration not complete.
[wplc-wait-for-sync-to-complete] Distribution of Dojo_Resources not complete.

Workaround 1:

1. Try to increase maxAppTimeToWait="5" to 15 min. in wp_cluster_cfg.xml and run the installation again.
If doesn't work and you notice that wp_cluster_cfg.xml parameter revert back to lower value, then try the workaround below:

Workaround 2:

1. Edit the PortalServer/ file and remove the lines for:

Do not just comment them out; remove them completely.

2. Install CF03. It should go through successfully this time. It will NOT update the profile so it should skip the failing script.
3. Make the same timeout changes to wp_cluster_cfg.xml again.
4. Add ProfileName and ProfileDirectory back to
5. Execute this ConfigEngine script to update the profile:
./ CONFIG-WP-PTF-CF -DWasPassword= -DPortalAdminPwd=

If doesn't work, continue to next step.

Workaround 3:

1 Edit the Installation Manager configuration file, found at /opt/IBM/InstallationManager/eclipse/configuration/config.ini
2. Add the following line to this file: WP_UPDATE_SKIP_CONFIG=true
3. Save the file and run the portal installation again. You will need to run ./ CONFIG-WP-PTF-CF -DWasPassword= -DPortalAdminPwd= to finish update.

Technote reference :

Friday, August 8, 2014

EJPSG0002E: Requested Member does not exist

You receive the following error message in the WebSphere Portal log: EJPSG0002E: Requested Member does not exist. uid=wpsadmin,o=defaultWIMFileBasedRealm/nullcauses initialization failures for portal server startup.

SystemOut.log says :

PortalCollect E PortalCollectionsService EJPJO0119E: Failed to initialize portal collections services.
java.lang.NoClassDefFoundError: (initialization failure)
Caused by: java.lang.RuntimeException: Exception while loading dynamically re-ordered service class
Caused by: EJPSB0107E: Exception occurred while retrieving the identity of the domain admin user/admingroup uid=wpsadmin,o=defaultWIMFileBasedRealm.
Caused by: EJPSG0002E: Requested Member does not exist.uid=wpsadmin,o=defaultWIMFileBasedRealm
Caused by: EJPSG0002E: Requested Member does not exist.uid=wpsadmin,o=defaultWIMFileBasedRealm/null
Caused by: EJPSG0002E: Requested Member does not exist.uid=wpsadmin,o=defaultWIMFileBasedRealm/null

Accessing the Portal from a browser returns the following error:

404: Initialization of one or more services failed


Any Portal 6.1, 7.0 or 8.0 environment
Resolving the problem

Change the administrative user and group by running the ConfigEngine task wp-change-portal-admin-user. Specify an administrative user and group that exist in the repository.

Only if the ConfigEngine task fails and you need a manual work-around that enables WebSphere Portal to start:
Update all AdminUser and AdminGroup properties for the Access Control Data Management Service with the appropriate administrative user and group from the LDAP. 

Technote :

SECJ0371W: Validation of the LTPA token failed

WebSphere Portal received a request with an expired or otherwise invalid LTPA token for which it needed to generate one or more URLs. By default, the LTPA token timeout is two hours. This timeout is encoded into the token itself. With the default settings, this condition could occur if someone logged in and worked in WebSphere Portal for longer than two hours.

Check your SystemOut.log for :


Such stack traces indicate that WebSphere Portal explicitly requests LTPA token validation from WebSphere Application Server during URL generation. This is by design.

You may either ignore the warning or suppress them. To suppress such warnings, set:

per the WebSphere Application Server InfoCenter section on Log level settings.