Application server load balancing and failover

Application server load balancing and failover

In a previous blog’s entry, I explained about Multiple application messaging servers on Peopletools 8.51 and facilities for load balance and failover on messaging domain server.
In a very similar way, the application server can also be used in load balance and failover configuration.
First of all, you need at least two applications servers, please note that if you are on the same server, then domain name and JSL (jolt) port should be different from each other. Of course, there are not many advantages to have this configuration if working on one and only one server, but rather good to have different server, if one crash, the other one takes place.
Here we go.
First application server :
 
And the second one :

Please, note that whether I name one “first” and the other one “second”, there is no precedence of one compared to the other by default (except if you define a weight in configuration.properties, but let’s leave it for later explanation).
Then starting each application server
 
Note that the number of listened ports regarding Jolt is defined is the psappsrv.cfg of the domain as following :
[JOLT Listener]
;=========================================================================
; Settings for JOLT Listener
;=========================================================================
Address=%PS_MACH%
Port=9000
Encryption=0
Min Handlers=5
Max Handlers=7 In the current case, the ports between 9000 and 9005 will be immediately in use after started :
[psadm2@psovmfscmfp1 APPDOM]$ netstat -an|grep 900
tcp        0      0 192.168.1.150:9000          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9001          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9002          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9003          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9004          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9005          0.0.0.0:*                   LISTEN
[psadm2@psovmfscmfp1 APPDOM]$ It means the second domain needs to be configured with a port above 9005.
 

Here’s the output from the database side, since my two apps are on the same machine, there’s nothing really useful so far, cannot distinguished which domain a database session is coming from :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
SQL> /
USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSQRYSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSQRYSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
21 rows selected.

Now, webserver configuration to be done in configuration.properties file ($PS_HOME/webserv/peoplesoft//applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps).
This is a easy configuration, put the application servers separated by simple comma as following :
[psadm2@psovmfscmfp1 peoplesoft]$ more ./applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps/configuration.properties
<<...snipped...>>
# psserver must point to your application server machine name (recommended) or IP address
# and JOLT port.  The PeopleTools release of the application server must match the PeopleTools
# release of the PeopleSoft files on the web server.
#
# To enable jolt failover and load balancing, provide a list of application server
# domains in the format of;  psserver=AppSrvr:JSLport,...
# For example:  psserver=SERVER1:9000,SERVER2:9010,SERVER3:9020
## sd="Application Server Connect String", ld="This string embodies the failover and load balancing string for PIA to connect to Application Server domains.  Consult the PeopleB
ooks recommendations on load balancing and failover for more information.  The format is psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name.   Multiple d
omains should be comma separated", tip="Format: psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name, e.g  psserver=SERVER1.oracle.com:9000,SERVER2.oracle.
com:9010"
psserver=psovmfscmfp1:9000,psovmfscmfp1:9010 ## sd="Jolt Pooling", dt="c", ld="Enable Jolt Pooling"
joltPooling=true
<<...snipped...>> Configured as above, all the application servers have the same weight (the same priority) in the round-robin algorithm. This algorithm affect ingoing transaction randomly to one or the other configured application server, there’s no priority, no load checkup, no network checkup, it’s just random (find out more in the My Oracle Support note E-WS: Load Balancing Options for Web Server and Application Server [ID 618367.1]).
But we could also introduced weight (from 1 to 10) on application server to make one precedence on the other as following :
psserver=psovmfscmfp1:9000#10,psovmfscmfp1:9010#1 In that case, the first server can takes extra work for an heavy load compared to the second one. 

Not only the load balance will be set, but the failover as well.

After the first connection to the front end application, we can check the client status of both application servers :
 
We can see a first pool as been instantiated.
Since round-robin algorithm is randomly given connection to the application server, it’s not always easy to show when alone, furthermore, the pool connection are making it even less simple.
But you probably can check it from the back end :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
  2* order by 2
SQL> /
USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,
… Here, from the same client machine, different users are in connected through the 2 applications servers (APPDOM and APPDOM2).

Now, let’s simulate a application server crash :

The online users won’t be disconnected, and the opened connections are automatically redirected to the second application server (failover) :

A second pool is now created, but that one on APPDOM2. Everything will be transparent to the front end users. 

Cloning a PeopleSoft Database

Cloning a PeopleSoft Database

One of the most common tasks of a PeopleSoft administrator or DBA is to clone a development or test database from a production database. Here are some of the steps you'll want to consider when refreshing PeopleSoft databases that are running PeopleTools 8.4x.

Note:  I'm sure this list isn't complete for every installation and it can certainly be improved, but that's the purpose of this Wiki -- Please incorporate your corrections, comments or suggestions so everyone can benefit. For purposes of discussion, the source database will be production, and the target database will be the dev/test database that you're overlaying.

1.     Export any security that you need to preserve from the target database to a flat file. Data Mover is a nice tool to use for this since you can qualify each table with a list of operator ID's to export. The tables you should consider exporting for specific users are PSOPRDEFN, PSOPRALIAS, PSROLEUSER, PSUSERATTR, PSUSEREMAIL, PSUSERPRSNLOPTN, PS_ROLEXLATOPR and PS_RTE_CNTL_RUSER.

2.     Stop the target application environment, including application servers and process schedulers. Stopping the web server is optional since it doesn't connect directly to the database. Be sure to clear cache.

3.     Overlay the target database with a recent backup of the Production database. The exact process will differ for your database platform, but your DBA should be able to do this with minimal direction.

4.     After the database comes back on-line, set DBNAME in PSDBOWNER back to the target database name.

5.     Set GUID to ' ' in the PSOPTIONS table. This will cause PeopleSoft to generate a new GUID so that change assistant can track it separately from the source database.

6.     Delete the data from the reporting tables, process scheduler tables and application messaging tables since this data isn't relevant in the target database. Run prcsclr.dms, rptclr.dms and appmsgpurgeall.dms. In addition, I also delete from the report manager tables PSRF_RATTR_TBL; PSRF_RSCRTY_TBL; PSRF_RINFO_TBL; and delete from PSRF_FINFO_TBL where PSRF_PRNT_FLDR_ID <> 0;

7.     If you have a script to reset everyone's e-mail address to a pre-defined value so that workflow messages from the Test environment don't get sent to real users, run it now. It should update the PSUSEREMAIL and PS_ROLEXLATOPR tables.

8.     Log on to the target database with data mover and run the following command to set the SYSADM password and any other back to it's pre-refreshed value:
-- Author: Brent Martin, ERP Associates Inc.
-- Date: February 10, 2006
-- Purpose: Reset application passwords and purge PeopleTools tables after a database refresh so that 
-- the application can be started with minimal reconfiguration.
-- Compatibility: This works with PeopleTools 8.4x releases
-- Notice: This script modifies data in your PeopleSoft database. It changes account passwords, and 
-- it deletes data from process scheduler, report repository and integration broker tables.
-- It as provided for reference purposes only. Use it at your own risk.
set log c:\temp\dbrefresh.log;
-- Change Access Password (SYSADM)
CHANGE_ACCESS_PASSWORD SYSADM1 CHANGEME1; 
-- Change Application Password (PSAPPS)
update psoprdefn set OPERPSWD = 'changeme1', encrypted = 0 where oprid = 'PSAPPS';
encrypt_password PSAPPS;
-- Set GUID to blank so PSEMAgent and Change Assistant doesn't get confused
update psoptions set guid = ' ';
-- Purge Process Scheduler
RUN h:\fdmo881\scripts\PRCSCLR.DMS;
-- Purge Report Repository Tables
RUN h:\fdmo881\scripts\RPTCLR.DMS;
-- Purge Application Messaging RUN h:\fdmo881\scripts\APPMSGPURGEALL.DMS;

9.     Import the security that you exported in step 1.

10.   While you're in data mover, change the user passwords that are configured in your application server, process scheduler, and integration broker configurations back to the pre-refresh values:
update psoprdefn set OPERPSWD = 'devpswd', encrypted = 0 where oprid = ('PSAPPS');
encrypt_password PSAPPS;
update psoprdefn set OPERPSWD = 'devpswd', encrypted = 0 where oprid = ('PTWEBSERVER');
encrypt_password PTWEBSERVER;

11.   Clear application server, web server and process scheduler cache if you haven't already done it. Start the target application environment.

12.   Log on through the web front end.

13.   Update the Report Node configuration and verify your Process Scheduler Servers are using the correct configuration. Make any Web Profile changes if needed.

14.   Navigate to PeopleTools > Utilities > Options and update the database name and description.

15.   Change password rules as appropriate for a development environment.

16.   Change your default local node if it should be named differently than the default local node from the source database. If it's not named differently, you may be at risk for a single sign-on vulnerability.

17.   Reconfigure attachment servers (if used)

18.   Reconfigure REN servers and clusters if the application server didn't configure it correctly at startup.

19.   Make sure you copy all of the batch objects (SQR's, Crystal Reports, COBOL programs, etc.) from your source to your target environment to keep the environment in synch. Don't forget to do this on both UNIX and Windows servers.

20.   Perform some sanity checks to make sure the environment is behaving itself before you let users back into it. At a minimum you should verify you can log on and run a report. Verify the report runs to completion, posts, and allows you to view the report without having to sign in a second time.