For purpose of this blog, consider a simple SOA composite that does the following:
- Read entries from a database
- Create XML from the entries
- Store XML entries to another database
- Read XML entries
- Post each XML entry to a web service
There is good amount of workflow and business logic that makes it a perfect use case for doing it with BPEL, but that is beyond the purpose of this entry.
Sample SOA composite had following dependencies:
- Couple of Datasource (one to read source data and other to store XML Entries)
- JMS Queue (Weblogic’s Uniform Distributed Queues)
Create Generic Datasources pointing to individual Oracle RAC node and a Multi Datasource using WLST scripts. (Most of the scripts are written for Weblogic 10.3.3, so haven’t explored newly introduced GridLinked Datasource in Weblogic 10.3.4.)
2. JMS Resources
Create JMS Server, JMS Module, Sub deployment, Connection Factory and JMS Queues using WLST scripts.
The composite uses DBAdapter connector to interact with databases, so add outbound connection pools the DBAdapter.rar pointing to actual Weblogic Datasources and redeploy application
4. Deploy Composite
Finally, deploy the composite using Enterprise Manager Console.
1. Server Start parameters
Using Enterprise Manager to deploy, the deployment window got stuck and never returned back. The issue, it turns out, is that the deployment was not able to communicate among instance servers. If SOA cluster is using multicast to communicate across weblogic instances, you will required following parameters on each server instance. The parameter adds well known address
-Dtangosol.coherence.wka1=soa01.mycompany.com -Dtangosol.coherence.wka2=soa02.mycompany.com -Dtangosol.coherence.localhost=soa01.mycompany.com -Xmx2048m
2. Failure updating DBAdapter.rar connector application
On attempt to create outbound connection pool pointing to weblogic datasource, activation fails complaining FileNotFoundException: Plan.xml. The file is present on the primary server instance, but it isn't replicated on other nodes. I am not sure why but for some reason the Plan.xml was required to present on all the nodes, even if deployment is being done from machine with AdminServer.
3. Failure on interaction with DB on some instances.
JCA Binding Component connection issue.
JCA Binding Component is unable to create an outbound JCA (CCI) connection.
apptools:InsertLogDB [ InsertLogDB_ptt::insert(TestLogCollection,TestLogCollection) ] : The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: BINDING.JCA-12510 JCA Resource Adapter location error.
Unable to locate the JCA Resource Adapter via .jca binding file element <connection-factory/> The JCA Binding Component is unable to startup the Resource Adapter specified in the <connection-factory/> element: location='eis/DB/application-ds'. The reason for this is most likely that either 1) the Resource Adapters RAR file has not been deployed successfully to the WebLogic Application server or 2) the '<jndi-name>' element in weblogic-ra.xml has not been set to eis/DB/application-ds. In the last case you will have to add a new WebLogic JCA connection factory (deploy a RAR). Please correct this and then restart the Application Server
Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections. Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections.
- Copy the Plan.xml containing all outbound connection pool to all nodes
- Update the connector application DBAdapter.rar
- Restart DBAdapter.rar and Application server instance
4. BAM Cluster Multicast misconfiguration
A misconfiguration on the Multicast address of BAM cluster resulted in Access Forbidden 403. The issue was that BAM server was trying to communicate with another environment cluster. (i.e. prod bam cluster trying to communicate with test bam cluster).
Here is the error:
<BEA-000141> <TCP/IP socket failure occurred while fetching statedump over HTTP from 142374575950937656S:10.50.XX.XXX:[XXXXX,XXXXX,-1,-1,-1,-1,-1]:soa:soa_server1.
java.io.FileNotFoundException: Response: '403: Forbidden' for url: ' http://10.50.XXX.XXX:16101/bea_wls_cluster_internal/psquare/p2?senderNum=3&lastSeqNum=0&PeerInfo=10,3,4&ServerName=soa_server1'
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
So make sure that the Mutlicast address and port combination used for BAM cluster and SOA cluster is unique. There is a Multicast test utility that you can use to test communication between two instance. More information about the utility is available here. Also refer this to troubleshoot Multicast configuration.
On Machine A:java utils.MulticastTest -A [multicast address] -P [multicast port] -N TestServer1
On Machine B:java utils.MulticastTest -A [multicast address] -P [multicast port] -N TestServer2
Hope this helps!.