*Customer names obfuscated to protect the innocent (and the guilty!)
This post covers a particular customer’s recovery scenario. They had reverted their config.xml to a previous version due to a failed attempt to integrate LDAP into WLS myrealm, and ever since, they have been unable to start BI Publisher via the emctl command. Although they can start it from the WLS Admin Console, the ’emctl status oms -details’ shows BI Publisher Server is Down, and they can’t even connect to it via the https://em12c.example.com:9702/xmlpserver/. They then attempted to run the emcli setup_bipublisher to re-register it with EM but get:
$ ./emcli setup_bipublisher -proto=https -host=em12c.example.com -port=9702 -uri=xmlpserver -force<br>
Error: BI Publisher could not be setup.:<br>
Error inserting BIPInstance row for :https://em12c.example.com:9702/xmlpserver: javax.management.MBeanException: JPS-01051: <br>
Credential audit events cannot be logged. Reason oracle.security.jps.service.audit.AuditException: JPS-00054: Failed to <br>
create the auditor for JPS.
And the BIP.out file shows:
<Servlet: "wsmtest.jaxws.diagnostic.Diagnostic" failed to preload on startup in Web application: "Diagnostic".
oracle.webservices.provider.ProviderException: java.lang.RuntimeException: java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission IdentityAssertion)
So what went wrong?
Well, we don’t have the details to understand what went wrong with their attempt to integrate LDAP into WLS, but what we can tell is that the config.xml that they recovered was from an intermediate stage of running ‘configureBIP’. This can be seen by these lines in the log files:
You would only ever see a listen port of 9700 after the WebLogic domain has been extended with BIP, but before further configuration occurs.
I see no viable approach to some sort of ‘resumption’ of configureBIP from where it left off, as shown in the above config.xml. The inherent problem with manipulating config.xml files directly is that there are many other WebLogic artifacts that are tied to the config.xml. Some of these are utilized by BIP. Some of these, for example (there are numerous others), include:
- JMS Queues
- JNDI Datasources
- Encrypted WebLogic files that are part of WebLogic, which require a private, embedded key.
Even if it was possible to somehow hand-rebuild their environment, it would set this customer up for all kinds of potential issues down the road e.g. when it is time to perform some of these follow-on tasks:
- Adding additional OMSs (including BIPs)
- Upgrading EM12c to EM13c
- Upgrading the repository database
How does this customer recover from this scenario?
For all intents and purposes, this needs to be treated as a corruption of the EM middleware and WebLogic domain. Similar to filesystem corruption or hardware failure, we have a whole section of the Enterprise Manager Cloud Control Advanced Installation and Configuration Guide on backing up and restoring Enterprise Manager. Thankfully, they will have a backup of the OMS, that was done prior to running configureBIP. This was made with:
$ emctl exportconfig oms
which we perform automatically for them. They should use this as the restore point to rebuild EM with, following the relevant parts of section 21.6.3 of the doc.