BI Publisher will not start (using emctl oms start and emctl oms start -bip_only)

*Customer names obfuscated to protect the innocent (and the guilty!)

This post covers a particular customer’s recovery scenario. They had reverted their config.xml to a previous version due to a failed attempt to integrate LDAP into WLS myrealm, and ever since, they have been unable to start BI Publisher via the emctl command. Although they can start it from the WLS Admin Console, the ’emctl status oms -details’ shows BI Publisher Server is Down, and they can’t even connect to it via the https://em12c.example.com:9702/xmlpserver/. They then attempted to run the emcli setup_bipublisher to re-register it with EM but get:

And the BIP.out file shows:
<Servlet: "wsmtest.jaxws.diagnostic.Diagnostic" failed to preload on startup in Web application: "Diagnostic".
oracle.webservices.provider.ProviderException: java.lang.RuntimeException: java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission IdentityAssertion)
at oracle.j2ee.ws.server.provider.ProviderProcessor.configureProviderManagement(ProviderProcessor.java:367)
at oracle.j2ee.ws.server.provider.ProviderProcessor.init(ProviderProcessor.java:198)

So what went wrong?

Well, we don’t have the details to understand what went wrong with their attempt to integrate LDAP into WLS, but what we can tell is that the config.xml that they recovered was from an intermediate stage of running ‘configureBIP’. This can be seen by these lines in the log files:

You would only ever see a listen port of 9700 after the WebLogic domain has been extended with BIP, but before further configuration occurs.

I see no viable approach to some sort of ‘resumption’ of configureBIP from where it left off, as shown in the above config.xml. The inherent problem with manipulating config.xml files directly is that there are many other WebLogic artifacts that are tied to the config.xml. Some of these are utilized by BIP. Some of these, for example (there are numerous others), include:

  • JMS Queues
  • JNDI Datasources
  • Encrypted WebLogic files that are part of WebLogic, which require a private, embedded key.

Even if it was possible to somehow hand-rebuild their environment, it would set this customer up for all kinds of potential issues down the road e.g. when it is time to perform some of these follow-on tasks:

  • Adding additional OMSs (including BIPs)
  • Upgrading EM12c to EM13c
  • Upgrading the repository database
  • etc.

How does this customer recover from this scenario?

For all intents and purposes, this needs to be treated as a corruption of the EM middleware and WebLogic domain. Similar to filesystem corruption or hardware failure, we have a whole section of the Enterprise Manager Cloud Control Advanced Installation and Configuration Guide on backing up and restoring Enterprise Manager. Thankfully, they will have a backup of the OMS, that was done prior to running configureBIP. This was made with:
$ emctl exportconfig oms

which we perform automatically for them. They should use this as the restore point to rebuild EM with, following the relevant parts of section 21.6.3 of the doc.

Pete

After 22 years of working at Oracle in just about every role except Marketing and Support, I am now working as a Senior Managed Services Consultant with Data Intensity, specializing in Oracle Database technology, High Availability and Disaster Recovery solutions. I am also a member of the OakTable Network, and have presented at RMOUG Training Days, Hotsos Symposia, Oracle OpenWorld conferences, and other user group events. I have co-authored the Expert Oracle Enterprise Manager 12c and Practical Oracle Database Appliance books published by Apress, and am one of the authors of the Building Database Clouds in Oracle Database 12c book published by Addison Wesley.

Comments are closed.