Middleware Diagnostics Advisor (MDA) Setup

One of the areas I’ve been exploring recently is the setup and use of the Middleware Diagnostics Advisor, more commonly known as MDA. MDA is an advisor in EM12c that analyzes the entire stack and provides diagnostic findings by identifying root causes for any problems it discovers. It correlates and analyzes the input and offers advice on how to resolve the problem. For example, it can help you identify that slow SQL statements or a JDBC connection pool is causing a performance bottleneck. As described in the documentation, MDA is currently supported for Oracle WebLogic Server 10g Release 3 (10.3) and higher. MDA monitors JDBC DataSources, EJBs, and JMS Queues.

MDA enables you to easily identify the underlying states in the application server environment that are causing degraded performance. These underlying states can manifest themselves as slow responses for requests, hung servers, slow servers, high memory utilization, high disk I/O, and so on.

MDA analyses the overall performance of an aspect in a runtime environment. When the overall performance of the aspect degrades beyond a certain limit, MDA diagnoses the issue to find the underlying cause. However, individual one off issues, which do not affect the overall performance, are not isolated by MDA.

MDA diagnoses performances issues in the following areas:

  • JDBC findings:
    • Checks if the SQL execution takes a long time.
    • Checks if the JDBC Pool size is small, and if the wait time for connections is high.
    • Checks if reclaimed connections are found for data sources, and if the effective pool size is small.
  • JMS findings:
    • Checks if the message processing is slow.
    • Checks if the number of messages reprocessed due to transaction timeout is high.
    • Checks if the number of messages reprocessed due to transaction rollback is high.
    • Checks if the message delivery is delayed.
    • Checks if the queue slowed down due to large number of messages.
    • Checks if the queue slowed down due to large size of messages.
  • EJB findings:
    • Checks if the remote call made by the EJB takes too long to return.
    • Checks if the EJB takes too long to execute.
  • Thread findings:
    • Checks if there are locks that are being waited on by other threads.

If you want to see a video of MDA in action, follow this link to see MDA sizing the JDBC connection pool.

Under the Covers

Now that you know a little bit more about what it does, let’s have a look at how you set up MDA and what happens under the covers.

Enabling and Disabling MDA

Firstly, let’s look at how MDA is enabled. That’s pretty straightforward, as it’s enabled out of the box when you deploy the JVMD agent. Once that’s done, you can disable it altogether if you want, or indeed enable it again, from the WebLogic Domain menu. Go to All Targets and select the WebLogic Domain you want to enable/disable MDA for, then from the WebLogic Domain menu select “Diagnostics” followed by “Middleware Diagnostics Advisor” as shown in the screenshot below:

MDA_Configuration1

That will take you to the page shown in the next screenshot. In this example, I’ve drilled into the WebLogic Domain installed with the OMS, and selected the first OMS in the two OMS configuration I have in this example. Once you select the target name, the bottom half of the screen appears as shown, prompting you for login information. In this example, OMS1 already has MDA enabled, so the “Disable” button is now available to me:

MDA_Configuration2

Now that enabling and disabling is at a fairly broad level – the WebLogic domain. What if you wanted to be a bit more granular than that? Well, you can do that too. So for each of the findings categories listed above (JDBC, JMS, EJB, and Thread) there’s a specific emctl command to either enable or disable the checks. Here’s an example for disabling the check for JMS findings:

bash-3.2$ which emctl
/u01/app/oracle/product/middleware/oms/bin/emctl
bash-3.2$ emctl set property -name oracle.sysman.emas.mda.disableJmsAnalysisForTargets -value all
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.emas.mda.disableJmsAnalysisForTargets has been set to value all for all Management Servers
OMS restart is not required to reflect the new property value

OK, that was pretty easy. How about re-enabling it? Well, that’s not listed in the documentation and switching the property name from disableJmsAnalysisForTargets to enableJmsAnalysisForTargets isn’t the easy answer you might expect. 😉

bash-3.2$ emctl set property -name oracle.sysman.emas.mda.enableJmsAnalysisForTargets -value all
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
SYSMAN password:
Definition does not exist for property "oracle.sysman.emas.mda.enableJmsAnalysisForTargets". Can not set property value

So how do you get rid of the property? You use the delete property verb instead:

bash-3.2$ emctl delete property -name oracle.sysman.emas.mda.disableJmsAnalysisForTargets
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.emas.mda.disableJmsAnalysisForTargets deleted successfully for all Management Servers

Why Disable MDA?

The next question to arise, of course, is why would you want to disable MDA? After all, if it’s doing such a useful job at diagnosing performance issues, surely you’d want to keep it enabled? Well, the answer to that is of course yes, you would want it to remain enabled. But there may be times when you want to disable some subset of the checks. Let’s say you left it enabled across all the checks it runs. What happens is that each check runs a job, once an hour. That means a lot of jobs being executed for a single middleware target, which in turn can result in a couple of different issues:

  • The jobs interface can become somewhat overcrowded with job names that start with “MDAANALYSIS”. That can make it difficult to find other jobs that you might be more interested in.
  • If your OMS machine is already busy, the overhead of starting so many separate jobs for each middleware target may become an issue, particularly if there are many middleware targets.

These are both valid issues, so let me address each one in turn.

The MDAANALYSIS* jobs are part of a category of jobs that can loosely be turned “system” jobs. These are jobs that Enterprise Manager uses either out of the box or once you have turned on certain functionality. Oracle has recognized that these jobs can clutter the EM interface and make it more difficult for you as a user to find your jobs. However, you need to realize these jobs will only be seen by super-administrators, so normal administrators will not see any of these jobs at all. In the screenshot below, you can see I’m logged in as PSHARMAN, a super-administrator, and I can see all the job activity in my EM environment, including an MDAANALYSIS job highlighted at the bottom of the screen:

MDA_Jobs1

In this next screenshot, taken a couple of minutes later, I’m logged in as the DEMO1 account, which is set up as a normal administrator, and I can’t see any of these “system” jobs (in fact, I can’t see any because none have been created using this DEMO1 account):

MDA_Jobs2

You also have the ability to skip these analysis runs for all MDA-enabled servers. To do this, follow the path Setup -> Middleware Management -> Middleware Diagnostics Advisor, then check the “Skip Analysis Runs for All MDA-Enabled Servers” and click “Apply”:

MDA_Jobs3

Note that this does not stop the MDA Data Collector on the target server.

In a future release, you will find there is a way to hide these system jobs, even for the super-administrator users.

On the second issue, the overhead becoming an issue one, while this at first glance seems to be something that may be a problem if you have lots of middleware targets, what actually happens under the scenes is that each job can handle 1,000 targets, so the job system overhead would not be an issue even when the OMS is already busy until you have many, many thousands of targets.

Other Tuning / Configuration Options

There are another couple of options you can use for tuning or configure MDA, again reached by following the path Setup -> Middleware Management -> Middleware Diagnostics Advisor. That will bring up the same page as we saw before, but this time you should focus on the bottom half of the page, as shown in the screenshot below:

MDA_Jobs4

The additional options you can set are:

  • Purge Policy – By default, an MDA job runs every 24 hours which purges data from the repository, deleting any data older than 31 days. You can increase or decrease this number of days to suit your requirements. This is a global parameter which applies to all targets.
  • Violations Percentage – This defaults to 10%, and is one of the two measures you can use to tune when violations should result in a finding. Again, this is a global setting.
  • For JMS wait time findings specifically, you can also set the wait time (in minutes) beyond which any messages picked up will be considered as violations. This defaults to 5 minutes.

Summary

So there you are, that’s how to set up, configure and tune MDA. It’s an advisor that analyzes the entire stack and provides diagnostic findings by identifying root causes for any problems it discovers, then correlates and analyzes the input and offers advice on how to resolve the problem. There are a few tuning / configuration options you can play around with. Test it out at your site and see what valuable insights it can provide!

Pete

After 22 years of working at Oracle in just about every role except Marketing and Support, I am now working as a Senior Managed Services Consultant with Red Stack Tech, specializing in Oracle Database technology, High Availability and Disaster Recovery solutions.

I am also a member of the OakTable Network, and have presented at RMOUG Training Days, Hotsos Symposia, Oracle OpenWorld conferences, and other user group events. I have co-authored the Expert Oracle Enterprise Manager 12c and Practical Oracle Database Appliance books published by Apress, and am one of the authors of the Building Database Clouds in Oracle Database 12c book published by Addison Wesley.

Comments are closed