The Cluster Verification Utility (CVU) can assist you in diagnosing a wide variety of configuration problems. As part of Autonomous Health Framework, CVU can perform these checks periodically by running autonomously on a scheduled basis.

Cluster Verification Utility (CVU) performs system checks in preparation for installation, patch updates, or other system changes. Using CVU ensures that you have completed the required system configuration and preinstallation steps so that your Oracle Grid Infrastructure or Oracle Real Application Clusters (Oracle RAC) installation, update, or patch operation, completes successfully.

Oracle Universal Installer is fully integrated with CVU, automating many CVU prerequisite checks. Oracle Universal Installer runs all prerequisite checks and associated fixup scripts when you run the installer.

Topics:

• Verifying the Existence of Node Applications
  You use the CVU comp nodeapp command to verify the existence of node applications, namely the virtual IP (VIP), Oracle Notification Services (ONS), and Global Service Daemon (GSD), on all the nodes.
• Verifying the Integrity of Oracle Clusterware Components
  You use the CVU comp crs command to verify the existence of all the Oracle Clusterware components.
• Verifying the Integrity of the Oracle Cluster Registry
  You use the CVU comp ocr command to verify the integrity of the Oracle Clusterware registry.
• Verifying the Integrity of Your Entire Cluster
  Use the CVU comp clu command to check that all nodes in the cluster have the same view of the cluster configuration.
• Checking the Settings for the Interconnect
  Use the CVU comp nodereach or comp nodeconcommand to check the settings for the interconnect.
• Enabling Tracing
  CVU does not generate trace files unless you enable tracing.

As part of Autonomous Health Framework, ORAchk is configured to run in Daemon Mode. ORAchk proactively scans for known problems within a wide range of products and autonomously reports on these issues.

ORAchk replaces the popular RACcheck tool, extending the coverage based on prioritization of top issues reported by users. You can configure ORAchk to upload its findings to the Collection Manager Repository for centralized examination and analysis.

To obtain the most out of ORAchk, Oracle recommends that you run ORAchk as the root user (via password or SUDO).

Topics:

• Overview of Oracle ORAchk
  Oracle ORAchk provides a lightweight and non-intrusive health check framework for the Oracle stack of software and hardware components.
• Running ORAchk
  You can schedule ORAchk to run at periodic intervals or you can run it as needed.
• ORAchk HTML Report Output
  Oracle ORAchk generates a detailed HTML report with findings and recommendations.

Cluster Health Monitor is integrated part of Oracle Grid Infrastructure. Cluster Health Monitor collects operating system statistics using operating system APIs to gain performance and reduce the CPU usage overhead.

Cluster Health Monitor collects operating system statistics (system metrics) such as memory and swap space usage, processes, IO usage, and network related data. The Cluster Health Monitor collects information in real time and usually once a second. Cluster Health Monitor collects as much of system metrics and data as feasible that is restricted by the acceptable level of resource consumption by the tool.

Cluster Health Monitor provides system metrics and data for troubleshooting many different types of problems such as node reboot, instance eviction, server hang, severe performance degradation, and any other problems that need the system metrics and data. By monitoring the data constantly, users can use Cluster Health Monitor detect potential problem areas such as CPU load, memory constraints, and spinning processes before the problem causes an unwanted outage.

The information collected by Cluster Health Monitor is stored in the Grid Infrastructure Management Repository (GIMR). GIMR is a centralized infrastructure database for diagnostic and performance data and resides in the Grid home. It is a single instance multitenant database with a single pluggable database (PDB). It includes partitioning (for data lifecycle management). GIMR clients include: Cluster Health Monitor, Rapid Home Provisioning, Enterprise Manager Cloud Control, and Trace File Analyzer. Cluster Health Monitor puts all of its operating system metric data in GIMR. Rapid Home Provisioning uses GIMR to persist metadata about each of the database homes that it services. Enterprise Manager Cloud Control and Trace File Analyzer retrieve data from GIMR, predominantly Cluster Health Monitor data.

You can review the information collected by Cluster Health Monitor using either diagcollection.pl or oclumon:

• Diagcollection.pl: As the grid user, using command Grid_home/bin/diagcollection.pl --collect --chmos produces output for all the data collected in the repository. Because there might be a lot of data, this could take a long time. It is better to limit the query to an interesting time interval.

  Grid_home/bin/diagcollection.pl --collect --crshome Grid_home 
    --chmos --incidenttime start_time --incidentduration 05:00

  The above command generates a report that covers 5 hours from the time specified by incidenttime. The incidenttime must be in the form MM/DD/YYYYHH:MN:SS where MM is month, DD is date, YYYY is year, HH is hour in 24 hour format, MN is minute, and SS is second. For example, if you want to put the incident time to start from 10:15 PM on June 01, 2016, the incident time is 06/01/201622:15:00. The incidenttime and incidentduration can be changed to capture more data.

• oclumon: Alternatively, you can use the command oclumon dumpnodeview -allnodes -v -last "11:59:59" > output-filename’ if diagcollection.pl fails with any reason. This command generates a report from the repository up to last 12 hours. The -last value can be changed to get more or less data.

The Cluster Health Advisor daemon collects data from Oracle Real Application Clusters (Oracle RAC), Oracle RAC One Node databases, and the host operating system and hardware resources, to analyze and produce a root cause and action after finding a database or host performance issue.

The user can initiate a calibration process to generate data from which a predictive analysis data set is created. This baseline operation model is then evaluated every five seconds to determine whether an abnormal condition has occurred and is progressing, along with its root cause, details, and corrective actions. This data evidence persists in the Grid Infrastructure Management Repository database for reporting to Oracle Enterprise Manager Cloud Control.

The Cluster Health Advisor daemon provides system and database administrators with early warning of pending performance issues, and root causes and corrective actions for Oracle RAC databases and cluster nodes. This advanced proactive diagnostic capability will increase availability and performance management.

To track the online health of an Oracle RAC system, Cluster Health Advisor analyzes the system’s measurements (taken as samples of the operational state of a running system) to detect whether the measured values are within an established normal base line, or if they are drifting anomalously beyond their acceptable perimeter. If Cluster Health Advisor detects irregularities in any of the measured values, it invokes an incident diagnostic engine to determine the identity of the underlying issue and generates incident details. The analysis cycle repeats periodically with a fresh set of sampled measurements to track the health status of the system over time.

To view the findings of Cluster Health Advisor, you can use:here is a and a command-line interface. Graphical interfaces include:

• Graphical interface

  • Enterprise Manager Cloud Control

  • A standalone specialized console dedicated to depicting Cluster Health Advisor analysis

• Command line interface — CHACTL

$ chactl query diagnosis –start start_time -end end_time 
10:05:15      	Database A	    Disk Failure   (instance1) [detected]
				                      Storage Bandwidth Saturation (instance 1) [detected]
10:05:30        Database A     Disk Failure (instance 1)  [cleared]
                               Storage Bandwidth Saturation (instance 2) [detected]
                               Storage Bandwidth Saturation (instance 1) [cleared]
		           Database B      Storage Bandwidth Saturation (instance 1 )
		           Host A		       Disk Failure

Trace File Analyzer centralizes and automates the collection of diagnostic information.

Topics:

• About the Trace File Analyzer Collector (tafctl)
  Trace File Analyzer Collector (TFA) is a diagnostic collection utility to simplify diagnostic data collection on Oracle Grid Infrastructure, Oracle RAC and single-instance Oracle Database systems.
• tfactl Command Summary
  tfactl is a command-line interface for managing the Trace File Analyzer Collector (TFA) utility.
• Using tfactl to Collect Diagnostic Logs for a Specific Time Period
  The diagcollection module of TFA has the capability to take a number of different parameters to allow the user to control how large or detailed the required evidence set is.

Hang Manager can reliably detect and resolve hung database sessions in a timely manner.

Hang Manager is only active when Oracle RAC is enabled. You can determine whether an Oracle RAC database is used / enabled by querying the CLUSTER_DATABASE initialization parameter. If this parameter is set to TRUE, then Oracle RAC is enabled on this database.

Since Oracle Database 11g Release 2 (11.2.0.2), Hang Manager can resolve hangs it detects by terminating the root session or process.  By default, Hang Manager will not terminate an instance or evict a node. Hang Manager does not resolve all hangs that it detects. For example, hangs which involve a possible application issue are left to the user to determine the correct course of action.  Also, if Hang Manager determines that the instance on which the source of a hang resides is experiencing high CPU or IO load, then hang resolution is delayed.  This allows the source session or process time to progress and for the hang to resolve on its own. Hang Manager only resolves hangs on Oracle Database instances; Hang Manager does not resolve any hangs on Oracle ASM instances.

Starting with Oracle Database 12c Release 1, if Oracle Database Quality of Service (QoS) Management is active on the cluster, then Hang Manager uses additional information provided by Oracle Database QoS Management to determine if a hang should be ignored or resolved.

When Hang Manager resolves a hang, it signals an ORA-32701 incident which is visible in the alert log. The alert log message is similar to the following:

ORA-32701: Possible hangs up to hang ID=24 detected
Incident details in: /ee/oracle/oracle_base/diag/rdbms/orcl/orcl1/incident/incdir_1944098/orcl1_dia0_34930694_i1944098.trc
DIA0 terminating blocker (ospid: 28311778 sid: 3398 ser#: 1) of hang with ID = 24
     requested by master DIA0 process on instance 2
     Hang Resolution Reason: Automatic hang resolution was performed to free a
    critical database process.
     by terminating session sid:3398 with serial # 1 (ospid:28311778)

The above message describes which hang is about to be resolved; the hang with ID 24. It also lists the root session ID, serial number and Operating System process ID and on which instance it resides. Finally, Hang Manager gives a short reason as to why this hang will be resolved.  In this case, a regular user session is blocking a critical database process so the user session will be terminated to allow the critical database process to run.

You can monitor Hang Manager activity by querying the following data dictionary views:

• V$HANG_INFO: Contains the active hangs and deadlocks detected by Hang Manager. It does not include any hangs which have been resolved by Hang Manager or have self-resolved. Only the latest 32 active hangs are in this view.

• V$HANG_SESSION_INFO: Contains the sessions in the main chain for all of the hangs included in the V$HANG_INFO view. Only the first 20 sessions of the main chain, including the root of the hang, for each hang are kept in this view.

• GV$HANG_STATISTICS or V$HANG_STATISTICS: Contains various Hang Manager statistics regarding hangs or deadlocks which it has detected. These statistics include: the number of hangs detected, the number of deadlocks detected, the number of hangs resolved by Hang Manager, the number of hangs ignored broken down by the reason for ignoring the hang, the number of hangs which self-resolved, and so on.

Memory Guard detects memory pressure on a server in real time and redirects new sessions to other servers to prevent using all available memory on the stressed server.

Enterprise database servers can use all available memory due to too many open sessions or runaway workloads. Running out of memory can result in failed transactions or, in extreme cases, a restart of the server and the loss of a valuable resource for your applications.

Rerouting new sessions to different servers protects the existing workloads on the memory-stressed server and enables the server to remain available. Memory Guard is a feature of Oracle RAC that manages the memory pressure for servers, adding a new resource protection capability in managing service levels for applications hosted on Oracle RAC databases.

When Oracle Database Quality of Service Management is enabled and managing an Oracle Clusterware server pool, Cluster Health Monitor sends a metrics stream that provides real-time information about memory resources for the cluster servers to Memory Guard. This information includes:

• Amount of available memory

• Amount of memory currently in use

If Memory Guard determines that a node is experiencing memory pressure, then the database services managed by Oracle Clusterware are stopped on that node, preventing new connections from being created. After the memory stress is relieved, the services on that node are restarted automatically, and the listener starts sending new connections to that server. The memory pressure can be relieved in several ways (for example, by closing existing sessions or by user intervention).

Oracle Database Quality of Service (QoS) Management is an automated, policy-based product that monitors the workload requests for an entire system.

Oracle Database QoS Management manages the resources that are shared across applications, and adjusts the system configuration to keep the applications running at the performance levels needed by your business. Oracle Database QoS Management responds gracefully to changes in system configuration and demand, thus avoiding additional oscillations in the performance levels of your applications.

Oracle Database QoS Management monitors the performance of each work request on a target system. Oracle Database QoS Management starts to track a work request from the time a work request requests a connection to the database using a database service. The amount of time required to complete a work request, or the response time (also known as the end-to-end response time, or round-trip time), is the time from when the request for data was initiated and when the data request is completed. By accurately measuring the two components of response time, which are the time spent using resources and the time spent waiting to use resources, Oracle Database QoS Management can quickly detect bottlenecks in the system. Oracle Database QoS Management then makes suggestions to reallocate resources to relieve a bottleneck, thus preserving or restoring service levels.

Oracle Database QoS Management manages the resources on your system so that:

• When sufficient resources are available to meet the demand, business-level performance requirements for your applications are met, even if the workload changes.

• When sufficient resources are not available to meet the demand, Oracle Database QoS Management attempts to satisfy the more critical business performance requirements at the expense of less critical performance requirements.