7 Administering Oracle Clusterware Components
The main components to manage in your Oracle Clusterware environment are the voting disks and the Oracle Cluster Registry (OCR).
Topics:
- About Oracle Clusterware
Oracle Clusterware includes a high availability framework for managing any application that runs on your cluster. Oracle Real Application Clusters (Oracle RAC) uses Oracle Clusterware as the infrastructure that binds multiple nodes that then operate as a single server. - Managing the Oracle Clusterware Stack
By default, Oracle Clusterware is configured to restart whenever the server it resides on is restarted. During certain maintenance operations, you may be required to stop or start the Oracle Clusterware stack manually. - Administering Voting Disks for Oracle Clusterware
Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster. - Backing Up and Recovering the Oracle Cluster Registry
Oracle Clusterware automatically creates Oracle Cluster Registry (OCR) backups every four hours. At any one time, Oracle Clusterware always retains the latest three backup copies of the OCR that are four hours old, one day old, and one week old. - Changing the Oracle Cluster Registry Configuration
To ensure proper operation of Oracle Clusterware, make sure the OCR is highly available. - Troubleshooting the Oracle Cluster Registry
The Oracle Cluster Registry (OCR) manages Oracle Clusterware and Oracle RAC database configuration information.
7.1 About Oracle Clusterware
Oracle Clusterware includes a high availability framework for managing any application that runs on your cluster. Oracle Real Application Clusters (Oracle RAC) uses Oracle Clusterware as the infrastructure that binds multiple nodes that then operate as a single server.
In an Oracle RAC environment, Oracle Clusterware monitors all Oracle components (such as instances and listeners). If a failure occurs, then Oracle Clusterware automatically attempts to restart the failed component and also redirects operations to a surviving component.
Oracle Clusterware manages applications to ensure they start when the system starts. Oracle Clusterware also monitors the applications to ensure they are always available. For example, if an application process fails, then Oracle Clusterware attempts to restart the process based on scripts that you customize. If a node in the cluster fails, then you can program application processes that typically run on the failed node to restart on another node in the cluster.
Topics:
- About the Voting Disks
Voting disks are one of the key files for Oracle Clusterware. - About Oracle Cluster Registry
The Oracle Cluster Registry is a file that contains information about the cluster node list and instance-to-node mapping information. - About High Availability of Oracle Clusterware Files
High availability configurations have redundant hardware and software that maintain operations by avoiding single points of failure.
7.1.1 About the Voting Disks
Voting disks are one of the key files for Oracle Clusterware.
The voting disk records node membership information. A node must be able to access more than half the voting disks at any time. To avoid simultaneous loss of multiple voting disks, each voting disk should be on a storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other voting disks.
For example, if you have five voting disks configured, then a node must be able to access at least three of the voting disks at any time. If a node cannot access the minimum required number of voting disks, then it is evicted, or removed, from the cluster. After the cause of the failure has been corrected and access to the voting disks has been restored, you can instruct Oracle Clusterware to recover the failed node and restore it to the cluster.
7.1.2 About Oracle Cluster Registry
The Oracle Cluster Registry is a file that contains information about the cluster node list and instance-to-node mapping information.
The Oracle Cluster Registry (OCR) also contains information about Oracle Clusterware resource profiles for resources that you have customized. The voting disk data is also backed up in OCR.
Each node in a cluster also has a local copy of the OCR, called an Oracle Local Registry (OLR), that is created when Oracle Clusterware is installed. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, whether Oracle Clusterware is fully functional. By default, OLR is located at Grid_home/cdata/$HOSTNAME.olr
7.1.3 About High Availability of Oracle Clusterware Files
High availability configurations have redundant hardware and software that maintain operations by avoiding single points of failure.
When a component is down, Oracle Clusterware redirects its managed resources to a redundant component. However, if a disaster strikes, or a massive hardware failure occurs, then having redundant components might not be enough. To fully protect your system it is important to have backups of your critical files.
The Oracle Clusterware installation process creates the voting disk and the OCR on shared storage. If you select the option for normal redundant copies during the installation process, then Oracle Clusterware automatically maintains redundant copies of these files to prevent the files from becoming single points of failure. The normal redundancy feature also eliminates the need for third-party storage redundancy solutions. When you use normal redundancy, Oracle Clusterware automatically maintains two copies of the OCR file and three copies of the voting disk file.
See Also:
Oracle Clusterware Administration and Deployment Guide for more information about managing voting disks
7.2 Managing the Oracle Clusterware Stack
By default, Oracle Clusterware is configured to restart whenever the server it resides on is restarted. During certain maintenance operations, you may be required to stop or start the Oracle Clusterware stack manually.
Note:
Do not use Oracle Clusterware Control (CRSCTL) commands on Oracle entities (such as resources, resource types, and server pools) that have names beginning with ora
unless you are directed to do so by Oracle Support. The Server Control utility (SRVCTL) is the correct utility to use on Oracle entities.
Topics:
- Starting Oracle Clusterware
You use the CRSCTL utility to manage Oracle Clusterware. - Stopping Oracle Clusterware
You use the CRSCTL utility to manage Oracle Clusterware.
See Also:
7.3 Administering Voting Disks for Oracle Clusterware
Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster.
Topics:
- Adding and Removing Voting Disks
You must manually add and remove voting disks if they are stored on local storage. - Backing Up and Recovering Voting Disks
You do not have to back up the voting disk. The voting disk data is automatically backed up in the Oracle Cluster Registry (OCR) as part of any configuration change and is automatically restored to any voting disk added. - Migrating Voting Disks to Oracle ASM Storage
You should store the Oracle Clusterware voting disk files in an Oracle ASM disk group.
7.3.1 Adding and Removing Voting Disks
You must manually add and remove voting disks if they are stored on local storage.
If you choose to store Oracle Clusterware files on Oracle ASM and use redundancy for the disk group, then Oracle ASM automatically maintains the ideal number of voting files based on the redundancy of the disk group.
If you use a different form of shared storage to store the voting disks, then you can dynamically add and remove voting disks after installing Oracle RAC. Do this using the following commands where path
is the fully qualified path for the additional voting disk.
To add or remove a voting disk that is stored on disk:
7.3.2 Backing Up and Recovering Voting Disks
You do not have to back up the voting disk. The voting disk data is automatically backed up in the Oracle Cluster Registry (OCR) as part of any configuration change and is automatically restored to any voting disk added.
Topics:
- Backing Up Voting Disks
The voting disk data is automatically backed up in Oracle Cluster Registry (OCR) as part of any configuration change so you do not have to perform manual backups of the voting disk. - Replacing Voting Disks
You replace a voting disk by deleting the unusable voting disk and then adding a new voting disk to your configuration. - Restoring Voting Disks
If all voting disks are corrupted, then you can restore them
See Also:
-
Oracle Clusterware Administration and Deployment Guide for more information about managing voting disks
7.3.2.1 Backing Up Voting Disks
The voting disk data is automatically backed up in Oracle Cluster Registry (OCR) as part of any configuration change so you do not have to perform manual backups of the voting disk.
The voting disk data is automatically backed up in OCR as part of any configuration change so you do not have to perform manual backups of the voting disk. The voting disk files are backed up automatically by Oracle Clusterware if the contents of the files have changed in the following ways:
-
Configuration parameters, for example
misscount
, have been added or modified -
After performing voting disk
add
ordelete
operations
See Also:
Oracle Clusterware Administration and Deployment Guide for more information about managing voting disks
7.3.2.2 Replacing Voting Disks
You replace a voting disk by deleting the unusable voting disk and then adding a new voting disk to your configuration.
If a voting disk is damaged, and no longer usable by Oracle Clusterware, then you can replace or re-create the voting disk. The voting disk contents are restored from a backup when a new voting file is added; this occurs regardless of whether the voting disk file is stored in Oracle Automatic Storage Management (Oracle ASM).
To replace a corrupt, damaged, or missing voting disk that is not stored in Oracle ASM:
See Also:
Oracle Clusterware Administration and Deployment Guide for more information about managing voting disks
7.3.3 Migrating Voting Disks to Oracle ASM Storage
You should store the Oracle Clusterware voting disk files in an Oracle ASM disk group.
If you choose to store your voting disks in Oracle ASM, then Oracle ASM stores all the voting disks for the cluster in the disk group you choose. You cannot combine voting disks stored in Oracle ASM and voting disks not stored in Oracle ASM in the same cluster.
The number of voting files you can store in a particular Oracle ASM disk group depends upon the redundancy of the disk group. By default, Oracle ASM puts each voting disk in its own failure group within the disk group. A normal redundancy disk group must contain at least two failure groups but if you are storing your voting disks on Oracle ASM, then a normal redundancy disk group must contain at least three failure groups. A high redundancy disk group must contain at least three failure groups.
Once you configure voting disks on Oracle ASM, you can only make changes to the voting disks' configuration using the crsctl replace votedisk
command. This is true even in cases where there are no working voting disks. Despite the fact that the crsctl query css votedisk
command reports zero voting disks in use, Oracle Clusterware remembers the fact that Oracle ASM was in use and the replace
verb is required. Only after you use the replace
verb to move voting disks back to non-Oracle ASM storage are the CRSCTL commands add css votedisk
and delete css votedisk
again usable.
To move voting disks from shared storage to an Oracle ASM disk group:
See Also:
-
Oracle Automatic Storage Management Administrator's Guide for more information about disk group compatibility attributes
-
Oracle Clusterware Administration and Deployment Guide for more information about managing voting disks
7.4 Backing Up and Recovering the Oracle Cluster Registry
Oracle Clusterware automatically creates Oracle Cluster Registry (OCR) backups every four hours. At any one time, Oracle Clusterware always retains the latest three backup copies of the OCR that are four hours old, one day old, and one week old.
You cannot customize the backup frequencies or the number of files that Oracle Clusterware retains. You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR file resides.
Topics:
- Viewing Available OCR Backups
Use theocrconfig
utility to view the backups generated automatically by Oracle Clusterware. - Manually Backing Up the OCR
Use theocrconfig
utility to force Oracle Clusterware to perform a backup of Oracle Cluster Registry (OCR) at any time, rather than wait for the automatic backup that occurs at four-hour intervals. - Recovering the OCR
There are two methods for recovering the OCR. The first method uses automatically generated OCR file copies and the second method uses manually created OCR export files.
7.4.1 Viewing Available OCR Backups
Use the ocrconfig
utility to view the backups generated automatically by Oracle Clusterware.
To find the most recent backup of the OCR:
7.4.2 Manually Backing Up the OCR
Use the ocrconfig
utility to force Oracle Clusterware to perform a backup of Oracle Cluster Registry (OCR) at any time, rather than wait for the automatic backup that occurs at four-hour intervals.
This option is especially useful when you want to obtain a binary backup on demand, such as before you make changes to OCR. The Oracle Local Registry (OLR) only supports manual backups.
To manually backup the contents of the OCR:
The default location for generating backups on Oracle Linux systems is Grid_home/cdata/cluster_name
where cluster_name is the name of your cluster and Grid_home is the home directory of the Oracle Grid Infrastructure for a cluster installation. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the ocrconfig
command as part of your operating system backup using standard operating system or third-party tools.
Note:
You can use the ocrconfig -backuploc
command to change the location where the OCR backups are created.
7.4.3 Recovering the OCR
There are two methods for recovering the OCR. The first method uses automatically generated OCR file copies and the second method uses manually created OCR export files.
Topics:
- Checking the Status of the OCR
In event of a failure, before you attempt to restore the Oracle Cluster Registry (OCR), ensure that the OCR is unavailable. - Restoring the OCR from Automatically Generated OCR Backups
When restoring the OCR from automatically generated backups, you first have to determine which backup file to use for the recovery.
7.4.3.1 Checking the Status of the OCR
In event of a failure, before you attempt to restore the Oracle Cluster Registry (OCR), ensure that the OCR is unavailable.
To check the status of the OCR:
See Also:
7.5 Changing the Oracle Cluster Registry Configuration
To ensure proper operation of Oracle Clusterware, make sure the OCR is highly available.
The OCR contains information about the cluster node list, which instances run on which nodes, and information about Oracle Clusterware resource profiles for applications that have been modified to be managed by Oracle Clusterware.
Note:
The operations in this section affect the OCR for the entire cluster. However, the ocrconfig
command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running. Avoid shutting down nodes while modifying the OCR using the ocrconfig
command.
Topics:
- Adding an OCR Location
You can add an Oracle Cluster Registry (OCR) location after an upgrade or after completing the Oracle RAC installation. - Migrating the OCR to Oracle ASM Storage
You should store the Oracle Cluster Registry (OCR) in an Oracle Automatic Storage Management (Oracle ASM) disk group. - Replacing an OCR
If you must change the location of an existing Oracle Cluster Registry (OCR), or change the location of a failed OCR to the location of a working one, then you can use this procedure if one OCR file remains online. - Removing an OCR
To remove an Oracle Cluster Registry (OCR) file, at least one copy of the OCR must be online. - Repairing an OCR Configuration on a Local Node
If a node in your cluster was not available when you modified the OCR configuration, then you must repair the OCR configuration on that node before it is restarted.
7.5.1 Adding an OCR Location
You can add an Oracle Cluster Registry (OCR) location after an upgrade or after completing the Oracle RAC installation.
Oracle Clusterware supports up to five OCR copies. Additional OCR copies provide greater fault tolerance.
To add an OCR file:
7.5.2 Migrating the OCR to Oracle ASM Storage
You should store the Oracle Cluster Registry (OCR) in an Oracle Automatic Storage Management (Oracle ASM) disk group.
To improve Oracle Clusterware storage manageability, OCR is configured, by default, to use Oracle ASM in Oracle Database 12c. However, if you upgrade from a previous release, then you can migrate OCR to reside on Oracle ASM, and take advantage of the improvements in managing Oracle Clusterware storage.
The OCR inherits the redundancy of the disk group. If you want high redundancy for the OCR, then you must create an Oracle ASM disk group with high redundancy. You should use a disk group with at least normal redundancy, unless you have an external mirroring solution. If you store the OCR in an Oracle ASM disk group, and the Oracle ASM instance fails on a node, then the OCR becomes unavailable only on that node. The failure of one Oracle ASM instance does not affect the availability of the entire cluster.
Oracle does not support storing the OCR on different storage types simultaneously, such as storing the OCR on both Oracle ASM and a shared file system, except during a migration. After you have migrated the OCR to Oracle ASM storage, you must delete the existing OCR files.
To move the OCR from shared storage to an Oracle ASM disk group:
See Also:
-
Oracle Automatic Storage Management Administrator's Guide for more information about disk group compatibility attributes
-
Oracle Clusterware Administration and Deployment Guide for more information about migrating the OCR to Oracle ASM
7.5.3 Replacing an OCR
If you must change the location of an existing Oracle Cluster Registry (OCR), or change the location of a failed OCR to the location of a working one, then you can use this procedure if one OCR file remains online.
To change the location of an OCR or replace an OCR file:
7.5.4 Removing an OCR
To remove an Oracle Cluster Registry (OCR) file, at least one copy of the OCR must be online.
You can remove an OCR location to reduce OCR-related overhead or to stop mirroring your OCR because you moved the OCR to a redundant storage system, such as a redundant array of independent disks (RAID).
To remove an OCR location from your cluster:
7.6 Troubleshooting the Oracle Cluster Registry
The Oracle Cluster Registry (OCR) manages Oracle Clusterware and Oracle RAC database configuration information.
Topics:
- About the OCRCHECK Utility
The OCRCHECK utility displays the data block format version used by the Oracle Cluster Registry (OCR), the available space and used space in the OCR, the ID used for the OCR, and the locations you have configured for the OCR. - Common Oracle Cluster Registry Problems and Solutions
The following table describes common Oracle Cluster Registry (OCR) problems and their corresponding solutions.
7.6.1 About the OCRCHECK Utility
The OCRCHECK utility displays the data block format version used by the Oracle Cluster Registry (OCR), the available space and used space in the OCR, the ID used for the OCR, and the locations you have configured for the OCR.
The OCRCHECK utility calculates a checksum for all the data blocks in all the OCRs that you have configured to verify the integrity of each block. It also returns an individual status for each OCR file and a result for the overall OCR integrity check. The following is a sample of the OCRCHECK output:
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262144
Used space (kbytes) : 16256
Available space (kbytes) : 245888
ID : 570929253
Device/File Name : +CRS_DATA
Device/File integrity check succeeded
...
Decive/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
The OCRCHECK utility creates a log file in the following directory, where Grid_home
is the location of the Oracle Grid Infrastructure for a cluster installation, and hostname
is the name of the local node: Grid_home/log/hostname/client
The log files have names of the form ocrcheck_nnnnn.log
, where nnnnn is the process ID of the operating session that issued the ocrcheck
command.
Related Topics
7.6.2 Common Oracle Cluster Registry Problems and Solutions
The following table describes common Oracle Cluster Registry (OCR) problems and their corresponding solutions.
Table 7-1 Common OCR Problems and Solutions
Problem | Solution |
---|---|
The OCR is not mirrored. |
Run the |
A copy of the OCR has failed and you must replace it. Error messages are being reported in Enterprise Manager or the OCR log file. |
Run the |
OCRCHECK does not find a valid OCR, or all copies of the OCR are corrupted. |
Run the |
The OCR configuration was updated incorrectly. |
Run the |
You are experiencing a severe performance effect from updating multiple OCR files, or you want to remove an OCR file for other reasons. |
Run the |
You want to change the location or storage type currently used by the OCR. |
Run the |