ILOM snapshot generally require for investigation of any hardware issue of node or storage server and it require to upload on oracle support.
Login to the interested compute/storage ilom of exadata with root user.
Extend ILOM Administration and click maintenance
---> Click Snapshot ----> Click Run
Snapshot automatically get downloaded ,You can see snapshot downloaded path
on the top of the right side page.
Upgrade TFACTL consist of two parts.
Uninstall the older version of TFA.
Install new version of TFA.
Status check of newly installed TFA
Uninstall the older version of TFA
root@pcsdbadm06:/u01/app/12.2.0.1/grid/bin/tfactl uninstall
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
Unable to determine a host list from TFA. So running local uninstall.
TFA will be uninstalled on node pcsdbadm06 :
Removing TFA from pcsdbadm06 only
Please remove TFA locally on any other configured nodes
Notifying Other Nodes about TFA Uninstall...
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.
FAIL
Sleeping for 10 seconds...
Stopping TFA Support Tools...
Stopping TFA in pcsdbadm06...
Shutting down TFA
oracle-tfa stop/waiting
. . . . .
Killing TFA running with pid 215885
. . .
Successfully shutdown TFA..
Deleting TFA support files on pcsdbadm06:
Removing /u01/app/oracle/tfa/pcsdbadm06/database...
Removing /u01/app/oracle/tfa/pcsdbadm06/log...
Removing /u01/app/oracle/tfa/pcsdbadm06/output...
Removing /u01/app/oracle/tfa/pcsdbadm06...
Removing /u01/app/oracle/tfa...
Removing /etc/rc.d/rc0.d/K17init.tfa
Removing /etc/rc.d/rc1.d/K17init.tfa
Removing /etc/rc.d/rc2.d/K17init.tfa
Removing /etc/rc.d/rc4.d/K17init.tfa
Removing /etc/rc.d/rc6.d/K17init.tfa
Removing /etc/init.d/init.tfa...
Removing /u01/app/12.2.0.1/grid/bin/tfactl...
Removing /u01/app/12.2.0.1/grid/tfa/bin...
Removing /u01/app/12.2.0.1/grid/tfa/pcsdbadm06...
Removing /u01/app/12.2.0.1/grid/tfa...
Install new version of TFA
Copy the software on the server and unzip it (/home/oracle)
root@pcsdbadm06:/home/oracle/tfa/installTFA-LINUX -crshome /u01/app/12.2.0.1/grid
TFA Installation Log will be written to File : /tmp/tfa_install_193064_2022_07_13-28_19_09.log
Starting TFA installation
TFA Version: 182100 Build Date: 202201291110
Running Auto Setup for TFA as user root...
Installing TFA now...
TFA Will be Installed on pcsdbadm06...
TFA will scan the following Directories
++++++++++++++++++++++++++++++++++++++++++++
.-------------------------------------------------------.
| pcsdbadm06 |
+--------------------------------------------+----------+
| Trace Directory | Resource |
+--------------------------------------------+----------+
| /u01/app/12.2.0.1/grid/OPatch/crs/log | CRS |
| /u01/app/12.2.0.1/grid/cfgtoollogs | CFGTOOLS |
| /u01/app/12.2.0.1/grid/crf/db/pcsdbadm06 | CRS |
| /u01/app/12.2.0.1/grid/crs/log | CRS |
| /u01/app/12.2.0.1/grid/cv/log | CRS |
| /u01/app/12.2.0.1/grid/evm/admin/log | CRS |
| /u01/app/12.2.0.1/grid/evm/admin/logger | CRS |
| /u01/app/12.2.0.1/grid/evm/log | CRS |
| /u01/app/12.2.0.1/grid/install | INSTALL |
| /u01/app/12.2.0.1/grid/log | CRS |
| /u01/app/12.2.0.1/grid/network/log | CRS |
| /u01/app/12.2.0.1/grid/oc4j/j2ee/home/log | DBWLM |
| /u01/app/12.2.0.1/grid/opmn/logs | CRS |
| /u01/app/12.2.0.1/grid/racg/log | CRS |
| /u01/app/12.2.0.1/grid/rdbms/log | ASM |
| /u01/app/12.2.0.1/grid/rdbms/trace | ASM |
| /u01/app/12.2.0.1/grid/scheduler/log | CRS |
| /u01/app/12.2.0.1/grid/srvm/log | CRS |
| /u01/app/oraInventory/ContentsXML | INSTALL |
| /u01/app/oraInventory/logs | INSTALL |
| /u01/app/oracle/crsdata/pcsdbadm06/acfs | ACFS |
| /u01/app/oracle/crsdata/pcsdbadm06/core | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/crsconfig | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/crsdiag | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/cvu | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/evm | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/output | CRS |
| /u01/app/oracle/crsdata/pcsdbadm06/rhp | RHP |
| /u01/app/oracle/crsdata/pcsdbadm06/trace | CRS |
'--------------------------------------------+----------'
Installing TFA on pcsdbadm06:
HOST: pcsdbadm06 TFA_HOME: /u01/app/12.2.0.1/grid/tfa/pcsdbadm06/tfa_home
.------------------------------------------------------------------------------.
| Host | Status of TFA | PID | Port | Version | Build ID |
+----------+---------------+--------+------+------------+----------------------+
| pcsdbadm06 | RUNNING | 240560 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db01 | RUNNING | 160373 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db02 | RUNNING | 353976 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db03 | RUNNING | 335037 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db04 | RUNNING | 190206 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db05 | RUNNING | 43582 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db06 | RUNNING | 307943 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
| pr01db07 | RUNNING | 204143 | 5000 | 18.2.1.0.0 | 18210020180529111033 |
'----------+---------------+--------+------+------------+----------------------'
Running Inventory in All Nodes...
Enabling Access for Non-root Users on pcsdbadm06...
Adding default users to TFA Access list...
Summary of TFA Installation:
.--------------------------------------------------------------------.
| pcsdbadm06 |
+---------------------+----------------------------------------------+
| Parameter | Value |
+---------------------+----------------------------------------------+
| Install location | /u01/app/12.2.0.1/grid/tfa/pcsdbadm06/tfa_home |
| Repository location | /u01/app/oracle/tfa/repository |
| Repository usage | 0 MB out of 10240 MB |
'---------------------+----------------------------------------------'
Installing oratop extension..
TFA is successfully installed...
Status check of newly installed TFA
root@pcsdbadm06:/u01/app/12.1.0.2/grid/bin/tfactl status
.-------------------------------------------------------------------------------------------------.
| Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status |
+----------+---------------+--------+------+------------+----------------------+------------------+
| pcsdbadm11 | RUNNING | 377483 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm01 | RUNNING | 393184 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm02 | RUNNING | 119709 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm03 | RUNNING | 88747 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm04 | RUNNING | 295305 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm05 | RUNNING | 482 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm06 | RUNNING | 212883 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm07 | RUNNING | 110574 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| pcsdbadm08 | RUNNING | 354894 | 5000 | 18.2.1.0.0 | 18210020180543111033 | COMPLETE |
| |
'----------+---------------+--------+------+------------+----------------------+------------------'
Oracle Platinum Support that provides auto-case generation when specific hardware faults occur for each components in Exadata or other oracle engineered system like super cluster ,zdlra(zero data loss appliance).On brand new system first ASR require to configure to generate alerts when hardware issue occurs. Those alerts will be received by oracle platinum support team through oracle gateway, and it will generate the service request automatically (ASR). There is separate gateway server which accepts fault telemetry data sent from one or more assets.
You will find the gateway server or IP information from /etc/hosts files of database server. We should perform ASR validation test for each components periodically to ensure ASR alert triggered after fault in components.
Now the question is how to perform ASR validation test.
First check whether exadata or engineered system are registered over https://support.oracle.com going to tab System and check the column ASR status.
ASR status either Active / Asset is qualified for ASR,but has not been activated or Pending.
If it is Active ,then you suppose to get alert email/Automated ASR creation or every fault in exadata component.
Now lets discuss about "How to perform manual ASR validation test"
Before ASR validation, Perform below three actions.
Create an email to direct all ASR test emails.
2. Check email is registered for the component you are going to test by clicking the system from the Asset Type (support.oracle.com).
3. Check "smtpToAddr" for database server or storage server have same email.
dcli -g /shared/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep smtpToAddr"
ASR test at cluster rack level.
1. Enable direct root login if not enable.
PermitRootLogin should be yes (edit /etc/ssh/sshd_config )
2. Go to exachk folder.
cd /opt/oracle.ahf/exachk
./exachk - profile hardware
You need to have latest version of exachk.
3. Wait for the completion of exachk and validate email received from each assets of the cluster.
4. Count emails received from each assets of the cluster, to ensure alert should be send out from each assets.
If there is no alert generate from assets ,then test it individually from the assets
ASR test for single server.
Storage server :
1. Login to the first database server(Root user) and so ssh to storage server.
2. # cellcli -e "alter cell validate snmp type=asr"
you will receive email alert from this storage server.
Database server :
1. Login to the the database server from root user.
2. Execute dbmcli -e "alter dbserver validate snmp type=asr"
you will receive email alert from this storage server.
