About 2 weeks ago I did run this patch PSU 12311357 on my UAT environment which having about the same setup with my Pre -production , 2 node RAC running on 11.2.0.2. Everything was went fine on UAT. Then the next day I applied this patch on Production Environment include single instance Dataguard.
First patch on Dataguard site was running very smooth, then continue with the Pre-Production on 1st node which I got stuck when doing "rootcrs.pl -unlock", I wait for about 1 hour and still not moving. Well there must be something wrong with this process. I checked all my CRS stack are gone and shutdown properly. On which file it still trying hard to unlock then ?
What I did that time is, I abort that session (CTRL+C). Then I try to run again the unlock command. Still it stucks there.
But this time I keep moving on to apply the patch since I don't have anymore time to open an SR, due to a critical fix that we need from this patch(I believe many people don't agree with me on my decision here). Well the patch was running fine without any issue. It also happen to 2nd node.
I open SR about this issue and turns out that it was hang due to many log files under GRID_HOME to unlock.
Here is the error message when that unlock command hang :
rootcrs_prd05istl1swmadb.log
....
2011-05-29 18:40:45: ###### Begin DIE Stack Trace ######
2011-05-29 18:40:45: Package File Line Calling
2011-05-29 18:40:45: --------------- -------------------- ---- ----------
2011-05-29 18:40:45: 1: main rootcrs.pl 325 crsconfig_lib::dietrap
2011-05-29 18:40:45: 2: crsconfig_lib s_crsconfig_lib.pm 350 main::__ANON__
2011-05-29 18:40:45: 3: crsconfig_lib s_crsconfig_lib.pm 350 (eval)
2011-05-29 18:40:45: 4: File::Find Find.pm 886 crsconfig_lib::reset_perms
2011-05-29 18:40:45: 5: File::Find Find.pm 700 File::Find::_find_dir
2011-05-29 18:40:45: 6: File::Find Find.pm 1229 File::Find::_find_opt
2011-05-29 18:40:45: 7: crsconfig_lib s_crsconfig_lib.pm 332 File::Find::finddepth
2011-05-29 18:40:45: 8: crsconfig_lib crsconfig_lib.pm 11198 crsconfig_lib::s_reset_crshome1
2011-05-29 18:40:45: 9: crsconfig_lib crsconfig_lib.pm 11209 crsconfig_lib::unlockCRSHomefordeinstall
2011-05-29 18:40:45: 10: main rootcrs.pl 382 crsconfig_lib::unlockCRSHome
2011-05-29 18:40:45: ####### End DIE Stack Trace #######
I still hold this issue, since so far I checked there is no way to just simulate running this unlock command without affecting the GRID stack. I will update again this article after I apply my next patch to see if this is related to my previous post "Oracle 11g R2 ASM Audit log generate a lot of files"
Update:
It is confirm due to a lot of file on Audit log cause the issue. After removing the files, I applied the next patch everything went very smooth. The rootcrs.pl -unlock actually doing permission change on files listed under GRID_HOME.
Update:
It is confirm due to a lot of file on Audit log cause the issue. After removing the files, I applied the next patch everything went very smooth. The rootcrs.pl -unlock actually doing permission change on files listed under GRID_HOME.
Articles Updated
ReplyDelete