For details please see this note on Oracle Support (361284.1).
On one of our system, this issue suddenly come up when some application that calling tnsnames to use the loadbalancing intermittently failed with ORA-12541. Initially I thought it is a connection issue, but further I do a trace from client side and found below :
....
[000001 05-APR-2011 14:55:15:106] nspsend: 01 09 00 00 06 00 00 00 |........|
[000001 05-APR-2011 14:55:15:106] nspsend: 00 00 28 44 45 53 43 52 |..(DESCR|
[000001 05-APR-2011 14:55:15:106] nspsend: 49 50 54 49 4F 4E 3D 28 |IPTION=(|
[000001 05-APR-2011 14:55:15:106] nspsend: 41 44 44 52 45 53 53 3D |ADDRESS=|
[000001 05-APR-2011 14:55:15:106] nspsend: 28 50 52 4F 54 4F 43 4F |(PROTOCO|
[000001 05-APR-2011 14:55:15:106] nspsend: 4C 3D 54 43 50 29 28 48 |L=TCP)(H|
[000001 05-APR-2011 14:55:15:106] nspsend: 4F 53 54 3D 6D 6E 65 74 |OST=mnet|
[000001 05-APR-2011 14:55:15:106] nspsend: 64 62 32 2D 76 69 70 29 |db2-vip)|
[000001 05-APR-2011 14:55:15:106] nspsend: 28 50 4F 52 54 3D 31 35 |(PORT=15|
[000001 05-APR-2011 14:55:15:106] nspsend: 32 31 29 29 28 4C 4F 41 |21))(LOA|
[000001 05-APR-2011 14:55:15:106] nspsend: 44 5F 42 41 4C 41 4E 43 |D_BALANC|
[000001 05-APR-2011 14:55:15:106] nspsend: 45 3D 79 65 73 29 28 43 |E=yes)(C|
[000001 05-APR-2011 14:55:15:106] nspsend: 4F 4E 4E 45 43 54 5F 44 |ONNECT_D|
[000001 05-APR-2011 14:55:15:106] nspsend: 41 54 41 3D 28 53 45 52 |ATA=(SER|
[000001 05-APR-2011 14:55:15:106] nspsend: 56 45 52 3D 44 45 44 49 |VER=DEDI|
[000001 05-APR-2011 14:55:15:106] nspsend: 43 41 54 45 44 29 28 53 |CATED)(S|
[000001 05-APR-2011 14:55:15:106] nspsend: 45 52 56 49 43 45 5F 4E |ERVICE_N|
[000001 05-APR-2011 14:55:15:106] nspsend: 41 4D 45 3D 43 48 4E 4E |AME=CHNN|
[000001 05-APR-2011 14:55:15:106] nspsend: 45 4C 29 28 46 41 49 4C |EL)(FAIL|
[000001 05-APR-2011 14:55:15:106] nspsend: 4F 56 45 52 5F 4D 4F 44 |OVER_MOD|
[000001 05-APR-2011 14:55:15:106] nspsend: 45 3D 28 54 59 50 45 3D |E=(TYPE=| ----> Here the client has sent connect packet to the listener on mnetdb2-vip
[000001 05-APR-2011 14:55:15:106] nspsend: 53 45 4C 45 43 54 29 28 |SELECT)(|
[000001 05-APR-2011 14:55:15:106] nspsend: 4D 45 54 48 4F 44 3D 42 |METHOD=B|
[000001 05-APR-2011 14:55:15:106] nspsend: 41 53 49 43 29 28 52 45 |ASIC)(RE|
[000001 05-APR-2011 14:55:15:106] nspsend: 54 52 49 45 53 3D 31 38 |TRIES=18|
[000001 05-APR-2011 14:55:15:106] nspsend: 30 29 28 44 45 4C 41 59 |0)(DELAY|
[000001 05-APR-2011 14:55:15:106] nspsend: 3D 35 29 29 28 43 49 44 |=5))(CID|
[000001 05-APR-2011 14:55:15:106] nspsend: 3D 28 50 52 4F 47 52 41 |=(PROGRA|
[000001 05-APR-2011 14:55:15:106] nspsend: 4D 3D 73 71 6C 70 6C 75 |M=sqlplu|
[000001 05-APR-2011 14:55:15:106] nspsend: 73 29 28 48 4F 53 54 3D |s)(HOST=|
[000001 05-APR-2011 14:55:15:106] nspsend: 6D 6E 65 74 61 70 70 33 |mnetapp3|
[000001 05-APR-2011 14:55:15:106] nspsend: 29 28 55 53 45 52 3D 6F |)(USER=o|
[000001 05-APR-2011 14:55:15:106] nspsend: 72 61 63 6C 65 29 29 29 |racle)))|
.....
[000001 05-APR-2011 14:55:15:110] nscall: connecting...
[000001 05-APR-2011 14:55:15:110] nsc2addr: entry
[000001 05-APR-2011 14:55:15:110] nsc2addr: (ADDRESS=(PROTOCOL=TCP)(HOST=mnetdb1-vip)(PORT=33949))
[000001 05-APR-2011 14:55:15:110] nttbnd2addr: entry
[000001 05-APR-2011 14:55:15:110] snlinGetAddrInfo: entry
[000001 05-APR-2011 14:55:15:110] snlinGetAddrInfo: Invalid IP address string mnetdb1-vip ----> Here the transport was unable to open to the PORT 33949 on mnetdb1-vip
[000001 05-APR-2011 14:55:15:110] snlinFreeAddrInfo: entry
At that point we notice it assign to port number 33949 which wrong port number. We only use port 1521.
The challenge now is to ask the network engineer to change the firewall rule. In our case the firewall use by many other subsytem. Well this issue never been an issue for all other application that using Weblogic.
