Thursday, October 3, 2013

Configuring NIC network interface on RHEL

Note: ipconfig is deprecated in favour of ip, however a lot of legacy code still uses it.
Very interesting tutorial here and here
This is a UI to edit all configuration without requiring root privileges, bur probably if you are not root you won'et even be able to see any configuration values - forget about changing them:

/usr/sbin/system-config-network-tui

This is what you get:



ââââââââââ⤠Network Configuration âââââââââââ
â                                           â
â                                           â
â Name                 eth0________________ â
â Device               eth0________________ â
â Use DHCP             [*]                  â
â Static IP            ____________________ â
â Netmask              ____________________ â
â Default gateway IP   ____________________ â
â Primary DNS Server   ____________________ â
â Secondary DNS Server ____________________ â
â                                           â
â       ââââââ            ââââââââââ        â
â       â Ok â            â Cancel â        â
â       ââââââ            ââââââââââ        â
â                                           â
â                                           â
âââââââââââââââââââââââââââââââââââââââââââââ



this is equivalent, but requires root privileges:

/usr/bin/system-config-network

They don't require a X-terminal session, they support a text-ui mode.

To manually hack the configuration:
cd /etc/sysconfig/network-scripts/
less ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="08:00:27:29:06:6F"
IPV6INIT="yes"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="ca6ef823-ed13-46ad-8a83-9d688f6cf239"


Some other "global" info is here:
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=osb-vagrant.acme.com

"/sbin/ip -o addr" displays all IPs
to add an extra IP, you must specify an interface eth0:N where N > 0:
/sbin/ifconfig eth0:1 10.0.2.16 netmask 255.255.255.0

(there is an implicit "up" at the end)
Entering "/sbin/ifconfig eth0 10.0.2.16 netmask 255.255.255.0" will crash the NIC, removing the current primary address to replace with the new one.



Wednesday, October 2, 2013

oracle.repackaged.ucp.jdbc.oracle.RACCallbackGuard

today we saw this in the logs:

####<Oct 2, 2013 8:47:23 AM CEST> <Warning>
 <oracle.repackaged.ucp.jdbc.oracle.RACCallbackGuard> <acme111>
 <osbpr1ms4> <Thread-100> <<anonymous>> <> 
<0000K5o7QgvE4Uk5ozS4yY1IIgWK000003> <1380696443473> <BEA-000000> 
<RAC callback: guarded method threw exception>



no idea what happened.... it could be a RAC instance dead and TX being switched to another instance... still under investigation...

Ok, what happened is that the DBA team started the second node of the RAC cluster, and the JDBCDrivers somehow got the notification, but something must have gone wrong.


Tuesday, October 1, 2013

Consensus-based server migration: caveat

I used to have a Database-based lease mechanism for Server Migration, but occasionally it was failing ("unable to contact DB", no clue why...) and the server would restart itself.
We changed to Consensus, hoping the network would be more robust. However, due to a network reconfiguration, some IPs were left undefined and the cluster broke:

<[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <f56960fb001bca18:-33f939f3:1413c1039a1:-8000-0000000000065ea0> <1380619475997> <BEA-000802> <ExecuteRequest failed
java.lang.AssertionError: Invalid state transition from failed to stable_leader.
java.lang.AssertionError: Invalid state transition from failed to stable_leader
        at weblogic.cluster.leasing.databaseless.ClusterState.setState(ClusterState.java:100)
        at weblogic.cluster.leasing.databaseless.ClusterState.setState(ClusterState.java:59)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.leaderInitialization(ClusterFormationServiceImpl.java:318)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.formClusterInternal(ClusterFormationServiceImpl.java:148)
        at weblogic.cluster.leasing.databaseless.ClusterFormationServiceImpl.timerExpired(ClusterFormationServiceImpl.java:339)
        at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>


The issue is that once the network has been fixed, the cluster didn't recover and we had to restart the servers... however this could simply be because when we restart the server, the Virtual IP associated to each server is readded to the NIC (/sbin/ifconfig -addif). Instead of restarting the servers I should have tried to add the IP manually... one should really monitor continuously the availability of those IPs...