WebLogic graceful shutdown waiting forever

thread A:

"[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.lang.Object@2eed24e4 WAITING java.lang.Object.wait(Native Method) weblogic.t3.srvr.GracefulShutdownRequest.waitForCompletion(GracefulShutdownRequest.java:62) weblogic.t3.srvr.ServerRuntime.shutdown(ServerRuntime.java:315) weblogic.server.RemoteLifeCycleOperationsImpl.shutdown(RemoteLifeCycleOperationsImpl.java:51) weblogic.server.RemoteLifeCycleOperationsImpl_WLSkel.invoke(Unknown Source)

thread B:

"[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.deploy.internal.targetserver.BasicDeployment$AdminModeCallback@53110ccd WAITING java.lang.Object.wait(Native Method) weblogic.deploy.internal.targetserver.BasicDeployment$AdminModeCallback.waitForCompletion(BasicDeployment.java:549) weblogic.deploy.internal.targetserver.BasicDeployment.gracefulProductionToAdmin(BasicDeployment.java:212) weblogic.deploy.internal.targetserver.BasicDeployment.productionToAdminFromServerLifecycle(BasicDeployment.java:431) weblogic.management.deploy.internal.DeploymentAdapter$1.gracefulProductionToAdmin(DeploymentAdapter.java:59) weblogic.management.deploy.internal.AppTransition$4.transitionApp(AppTransition.java:48) weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240) weblogic.management.deploy.internal.ConfiguredDeployments.productionToAdmin(ConfiguredDeployments.java:182) weblogic.management.deploy.internal.DeploymentPostAdminServerService.stop(DeploymentPostAdminServerService.java:36) weblogic.t3.srvr.ServerServicesManager.stopInternal(ServerServicesManager.java:495) weblogic.t3.srvr.ServerServicesManager.stop(ServerServicesManager.java:309) weblogic.t3.srvr.T3Srvr.suspend(T3Srvr.java:1155) weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1106) weblogic.t3.srvr.T3Srvr.gracefulShutdown(T3Srvr.java:1017) weblogic.t3.srvr.GracefulShutdownRequest.run(GracefulShutdownRequest.java:41) weblogic.work.ContextWrap.run(ContextWrap.java:41) weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528) weblogic.work.ExecuteThread.execute(ExecuteThread.java:209) weblogic.work.ExecuteThread.run(ExecuteThread.java:178)

We find out that

stopInternal(servicesAfterAdminState.toArray(), flag);

 cycles over all services.

 To observe which service is stuck we have to enable a debug flag:

private static final DebugCategory debugSLC = Debug.getCategory("weblogic.slc");
private static final DebugLogger debugSLCWLDF = DebugLogger.getDebugLogger("DebugServerLifeCycle");

This flag is under t3.srvr in weblogic server debug tab
(read this fascinating post on the topic)

If you enable the flags, and do the graceful shutdown, you will see stuff like:

BEA-000000 calling stop on weblogic.management.deploy.internal.DeploymentPostAdminServerService@7fead4cc


BEA-149059 Module jejb_transport.jar of application JEJB Transport Provider is transitioning from STATE_ACTIVE to STATE_ADMIN on server osbdv1ms1.

BEA-149060 Module jejb_transport.jar of application JEJB Transport Provider successfully transitioned from STATE_ACTIVE to STATE_ADMIN on server osbdv1ms1

In fact, the call to DeploymentPostAdminServerService.stop()  invokes a
ConfiguredDeployments.getConfigureDeploymentsHandler().productionToAdmin(true);

which again calls a transitionApps(AppTransition.GRACEFUL_PRODUCTION_TO_ADMIN)


You should also grep for "RunningToAdmin for app" (traced in BasicDeployment.productionToAdminFromServerLifecycle() )

so you should also enable the "DeploymentDebug" flag:
DebugDeploy DebugDeployment DebugDeploymentService DebugDeploymentServiceInternal DebugDeploymentServiceStatusUpdates DebugDeploymentServiceTransport DebugDeploymentServiceTransportHttp
having enabled these flags, you will see Sep 28, 2012 6:04:15 PM CEST Debug Deployment DmsThread-1 WLS Kernel 0000JcD_5_bE8Tk5ozx0iY1GPSV1000002 1348848255372 BEA-000000 Targets for app, DbAdapter
In practice, if you untarget the "JCA Transport Provider" the Server will shut down gracefully.

WebLogic graceful shutdown command

I believe most of us shutdown WebLogic with a kill "pid" .

If the process doesn't want to die, we do a kill -9 "pid" and we mutter "die, bastard!"

This is not very elegant and ladylike.

The alternative is through the Web Console, you can control each instance inclusive the Admin.

Otherwise you can use:
java weblogic.Admin -url t3://localhost:7001/ -username weblogic -password weblogic FORCESHUTDOWN yourserver

or the $DOMAIN_HOME/stopWebLogic.sh which in turns uses a WLST script invoking the WLST shutdown command

Here
http://docs.oracle.com/cd/E15523_01/web.1111/e13708/overview.htm
the official documentation on how to startup and shutdown