FYI: OPMNCTL Failed to Start Managed Process After Maximum Retry Limit on Solaris 10, OBIEE 11g

If you've happened to glance at Oracle Technical Network, it appears that alot of practitioners are encountering an issue when upgrading their 11g platform to the newest 11.1.1.6.4 version if they're using a Solaris or Linux distribution.

After upgrading to 11.1.1.6.4 as outlined in my previous blog post some users encounter the following error(s) when trying to start their BI services using opmnctl (Oracle Process Manager and Notification Server) with the 'opmnctl startall' command:

================================================================================
Response: 1 of 5 processes started.

ias-instance id=instance1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---

ias-component/process-type/process-set:
coreapplication_obis1/OracleBIServerComponent/coreapplication_obis1/

Error
--> Process (index=1,uid=1445883997,pid=18522)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log

ias-component/process-type/process-set:
coreapplication_obips1/OracleBIPresentationServicesComponent/coreapplication_obips1/

Error
--> Process (index=1,uid=1445883998,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
failed to start a managed process after the maximum retry limit
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
Log:
none

ias-component/process-type/process-set:
coreapplication_obisch1/OracleBISchedulerComponent/coreapplication_obisch1/

Error
--> Process (index=1,uid=1445884000,pid=18523)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

ias-component/process-type/process-set:
coreapplication_obiccs1/OracleBIClusterControllerComponent/BIClusterController/

Error
--> Process (index=1,uid=1445884001,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/server/bin/nqsclustercontroller
failed to start a managed process after the maximum retry limit

 The important things to note are:

• You may encounter all, some, or just 1 of your bi_services which fail to start. In the above example, the following services failed to start:
• BI Server
• BI Publisher
• BI Cluster Controller
• BI Scheduler
• There are actually two issues:
• Managed Processes failed to start after the maximum retry limit
• Certain executable files do not have the execute permission enabled

Issue 1: Fix Maximum Retry Limit

This can be easily resolved by setting the ulimit at the user level to : ulimit -d unlimited , as outlined in Oracle's note ID: 1292887.1 . Anyone who has worked in OBIEE 10g should be familiar with this as it is required in 10g's installation process as well.

Issue 2: Enable read access on executable files

If you tried re-running opmnctl startall after resolving issue 1, you'll notice that the BI services still fail to run despite increasing the maximum retry limit. Upon examining the log files as outlined in the above error:

• u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log
• u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

You'll notice that OPMNCTL still fails because it can't execute: nqsserver, nqscheduler and nqsclustercontroller. Below is the log from the above bullet point #1 :

/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute--------
12/11/06 15:48:06 Start process
--------
/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute

 During the 11.1.1.6.4 upgrade on your Solaris or Linux box, these files became read only!

How to resolve this read only issue:

Navigate to your $ORACLE_HOME/bifoundation/server/bin folder and make the the following files executable by using the chmod +x command:  nqscheduler, nqsclustercontroller, nqsserver. These files should have read/write privileges as outlined in the image below:

Now when you run opmnctl startall, all  services should execute successfully:

Important:

After the upgrade, it is critical that you follow Oracle's recommended start up & shutdown sequence for weblogic & OBIEE 11g

Start up:

• $DOMAIN_HOME/bin/startWebLogic.sh (Weblogic)
• $WL_HOME/server/bin/startNodeManager.sh (Node Manager)
• $DOMAIN_HOME/bin/startManagedWebLogic.sh bi_server1 http://server.domain:7001 (Managed Server)
• $ORACLE_INSTANCE/bin/opmnctl startall (OPMN)

Shutdown:

• $ORACLE_INSTANCE/bin/opmnctl stopall (OPMN)
• $DOMAIN_HOME/bin/stopManagedWebLogic.sh bi_server1 t3://host.domain:7001 username password (Managed Server)
• $DOMAIN_HOME/bin/stopWebLogic.sh username password t3://host.domain:7001 (Admin Server)

In my upcoming blog post i'll discuss how to automate this entire process.

keywords: opmn , opmn start failed, obiee 11g, opmn obiee weblogic, bi_server