Friday, November 30, 2012

FYI: Log could not be retrieved - nQSError: 43113, nQSError: 10058

I've noticed something a little surprising with the OBIEE 11g upgrade assistant & opatch tool. After you apply a patch set (e.g. 11.1.1.6.4), core BIServer components become 'locked' or a default file privilege gets applied that is too restrictive.

In fact, in my last post I covered a similar issue with Oracle Process Manager and Notification Server (OPMN) failing to start after applying a bundle patch, you can read it here .

This post follows a similar issue, in that logging in OBIEE 11g is non-functional after applying a patch set:

Log Could Not Be Retrieved
  Odbc driver returned an error (SQLExecDirectW).
  Error Details
Error Codes: OPR4ONWY:U9IM8TAC
State: HY000. Code: 10058. [NQODBC] [SQL_STATE: HY000] [nQSError: 10058] A general error has occurred. [nQSError: 43113] Message returned from OBIS. [nQSError: 43100] Log Viewer process reported error 2:No such file or directory. (HY000)
Oracle Technical Network has, unfortunately, no documentation on this issue of BIServer components file privileges becoming too restrictive. If an OBIEE Administrator wants to enable logging in their Answers environment, it is usually achieved by setting the LOGLEVEL to a value greater than 0 or less than 6:



This will not work IF your Answers environment is unable to identify or access the log file or directory. So the question is:

Why is access to the log file & directory disabled in my Answers environment and how do I enable access?

First: Understand that Presentation Logging in OBIEE 10g and OBIEE 11g is captured in your nqquery.log file located at: $ORACLE_INSTANCE_HOME\diagnostics\OracleBIServerComponent\..\..\nqquery.log

Second: The mechanism Answers uses for reading a log file is the nqlogviewer executable which is located in your $ORACLE_HOME\bifoundation\server\bin folder

So the error nQSError:43113 leads me to believe that insufficient privleges has been set for the nqlogviewer executable.  I assume this is true because similiarly in my previous post core BI components nqserver, nqscheduler, etc, were all inaccessible.

So to resolve this, apply the following command to your nqlogviewer : chmod 777 nqlogviwer (777 enables access to ALL USERs, so make sure to replace it with the parameter that allows read/write access to your obiee account):



Now go back to the same report, and viola, the session log is visible - no restart required!



Keywords: nqserror, obiee 11g installation, obiee 11.1.1.6 upgrade, obiee upgrade assistant, obiee 11g, opatch, opmn start failed


Monday, November 26, 2012

How-to: OBIEE 11g Start up/Shut Down & Script in Solaris/Linux

'Start BI Services'

This simple, easy to use process for starting & stopping Weblogic & OBIEE 11g in Windows is something many developers take for granted. In personal development and proof of concept environments a Windows environment will suffice, but once we start talking about enterprise testing & production environments we quickly realize that a windows based Weblogic and OBIEE 11g environment just won't work.

We then start down the path of deployment on a Redhat or Solaris environment. After reading Oracle's official guide on starting and stopping OBIEE 11g, we quickly realize just how lucky we were. No longer do we have the option of executing a single file to start and shut down Weblogic and OBIEE, at least not out of the box.

This request is commonly echoed on Oracle Technical Network so today we'll outline how to create a single file that acts as both a start up and shutdown script.

First, let's go over the manual start up process for Weblogic & OBIEE 11g in Solaris & Linux:
  • Start Administration Server
    • located in your Weblogic Server Domain user_projects folder
    • /user_projects/domains/bifoundation/bin/startWebLogic.sh
    • example: ./startWebLogic.sh
  • Start Node Manager
    • located in your Weblogic Server Home Directory
    • /wlserver_10.3/server/bin/startNodeManager.sh
    • example: ./startNodeManager.sh
  • Start Managed Server
    • located in your Weblogic Server Domain user_projects folder
    • /user_projects/domains/bifoundation/bin/startManagedWebLogic.sh
    • example: ./startManagedWebLogic.sh bi_server1 t3://HOSTNAME:7001
  • Start BI Services using Oracle Process Manager and Notification Server (opmnctl)
    • located in your Orcle Instance folder
    • /instances/instance1/bin/opmnctl
    • example: opmnctl startall
Second, let's go over the manual shut down process of Weblogic & OBIEE 11g in Solaris & Linux:

  • Stop BI Services using Oracle Process Manager and Notification Server (opmnctl)
    • located in your Orcle Instance folder
    • /instances/instance1/bin/opmnctl
    • example: opmnctl stopall
  • Stop Managed Server
    • located in your Weblogic Server Domain user_projects folder
    • /user_projects/domains/bifoundation/bin/stopManagedWebLogic.sh
    • example: ./stopManagedWebLogic.sh bi_server1 t3://HOSTNAME:7001
  • Stop Node Manager
    • There is no 'stopNodeManager', must kill associated proccesses
    • example: pkill -TERM -u USERNAME -f "/startNodeManager.sh"
    • example: pkill -TERM -u USERNAME -f "/java"
  • Stop Admin Server
    • located in your Weblogic Server Domain user_projects folder
    • /user_projects/domains/bifoundation/bin/stopWebLogic.sh
    • example: ./stopWebLogic.sh

So how do we automate this process?

By customizing the shell script below, you'll be able to automate the start up and shutdown sequence in OBIEE 11g.

The lines highlighted in red require customization.

#!/bin/bash
#
# File:    obiee.sh

# Purpose: Start and stop Oracle Business Intelligence 11g components.
#

# description: Start up and shutdown sequence for OBIEE 11g and Weblogic
#

# These values must be adapted to your environment.

ORACLE_OWNR=username                  # Local Unix user running OBIEE
ORACLE_FMW=/export/obiee/11g      # Deployment root directory
                                  
BIEE_USER=weblogic                # BIEE administrator name
BIEE_PASSWD=weblogic              # BIEE administrator password              
BIEE_DOMAIN=bifoundation_domain         # Domain name
BIEE_INSTANCE=instance1             # Instance name
BIEE_SERVER=bi_server1              # Server name
BIEE_MANAGER_URL=hostname:7001    # Admin server URL (hostname:port)
 

WL_PATH=/export/obiee/11g/wlserver_10.3/server/bin
BIEE_PATH=/export/obiee/11g/user_projects/domains/bifoundation_domain/bin
ORACLE_INSTANCE=/export/obiee/11g/instances/instance1

export ORACLE_INSTANCE

START_LOG=/export/obiee/11g/obiee_startup_log/obiee-start.log
STOP_LOG=/export/obiee/11g/obiee_startup_log/obiee-stop.log


# SUBSYS=obiee

start() {
    echo "********************************************************************************"
    echo "Starting Admin Server on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$BIEE_PATH/startWebLogic.sh" &
    wait_for "Server started in RUNNING mode"
   
    echo "********************************************************************************"
    echo "Starting Node Manager on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$WL_PATH/startNodeManager.sh" &
    wait_for "socket listener started on port"

    echo "********************************************************************************"
    echo "Starting Managed Server $BIEE_SERVER on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$BIEE_PATH/startManagedWebLogic.sh $BIEE_SERVER t3://$BIEE_MANAGER_URL" &
    wait_for "Server started in RUNNING mode"

    echo "********************************************************************************"
    echo "Starting BI components on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$ORACLE_INSTANCE/bin/opmnctl startall"

    echo "********************************************************************************"
    echo "OBIEE start sequence completed on $(date)"
    echo "********************************************************************************"
}

stop() {
    echo "********************************************************************************"
    echo "Stopping BI components on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$ORACLE_INSTANCE/bin/opmnctl stopall"

    echo "********************************************************************************"
    echo "Stopping Managed Server $BIEE_SERVER on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$BIEE_PATH/stopManagedWebLogic.sh $BIEE_SERVER t3://$BIEE_MANAGER_URL $BIEE_USER $BIEE_PASSWD"

    echo "********************************************************************************"
    echo "Stopping Node Manager on $(date)"
    echo "********************************************************************************"
   # pkill -TERM -u $ORACLE_OWNR -f "weblogic\\.NodeManager"
    pkill -TERM -u $ORACLE_OWNR -f "/bin/sh/export/obiee/11g/wlserver_10.3/server/bin/startNodeManager.sh"
 pkill -TERM -u $ORACLE_OWNR -f "/export/obiee/11g/Oracle_BI1/jdk/bin/sparcv9/java"
    echo "********************************************************************************"
    echo "Stopping Admin Server on $(date)"
    echo "********************************************************************************"
    su $ORACLE_OWNR -c "$BIEE_PATH/stopWebLogic.sh"
   
    echo "********************************************************************************"
    echo "OBIEE stop sequence completed on $(date)"
    echo "********************************************************************************"
}

wait_for() {
    res=0
    while [[ ! $res -gt 0 ]]
    do
        res=$(tail -5 "$START_LOG" | fgrep -c "$1")
        sleep 5
    done
}

case "$1" in
    start)
        echo "********************************************************************************"
        echo "Starting Oracle Business Intelligence on $(date)"
        echo "Logs are sent to $START_LOG"
        echo "********************************************************************************"
        start &> $START_LOG
       # touch /var/lock/subsys/$SUBSYS
    ;;
    stop)
        echo

"********************************************************************************"
        echo "Stopping Oracle Business Intelligence on $(date)"
        echo "Logs are sent to $STOP_LOG"
        echo "********************************************************************************"
        stop &> $STOP_LOG
       # rm -f /var/lock/subsys/$SUBSYS
    ;;
    status)
        echo "********************************************************************************"
        echo "Oracle BIEE components status...."
        echo "********************************************************************************"
        su $ORACLE_OWNR -c "$ORACLE_INSTANCE/bin/opmnctl status"
    ;;
    restart)
        $0 stop
        $0 start
    ;;
    *)
        echo "Usage: $(basename $0) start|stop|restart|status"
        exit 1
esac

exit 0



You must add your admin credentials to the following files:

  • /user_projects/domains/bifoundation_domain/bin/startManagedWebLogic.sh


  • /user_projects/domains/bifoundation_domain/servers/AdminServer/security/boot.properties

Start the script as follows:


The script will generate the following message after completion:



You can validate all services are running by logging into Fusion Middleware and Weblogic to check AdminServer, ManagedServer, and BI Services:


AdminServer & ManagedServer

BI Services



Stop OBIEE 11g and Weblogic by running the following command:



The script will generate the following message after completion:


You can validate that AdminServer, ManagedServer and BI Services are all shutdown by running ps -ef | grep obiee in your terminal. No weblogic & OBIEE processes should be active:



Note: if you are editing your shell script in a windows environment then uploading it to a Solaris/Linux environment via FTP, you will encounter whitespace issues when attempting to execute. This can be fixed by running the following command on the file:

perl -i -pe's/\r$//;' <file name here>






keywords: obiee start up script, obiee shut down script, obiee 11g installation, obiee admin

Monday, November 12, 2012

FYI: Error while running Upgrade Assistant OBIEE 11g - UPGAST-00055

When migrating our legacy OBIEE 10g webcat and RPD to Oracle's new 11g platform, we often don't have the privilege of using a GUI based operating system for assembly & system test as most production systems are Solaris or Red Hat.

We can all agree that a unix distribution is more powerful & flexible, yet leaves much to be desired when attempting to resolve configuration issue in a 'friendly UI environment'. This is certainly no exception when attempting to use Oracle's Upgrade Assistant Command Line Interface .

Many users have encountered error " UPGAST-00055: error reading the Universal Installer Inventory. The inventory pointer oraInst.loc is either not readable or doest not exist". A quick search of Oracle's Fusion Middleware Error Message Reference Guide makes me believe that Oracle's writers are playing a cruel joke in their attempt to 'help':








Action: "see the secondary error message?" You've got to be kidding me!!

Oracle's 'help' aside, the upgrade assistant is looking for an oraInst.loc file which contains 2 things:
  1. The location of your installation directory
  2. The install group of your user
But the upgrade assistant cant find your oraInst.loc file, why?
UA, by default, searches for the oraInst.loc file in your /var/opt/oracle folder. But many times, the file won't be located there - for a number of reasons , including:
  • custom installation directories
  • co-installation of multiple oracle products
  • de-installation and subsequent re-installation attempts
But you're not out of luck. Most likely, your oraInst.loc file will be in your $MW_HOME/oracle_common folder:















You can then use the UA CLI paramater -invPtrLoc to specify your custom oraInst.loc path.:




./ua MT -BIEEE -webcatdir /export/obiee/11g/Oracle_BI1/bin/webcatalog -invPtrLoc /export/obiee/11g/oracle_common/oraInst.loc -webCatDeliversDir /delivers -wlsPort 7001 -wlsUser weblogic
will generate the following:








followed by a series of postupgrade tasks that you can view at 'tail -f $ORACLE_HOME/upgrade/logs/postupgrade.txt'
and ultimately a 'completed successfully' message:






































But what if you can't find your oraInst.loc file? Don't worry! Oracle has a template you can use located at:
  • /$ORACLE_HOME//bifoundation/provision/scripts/bidomain/inst
  1. Rename it to oraInst.loc
  2. Copy it to your $ORACLE_HOME folder
  3. Populate the inventory_loc line with the path to your oraInventory folder
  4. Populate the inst_group with the group name of the account you're using to install & upgrade OBIEE
  5. Run the upgrade assistant as specified above

Problem solved!

keywords : obiee 11g upgrade, upgrade assistant,  obiee administration, obiee 11g administration, weblogic obiee

Wednesday, November 7, 2012

FYI: OPMNCTL Failed to Start Managed Process After Maximum Retry Limit on Solaris 10, OBIEE 11g

If you've happened to glance at Oracle Technical Network, it appears that alot of practitioners are encountering an issue when upgrading their 11g platform to the newest 11.1.1.6.4 version if they're using a Solaris or Linux distribution.

After upgrading to 11.1.1.6.4 as outlined in my previous blog post some users encounter the following error(s) when trying to start their BI services using opmnctl (Oracle Process Manager and Notification Server) with the 'opmnctl startall' command:

================================================================================
Response: 1 of 5 processes started.

ias-instance id=instance1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



ias-component/process-type/process-set:
coreapplication_obis1/OracleBIServerComponent/coreapplication_obis1/

Error
--> Process (index=1,uid=1445883997,pid=18522)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log

ias-component/process-type/process-set:
coreapplication_obips1/OracleBIPresentationServicesComponent/coreapplication_obips1/

Error
--> Process (index=1,uid=1445883998,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
failed to start a managed process after the maximum retry limit
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
Log:
none

ias-component/process-type/process-set:
coreapplication_obisch1/OracleBISchedulerComponent/coreapplication_obisch1/

Error
--> Process (index=1,uid=1445884000,pid=18523)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

ias-component/process-type/process-set:
coreapplication_obiccs1/OracleBIClusterControllerComponent/BIClusterController/

Error
--> Process (index=1,uid=1445884001,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/server/bin/nqsclustercontroller
failed to start a managed process after the maximum retry limit


 The important things to note are:
  • You may encounter all, some, or just 1 of your bi_services which fail to start. In the above example, the following services failed to start:
    • BI Server
    • BI Publisher
    • BI Cluster Controller
    • BI Scheduler
  • There are actually two issues:
    • Managed Processes failed to start after the maximum retry limit
    • Certain executable files do not have the execute permission enabled
Issue 1: Fix Maximum Retry Limit

This can be easily resolved by setting the ulimit at the user level to : ulimit -d unlimited , as outlined in Oracle's note ID: 1292887.1 . Anyone who has worked in OBIEE 10g should be familiar with this as it is required in 10g's installation process as well.

Issue 2: Enable read access on executable files

If you tried re-running opmnctl startall after resolving issue 1, you'll notice that the BI services still fail to run despite increasing the maximum retry limit. Upon examining the log files as outlined in the above error:

  • u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log
  • u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log
You'll notice that OPMNCTL still fails because it can't execute: nqsserver, nqscheduler and nqsclustercontroller. Below is the log from the above bullet point #1 :

/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute--------
12/11/06 15:48:06 Start process
--------
/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute


 During the 11.1.1.6.4 upgrade on your Solaris or Linux box, these files became read only!

How to resolve this read only issue:

Navigate to your $ORACLE_HOME/bifoundation/server/bin folder and make the the following files executable by using the chmod +x command:  nqscheduler, nqsclustercontroller, nqsserver. These files should have read/write privileges as outlined in the image below:



















Now when you run opmnctl startall, all  services should execute successfully:

Important:

After the upgrade, it is critical that you follow Oracle's recommended start up & shutdown sequence for weblogic & OBIEE 11g

Start up:
  • $DOMAIN_HOME/bin/startWebLogic.sh (Weblogic)
  • $WL_HOME/server/bin/startNodeManager.sh (Node Manager)
  • $DOMAIN_HOME/bin/startManagedWebLogic.sh bi_server1 http://server.domain:7001 (Managed Server)
  • $ORACLE_INSTANCE/bin/opmnctl startall (OPMN)
Shutdown:
  • $ORACLE_INSTANCE/bin/opmnctl stopall (OPMN)
  • $DOMAIN_HOME/bin/stopManagedWebLogic.sh bi_server1 t3://host.domain:7001 username password (Managed Server)
  • $DOMAIN_HOME/bin/stopWebLogic.sh username password t3://host.domain:7001 (Admin Server)

In my upcoming blog post i'll discuss how to automate this entire process.



keywords: opmn , opmn start failed, obiee 11g, opmn obiee weblogic, bi_server