How-to: Automated Web Catalog Deployment in OBIEE 11g

In a previous article I covered the process of how to automatically migrate the repository from a developer environment to production. Equally important is the process of migrating the web catalog.  Most BI Architects are familiar with the ability to archive and unarchive, but there are many unknowns that the Oracle documentation does not cover.

• What about version control?
• How do you migrate only select files?
• How do you migrate object level security?
• What if there are multiple people editing the web catalog?

A typical web catalog migration path consists of:

1. Developer checks out or download a local copy of the web catalog from version control
1. This is only done on an as needed basis
2. Developer makes changes to specific web catalog files
3. Developer archives changes
4. Developer checks out a folder (in this case called archive.zip) from version control that contains all production changes for the specific release
5. Developer adds their changes to the archive.zip
6. Developer updates 'change list' within the archive.zip to contain path of changed document
7. Developer checks-in archive.zip to version control
8. Deployment script automatically propagates through out each environment as outlined below:

This process uses the 'archive.zip' file as the directory to propagate changes throughout all environments. This is a preferred method because the archive.zip file only contains changes to the web catalog, and not the entire web catalog itself.

What's in the archive.zip?

The archive.zip will contain 

1. Only the modified web catalog objects for your specific release. 
2. A csv file that contains the destination path of each web catalog object
1. This is used as input for the shell script we're going to create for automatic deployment
2. I call this file 'Catalog_Deployment.csv' but it can be renamed if needed

Your archive.zip file might contain the following:

In the above example I have 5 files:

1. Financial_Reports.catalog is an archive of the 'Financial Reports' folder which contains multiple reports
2. Financial_Reports_Dashboard.catalog is an archive of the Financial Reports Dashboard that displays all of the financial reports
3. HR_Reports.catalog is an archive of the 'HR Reports' folder which contains multiple reports
4. HR_Reports_Dashboard.catalog is an archive of the HR Reports Dashboard that displays all of the HR reports
5. Catalog_Deployment.csv is a csv file that contains the target directory path when we unarchive each catalog file (via a shell script)

In the above example, the Catalog_Deployment.csv file would contain the following:

The Catalog_Deployment.csv contains only 2 columns:

1. The name of the archive file(s) in your archive.zip (Column A)
2. The unarchive target directory (Column B)

How do you identify the unarchive target directory (Column B)?

The unarchive target directory (column b) can be found in the 'location' section of the' properties' tab for the specific folder you're trying to archive:

Let's cover the steps required in order for the web catalog to make it out of the developer's box and into assembly test

Step 1.  Check out the archive.zip file from your version control software

'Check out' is the correct terminology because this will ensure the file is locked and no one can make any changes except for the specific developer. You should now have a locked version of archive.zip that you can modify.

Step 2. Archive the modified web catalog files

This is a straight forward step that can be achieved through Answers. The archive button can be found in the 'tasks' section as noted below:

Step 3. Modify the archive.zip file & Re upload to Version Control

Your modifications should include:

1. The addition of the modified .catalog files
2. Revisions to the Catalog_Deployment.csv 

Once the changes are made, you can upload back to Version Control and 'check the archive.zip' back in so other developers can add their changes. The key here is only one developer modifies the archive.zip file at a time!

Step 4. Pragmatically deploy the web catalog via a deployment shell script

The script we're going to use is going read each row in the Deployment_Catalog.csv file and and use Catalog Manager's runcat.sh script to perform command line based unarchiving.  Oracle has very little documentation on runcat.sh, but think of it as a way to launch Catalog Manager. You can launch Catalog Manager in either GUI mode or command line mode. Using the '.runcat.sh -cmd unarchive' parameters, we're telling Catalog Manager to unarchive in command line mode.

#!/bin/bash
#
# This file will read the archive files that are unzipped in /tmp/webcatmigration
#

/bin/dos2unix /tmp/webcatmigration/archive/Catalog_Deployment.csv /tmp/webcatmigration/archive/Catalog_Deployment.csv1
rm /tmp/webcatmigration/archive/Catalog_Deployment.csv
mv /tmp/webcatmigration/archive/Catalog_Deployment.csv1 /tmp/webcatmigration/archive/Catalog_Deployment.csv
while IFS=, read file path
do
echo ""
echo First Column in Catalog_Deployment - $file
echo Second Column in Catalog_Deployment - $path
echo ""
/export/obiee/11g/instances/instance1/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/catalogmanager/runcat.sh -cmd unarchive -offline /export/obishare/catalog/webcatalog -inputFile /tmp/webcatmigration/archive/$file -folder "$path"
done < /tmp/webcatmigration/archive/Catalog_Deployment.csv
~

This script does the following:

• Converts the .csv file from binary to unix via dos2unix
• Reads each row in Catalog_Deployment.csv and passes A(N) and B(N) for each row to the $file and $path parameters

A good option for automatic deployment would be to have this script run every Friday after hours as part of the deployment process. You would have one script in each environment for deployment. Ideally the script would be able to read directly from the file server of your version control software.

In summary, we've accomplished the following:

• Implemented a mechanism to manage web catalog modifications
• Automated the web catalog deployment process
• Minimized human error

Notes

How often should I back up the entire web catalog?

Notice that the only component of the web catalog being uploaded to version control are the actual changes to the web catalog objects. It is a good idea to take a back up of the entire web catalog prior to the 'go live' of deployment of each 'new release'

What about object level security?

The runcat.sh and archive/unarchive functionality will automatically migrate object level security of each object, but if the object level security is using new application roles that do not exist in the assembly/system/pre-production/production environments - the application roles must first be created in each environment's Enterprise Manager.

Don't forget about the GUIDs!

If you are migrating to a production environment that has different authentication providers than the non-production environment(s) - you must refresh the GUIDs first. You can read how to refresh the GUIDs in this Oracle note - How to refresh GUIDs for OBIEE 11g ? (Doc ID 1564006.1)

 

keywords: OBIEE 11g, deployment, archive, unarchive, migration, answers, runcat.sh web catalog, webcat