XOG - The Basics...

Discussion created by Kathy_Fisher Employee on Oct 12, 2010
Latest reply on Jul 31, 2017 by urmas
Hi Folks,

I've noticed a few posts out there from customers new to XOG and thought having this information on the board would help provide some basic guidelines for getting started. This is simply an internal bit of documentation which goes beyond what is offered in our standard Integration Guide. We use this here in support to help get new engineers started using XOG. Hope you find this helpful as well.


XOG – The Basics…
The XML Open Gateway (XOG) is a CA Clarity PPM web service interface that you can use to:
• Import data
• Export data
• Move configuration data from one system to another

The XOG client is a Java program that you can install on your computer and use to import and export data. The XOG client communicates with the CA Clarity PPM server on the standard HTTP port using the SOAP protocol. You must have Java JRE version 1.5 in order to use the XOG client. Using the client, you can:
• Log in to start an authenticated session
• Execute requests to read or write CA Clarity PPM data
• Log out to end the session

XOG client installation:
1. In the Clarity Admin Tool, select Client Downloads.
2. Download and install XML Open Gateway Client. This does not have to be done on the server; a workstation with access to the environments will work (if you can open an Internet browser and access Clarity, you should be fine). Be sure to have a Clarity user available with the XOG Administration global right as well as the rights required for your XOG activity. **See access rights section below
3. In the folder structure created by the XOG install, the two main folders you will need are \bin and \xml. The \bin contains the properties file needed for modifying your XOG settings. The \xml folder contains xml files used to tell XOG what data to read as well as example files for XOG writes. (Make a copy of the bin and xml folders for backup purposes)
4. In the \xml folder are numerous xml files that end in ‘read’. The naming convention tells you what each file is set to read. For example, prj_projects_read.xml reads project data from Clarity. If you are not sure what is actually being read from the database by xml file name, most ‘read’ files have corresponding ‘write’ files that you can look in to see an example of the read output. For example, the prj_projects_read.xml has a corresponding prj_projects_write.xml file that shows an example of the output from the XOG read of projects.
5. Content related data refers to those XOGable items such as Objects (and Object related information like Views, Lists, Filters, Lookups, etc), Processes, Portlets, Jobs (used for Report settings as well), Partition Models, and Lookups that are found in the content_pack_read.xml. Most of the XOGable items in the content_pack_read.xml have a separate xml file you can use. So, for example, the content_pack_read.xml file contains the tags for XOGing out Objects; however, there is also an object_read.xml that will read the same settings. This also applies for Jobs, Processes, Portlets, etc.
6. The xml read files provided give examples of how to set the filter section per item. Use the ID of the item to specify what it is that you want XOG to extract from the database. For example, if you are XOGing objects (instance data), the ID can be found by going to Studio -> Objects and looking at the ID column in the list of available Objects. For Lookups, you can go to Data Administration -> Lookups in the Admin Tool.

Example: XOG out the Object information for the Project Object. You would edit the object_read.xml filter to say:


<Filter name="object_code" criteria="EQUALS">project</Filter>


Instance related data refers to specific data stored in Clarity; for example, a specific project or a specific resource. In some cases, related data reads are broken up into multiple xml read files. As an example, the prj_project_read.xml will read a project’s general data as well as tasks, resources and assignments, and custom data; but that project’s financial planning information is XOG’d out via the financial_planning_read.xml.

Instance related data could be XOG’d out by filtering on the ID of the instance or by applying filters to read multiple instances. Arguments can also be utilized in some read xml files to specify which items associated with an instance you would like to XOG (a project’s tasks but not resources, for example).

Example: XOG out Project Property data and Custom Fields (only) for active projects starting after the first of this year. You would edit the prj_projects_read.xml file’s Header and Query sections:

<Header version="6.0.11" action="read" objectType="project" externalSource="NIKU">

<args name="order_by_1" value="name"/>

<args name="order_by_2" value="projectID"/>

<args name="include_tasks" value="false"/>

<args name="include_resources" value="false"/>

<args name="include_custom" value="true"/>

<args name="include_dependencies" value="false"/>



<Filter name="active" criteria="EQUALS">true</Filter>

<Filter name="approved" criteria="EQUALS">true</Filter>

<Filter name="start" criteria="AFTER">2005-01-01</Filter>


Important to remember:
• It is important to realize what exactly is being XOG’d out of Clarity when you run a read file. Be sure to check the example ‘write’ file to see what the output of your read will be.
• Dependencies will be honored when performing XOG reads. Therefore, if you are XOGing out an Object, and in that Object is an active Attribute that uses a certain Lookup, both the active Attribute and the Lookup will be XOG’d out and appear in your output file. If you are XOGing out a Portlet and that Portlet uses a specific Query that has an Attribute referencing a certain Lookup, the XOG will read out the Portlet, the Query and the Lookup.
• When you perform a read using XOG, the output file is formatted to be a write xml file. This means that if you XOG out a project, the output xml file will be XOG-ready as a write xml file.
• XOG installs with examples of each writeable item. Use these as guides to formatting your XOG input file if you are not using the XOG read output as your input file.

Access Rights
Before using the XOG client, you must have a valid CA Clarity PPM login name and password. You must also have one of the following access rights:
• Administration - Access
• Administration - XOG

XOG Access Rights for Individual Objects
Before a resource can use the XOG to import or export data for a particular object, you must assign the resource the XOG access right for that object (for example, Asset - XOG Access, Project - XOG Access, Resource - XOG Access, and so on).

For example, you can grant the Asset - XOG Access right to a resource to support a custom CA Clarity PPM desktop application that needs asset information. While the resource can import and export instance data that is associated with the asset object, the resource is not able to import or export data on any other objects. XOG access rights for objects are listed in the access rights list in the Administration Tool with other access rights. XOG access rights are global rights.

HOW to run XOG:
You can run the XOG in the following ways:
• From the command line you can type in the parameters required to import and export data directly on the command line or you can store the parameters in a .properties file and call the file from the command line.
• Using the XOG client as a shell
• Using SOAP

We will focus on how customer routinely run XOG (from the installed client)
To run the XOG from the client using command-line parameters:
1. Open a command prompt:
• (Windows) From the Windows Start menu, select All Programs, CA, Clarity, CA Clarity XML Open Gateway.
• (UNIX) Navigate to the XOG client home directory.
2. Type the xog command with the parameters for the operation you want to perform and press Enter.

To run the XOG client using the test.properties file (most often this is how customers are executing the XOG client):
In the bin folder, open your properties file (XOG installs with test.properties).
1. Set the servername = to the server you are reading to or writing from.
2. Check the settings for portnumber and sslnumber. They are commented out by default.
3. Set the output = whatever you want to name the output file. By default, the properties file sets the name to out.xml and the location to the bin folder.
a. If you are doing a read, your output will be the write version of the data you’ve read. You may want to name your output file accordingly. So, for example, if you are XOGing out an object, the output file will be the write file for XOGing that object into another environment. In this case, if you are reading the project object, you may want to name your output file project_object_fromdev.xml.
b. If you are doing a write, your output will be a results scorecard of how many items were updated, inserted and/or failed; as well as failure messages and warnings. So you may want to name the output file for a write something like write_results.xml.
4. Set the username and password to a user with appropriate rights (be sure there are not spaces at the end of the username or password)
5. Set the input = the read or write xml file you want to use.
a. Upon install, the input is set equal to almost all of the xml files, but commented out. It may be helpful to delete all but the first file (make sure to remove the # in front), and just change the name of that file as you XOG. You can even create a property file specific to that input/output file combination if it’s a XOG you will be performing regularly.
b. You may also leave all the input files, just keep the ‘not in use’ commented out (placed on separate lines would help).
6. Upon completion of the xml input and the properties file, either open XML Open Gateway via the Start ? Programs ? Clarity menu or open a command prompt and go to your XOG installation’s bin directory.
a. Type the command: xog –propertyfile test.properties
7. (Note that if you are using a properties file with a different name, replace ‘test.properties’ with the name of your properties file).
8. Once XOG has completed running, you will have an output file located and named whatever you specified in the properties file.
9. Keep in mind that not everything in Clarity is available for XOGing. It is handy to come up with a migration plan (see migration guidelines below) that details how items will be moved between environments (with XOG or manually). It is also important to update your XOG client installation with newer versions of Clarity (on upgrades) to keep up to date with what is available for XOG.

General Order for Migration XOG:
1. Entities, Departments, Locations and Fiscal Time periods
2. Any additional OBS structures other than Department or Location
3. Partitions
4. Investment Type (INV_TYPE) static dependent lookups – XOG individually. This type of lookup does not XOG out correctly when included in the Project Object and if included will throw an error.
5. Project Object. (very tricky must be done carefully - see details below)
6. Other Objects & Views. (much more simple than the project object). Same concept though. Scale down the objects and views to just what is needed.
7. Portlets and Portlet Pages
8. Report Definitions
9. Processes. It's important to separate out "template" processes from regular ones. We found at least at 8.1FP03 that if you combine say a process tied to an idea and a template process tied to the project object in the same XOG XML, the resulting idea process will be tied to the project template! So I keep them in separate XOG XML files. The safest bet by far is to have one file per process.
10. Instance data (users, resources, projects, etc…)