Clarity

Hi Folks,

I've noticed a few posts out there from customers new to XOG and thought having this information on the board would help provide some basic guidelines for getting started. This is simply an internal bit of documentation which goes beyond what is offered in our standard Integration Guide. We use this here in support to help get new engineers started using XOG. Hope you find this helpful as well.

Regards,
--Kathy

------------------------------------------------------------------------------------------------------------------
XOG – The Basics…
The XML Open Gateway (XOG) is a CA Clarity PPM web service interface that you can use to:
• Import data
• Export data
• Move configuration data from one system to another

The XOG client is a Java program that you can install on your computer and use to import and export data. The XOG client communicates with the CA Clarity PPM server on the standard HTTP port using the SOAP protocol. You must have Java JRE version 1.5 in order to use the XOG client. Using the client, you can:
• Log in to start an authenticated session
• Execute requests to read or write CA Clarity PPM data
• Log out to end the session

XOG client installation:
1. In the Clarity Admin Tool, select Client Downloads.
2. Download and install XML Open Gateway Client. This does not have to be done on the server; a workstation with access to the environments will work (if you can open an Internet browser and access Clarity, you should be fine). Be sure to have a Clarity user available with the XOG Administration global right as well as the rights required for your XOG activity. **See access rights section below
3. In the folder structure created by the XOG install, the two main folders you will need are \bin and \xml. The \bin contains the properties file needed for modifying your XOG settings. The \xml folder contains xml files used to tell XOG what data to read as well as example files for XOG writes. (Make a copy of the bin and xml folders for backup purposes)
4. In the \xml folder are numerous xml files that end in ‘read’. The naming convention tells you what each file is set to read. For example, prj_projects_read.xml reads project data from Clarity. If you are not sure what is actually being read from the database by xml file name, most ‘read’ files have corresponding ‘write’ files that you can look in to see an example of the read output. For example, the prj_projects_read.xml has a corresponding prj_projects_write.xml file that shows an example of the output from the XOG read of projects.
5. Content related data refers to those XOGable items such as Objects (and Object related information like Views, Lists, Filters, Lookups, etc), Processes, Portlets, Jobs (used for Report settings as well), Partition Models, and Lookups that are found in the content_pack_read.xml. Most of the XOGable items in the content_pack_read.xml have a separate xml file you can use. So, for example, the content_pack_read.xml file contains the tags for XOGing out Objects; however, there is also an object_read.xml that will read the same settings. This also applies for Jobs, Processes, Portlets, etc.
6. The xml read files provided give examples of how to set the filter section per item. Use the ID of the item to specify what it is that you want XOG to extract from the database. For example, if you are XOGing objects (instance data), the ID can be found by going to Studio -> Objects and looking at the ID column in the list of available Objects. For Lookups, you can go to Data Administration -> Lookups in the Admin Tool.

Example: XOG out the Object information for the Project Object. You would edit the object_read.xml filter to say:


<ObjectQuery>



<Filter name="object_code" criteria="EQUALS">project</Filter>


</ObjectQuery>

Instance related data refers to specific data stored in Clarity; for example, a specific project or a specific resource. In some cases, related data reads are broken up into multiple xml read files. As an example, the prj_project_read.xml will read a project’s general data as well as tasks, resources and assignments, and custom data; but that project’s financial planning information is XOG’d out via the financial_planning_read.xml.

Instance related data could be XOG’d out by filtering on the ID of the instance or by applying filters to read multiple instances. Arguments can also be utilized in some read xml files to specify which items associated with an instance you would like to XOG (a project’s tasks but not resources, for example).

Example: XOG out Project Property data and Custom Fields (only) for active projects starting after the first of this year. You would edit the prj_projects_read.xml file’s Header and Query sections:

<Header version="6.0.11" action="read" objectType="project" externalSource="NIKU">


<args name="order_by_1" value="name"/>


<args name="order_by_2" value="projectID"/>


<args name="include_tasks" value="false"/>


<args name="include_resources" value="false"/>


<args name="include_custom" value="true"/>


<args name="include_dependencies" value="false"/>

</Header>

<Query>


<Filter name="active" criteria="EQUALS">true</Filter>


<Filter name="approved" criteria="EQUALS">true</Filter>


<Filter name="start" criteria="AFTER">2005-01-01</Filter>

</Query>


Important to remember:
• It is important to realize what exactly is being XOG’d out of Clarity when you run a read file. Be sure to check the example ‘write’ file to see what the output of your read will be.
• Dependencies will be honored when performing XOG reads. Therefore, if you are XOGing out an Object, and in that Object is an active Attribute that uses a certain Lookup, both the active Attribute and the Lookup will be XOG’d out and appear in your output file. If you are XOGing out a Portlet and that Portlet uses a specific Query that has an Attribute referencing a certain Lookup, the XOG will read out the Portlet, the Query and the Lookup.
• When you perform a read using XOG, the output file is formatted to be a write xml file. This means that if you XOG out a project, the output xml file will be XOG-ready as a write xml file.
• XOG installs with examples of each writeable item. Use these as guides to formatting your XOG input file if you are not using the XOG read output as your input file.

Access Rights
Before using the XOG client, you must have a valid CA Clarity PPM login name and password. You must also have one of the following access rights:
• Administration - Access
• Administration - XOG

XOG Access Rights for Individual Objects
Before a resource can use the XOG to import or export data for a particular object, you must assign the resource the XOG access right for that object (for example, Asset - XOG Access, Project - XOG Access, Resource - XOG Access, and so on).

For example, you can grant the Asset - XOG Access right to a resource to support a custom CA Clarity PPM desktop application that needs asset information. While the resource can import and export instance data that is associated with the asset object, the resource is not able to import or export data on any other objects. XOG access rights for objects are listed in the access rights list in the Administration Tool with other access rights. XOG access rights are global rights.

HOW to run XOG:
You can run the XOG in the following ways:
• From the command line you can type in the parameters required to import and export data directly on the command line or you can store the parameters in a .properties file and call the file from the command line.
• Using the XOG client as a shell
• Using SOAP

We will focus on how customer routinely run XOG (from the installed client)
To run the XOG from the client using command-line parameters:
1. Open a command prompt:
• (Windows) From the Windows Start menu, select All Programs, CA, Clarity, CA Clarity XML Open Gateway.
• (UNIX) Navigate to the XOG client home directory.
2. Type the xog command with the parameters for the operation you want to perform and press Enter.

To run the XOG client using the test.properties file (most often this is how customers are executing the XOG client):
In the bin folder, open your properties file (XOG installs with test.properties).
1. Set the servername = to the server you are reading to or writing from.
2. Check the settings for portnumber and sslnumber. They are commented out by default.
3. Set the output = whatever you want to name the output file. By default, the properties file sets the name to out.xml and the location to the bin folder.
a. If you are doing a read, your output will be the write version of the data you’ve read. You may want to name your output file accordingly. So, for example, if you are XOGing out an object, the output file will be the write file for XOGing that object into another environment. In this case, if you are reading the project object, you may want to name your output file project_object_fromdev.xml.
b. If you are doing a write, your output will be a results scorecard of how many items were updated, inserted and/or failed; as well as failure messages and warnings. So you may want to name the output file for a write something like write_results.xml.
4. Set the username and password to a user with appropriate rights (be sure there are not spaces at the end of the username or password)
5. Set the input = the read or write xml file you want to use.
a. Upon install, the input is set equal to almost all of the xml files, but commented out. It may be helpful to delete all but the first file (make sure to remove the # in front), and just change the name of that file as you XOG. You can even create a property file specific to that input/output file combination if it’s a XOG you will be performing regularly.
b. You may also leave all the input files, just keep the ‘not in use’ commented out (placed on separate lines would help).
6. Upon completion of the xml input and the properties file, either open XML Open Gateway via the Start ? Programs ? Clarity menu or open a command prompt and go to your XOG installation’s bin directory.
a. Type the command: xog –propertyfile test.properties
7. (Note that if you are using a properties file with a different name, replace ‘test.properties’ with the name of your properties file).
8. Once XOG has completed running, you will have an output file located and named whatever you specified in the properties file.
9. Keep in mind that not everything in Clarity is available for XOGing. It is handy to come up with a migration plan (see migration guidelines below) that details how items will be moved between environments (with XOG or manually). It is also important to update your XOG client installation with newer versions of Clarity (on upgrades) to keep up to date with what is available for XOG.

General Order for Migration XOG:
1. Entities, Departments, Locations and Fiscal Time periods
2. Any additional OBS structures other than Department or Location
3. Partitions
4. Investment Type (INV_TYPE) static dependent lookups – XOG individually. This type of lookup does not XOG out correctly when included in the Project Object and if included will throw an error.
5. Project Object. (very tricky must be done carefully - see details below)
6. Other Objects & Views. (much more simple than the project object). Same concept though. Scale down the objects and views to just what is needed.
7. Portlets and Portlet Pages
8. Report Definitions
9. Processes. It's important to separate out "template" processes from regular ones. We found at least at 8.1FP03 that if you combine say a process tied to an idea and a template process tied to the project object in the same XOG XML, the resulting idea process will be tied to the project template! So I keep them in separate XOG XML files. The safest bet by far is to have one file per process.
10. Instance data (users, resources, projects, etc…)

[color=#121E9C][size=4][font=Arial]What?!? One needs to be an [color=#FF0303]ENGINEER[color] to run the XOG?!? :grin: True, but that was not in the user guide!

[font][size][color]

Hi Jonathan,

I find it helpful to read (XOG) out an existing resource record in its entirety in order to find out the exact syntax for a successful XOG. In your case, you could simply run the cmn_users_read.xml (make sure you update <Filter name="userName" criteria="EQUALS">Update with userName here </Filter>) within the cmn_users_read.xml file.

In my company we have several OBS associations that we can XOG out on the CMN_User.

I think you should have something along the same if you execute the above file with a valid user name, which will allow you to sequentially XOG out your resources for import into V12.

<OBSAssocs completed="false">
<OBSAssoc id="xxxxxxx"
name="Thomson Reuters – Department" unitPath="/xxxxxxxx"/>
<OBSAssoc id="xxxxxxx"
name="*********“ Location" unitPath="*********"/>
<OBSAssoc id="xxxxxxx" name="Department" unitPath="/xxxxx"/>
<OBSAssoc id="xxxxx" name="xxxxx" unitPath="xxxxxxx"/>
</OBSAssocs>

Hope that helps :-)

Hi Kathy,

Great XOG Basic steps guidelines, I found them to be very informative. I have a question, I am xogging from 7.5.3 to 12 and I need to xog out cmn_users, I would like to only xog out resources by OBS level, is that possible to filter on? I am able to xog out by userName or isActive "true" (isActive actually crash's the Clarity App server we think, so we don't do that one). I can't seem to find the right filter name="" field to use to then say "EQUALS" the actual obs department name. Can you help?

Jonathan P. Thompson
Brown Brothers Hariman
PMO Rerporting & Data Mgmt Coordinator

Hi Kathy,

This post is quite helpful. Thanks a lot.

Regards,
Murali

Kathy,
Your instructions are very helpful. Thank you so much. I do have a question however. Is it correct to assume that data is xogged out as well? If not how do i do this piece? We have a custom object that I was able to xog out. Upon examining the file it doesn't seem like i captured the data.

How do I capture object and data? I looked at the content pack file, however, there are so many parameters that I did not know where to start.

Thank you in advance.

Hello,

To answer your question, YES data can be XOGed out but NO it will not be brought out with the object itself as you have seen. That will require an "instance" XOG read. The Content Pack XML files are used for object content. So, in your case since you are needing to read out instance data for a custom object, you'll want to use the "custom_object_instances_read.xml" file.

If you take a look at the example XML files which get installed with the XOG client you'll see this one listed. You'll simply replace the objectCode and instanceCode filters with your custom names as needed. You can choose to filter by object (and bring out all instance data) or by instance and bring out only a select resultset using the OR criteria. Take a peek at the "custom_object_instances_write.xml" to get an idea of what your resulting output will look like.

Here's the code from the read XML:

<?xml version="1.0" encoding="UTF-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd">
<Header version="8.0" action="read" objectType="customObjectInstance" externalSource="NIKU"/>
<CustomObjectInstanceQuery>
<Filter name="objectCode" criteria="EQUALS">manufacturer</Filter>
<Filter name="instanceCode" criteria="OR">sony,nikon</Filter>
</CustomObjectInstanceQuery>
<CustomObjectInstanceQuery>
<Filter name="objectCode" criteria="EQUALS">shoe</Filter>
<Filter name="instanceCode" criteria="OR">brooks,nike</Filter>
</CustomObjectInstanceQuery>
</NikuDataBus>

Hope that helps!
--Kathy

Hey Kathy,


Thanks a lot for sharing... this is pretty much useful :)

Thanks,
Bhanu

Hello,
I want read a project with xog: task, dependencies, resources, risk, baseline line, financial... All data.
I see the folder xml (xog client), ¿What xml i need for read all data from a project? ¿What is the sequence of data?
What happens, if then, i writte this project in other environment and, for example, no exist one resource?
Thanks a lot.

Hi

if you want to xog-out for all the project information like resources, dependencies, etc, just set the value to true. if not require, just change the value to false.

<?xml version="1.0" encoding="UTF-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd">
<Header version="6.0.11" action="read" objectType="project" externalSource="NIKU">
<!-- you change the order by simply swap 1 and 2 number in the name attribute -->
<args name="order_by_1" value="name"/>
<args name="order_by_2" value="projectID"/>
<args name="include_tasks" value="true"/>
<args name="include_dependencies" value="true"/>
<args name="include_subprojects" value="true"/>
<args name="include_resources" value="true"/>
<args name="include_baselines" value="true"/>
<args name="include_allocations" value="true"/>
<args name="include_estimates" value="true"/>
<args name="include_actuals" value="true"/>
<args name="include_custom" value="true"/>
</Header>
<Query>

<Filter name="projectID" criteria="EQUALS">test</Filter>
</Query>
</NikuDataBus>


Thanks
Senthil

Hi All,

We have 4 partitions in our project.
and the number of attributes for PROJECT OBJECT in one of the partitions is approx. 200
and the total number of attributes in project object is around 1000.

I want to xog out the attributes based on the partition but couldnot write condition.
Please let me know how to write condition to xog out.

Thanks in advance.


Aish

Hi there,
Have you tried finding a specific project in a particular partition and then reading out that project id? That is what I do when working with multiple partitions. So for example, I find a project that has customized values in a specific partiton, and note the project ID. Then I go to the Project Read (prj_projects_read.xml) and insert the following condition: <Filter name="projectID" criteria="EQUALS">insert project id here</Filter> Does that help?

Hi Kery,

Thanks for your reply.
But i wanted to Xog out the object's ATTRIBUTE and not the instance based on partition.

to explain further,

I wanted to edit custom_object_read xml to read attributes based on partition,

<ObjectQuery>


<Filter name="object_code" criteria="EQUALS">object_id_here</Filter>
[color=#FD6363]______________ how can i add condition here to filter based on partiton?[color]

</ObjectQuery>

Thanks,
Aish

Hi Aish,
If you do a read out of an example of the custom object- you can get that syntax. When I do a read out on a project (as an example), I see that the partition info on a project is stored in the Custom Information rather than the general properties that you currently have syntax for:

<CustomInformation>
<ColumnValue name="partition_code">partition_id</ColumnValue>

Because of where I see the partition attribute lies in the read out, I can now easily add to the read out condition. Because you are looking at the a custom object, I don't know where the partition attribute is stored, hence my recommendation to read out one example of the custom object you are dealing with. For the above example, I would use the following syntax in my 'read':

<FilterByCustomInfo name="partition_code" criteria="EQUALS">[color=#FFFFFF]partition ID Here[color]</FilterByCustomInfo>

Does that answer your question?

Hi Aishwarya,

You can use the following to xog out the object. The output xml file generated by such xog is too large. There was similar query in linkedin forum, I am pasting the same response here as well. The title of the query is following "Is it possible to restrict a content pack XOG to a single partition? "

The important point is placing the partition filter in <ViewQuery> tag

<?xml version="1.0" encoding="UTF-8"?>
<NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd">
<Header version="8.0" action="read" objectType="contentPack" externalSource="NIKU">
<args contentType="object" name="order_by_1" value="code" />
</Header>
<ViewQuery>
<Filter name="code" criteria="EQUALS">list</Filter>
<Filter name="object_code" criteria="EQUALS">project</Filter>
<Filter name="partition_code" criteria="EQUALS">PARTITION_CODE</Filter>
</ViewQuery>
<ViewQuery>
<Filter name="code" criteria="EQUALS">property</Filter>
<Filter name="object_code" criteria="EQUALS">project</Filter>
<Filter name="partition_code" criteria="EQUALS">PARTITION_CODE</Filter>
</ViewQuery>
<ObjectQuery>
<Filter name="object_code" criteria="EQUALS">project</Filter>
</ObjectQuery>
</NikuDataBus>

Thanx
Satish

An old thread on the main board to supplement this

Clarity (XOG) for beginners
NSA language

Martti K.

Hi,

is there a way to remove a given object from the database?
I would like to remove objects (type "product") using a SOAP message filtering by ID.

Furthermore, is there an official guide of XOG?

Thanks in advance

splogad wrote:

is there a way to remove a given object from the database?

No - XOG is generally INSERT / UPDATE only (there are a couple of minor exceptions to this, but not for actual object instances)

splogad wrote:

Furthermore, is there an official guide of XOG?

The "official guide" would be the manuals that come with Clarity - e.g. "CAClarityPPM_XOG_DeveloperGuide_ENU" (the actual name will vary dependant on your version / language)

Thanks!

Leaving aside XOG, is there a different way to remove a record from the db?

splogad wrote:

Leaving aside XOG, is there a different way to remove a record from the db?

Through the application!
Or mark the record as "inactive" (which you can do through XOG). Inactive is a "logical delete" - ask yourself why you really need to "delete" a record and why that could not just be an inactive record.

is there a way to remove a given object from the database?

I would like to remove objects (type "product") using a SOAP message filtering by ID.

Are you talking about object or object instances?
You cannot remove OOTB objects like investments (projects and NPIO's like products).
You can mark their instances for deletion and rung the purge investments job.

Martti K.

Well, I undestand the difference between "inactivate a given record" vs "delete a given record".
My situation:
1) A staging table is written by an external tool
2) On Create -> a given process starts and a script is executed.
3) if an error is thrown, then I want to remove this record, so that I can run again the "On Create script"
4) I can't update the record otherwise a different script will start
5) I can't change this integration logic.

If I can inactivate an object and save it again with THE SAME CODE, then it is ok.
In that case could you please show how to inactivate an item of a customobject via XOG?

thanks

If it is custom object then

If I can inactivate an object and save it again with THE SAME CODE, then it is ok.

In that case could you please show how to inactivate an item of a customobject via XOG?


is a challenge.

OOTB objects like projects and users/resources have active attribute the value of which has certain consequences.

However, I don't have active attribute by default in my custom objects.
If I added one I don't think that will work the same way as in projects or users/resources.


Martti K.

If I understand right, you are using a custom Clarity sub-object as this "staging table"? (If not, i.e. it is a bespoke table, then why can you not just manipulate it however you like?)

So you need to change your processes... perhaps to pick up a different column to use as the "code" (thereby removing your uniqueness problems)?

and I know that you have said you can't change your "integration logic", but that is what your talking about anyway with your "delete" option?

Your "problems" seems to be caused by how your integration logic/process has been designed&built rather than anything "technical" to do with how Clarity works - i.e. your integration logic has not satisfied its requirements?

If I can inactivate an object and save it again with THE SAME CODE, then it is ok.

In that case could you please show how to inactivate an item of a customobject via XOG?
The CODE is the problem, that belongs to "Clarity" - so if your process uses another field as the "code" then you would be able to create multiple records in your object just fine. :wacko:

(Sorry I'm not being much help - but you are going to have to change SOMETHING to resolve your issues and the thing that needs fixing seems to be your "integration logic" :sad )

Hi Dave,

your hint was really precious.
I changed my "integration logic", now it works.

giacomo

Good to know! :grin:

This thread seems to be missing some posts.

Martti K.

Fixed :happy

Hey Kathy,

Thanks for a useful post :)

Uruj

Kathy,

I am needing to XOG "IN" custom attributes to the project attribute on existing projects.  What is the best xml sample to use in this function?

Thank you

prj_projects_write.xml

 

Martti K.

 

Hi,

I want to migrate custom lookups from one environment to another but I am not able to read a bunch of lookups from the sourve environment. Do I have to create an xml for each lookup?

I appreciate your advice.

Thanks!!

Flo

flogoya:

Hi,

I want to migrate custom lookups from one environment to another but I am not able to read a bunch of lookups from the sourve environment. Do I have to create an xml for each lookup?

No; we can have one XML file that contains multiple lookups.  (XOG out or in)

The query for the XOG out would look like this (the important bits being the "OR" in the criteria and then the comma-delimited list of lookups);

  <LookupQuery>
    <Filter name="code" criteria="OR">MY_LOOKUP_1,MY_LOOKUP_2</Filter>
  </LookupQuery>

 

Any insight as to why when I run the following XOG Read I can't pull out my custom lookup?

 <?xml version="1.0" encoding="UTF-8"?>
  <NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd">
   <Header version="8.0" action="read" objectType="contentPack" externalSource="NIKU">
   </Header>
    <LookupQuery>
     <Filter name="code" criteria="EQUALS">Lookup_ID_Here</Filter>
    </LookupQuery>
 </NikuDataBus>
 

For me it works fine in v13.2 if I leave the first line out. (Don't ask me why)

 

  <NikuDataBus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../xsd/nikuxog_read.xsd">
   <Header version="8.0" action="read" objectType="contentPack" externalSource="NIKU">
   </Header>
    <LookupQuery>
     <Filter name="code" criteria="EQUALS">ZTEST</Filter>
    </LookupQuery>
 </NikuDataBus>

 

Martti K.

Hi Kathy,

We have encountered one bug when we xog the process from lower enviroment.

The process includes steps to execute the few jobs in a row.

The process was working fine in lower enviroment but when xogged the same process to higher environent. Job did not get triggered.

Once we removed the step from Process and added it. Then it worked fine.

Kindly let us know if this known bug in clarity 13.1

 

Thanks,

Sandy

Hi Kathy,

Thank you for the detailed information on Xog.

Would you also please eloborate any steps to be taken to Xog, in environments where SSL is enabled.

Thanks and Regards,

Rajini