ca.portal.admin

Re:Re: IDMS Data Replication Via TCP/IP

Discussion created by ca.portal.admin on Nov 6, 2008
A couple of, ahem, not unbiased comments:

- the code required in support of this is not insubstantial, it may
require maintenance when and if CA changes the TCP/IP interface

- replication over TCP/IP is not going to be recoverable in the same way
as real time replication over MQ or offloaded journal based replication
using ftp

- if the code is embedded in an application, there is risk of the
application failing when an outboard component is unavailable

- if the code is independent of the application, say in an exit, then CV
performance and stability are issues

- a means will be needed to replicate during database recovery, perhaps
some means of reading the journal

A third party replication tool might address these issues at a
reasonable cost.

Jim Phillips
ISP
(800) 295-7608 x223

Date: Thu, 6 Nov 2008 11:35:35 -0500> From:
Harold.Govan@REEDELSEVIER.COM> Subject: IDMS Data Replication Via
TCP/IP> To: IDMSVENDOR-L@LISTSERV.IUASSN.COM> > Hi Everyone:> > I have a
question for folks using IDMS TCP/IP under R16.0.> > We are currently
using IDMS TCP/IP to support web-based apps coming in> from WebSphere.
We are contemplating doing selective data replication> to ORACLE and MS
SQL Server databases using TCP/IP enhancements to> existing IDMS-DC
applications.> > Has anyone out there done this type of replication
without the benefit> of a third-part product ? If so, I would appreciate
any information> you could share regarding your experiences.> > TIA> > >
Hal Govan> Senior Database Administrator> Reed Elsevier - Technology
Services> harold.govan@reedelsevier.com
<mailTo:harold.govan@lexisnexis.com>=20> Phone: (937) 865-7820
_________________________________________________________________
"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: IDMS Data Replication Via TCP/IP
"This probably doesn't help your situation any, but I can tell you what we
did.

We are trying to get off the mainframe (and IDMS).
We have a need to keep many databases synchronized real-time throughout the
day while development continues to retire IDMS applications and replace with
brand new non-IDMS systems.

We wrote our own message exchange system.
It handles both PUBLISH and SUBSCRIBE messages.
IDMS can be updated from an update that originates on another database and
another database can be updated when an update originates in IDMS.

On the PUBLISH end:
1. An update occurs in IDMS.
2. A database procedure gets called for the record and writes a queue
record to any one of over a hundred queues.
3. A DC-COBOL program fires up as a result of queue trigger of 1.
4. DC-COBOL program re-obtains the record or (in case of an ERASE, captures
the key from the queue record) and writes appropriate fields for the record
to message exchange database on IDMS.
5. TCP/IP Socket program idles looking for PUBLISH records to send to
server.
6. Server routes message to appropriate system and processes update
database.

On the SUBSCRIBE end:
1. TCP/IP Socket program sends message to message exchange database on IDMS
from open-systems side.
2. Another non-socket idler program looks for SUBSCRIBE messages to
process.
3. Idler program reads the tag to figure out what message it is.
4. Idler attaches the appropriate task to process the message and update
IDMS.

The above is a simplification of what happens.
There are called programs for handling writing and reading messages and a
lot of tables are involved so we don't have to recompile / reassemble every
time there is a change or a new message.

It took a while to develop ( 1 year +) and it was a lot of fine-tuning
effort along the way for the first couple of years.
Also, it is possible to end up with ""Dead Messages"" for any of a number of
reasons.
We have a process in place to handle those so we don't lose updates and get
databases out-of-sync.

Thanks.
Jon Gocher

----- Original Message -----
From: ""Govan, Hal (RET-DAY)"" <Harold.Govan@REEDELSEVIER.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Thursday, November 06, 2008 11:35 AM
Subject: IDMS Data Replication Via TCP/IP


Hi Everyone:

I have a question for folks using IDMS TCP/IP under R16.0.

We are currently using IDMS TCP/IP to support web-based apps coming in
from WebSphere. We are contemplating doing selective data replication
to ORACLE and MS SQL Server databases using TCP/IP enhancements to
existing IDMS-DC applications.

Has anyone out there done this type of replication without the benefit
of a third-part product ? If so, I would appreciate any information
you could share regarding your experiences.

TIA


Hal Govan
Senior Database Administrator
Reed Elsevier - Technology Services
harold.govan@reedelsevier.com <mailTo:harold.govan@lexisnexis.com>
Phone: (937) 865-7820
"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: IDMS Data Replication Via TCP/IP
"This probably doesn't help your situation any, but I can tell you what we
did.

We are trying to get off the mainframe (and IDMS).
We have a need to keep many databases synchronized real-time throughout the
day while development continues to retire IDMS applications and replace with
brand new non-IDMS systems.

We wrote our own message exchange system.
It handles both PUBLISH and SUBSCRIBE messages.
IDMS can be updated from an update that originates on another database and
another database can be updated when an update originates in IDMS.

On the PUBLISH end:
1. An update occurs in IDMS.
2. A database procedure gets called for the record and writes a queue
record to any one of over a hundred queues.
3. A DC-COBOL program fires up as a result of queue trigger of 1.
4. DC-COBOL program re-obtains the record or (in case of an ERASE, captures
the key from the queue record) and writes appropriate fields for the record
to message exchange database on IDMS.
5. TCP/IP Socket program idles looking for PUBLISH records to send to
server.
6. Server routes message to appropriate system and processes update
database.

On the SUBSCRIBE end:
1. TCP/IP Socket program sends message to message exchange database on IDMS
from open-systems side.
2. Another non-socket idler program looks for SUBSCRIBE messages to
process.
3. Idler program reads the tag to figure out what message it is.
4. Idler attaches the appropriate task to process the message and update
IDMS.

The above is a simplification of what happens.
There are called programs for handling writing and reading messages and a
lot of tables are involved so we don't have to recompile / reassemble every
time there is a change or a new message.

It took a while to develop ( 1 year +) and it was a lot of fine-tuning
effort along the way for the first couple of years.
Also, it is possible to end up with ""Dead Messages"" for any of a number of
reasons.
We have a process in place to handle those so we don't lose updates and get
databases out-of-sync.

Thanks.
Jon Gocher

----- Original Message -----
From: ""Govan, Hal (RET-DAY)"" <Harold.Govan@REEDELSEVIER.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Thursday, November 06, 2008 11:35 AM
Subject: IDMS Data Replication Via TCP/IP


Hi Everyone:

I have a question for folks using IDMS TCP/IP under R16.0.

We are currently using IDMS TCP/IP to support web-based apps coming in
from WebSphere. We are contemplating doing selective data replication
to ORACLE and MS SQL Server databases using TCP/IP enhancements to
existing IDMS-DC applications.

Has anyone out there done this type of replication without the benefit
of a third-part product ? If so, I would appreciate any information
you could share regarding your experiences.

TIA


Hal Govan
Senior Database Administrator
Reed Elsevier - Technology Services
harold.govan@reedelsevier.com <mailTo:harold.govan@lexisnexis.com>
Phone: (937) 865-7820
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: Problem with index contention and replication a record
"Javier -

We used to have severe deadlocking problems similar to what you've
described - hundreds each day.
Now we get maybe no more than 10 a day.

We resolved this through programming efforts.
I don't know how many programs are affected in your shop, but it may be
easier to just handle it from the applications side.

Start with changing OBTAIN to OBTAIN KEEP EXCLUSIVE.
We found this took care of about 25% of our problems.
We had better luck calling a DC-COBOL program from ADS to do ENQUEUE /
DEQUEUE at appropriate spots in the code.
Sometimes the appropriate spots ended up being right upon entry and just
before map out.
Every dialog or DC program that hits this index would need to ENQUEUE /
DEQUEUE on the same resource name.
This worked well for us.

If you have very high online task volume, this will elongate your response
times because now you are single-threading through your updates.
The effort wasn't that tough to do and it made a big difference for us.
We never went through the effort of measuring average response time
afterward.
All I know is nobody complained about it.
If anything, they were happy to get rid of the deadlocks.

Thanks.
Jon Gocher


----- Original Message -----
From: ""javier sotela"" <jsotela@HOTMAIL.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Sunday, October 26, 2008 12:09 PM
Subject: Problem with index contention and replication a record


Hi Everyone:

We have a problem with an apllication and specifically with a record. We
have a record calc with the cal key is a sequence number 8 bytes, this
represent order number. In the same record we have the phone number and we
defined an index with phone number ten bytes + random number 8 bytes. This
is a heavily used record in the application, and there is lot of user
udating the record, with the problem that when they create record, the order
number is a sequence number and the phone number is almost sequence number,
then the key in the index its very similar and normaly goes almost on the
same sr8, no matter how different is the random number attached at the end
of the index key. Then we have a lot of deadlocks because of the index key.
We are trying to create a new record calc, with the calc key the phone
number and putting in there the order number, just to preserve the chance to
retrieve the information by phone number or by order number. The idea its
try to replicate some of the information when the original record its
created without changing programming. Does anybody can tell me how to do the
data replication of the record without changing programming??? I know that
there is database procedures, but its suppose that you don't have to use dml
verb in there, then I'm looking for any solution or on the other hand any
ideas in how to solve the problem with the index contention ???

Thanks.

J. Sotela
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: Problem with index contention and replication a record
"Javier -

We used to have severe deadlocking problems similar to what you've
described - hundreds each day.
Now we get maybe no more than 10 a day.

We resolved this through programming efforts.
I don't know how many programs are affected in your shop, but it may be
easier to just handle it from the applications side.

Start with changing OBTAIN to OBTAIN KEEP EXCLUSIVE.
We found this took care of about 25% of our problems.
We had better luck calling a DC-COBOL program from ADS to do ENQUEUE /
DEQUEUE at appropriate spots in the code.
Sometimes the appropriate spots ended up being right upon entry and just
before map out.
Every dialog or DC program that hits this index would need to ENQUEUE /
DEQUEUE on the same resource name.
This worked well for us.

If you have very high online task volume, this will elongate your response
times because now you are single-threading through your updates.
The effort wasn't that tough to do and it made a big difference for us.
We never went through the effort of measuring average response time
afterward.
All I know is nobody complained about it.
If anything, they were happy to get rid of the deadlocks.

Thanks.
Jon Gocher


----- Original Message -----
From: ""javier sotela"" <jsotela@HOTMAIL.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Sunday, October 26, 2008 12:09 PM
Subject: Problem with index contention and replication a record


Hi Everyone:

We have a problem with an apllication and specifically with a record. We
have a record calc with the cal key is a sequence number 8 bytes, this
represent order number. In the same record we have the phone number and we
defined an index with phone number ten bytes + random number 8 bytes. This
is a heavily used record in the application, and there is lot of user
udating the record, with the problem that when they create record, the order
number is a sequence number and the phone number is almost sequence number,
then the key in the index its very similar and normaly goes almost on the
same sr8, no matter how different is the random number attached at the end
of the index key. Then we have a lot of deadlocks because of the index key.
We are trying to create a new record calc, with the calc key the phone
number and putting in there the order number, just to preserve the chance to
retrieve the information by phone number or by order number. The idea its
try to replicate some of the information when the original record its
created without changing programming. Does anybody can tell me how to do the
data replication of the record without changing programming??? I know that
there is database procedures, but its suppose that you don't have to use dml
verb in there, then I'm looking for any solution or on the other hand any
ideas in how to solve the problem with the index contention ???

Thanks.

J. Sotela
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
CA-World 2008 - Information for IDMS Attendees
"Dear all,

CA-World 2008 starts next weekend and, as usual, it's going to be large,
sprawling conference. Here is some information which we hope will help
you, as an IDMS professional, find your way around.

Steve Rundle
IUA/CA-World planning committee

CA-World 2008 Information
=========================

Headlines
---------
CA-World is large. There's no avoiding the fact. Sometimes it is
daunting simply looking for the right places to go and the best sessions
to attend. So, to assist IDMS-ers here is information to help plan your
itinerary while you are there.

Note: All the following information is also available in a formatted
style on the IUA website at: www.iuassn.org

Mix and Munch Lunch - Tuesday and Wednesday
------------------------------------------
This is a very cute name for a table set aside for IDMS track attendees
to eat lunch on Tuesday and Wednesday. No more trekking to a cavernous
hall for lunch and then not being able to see anyone you know. This time
you should be able to easily find the IDMS people. This table will be in
room San Polo 3405-6 and the lunch will be provided in buffet style. So
you can get your food and while you are munching your lunch, you can mix
with IDMS people. Yay!

IDMS User group meetings
------------------------
Most of the IUA board members will be at CA-World this year, and there
will be several IUA meetings through the week. The most important is the
General Meeting on Monday at 2:15 immediately after the IDMS Status and
Plans session.

CA-World Event night. - Tuesday Night
-------------------------------------------------
This will be, as usual, a spectacular event. This year they are
promising a feast for the senses, with Circus performers, a DJ, food
from all over the world, and some contemporary games (No I don't know
what that is, I'm guessing Xbox Games and similar) .

Mainframe Event Night - Wednesday Night
------------------------------------------------------
This year the entire Mainframe focus area is holding an Event Night on
the Wednesday night at Carnvale (next to Hurrah's). It promises to be an
exciting evening with a band, dancers and food. Entry will be invitation
only for mainframe attendees. Make sure you pick up an invitation at one
of the Mainframe stands in the EC.

Keynotes:
----------
Sunday 6pm: John Swainson CA CEO. Keynote address
Monday 8:30am: Al Nugent CA EVP and CTO : Technology Keynote
Monday 4pm: Jack Welch Former CEO General Electric: Guest Keynote

International Pavilion:
------------------------
For those travelling from outside North America CA are, again, providing
an International Pavilion. Here CA can provide local language support,
and other international needs. It's also a comfortable place to meet
with others from the same country.

Session Highlights.
===================
Here are some highlights which I have selected from the huge list of
sessions and events around CA-World this year.

Pre con education.
------------------
These two full days of free pre-conference education sessions are held
on Saturday and Sunday in classrooms. They are free to anyone attending
CA-World but, importantly, you must pre-register for the classes. To
pre-register, go to the CA-World registration site at:
https://secure1.jackmorton.com/caworld/Login enter you details,
including your registration number which you will have on your CA World
2008 confirmation email. Then you should see a pre-con education button.
The session ids are:
EC802ENA: CA IDMS: Implementing and Maintaining the Database
EC801ENA: CA IDMS Performance and Tuning

IDMS r17 sessions
-----------------
Among the usual great sessions, such as the IDMS Status and Plans and a
Mainframe opening session, there are sessions referring to the recent
announcement of IDMS r17. There are no less than 5 sessions on new
features of IDMS r17 (including zIIP exploitation and SQL features).
Ivory Service Architect
GT Software are showing Ivory Service Architect and how it can be used
to deploy a web service in an IDMS (or a Datacom) environment, including
a Hands-on Lab, which will be enlightening.

IUA IDMS PLC general meeting
----------------------------
Don't forget the IUA/EIUA IDMS PLC General meeting held at 2:15 on
Monday (immediately after the IDMS Status and Plans session and in the
same room) The final voting for this year's IUA board members elections
takes place.

Sessions in my list
-------------------
All sessions are listed in the agenda builder and session list at:
http://www.caworld.com/agendas but there are so many sessions that I
thought I'd make a short list for your to start from, so here are all
the sessions I have collected and collated and put into my calendar
which could be useful for IDMS attendees. (Note: Most IDMS sessions are
in room Bellini 2006)

Saturday and Sunday
----------------------
Pre-con education:
EC802ENA: CA IDMS: Implementing and Maintaining the Database
EC801ENA: CA IDMS Performance and Tuning

Monday
---------
MA001SN Welcome to the Mainframe: Opening Session
MI002SN Extending Your Mainframe in an SOA Environment
MI200SN CA IDMS Status and Plans
IUA IDMS PLC Meeting All Members
MI280SN Encrypting CA IDMS Data

Tuesday
--------
MI220SN CA IDMS Business Value in the SOA World
MI310SN Modernizing CA IDMS for SOA
MI330SN: Exploiting CA IDMS Server and CA IDMS SQL
MI230SN CA IDMS r17 Features Overview
MI420SN .NET for Mainframe Professionals
MI250SN CA IDMS r17 and Exploitation of the zIIP
MI240SN CA IDMS r17 Beta Experience

Wednesday
----------
MI340SN Modernize Your CA IDMS Database: Best Practices
MI400SN Mainframe SOA with Ivory Service Architect
MI320SN Department of Homeland Security: Leveraging CA IDMS in a SOA
Environment
MI440LN: Lab - Hands-On SOA Deployment in a CA Datacom or CA IDMS
Environment
MI260SN CA IDMS Tuning Options for Performance
MI270SN What's New with SQL? CA IDMS r17 SQL Features
MI350SN .Net Development for CA IDMS
MI300SN 21st Century CA IDMS DBA Modernization Techniques
MI210SN CA IDMS Birds of a Feather (BOF)

Thursday
-----------
MI290SN Audit Trail for CA IDMS Security
MI430SN Preparing for a New Generation on the Mainframe
MI360SN: Improving CA IDMS Based Application Flexibility and Quality
MI370SN Cost Effective SOA Enablement of CA IDMS Systems using zIIP/zAAP
Specialty Engines


I hope you find this useful.

See you there folks.

Any questions on any of this stuff? Then get back to me.
Regards
Steve Rundle
IUA/CA-World planning committee
"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: IDMS Data Replication Via TCP/IP
"Are you using WebSphere MQSeries software to do this?

Outcomes