ca.portal.admin

Re:Re: IDMS Data Replication Via TCP/IP

Discussion created by ca.portal.admin on Nov 9, 2008
This probably doesn't help your situation any, but I can tell you what
we did.

We are trying to get off the mainframe (and IDMS).
We have a need to keep many databases synchronized real-time throughout
the day while development continues to retire IDMS applications and
replace with brand new non-IDMS systems.

We wrote our own message exchange system.
It handles both PUBLISH and SUBSCRIBE messages.
IDMS can be updated from an update that originates on another database
and another database can be updated when an update originates in IDMS.

On the PUBLISH end:
1. An update occurs in IDMS.
2. A database procedure gets called for the record and writes a queue
record to any one of over a hundred queues.
3. A DC-COBOL program fires up as a result of queue trigger of 1.
4. DC-COBOL program re-obtains the record or (in case of an ERASE,
captures the key from the queue record) and writes appropriate fields
for the record to message exchange database on IDMS.
5. TCP/IP Socket program idles looking for PUBLISH records to send to
server.
6. Server routes message to appropriate system and processes update
database.

On the SUBSCRIBE end:
1. TCP/IP Socket program sends message to message exchange database on
IDMS from open-systems side.
2. Another non-socket idler program looks for SUBSCRIBE messages to
process.
3. Idler program reads the tag to figure out what message it is.
4. Idler attaches the appropriate task to process the message and
update IDMS.

The above is a simplification of what happens.
There are called programs for handling writing and reading messages and
a lot of tables are involved so we don't have to recompile / reassemble
every time there is a change or a new message.

It took a while to develop ( 1 year +) and it was a lot of fine-tuning
effort along the way for the first couple of years.
Also, it is possible to end up with ""Dead Messages"" for any of a number
of reasons.
We have a process in place to handle those so we don't lose updates and
get databases out-of-sync.

Thanks.
Jon Gocher

----- Original Message -----
From: ""Govan, Hal (RET-DAY)"" <Harold.Govan@REEDELSEVIER.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Thursday, November 06, 2008 11:35 AM
Subject: IDMS Data Replication Via TCP/IP


Hi Everyone:

I have a question for folks using IDMS TCP/IP under R16.0.

We are currently using IDMS TCP/IP to support web-based apps coming in
from WebSphere. We are contemplating doing selective data replication
to ORACLE and MS SQL Server databases using TCP/IP enhancements to
existing IDMS-DC applications.

Has anyone out there done this type of replication without the benefit
of a third-part product ? If so, I would appreciate any information
you could share regarding your experiences.

TIA


Hal Govan
Senior Database Administrator
Reed Elsevier - Technology Services
harold.govan@reedelsevier.com <mailTo:harold.govan@lexisnexis.com>
Phone: (937) 865-7820
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
CA-World 2008 - Information for IDMS Attendees
"Dear all,=20

CA-World 2008 starts next weekend and, as usual, it's going to be large,
sprawling conference. Here is some information which we hope will help
you, as an IDMS professional, find your way around.=20

Steve Rundle
IUA/CA-World planning committee

CA-World 2008 Information
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=


Headlines=20
---------
CA-World is large. There's no avoiding the fact. Sometimes it is
daunting simply looking for the right places to go and the best sessions
to attend. So, to assist IDMS-ers here is information to help plan your
itinerary while you are there.=20

Note: All the following information is also available in a formatted
style on the IUA website at: www.iuassn.org

Mix and Munch Lunch - Tuesday and Wednesday
------------------------------------------
This is a very cute name for a table set aside for IDMS track attendees
to eat lunch on Tuesday and Wednesday. No more trekking to a cavernous
hall for lunch and then not being able to see anyone you know. This time
you should be able to easily find the IDMS people. This table will be in
room San Polo 3405-6 and the lunch will be provided in buffet style. So
you can get your food and while you are munching your lunch, you can mix
with IDMS people. Yay!

IDMS User group meetings
------------------------
Most of the IUA board members will be at CA-World this year, and there
will be several IUA meetings through the week. The most important is the
General Meeting on Monday at 2:15 immediately after the IDMS Status and
Plans session.=20

CA-World Event night. - Tuesday Night=20
-------------------------------------------------
This will be, as usual, a spectacular event. This year they are
promising a feast for the senses, with Circus performers, a DJ, food
from all over the world, and some contemporary games (No I don't know
what that is, I'm guessing Xbox Games and similar) .

Mainframe Event Night - Wednesday Night
------------------------------------------------------
This year the entire Mainframe focus area is holding an Event Night on
the Wednesday night at Carnvale (next to Hurrah's). It promises to be an
exciting evening with a band, dancers and food. Entry will be invitation
only for mainframe attendees. Make sure you pick up an invitation at one
of the Mainframe stands in the EC.=20

Keynotes:
----------
Sunday 6pm: John Swainson CA CEO. Keynote address
Monday 8:30am: Al Nugent CA EVP and CTO : Technology Keynote
Monday 4pm: Jack Welch Former CEO General Electric: Guest Keynote

International Pavilion:
------------------------
For those travelling from outside North America CA are, again, providing
an International Pavilion. Here CA can provide local language support,
and other international needs. It's also a comfortable place to meet
with others from the same country.=20

Session Highlights.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Here are some highlights which I have selected from the huge list of
sessions and events around CA-World this year.=20

Pre con education.
------------------
These two full days of free pre-conference education sessions are held
on Saturday and Sunday in classrooms. They are free to anyone attending
CA-World but, importantly, you must pre-register for the classes. To
pre-register, go to the CA-World registration site at:
https://secure1.jackmorton.com/caworld/Login enter you details,
including your registration number which you will have on your CA World
2008 confirmation email. Then you should see a pre-con education button.
The session ids are:
EC802ENA: CA IDMS: Implementing and Maintaining the Database =20
EC801ENA: CA IDMS Performance and Tuning =20

IDMS r17 sessions
-----------------
Among the usual great sessions, such as the IDMS Status and Plans and a
Mainframe opening session, there are sessions referring to the recent
announcement of IDMS r17. There are no less than 5 sessions on new
features of IDMS r17 (including zIIP exploitation and SQL features).=20
Ivory Service Architect
GT Software are showing Ivory Service Architect and how it can be used
to deploy a web service in an IDMS (or a Datacom) environment, including
a Hands-on Lab, which will be enlightening.

IUA IDMS PLC general meeting
----------------------------
Don't forget the IUA/EIUA IDMS PLC General meeting held at 2:15 on
Monday (immediately after the IDMS Status and Plans session and in the
same room) The final voting for this year's IUA board members elections
takes place.=20

Sessions in my list
-------------------
All sessions are listed in the agenda builder and session list at:
http://www.caworld.com/agendas but there are so many sessions that I
thought I'd make a short list for your to start from, so here are all
the sessions I have collected and collated and put into my calendar
which could be useful for IDMS attendees. (Note: Most IDMS sessions are
in room Bellini 2006)

Saturday and Sunday
----------------------
Pre-con education:
EC802ENA: CA IDMS: Implementing and Maintaining the Database=20
EC801ENA: CA IDMS Performance and Tuning=20

Monday=20
---------
MA001SN Welcome to the Mainframe: Opening Session=20
MI002SN Extending Your Mainframe in an SOA Environment =20
MI200SN CA IDMS Status and Plans =20
IUA IDMS PLC Meeting All Members
MI280SN Encrypting CA IDMS Data

Tuesday=20
--------
MI220SN CA IDMS Business Value in the SOA World
MI310SN Modernizing CA IDMS for SOA
MI330SN: Exploiting CA IDMS Server and CA IDMS SQL
MI230SN CA IDMS r17 Features Overview
MI420SN .NET for Mainframe Professionals
MI250SN CA IDMS r17 and Exploitation of the zIIP
MI240SN CA IDMS r17 Beta Experience

Wednesday
----------
MI340SN Modernize Your CA IDMS Database: Best Practices
MI400SN Mainframe SOA with Ivory Service Architect
MI320SN Department of Homeland Security: Leveraging CA IDMS in a SOA
Environment
MI440LN: Lab - Hands-On SOA Deployment in a CA Datacom or CA IDMS
Environment =20
MI260SN CA IDMS Tuning Options for Performance
MI270SN What's New with SQL? CA IDMS r17 SQL Features
MI350SN .Net Development for CA IDMS
MI300SN 21st Century CA IDMS DBA Modernization Techniques
MI210SN CA IDMS Birds of a Feather (BOF)

Thursday
-----------
MI290SN Audit Trail for CA IDMS Security
MI430SN Preparing for a New Generation on the Mainframe
MI360SN: Improving CA IDMS Based Application Flexibility and Quality =20
MI370SN Cost Effective SOA Enablement of CA IDMS Systems using zIIP/zAAP
Specialty Engines


I hope you find this useful.

See you there folks.=20

Any questions on any of this stuff? Then get back to me.
Regards
Steve Rundle
IUA/CA-World planning committee
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
FW: IUA Board of Directors - Election Notice
"Hi everyone,

=20

Thursday, November 13th, is the last date to register your vote online =
for the IUA Board of Directors.

=20

If you're a member of the IUA, please visit www.iuassn.org, and sign in. =
Choose the ""Board of Directors Ballot"" under the Members Only section, =
and please vote to fill the available positions. =20

=20

Also for more information on CA-World, please see the article ""CA-World =
News for IDMS Users"" located on www.iuassn.org <http://www.iuassn.org/> =
home page.

=20

Thank you,

Diane Montstream

IUA Nominations Chair

=20

=20

=20
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
FW: IUA Board of Directors - Election Notice
"Hi everyone,



Thursday, November 13th, is the last date to register your vote online for the IUA Board of Directors.



If you're a member of the IUA, please visit www.iuassn.org, and sign in. Choose the ""Board of Directors Ballot"" under the Members Only section, and please vote to fill the available positions.



Also for more information on CA-World, please see the article ""CA-World News for IDMS Users"" located on www.iuassn.org <http://www.iuassn.org/> home page.



Thank you,

Diane Montstream

IUA Nominations Chair






"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: Problem with index contention and replication a record
"Javier,

I am not sure I understand the purpose of the random number and why the
phone number is almost sequential.
Adding a random number to the end of the index key will not change which
SR8 it will go to.
If the phone number is sequential because of user processing, e.g. they get
the orders in sequence by phone number, you may be able to split the index
so it is by number, exchange and area code instead of area code, exchange
and number, or split it even further. You may still need programming
changes.

Are you sure the index is the problem and not just the symptom. You could
end up spending a lot of time resolving the deadlock on the index just to
find out you will have the problem somewhere else afterwards.

Adding a new record as you suggest, may work, but I do not see a way of
doing it without making program changes.

My suggestion is to make the index manual, then when a new record is
stored, you write the calc key to a queue. Set a task to automatically
execute when the queue is written to and the only thing that task will do
is to connect your record to the index. There is a small lag between
storing the record and connecting it, but you can single thread the task
that does the connect and avoid the deadlocks on the index. If the called
program always reads everything in the queue, you do not risk that there
are records left that were not connected.

Tommy Petersen




Jon Gocher
<jgocher@FRONTIER
NET.NET> To
Sent by: IDMS IDMS-L@LISTSERV.IUASSN.COM
Public Discussion cc
Forum
<IDMS-L@LISTSERV. Subject
IUASSN.COM> Re: Problem with index contention
and replication a record

11/09/2008 01:46
PM


Please respond to
IDMS Public
Discussion Forum
<IDMS-L@LISTSERV.
IUASSN.COM>






Javier -

We used to have severe deadlocking problems similar to what you've
described - hundreds each day.
Now we get maybe no more than 10 a day.

We resolved this through programming efforts.
I don't know how many programs are affected in your shop, but it may be
easier to just handle it from the applications side.

Start with changing OBTAIN to OBTAIN KEEP EXCLUSIVE.
We found this took care of about 25% of our problems.
We had better luck calling a DC-COBOL program from ADS to do ENQUEUE /
DEQUEUE at appropriate spots in the code.
Sometimes the appropriate spots ended up being right upon entry and just
before map out.
Every dialog or DC program that hits this index would need to ENQUEUE /
DEQUEUE on the same resource name.
This worked well for us.

If you have very high online task volume, this will elongate your response
times because now you are single-threading through your updates.
The effort wasn't that tough to do and it made a big difference for us.
We never went through the effort of measuring average response time
afterward.
All I know is nobody complained about it.
If anything, they were happy to get rid of the deadlocks.

Thanks.
Jon Gocher


----- Original Message -----
From: ""javier sotela"" <jsotela@HOTMAIL.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Sunday, October 26, 2008 12:09 PM
Subject: Problem with index contention and replication a record


Hi Everyone:

We have a problem with an apllication and specifically with a record. We
have a record calc with the cal key is a sequence number 8 bytes, this
represent order number. In the same record we have the phone number and we
defined an index with phone number ten bytes + random number 8 bytes. This
is a heavily used record in the application, and there is lot of user
udating the record, with the problem that when they create record, the
order
number is a sequence number and the phone number is almost sequence number,
then the key in the index its very similar and normaly goes almost on the
same sr8, no matter how different is the random number attached at the end
of the index key. Then we have a lot of deadlocks because of the index key.
We are trying to create a new record calc, with the calc key the phone
number and putting in there the order number, just to preserve the chance
to
retrieve the information by phone number or by order number. The idea its
try to replicate some of the information when the original record its
created without changing programming. Does anybody can tell me how to do
the
data replication of the record without changing programming??? I know that
there is database procedures, but its suppose that you don't have to use
dml
verb in there, then I'm looking for any solution or on the other hand any
ideas in how to solve the problem with the index contention ???

Thanks.

J. Sotela
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
"
IDMS 3rd-party providers forum
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP
IDMSVENDOR-L@LISTSERV.IUASSN.COM
IDMSVENDOR-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: Problem with index contention and replication a record
"Javier,

I am not sure I understand the purpose of the random number and why the
phone number is almost sequential.
Adding a random number to the end of the index key will not change which
SR8 it will go to.
If the phone number is sequential because of user processing, e.g. they get
the orders in sequence by phone number, you may be able to split the index
so it is by number, exchange and area code instead of area code, exchange
and number, or split it even further. You may still need programming
changes.

Are you sure the index is the problem and not just the symptom. You could
end up spending a lot of time resolving the deadlock on the index just to
find out you will have the problem somewhere else afterwards.

Adding a new record as you suggest, may work, but I do not see a way of
doing it without making program changes.

My suggestion is to make the index manual, then when a new record is
stored, you write the calc key to a queue. Set a task to automatically
execute when the queue is written to and the only thing that task will do
is to connect your record to the index. There is a small lag between
storing the record and connecting it, but you can single thread the task
that does the connect and avoid the deadlocks on the index. If the called
program always reads everything in the queue, you do not risk that there
are records left that were not connected.

Tommy Petersen




Jon Gocher
<jgocher@FRONTIER
NET.NET> To
Sent by: IDMS IDMS-L@LISTSERV.IUASSN.COM
Public Discussion cc
Forum
<IDMS-L@LISTSERV. Subject
IUASSN.COM> Re: Problem with index contention
and replication a record

11/09/2008 01:46
PM


Please respond to
IDMS Public
Discussion Forum
<IDMS-L@LISTSERV.
IUASSN.COM>






Javier -

We used to have severe deadlocking problems similar to what you've
described - hundreds each day.
Now we get maybe no more than 10 a day.

We resolved this through programming efforts.
I don't know how many programs are affected in your shop, but it may be
easier to just handle it from the applications side.

Start with changing OBTAIN to OBTAIN KEEP EXCLUSIVE.
We found this took care of about 25% of our problems.
We had better luck calling a DC-COBOL program from ADS to do ENQUEUE /
DEQUEUE at appropriate spots in the code.
Sometimes the appropriate spots ended up being right upon entry and just
before map out.
Every dialog or DC program that hits this index would need to ENQUEUE /
DEQUEUE on the same resource name.
This worked well for us.

If you have very high online task volume, this will elongate your response
times because now you are single-threading through your updates.
The effort wasn't that tough to do and it made a big difference for us.
We never went through the effort of measuring average response time
afterward.
All I know is nobody complained about it.
If anything, they were happy to get rid of the deadlocks.

Thanks.
Jon Gocher


----- Original Message -----
From: ""javier sotela"" <jsotela@HOTMAIL.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Sunday, October 26, 2008 12:09 PM
Subject: Problem with index contention and replication a record


Hi Everyone:

We have a problem with an apllication and specifically with a record. We
have a record calc with the cal key is a sequence number 8 bytes, this
represent order number. In the same record we have the phone number and we
defined an index with phone number ten bytes + random number 8 bytes. This
is a heavily used record in the application, and there is lot of user
udating the record, with the problem that when they create record, the
order
number is a sequence number and the phone number is almost sequence number,
then the key in the index its very similar and normaly goes almost on the
same sr8, no matter how different is the random number attached at the end
of the index key. Then we have a lot of deadlocks because of the index key.
We are trying to create a new record calc, with the calc key the phone
number and putting in there the order number, just to preserve the chance
to
retrieve the information by phone number or by order number. The idea its
try to replicate some of the information when the original record its
created without changing programming. Does anybody can tell me how to do
the
data replication of the record without changing programming??? I know that
there is database procedures, but its suppose that you don't have to use
dml
verb in there, then I'm looking for any solution or on the other hand any
ideas in how to solve the problem with the index contention ???

Thanks.

J. Sotela
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
"
IDMS Public Discussion Forum
IDMS-L@LISTSERV.IUASSN.COM
SMTP
IDMS-L@LISTSERV.IUASSN.COM
IDMS-L@LISTSERV.IUASSN.COM
SMTP








Normal

Normal
Re: IDMS Data Replication Via TCP/IP
"Steve -

It's our own software, ....no purchased products.
It's customized for our internal use.

The SUBSCRIBE side works using WAIT / POST.
We don't read the next message off the queue until the task that is attached
to process the message completes.
We single-thread message processing so we don't flood the system and have
massive deadlocking problems.
On simple messages, we may process several dozen per second.
On more complex messages, we may process 1 message every couple of seconds.
All-in-all, databases are synchronized within about 1 to 10 seconds of one
another.
This is just fine for our purposes.

Jon Gocher


----- Original Message -----
From: ""Harmeson, Steve (Newport News)"" <Steve.Harmeson@NGC.COM>
To: <IDMS-L@LISTSERV.IUASSN.COM>
Sent: Monday, November 10, 2008 10:02 AM
Subject: Re: IDMS Data Replication Via TCP/IP


Are you using WebSphere MQSeries software to do this?

Outcomes