CA 1 Flexible Storage

Expand all | Collapse all

TMC / Audit Locks

  • 1.  TMC / Audit Locks

    Posted Dec 11, 2017 11:38 AM

    HI, I have seem too many enqueues recently on TMS-AUD and TMS-TMC, causing some medium impact to the systems, since sometimes it doesn't release the resource for some reason.

    I would like to know what causes a job to lock the TMC/AUDIT as exclusive. I understand that every Tape Processing will update AUDIT / TMC, but if THIS is the reason for the LOCKS I should have enqueues every minute with the number of Tape jobs running at same time. Can you please clarify why a job requires exclusive access to TMC/AUDIT?

    S=SYSTEM  TMSQNAME TMS-AUD                                           
    SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR   STATUS   
    XX00      ABC12345                   1234       007F83C0  EXCLUSIVE   OWN     
    XX00      XYZ12345                    0123       007F83C0  EXCLUSIVE   WAIT   



  • 2.  Re: TMC / Audit Locks

    Broadcom Employee
    Posted Dec 11, 2017 11:50 AM

    Hi,

    Do you have a TMSCOPY running at the same time?

    Do you use a blocket AUDIT?

     

    Depending on the answers, I will let you know if you need to open a case or not.

    Pierre



  • 3.  Re: TMC / Audit Locks

    Posted Dec 11, 2017 11:55 AM

    Hi Pierre

    There was no TMSCOPY running.

    I am not sure what would be a blocked AUDIT so assume I do not use blocked Audit. 

     

    Honestly the issue is fixed now, and what a most want here is to understand WHY a job requires exclusive access to Audit / TMC ? It is not pointed in JCL DISP=(OLD), it is not even referenced on JCL. Basically that's all I need for now. Understand what causes EXCLUSIVE access to such vital file to Tape Processing



  • 4.  Re: TMC / Audit Locks

    Broadcom Employee
    Posted Dec 11, 2017 12:11 PM

    Hi Marlm,

    This is the way we protect the TMC and AUDIT updates from one tape to the others.

    This is normally not measurable and very short in time except if you have a problem on tape devices or any other system tasks.

    You have in the documentation some explanations about QNAME but I recognize not so much detailed.

    Let me discuss with the labs to provide you a more detailed information.

    Thanks for asking.

    Pierre



  • 5.  Re: TMC / Audit Locks

    Posted Dec 11, 2017 12:47 PM

    Thank you Pierre



  • 6.  Re: TMC / Audit Locks
    Best Answer

    Posted Dec 11, 2017 12:31 PM

    As you said, the TMC is updated everytime there is a tape event (OPEN/CLOSE/EOV as well as a CATALOG/UN-CATALOG event). And, everytime the TMC is update we ALSO will update the AUDIT file. The order of the enq/deq is to first ENQ/LOCK the TMC, update the TMC, ENQ/LOCK the AUDIT, update the AUDIT, DEQ the AUDIT and finally DEQ the TMC. Now, some tape events (creating a secondary file for example) may cause 4 different updates to the TMC and AUDIT. So, creating many (thousands) of secondary-files in a short period of time will cause a LOT of I/O's to the TMC. And if each secondary-file is also cataloged; that will simply increase the number of I/O's and cause delays in the CATALOG address space.

     

    One way to avoid the delay in the CATALOG address space is to run the SMFQ subtask of the CTS address space. 

     

    How many systems are sharing the TMC and AUDIT file, and are you converting our RESERVE to a global enqueue using a GRS-STAR or MIM configuration? (GRS-RING should not be used, much to slow.)



  • 7.  Re: TMC / Audit Locks

    Posted Dec 11, 2017 12:52 PM

    Hi Russ;

    Thank you very much for your reply

    We usually have about 4 LPARS sharing AUDIT/TMC, in this specific system I had issues this morning it was 4 LPARS.

    We use GRS in this system.

     

    We had a situation like this:

     

    ISG343I 00.31.09 GRS STATUS 595 918                                  
    DEVICE:6101 VOLUME:XTTM05 RESERVED BY SYSTEM BT00                    
    S=SYSTEM  TMSQNAME TMS-AUD                                           
    SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR   STATUS   
    BT00      ZRPCPLD8           02AE       007F83C0 EXCLUSIVE   OWN     
    BT00      CLRQSTD8           02B3       007F83C0 EXCLUSIVE   WAIT   

     

    S=SYSTEMS SYSIGGV2 SYS1.TIRKST.CATALOG                    
    SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR
    GT00      CATALOG            0037       0078C148 EXCLUSIVE
    BT00      CATALOG /ZRPCPLD8  0037/02AE  0078C148   SHARE  
    AT00      CATALOG            0037       0078C148   SHARE 



  • 8.  Re: TMC / Audit Locks

    Posted Dec 11, 2017 02:11 PM

    Yes, that is showing the interaction between the MVS/Catalog and the TMC/AUDIT updates. You see, the logic I shared before (about the TMC and AUDIT being locked) gets longer and more complicated with the MVS/Catalogs. The reason is that the user-catalog (SYS1.TIRKST.CATALOG) is reserved while the MVS/Catalog is updating it (as expected); BUT it holds the reserve while it then writes an SMF record. Now, the CA 1 intercept then intercepts the SMF record being created, and updates the TMC. So now, the user-catalog is reserved UNTIL both the TMC and AUDIT have also been updated. This does cause a longer delay for the updating of the MVS/Catalog.

     

    The easiest solution to this is to start the SMFQ subtask of the CTS address space. The SMFQ subtask only affect MVS/Catalog/un-catalog operations on the system it is running on, so if you have 4 very active systems you would need to run the SMFQ subtask on all 4 systems. If only 2 systems have a lot catalog/un-catalog operations; then you could just run the SMFQ subtask on those 2 systems and let the other 2 systems run without it. The SMFQ subtask if active will eliminate the delay in the MVS/Catalog processing, so while the same number of updates are done to the TMC - they won't be causing any delay in MVS/Catalog processing.

     

    Of course, another option would be to enable Record-Level-Sharing for the MVS user-catalogs. But that is completely different solution and one that does not involve CA 1 directly.

     

    Russell



  • 9.  Re: TMC / Audit Locks

    Posted Dec 11, 2017 02:47 PM

    Thank you Russ.

    Just to make sure about how much time it takes in a perfect scenario.

    The lock at AUDIT/TMC happens at the OPEN ?

    The Unlock at Audit/TMC happens at the CLOSE ?

    When exactly each step Lock/Unlock happens ?

     

    BTW. Will check how to start SMFQ as suggested



  • 10.  Re: TMC / Audit Locks

    Posted Feb 01, 2018 11:07 AM

    Oh no. The AUDIT/TMC is locked ONLY while the update to the record is taking place. So, it is LOCKED, updated, un-locked at OPEN. Then it is LOCKED, updated, un-locked at CLOSE. Then, it is LOCKED, updated, un-locked at CATALOG time.

     

    What the SMFQ does is "move" the "LOCK, update,un-lock" from the CATALOG address space (done while the user-catalog itself is still locked) to the CTS address space and is done independently from the lock on the user-catalog.

     

    Russell



  • 11.  Re: TMC / Audit Locks

    Posted Jun 11, 2018 12:14 PM

    Again. Thank you for your contribution RussellWitt . I already started the implementation of SMFQ and it really seems to have fixed the enqueues we had.

    I am just out of curiosity now. Why does the update on TMC/AUDIT requires a lock on user-catalog ? The other question is if it would be reasonable to have the updates being done without the lock on user-catalog by default, without SMFQ jumping into it. What component would have to be changed for that ? Thanks again



  • 12.  Re: TMC / Audit Locks

    Posted Jun 11, 2018 12:34 PM

    It is not that the update on the TMC/AUDIT that requires a lock on the user-catalog; but rather that the user-catalog happens to be locked (by design) while the SMF-record is being created. And when CA 1 intercepts the SMF-record (after it is created, but before it is posted back to actual SMF-creator) it then updates the TMC/AUDIT. The reason that the MVS/CATALOG logic holds the enqueue while the SMF-record is being created is because the SMF records are used for recovery purposes. And MVS/Catalog wants to make sure that the SMF-record has been cut before releasing the enqueue and going on to the next update.

     

    I believe that IBM is coming out with an enhancement to SMF processing so that something like the IEFU83 exit is available AFTER the record has been cut (so the SMF-creator thinks it is all finished) but in the order of their creation and very shortly after it being created. It is very similar to the SMFQ however, and would basically do the same thing.

     

    Russell Witt

    Sr. Consulting Software Engineer