Endevor

  • 1.  Elibs "interesting" information

    Posted Jul 26, 2017 12:18 PM

     

    After experiencing a couple of serious Elib failures, I thought I would share this information with the community.

    We have now received confirmation from CA of currently (previously) undocumented size limitations for Elibs. 

     

    For an Elib INIT, the maximum permitted DIR PAGES is   32,767.  

    There is currently no restriction on specifying a larger number than this.  The initialisation will appear to work OK, but this will lead to a corrupted dataset   

     

    For a Elib DIR REORG, the maximum for DIR PAGES is 32,500.

    We became aware of this problem when we coded a REORG of 36,000 pages. This failed with an undocumented error.    

     

    CA will be supplying a PTF to ensure these limits are not exceeded.  This will be enforced when parsing the BC1PNLIB control cards.     

     

    Our largest Elib (a delta library)  over three quarters of a million members,  with 32,000 DIR PAGES.  This means we have exceeded the limit for optimal performance, according to the standard (rule-of-thumb) calculation: 32,767 * 24  =   786,400.  As we have no directory overflows, there is no impact yet.

     

    CA have also stated this restriction will have minimal performance impact.  They estimate that even with 3 million members, this limit, would only cause an average performance loss of 30% when reading a member.



  • 2.  Re: Elibs "interesting" information

    Posted Jul 26, 2017 04:23 PM

    Thanks for sharing this information Steve!



  • 3.  Re: Elibs "interesting" information

    Posted Jul 27, 2017 09:40 AM

    Thanks, Steve.... now I have to go back to the drawing board and see whether this affects my redesign project or not!



  • 4.  Re: Elibs "interesting" information

    Broadcom Employee
    Posted Aug 23, 2017 07:39 AM

    Hi Steve,

     

    We are still investigating this subject and will document this topics, in the meantime I would like to share these information with some confirming Steve's entries.

     

    1) The internal limit for directory pages is 32767.

     

    2) For INIT, the above limit should be adhered to.
       For BC1PNLIB REORG, the limit is slightly lower, depending on the number of map pages. Value 32500 should be always safe.

     

    3) We plan to document a new limit of 32500 and enforce it during parsing.

     

    4) So you can simply run reorg with correct value, with 32500 as recommended and we will document it.

    The doc says, for 4KB pages, one directory page can hold 48 members.
    But it also adds, that optimally, dir pages are only half full.
    So to calculate the optimal # of dir pages you will divide expected maximum number of members with number 24 (=48/2).

    The dir pages can play in performances.
    If you have too few dir pages, you get more overflows and more I/O.
    On the other hand, if you have too many dir pages, your performance will not grow beyond some point and you just waste space.
    The formula (constant 24 for 4k pages) is documented to be optimal.

     

    5) The new limit will be set to 32500 - is this too limiting?
    The number of members for which this value is optimal is 32500*24 = 780,000. But even if you store 3 million members in your ELIB your average performance for reading a member will go down only ~30%.
    ELIB processing is just a fraction of action processing so overall performance will probably drop roughly to single digit percentage. As you can see, the performance impact is low even for very high number of members, so the limit of 32k dir pages is generous enough and I believe you will be comfortable for while with that value.
    But number around 1,536,000 seems to be the limit for overflow-free directory, since directory page can store up to 48 directory members.
    However, this limit is not a wall you cannot go through. You can have more members with directory overflows and your ELIB will work just fine. You should worry only in case there is very high number of directory overflows. Roughly speaking, you can see measurable performance dropdown at about 2,000,000 members and you should start worrying about performance beyond 4,000,000 members.
     
    I'm not aware of ELIB storing such huge number of members and it would be interesting if the communities share their own situation regarding ELIBs. The BC1PNLIB INQUIRE will help you to get a status.

    BC1PNLIB OPTIONs
    None     
    -- If you omit this clause, CA Endevor SCM prints only data set header information.
    DIRECTORY
    --Tells CA Endevor SCM to print information about target directory page utilization.
    MEMBERS
    --Tells CA Endevor SCM to print footprint information plus member size in pages,
    records, and bytes.
    ANALYZE
    --Tells CA Endevor SCM to print an analysis of data set integrity.

    Also, we have already created some knowledge document(an other are coming soon), go directly to https://support.ca.com/us/search.html?q=ELIB or search for ELIB from https://support.ca.com to read them.

    Don't hesitate to evaluate and comment our knowledge document, your experience is important for us and we will review them if necessary.

    I hope this help.
     
    Regards,
    Ollivier



  • 5.  Re: Elibs "interesting" information

    Broadcom Employee
    Posted Jul 11, 2018 05:49 AM

    Hi Steve,

     

    I was currently investigating for another question about Elib page size and found this post. 

    In order to complete my answer, the solution is APAR #: RO97456 for Endevor 18.0

     

    Regards,

    Ollivier



  • 6.  Re: Elibs "interesting" information

    Posted Jul 11, 2018 06:48 PM

    The advice I received from CA employees when they visited to discuss their products was use PDSE format libraries.  They said the advantages of ELIB are gone now that we have the PDSE version 2 format.



  • 7.  Re: Elibs "interesting" information

    Posted Jul 12, 2018 10:55 AM

    I'd be interested to know if CA would echo that advice here?

     

    Are you setting PDSE ELIB's to VB to handle the differing type definitions?



  • 8.  Re: Elibs "interesting" information

    Posted Jul 12, 2018 11:04 AM

    I agree with Steve, Mathew. I kind of wonder if that was "endorsed" technical advice or not.... especially since I would assume you cannot put PDS/Es under VSAM/RLS control for performance enhancements whereas you can with an ELIB.