Plex 2E

Expand all | Collapse all

SQL BlockFetch going haywire

George Jeffcock

George JeffcockDec 10, 2014 09:56 AM

  • 1.  SQL BlockFetch going haywire

    Posted Feb 12, 2013 02:06 AM
    Hi All

    We are using the SQL BLockFetch on the Plex Wiki. Lately we have notice that in some conditions the fields values change, to a random value on our grids.
    We have notice that our numeric fields and our dates fields are the ones being effected.

    What typically happens, the first time we load our grid, the data is fine, if we add to our where clause in our SQL Query (we already have a where cause, just add more where statements to the query), and we see if the rows return are 7 and less, then our numeric and date fields display different values that are not in our database. It seem it display a random value.

    If we keep refreshing (reloading) our grid the values tend to jump to different random values, and something it jumps to the right value, and then again to the wrong value.

    As you can see, this is not normal behavior, we are not sure what the problem is, or where it could be, but it does seem like some over flow error.

    Any help would be greatly appreciated

    Devlyn

    Edit:
    Also found this thread now BlockFetchSql on iSeries OS v 6R1, and believe we having the same problem.


  • 2.  RE: SQL BlockFetch going haywire

    Posted Feb 12, 2013 04:25 AM
    Hi.

    I'm Arni, I invented the BlockfetchSQL pattern. I know I have made a few enhancements locally which I have not yet published, I don't have any specific info about what was updated, but I could send you an update to try. On the other hand, I would like to see if this is a problem with the sql and the fields you are putting into the SqlView variable. It the output of the sql does not match exactly with the fields in SqlView you can get problems.

    Would you like to try out an update?


  • 3.  RE: SQL BlockFetch going haywire

    Posted Feb 13, 2013 01:59 AM
    Hi Arni

    I would love to try the update. Where can I get it? Most of our SQLBlockFetches query from a custom SQL view, we use a entity in Plex to make a dummy view with all the right fields in sequence. We use this view in SqlView variable and the Output. But will flag this, so we can double check.

    If there is any other information you need, you are welcome to ask.


  • 4.  RE: SQL BlockFetch going haywire

    Posted Feb 13, 2013 11:00 AM
    Hi Devlyn.

    Try this update for me please. If you manage to use this update to update your model and all goes well, I will publish the update for all to use at http://wiki.plexinfo.net/index.php?title=My_BlockFetch_dynamically_using_SQL

    Update file at: http://www.filedropper.com/blockfetchsqlupdate_1

    Good luck.


  • 5.  RE: SQL BlockFetch going haywire

    Posted Feb 14, 2013 12:43 AM
    Hi Arni

    Thank you very much, I have downloaded it, little bit busy with other more pressing matters. But pretty soon, will be able to test the update out.

    I'll keep you posted how everything goes.

    Devlyn


  • 6.  RE: SQL BlockFetch going haywire

    Posted Feb 19, 2013 06:57 AM
    Hi Arni

    We tried the update, I have bad news, it did not go well, imported the update you gave us, and it broke our SQLBlockfecth. On every Call it returns 64 blank rows. We also notices that when we try and compile the SQLBlockFetch in SQLRPGIV then it does not want to compile, we change it to SQLRPG400 then it compiles but then returns 64 blank rows.

    When we compile in SQLRPGIV we see the following in the spool file:
              Field             Attributes         Defined                         
              No references in the source.                                         
        * * * * *   E N D   O F   E X T E R N A L   R E F E R E N C E S   * * * * *
      5761WDS V6R1M0  080215 RN        IBM ILE RPG             NEL60283/P6P7BF     
                               M e s s a g e   S u m m a r y                       
      Msg id  Sv Number Message text                                               
     *RNF5152 00      1 DUMP is used but Debug not specified in Control            
                        specification.                                             
     *RNF7031 00    190 The name or indicator is not referenced.                   
     *RNF0637 30      2 An operand was expected but was not found; specification is
                        ignored.                                                   
     *RNF5023 30      5 Factor 2 operand is not specified for this operation.      
     *RNF5030 30      5 Result-Field operand is required but not specified.        
     *RNF5359 30      9 The EVAL operation is missing a result for the assignment. 
            * * * * *   E N D   O F   M E S S A G E   S U M M A R Y   * * * * *    
      5761WDS V6R1M0  080215 RN        IBM ILE RPG             NEL60283/P6P7BF  
    And later in the spool file we see:
    =====================================================================
     *RNF5359 30 a      000719  The EVAL operation is missing a result for the assi 
        868 c                   Eval      Element=0                                 
        869 C     F0002H        IFEQ      CP                                        
        870  *                                                                      
        871  *                                                                      
        872 C                   EXSR      SZ000C                                    
        873 C     F0002I        IFEQ      CR                                        
        874 C     F0002I        OREQ      CS                                        
        875 C     F0002I        OREQ      CT                                        
        876 C     F0002I        OREQ      CU                                        
        877  *                                                                      
        878 C                   MOVEL (P) F0002F        F0002R                      
        879 C                   ELSE                                                
        880  *                                                                         
    We rolled back to before the update. Gonna double check if our Plex fields match with our SQL fields. But is there anything you may know that we can try to get the update working?

    Thank you for your help thus far.
    Devlyn


  • 7.  RE: SQL BlockFetch going haywire

    Posted May 02, 2013 04:37 AM
    I think your compile error indicates that a API call to the source "_SQL_RPGSource.PrepareSQL" is missing or this source is missing the line:
    C Z-ADD 0 Element 5 0

    The Element field is defined in this statement. Can you check why Element is not defined ? Is the above line missing ?

    It's been a long time since you got my update. Have you perhaps managed to sort this out ?

    Regards,
    Arni.


  • 8.  RE: SQL BlockFetch going haywire

    Posted Feb 12, 2013 04:32 AM


  • 9.  RE: SQL BlockFetch going haywire

    Posted Feb 12, 2013 05:00 AM
    We have already done this. It may also seem like it could be a casting issue.


  • 10.  RE: SQL BlockFetch going haywire

    Posted Feb 12, 2013 08:51 AM
    Hi Devlyn,

    You don't mention what platform you're experiencing this on.

    If it's the IBM i, then have you looked into debugging the server function and seeing if the corruption you're seeing is with the data being returned, or something else?

    The issue in the other thread you linked is something different, I believe. In that instance the IBM i Query engine is failing and dumping out.

    Crispin.


  • 11.  RE: SQL BlockFetch going haywire

    Posted Feb 13, 2013 12:53 AM
    Hi Crispin

    Sorry about that, the platform we are using is the IBM. I have no idea how to debug a RPG function, is there maybe a tutorial I can follow somewhere?

    Devlyn


  • 12.  RE: SQL BlockFetch going haywire

    Posted Feb 13, 2013 10:51 AM
    Hi Devlyn,

    From the green screen command line.

    1. Find the server job serving the client you wish to debug.
    2. STRSRVJOB jobnumber/user/jobname
    3. STRDBG Library/Program OPMSRC(*YES)
    4. Add breakpoints where you need them.

    You will need the source of you are generating RPGIII and manually use ADDBKP unless the programs are compiled with DBGVIEW(*ALL/*SOURCE/*STMT).

    Google search of debug rpgiii program may help you...

    Good luck,

    Crispin.


  • 13.  RE: SQL BlockFetch going haywire

    Posted Feb 14, 2013 12:40 AM
    Hi Crispin.

    If the update does not work, then I have another way to isolate the problem.

    Thanks for you help

    Enjoy
    Devlyn


  • 14.  RE: SQL BlockFetch going haywire

    Posted Aug 13, 2013 05:36 AM
    Hi Devlyn.

    Today I fixed a bug in my BlockfetchSQL pattern which may have effect on the output result if you are using Zoned Decimal numbers. My fix only affects Zoned Decimals. The fix involves adding two lines into the RPG source named Zoned2Ch.

    The first part of Zoned2Ch should be, leave the second SELECT statement unchanged:
    *==================================================================
    * ZONED2CH
    *==================================================================
    C MOVE &(1:) $PAK1 1
    C MOVE &(1:) $PAK2 1
    C BITOFF X'0F' $PAK1
    C BITOFF X'F0' $PAK2
    C SELECT
    C $PAK1 WHENEQ X'F0'
    C MOVE '+' $CHA1 1
    C $PAK1 WHENEQ X'D0'
    C MOVE '-' $CHA1
    C $PAK1 WHENEQ X'00'
    C MOVE 'E' $CHA1
    C $PAK1 WHENEQ X'40'
    C MOVE 'E' $CHA1
    C OTHER
    C MOVE '+' $CHA1
    C ENDSL

    The lines I added were these:
    C $PAK1 WHENEQ X'40'
    C MOVE 'E' $CHA1

    Good luck.


  • 15.  RE: SQL BlockFetch going haywire

    Posted Aug 13, 2013 06:43 AM
    Hi Arni

    Thank you for you efforts. I have been busy with other stuff, but this still remains a problem. The previous update you sent, did not even what to compile SQLBlockFeth. But I will give this fix of yours a go, when time permits and let you know.

    Enjoy
    Devlyn


  • 16.  RE: SQL BlockFetch going haywire

    Posted May 12, 2014 10:25 AM

    Hi Arni

    I was given time now to work on the SQLBlockfetch, I have tried your code change, and sad to say that it does not solve our problem.
    I will continue to scratch around and find out more about our problem.  But if you have anything suggestions it would be appericated.

    Enjoy
    Devlyn



  • 17.  RE: SQL BlockFetch going haywire

    Posted May 13, 2014 08:43 AM

    Hi Arni

    I did some testing, and I discover the pointer sqldata in the RPG API calls (GetFieldData_PackedDecimal, GetFieldData_PackedDecimal, GetFieldData_ZonedDecimal) sometimes reutrns a value that has concaternated to or more other fields togeter.  This is why our numbers sometime appear random on our WinC Grid screens.  Do you maybe have any idea as why this could be happening?

    Enjoy
    Devlyn



  • 18.  RE: SQL BlockFetch going haywire

    Posted May 15, 2014 09:25 AM

    Hi Devlyn,

    One of our developers just ran into a similar problem with this pattern.  We are on IBM I 7.1 and Plex 6.1.  He is having trouble with a zoned number.  Here is what he told me about the problem.  This was after he applied the fix that Ari mentioned for zoned numbers.

    BEGIN EMAIL

    Well, it sort of fixed it.  I thought I’d put it in an email to make it clearer.

    If the first character after the number has the 4 upper bits as x’40’ then it works.  If I change the “OTHER” part to put an ‘E’ in there it works better.  However I have an example where it’s just plain a longer number.  The zoned number I want is ‘02’ but what’s in the buffer is ‘0281’ or in hex F0F2F8F1000000… So you just can’t find the end by checking the upper bits.

    It looks like each variable returned from the sql stmt gets a 1000-byte allocated space.   It happens here:

    * API Call  Source code: MMA_CORE/DynamicSQLPrepare2          
    C                   Eval      SqlN=1                           
    C/exec sql                                                     
    C+ DESCRIBE SQLExtract INTO :SQLDA USING ANY                   
    C/end-exec                                                     
    C                   Eval      SqlN=SqlD                        
    C                   Eval      SQLABC=SqlN*%Len(SqlVar)+16       
    C/exec sql                                                     
    C+ Describe SQLExtract Into :SQLDA Using ANY                   
    C/end-exec                                                     
    C                   Z-ADD     0             Element           5 0
    C                   DoU       Element=SqlD                     
    C                   Eval      Element=Element+1                
    C                   Eval      SqlVar=SQL_VAR(Element)          
    C                   Alloc     10000         SqlData         
    C                   Alloc     1             SqlInd          
    C                   Eval      SQL_VAR(Element)=SqlVar       
    C                   EndDo                                    

    It also looks like when this fails, the 1000-byte SqlData area isn’t zeroed out.  Not knowing how to do that, I assigned blanks to it and since the other guy’s fix checks for x’40’, it catches it and stops at the right time.  It doesn’t break other fields either, but you should probably tell me how to put zeros in the buffer.  I declared a variable

    D $tmp            S           1024A   based(tmpptr)

    And then added this code after the Alloc statement:

    C                   Eval      tmpptr = sqldata  
    C                   eval      $tmp = ' '         

    Which fills it with x’40’ bytes.

    Okay, so if you tell me how to put zeros in there I’ll experiment a bit more and then you can decide what to do next. 

    END EMAIL

    I told him how to fill it with all x'00', but I haven't heard back from him yet.  I tried searching on the internet to see if I could find examples of other people using the SQLDA with dynamic fields.  About the best answer I could find was in this thread http://itknowledgeexchange.techtarget.com/itanswers/embedded-sql-in-rpg-ile/.  The poster named Sloopy said he had over 1700 lines of code that he used to handle the conversion of the data from the SQLDA.  He didn't post all of his code, but he said he could send it if someone requested it.  This thread is from 2008.

    Dean Eshleman,
    Everence Financial



  • 19.  RE: SQL BlockFetch going haywire

    Posted May 15, 2014 10:01 AM

    Hi Dean

    Thank for your post, it has been helpful.  I am currently reading up on the Embedded SQL in RPG, to get a better understanding how this SQLDA data structure is working.  Hopefuly it may give me a clue into how adjust the SQL Blockfetch patten so it can work well with zoned and pack deciamls (the problem we are facing).

    I sent an email to Sloopy, and hopefully I may get the source code, that I can study it and get a better understanding.

    But if I can maybe get in touch with your developer.  We can maybe knock heads, and come to a solution quicker.

    Enjoy
    Devlyn



  • 20.  RE: SQL BlockFetch going haywire

    Posted May 15, 2014 01:44 PM

    Devlyn,

    You can reach us at dean.eshleman@everence.com and mark.byler@everence.com.

    Dean Eshleman
    Everence Financial



  • 21.  RE: SQL BlockFetch going haywire
    Best Answer

    Posted May 19, 2014 03:07 AM

    Hi All

    I have found the solution.  The problem is that the SQLDATA variable is a buffer that returns the data fetched.  It's a fixed length, and never gets initialized before each fetch of each field so it gets garbage data after the data you want coming back.  This is a problem for numeric fields as packed and zone decimals.  As garbage characters gets included in the casting from string to packed or zone decimal.  This causes the random numbers to be displayed on our WinC grids.

    I needed to find a way to only work with the length of the data in the SQLDATA variable.  So it does not include the garbage characters.  I did a little reading up on embedded SQL in RPG.  And found out that you can Query a variable called SQLLEN to get the length of the data coming back:  You can refer to IBM's manual http://publib.boulder.ibm.com/iseries/v5r2/ic2924/info/sqlp/rbafymst.pdf

    But for packed or zone decimals, it returns a number that represents the scale and precision.  You can do a simple calculation to get the length, so you do not include the garbage characters.

    Here is what you need to do:
    First you need to create a field in plex I called mine DataLength and it inherits from FIELDS/Number.  Then you need to create a new source code GetFieldDataLength, which has DataLength a parameter.  You then add the following code to the GetFieldDataLength:

         D numLength       S             10I 0
     
          /Free
             numLength = %DIV(SqlLen:256);
             &(1:) = numLength;
          /End-Free
     
    Then you add the DataLength field to the Sql variable in the BlockFetchSQL function. Then you add an API Call to the new source code GetFieldDataLength just after the following after API Calls in the BlockFetchSQL function:
    • GetFieldData_PackedDecimal
    • GetFieldData_ZonedDecimal

    Then change the following while loops:

    • From: While  Sql<Sql.OutChar2> != <Sql.OutChar2.End> AND Sql<Sql.OutChar2> != <Sql.OutChar2.Minus> AND Sql<STRING Position> << <STRING Position.30>
      To: While  Sql<Sql.OutChar2> != <Sql.OutChar2.End> AND Sql<Sql.OutChar2> != <Sql.OutChar2.Minus> AND Sql<STRING Position> << <STRING Position.30> AND Sql<STRING Position> <= Sql<DataLength>
    • From: While Sql<Sql.OutChar1> != <Sql.OutChar1.End> AND Sql<STRING Position> << <STRING Position.30>
      To: While Sql<Sql.OutChar1> != <Sql.OutChar1.End> AND Sql<STRING Position> << <STRING Position.30> AND Sql<STRING Position> <= Sql<DataLength

    And that solves the blockfetch from going haywirewink

    Hope this can help you all out that is using this pattern
    Enjoy
    Devlyn



  • 22.  Re: SQL BlockFetch going haywire

    Posted Dec 10, 2014 09:56 AM

    Ditto so this posting will prove invaluable



  • 23.  Re: SQL BlockFetch going haywire

    Posted Jan 20, 2015 05:09 AM

    Thank you Dean and Devlyn.

     

    I had an example that I could recreate on demand and Devlyn code change fixed it.

     

    Well done all concerned



  • 24.  Re: SQL BlockFetch going haywire

    Posted May 26, 2015 04:33 AM

    I've ran into a similar problem as in original post. If I try compile a function that inherits _SQL.Fetch.BlockFetchSQL using language SQLRPGIV it fails, but changing to SQLRPG400 the compile succeeds.

    Since I'm only testing I use simple table with two fields specified as character.

    I downloaded the XML from My BlockFetch dynamically using SQL - The CA Plex Wiki yesterday and tried imported it into two different models ending up with same failure:

    *RNF5152 00      1 DUMP is used but Debug not specified in Control specification.                                    

    *RNF7031 00    114 The name or indicator is not referenced.          

    *RNF0604 20     10 The token is not valid; token is ignored.         

    *RNF3301 30      2 The Name entry is not valid; specification ignored.

    *RNF5309 30     37 Factor 1 operand is not valid.                    

    *RNF5311 30      4 Factor 2 operand is not valid.                    

    *RNF5313 30     51 Result-Field operand is not valid.                

    *RNF7030 30      3 The name or indicator is not defined.             

    *RNF7503 30      8 Expression contains an operand that is not defined.

     

    *RNF3301 30 a      000028  The Name entry is not valid; specification ignored.

        29 D $ZStr           S             30A   based(ZSqlPtr)                   

     

    *RNF5309 30 a      000649  Factor 1 operand is not valid.            

       825 C                   MOVE      '0'           $CHA1             1

     

    *RNF5313 30 a      000650  Result-Field operand is not valid.        

       826 C     $PAK1         WHENEQ    X'10'                           

     

     

    I hope someone can figure out what might be the problem, we run Plex 6.1 and IBM i is on 7.1.



  • 25.  Re: SQL BlockFetch going haywire

    Posted May 26, 2015 08:09 AM

    I had to change all the source by hand where  '$' was found to another character...I choose 'Z' and once replaced in all the source it worked.

     

    I was lazy and did not work out why so I cant help you on the why



  • 26.  Re: SQL BlockFetch going haywire

    Posted May 26, 2015 09:49 AM

    Indeed that worked!

    It might be some problem with character conversion in the communcation with the server from Plex dev env.