Service Virtualization

Expand all | Collapse all

Virtualize De-limited text with symbols ("control" characters)

  • 1.  Virtualize De-limited text with symbols ("control" characters)

    Posted Oct 26, 2017 09:43 AM

    Hi,

     

    I'm using Devtest v9.1 and trying to virtualize using the De-limited Text option. 

     

     

    The request text contains some symbols ("control" characters). hence while trying to create the image using the RR pairs option i'm getting an error - (Character reference "&#x7" is an invalid XML character)

     

    I got to know that XML does not support these symbols , hence the error.

    unicode - Why are "control" characters illegal in XML 1.0? - Stack Overflow 

     

    Please advice.

     

    Thank you.

     

    Regards,

    Isaac

     

     



  • 2.  Re: Virtualize De-limited text with symbols ("control" characters)

    Broadcom Employee
    Posted Nov 08, 2017 10:12 PM

    Hi,

        What is the protocol did you in Request datamanager while creating the service?. In latest version of Devtest 10.0, there is a protocol called "Null Dataprotocol" which will be useful for preventing content detection from running when no DPHs. Please try with this DPH and create virtual service.

     

    Thanks & Regards

    Srikanth Gajawada 



  • 3.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Jan 15, 2018 10:44 AM

    Hi Srikanth,

     

    We have upgraded to 10.2 now. Was trying out the Null Dataprotocol  as you had suggested. Was not sure on how we can assign the operation and argument in this case. Cos otherwise everything is assigned as <unknown-op> and not sure how we can track each transaction. Please advice. Thank you.



  • 4.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Nov 09, 2017 02:03 AM

    Hi Srikanth,

     

    Thank you very much for your valuable inputs. I tried using De-limited Text protocol and also the TCP protocol. Both failed in the last step when LISA tries to save it as XML. Sample Error (Character reference "&#x7" is an invalid XML character). We are in the process of upgrading to DevTest 10.1. So will definitely check it out.

     

    Thanks,

    Isaac



  • 5.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Jan 25, 2018 01:58 PM

    I am not certain that the community has all the information required to really provide a viable set of solution options.

     

    You might be able to use TCP recorder telling it to NOT treat the request as text and use a request delimiter of "Records are terminated with line endings" to at least get a request body built.

     

    Given what I see in your screenshot (copied below), whatever you do, you must handle the 0x02 character at the start of the request and the 0x07 characters that appear to separate one argument from the next. I don't see a specific delimiter on the end of the line. Perhaps, CRLF or LF are the end of line separators -- we do not have enough info in the post to know for sure.

     

    At a minimum, your first DPH in the chain is probably a custom DPH that: 

    - Gets the request body as a byte array

    - Iterates over the byte array and deals with the 0x02 and 0x07 bytes (maybe you strip the 0x02 and replace the 0x07 with a character that you can split fields on such as "~" or "|".

    - After dealing with the special characters, set the updated request back into the request body so it is passed to the next DPH  

    - The next DPH (perhaps Delimited Text DPH) parses based on the field separator you decide upon (i.e., ~ or |).

     

    For RR Recording, this will probably work. At run time however, the application under test might be sending a buffer and expect the TCP transport handler to know where a transaction begins (0x02) and where it ends (maybe some other special character). For this, you might still need a Custom Delimiter.



  • 6.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Jan 25, 2018 04:03 PM

    I am convinced that we do not have all the information necessary to help you complete a data parsing algorithm.

    However, I was able to get a VSI created using some sample data. Here is a shot of the VSI followed by a description of how I got this far. The circled argument (val5) is not parsed correctly - see further in this note:

    Step 1: Create some R/R Pairs (assume you have these already)

    Step 2: Configure the Recording to support TCP protocol

    I am not convinced terminating with line endings is accurate, but when using RR pairs it is. Run time execution may be expected to do something else.

    Step 3: Add DPHs. Note: I used the RDM DPH as a technique for stopping the recorder so I can see what it has done so far. It also gives the ability to back up, change things, and run forward.

    Step 4: Scriptable DPH - add logic to remove the x02 and replace the x07 characters with "|"

    Step 5: Configure the Delimited Text DPH to separate values based on the "|" symbol

    Step 6: RDM stops the recorder.

    a.  Notice you have no operation. This is not an XML or JSON based payload, so some logic needs to be added to set a valid operation. 

    b. Notice also that each parsed value aligns to an internally generated XML element name (i.e., argument) to satisfy the SAX parser and the VSI's Argument List. (e.g., val1, val2, val3, etc.). I recommend adding additional logic to set these arguments to meaningful names based on a better understanding of the incoming request payload. 

    c. val5 contains additional fields that are NOT separated by a 0x07 character. Some type of delimiter is missing here. (maybe tabs, maybe fixed length, maybe copybook DPH, who knows?)

    I will repeat that when the request enters from the SUT, you may still need a custom delimiter and you definitely need a custom Data Protocol Handler to parse things correctly and accurately. Good Luck!



  • 7.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Jan 31, 2018 09:40 AM

    Hi Joel,

     

    Thank you very much for info. We will try to work with the information you have provided.

     

    Thanks,

    Isaac



  • 8.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 01, 2018 11:00 AM

    Hi Joel,

     

    What i got to know is that all the responses do not have the control symbol. So i substituted the control symbol with another symbol in the request and created the model using the De-limited Text protocol to make things simpler. 

    Now I'm trying to change the symbol in the incoming request by using a Javascript step just  so that it can return the corresponding response .  

      

     

     

    Now when i send in the request ,

     

     

    the lisa.vse.request value got changed 

     

     

    But in the VS Response selection im getting an error 

     

     


    ============================================================================
    | java.lang.ArrayIndexOutOfBoundsException: -1
    ============================================================================
    | Step: VS Image Response Selection
    ----------------------------------------------------------------------------
    | Message: -1
    ----------------------------------------------------------------------------
    | Trapped Exception: -1
    | Trapped Message: java.lang.ArrayIndexOutOfBoundsException: -1
    ----------------------------------------------------------------------------
    STACK TRACE
    java.lang.ArrayIndexOutOfBoundsException: -1
    at com.thoughtworks.xstream.core.util.FastStack.pop(FastStack.java:42)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:125)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103)
    at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63)
    at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1061)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1052)
    at com.itko.lisa.vse.stateful.model.Streamer.fromXML(Streamer.java:92)
    at com.itko.lisa.vse.stateful.ConversationalStep.execute(ConversationalStep.java:453)
    at com.itko.lisa.test.TestNode.executeNode(TestNode.java:984)
    at com.itko.lisa.test.TestCase.execute(TestCase.java:1297)
    at com.itko.lisa.test.TestCase.execute(TestCase.java:1198)
    at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1183)
    at com.itko.lisa.test.TestCase.executeTest(TestCase.java:1124)
    at com.itko.lisa.coordinator.Instance.run(Instance.java:208)
    ============================================================================

     

    Please advice.

     

    Thanks,

    Isaac



  • 9.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 02, 2018 06:53 PM

    I am not sure what might cause this exception.

    What is the operation coded as in the VSI?  Does the VSI only have POST / and no additional information?

    I assume there are no Matches Script logic in the VSI.

    It would be best if we could see the VSI and a sample of the incoming request. Ideally, the incoming request in lisa.vse.request. In the log file it displays as: 

    2018-02-02 hh:mm:ss,sssZ (hh:mm)[ServiceName [VS_ServiceName_Run]/1] INFO  - Inbound Request {"id":0,"operation":"POST /sampleTest","arguments":{}}



  • 10.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 05, 2018 06:33 AM
      |   view attached

    Hi Joel,

     

    Please find the project attached. Please check and let us know.

     

    Thanks,

    Isaac

    Attachment(s)

    zip
    Test.zip   17 KB 1 version


  • 11.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 05, 2018 11:12 AM

    The issue is the JSR step in the VSM just prior to the Image Selection Step.

    The JSR code, see below, is overlaying the actual lisa_vse_request object. This object contains more than just the payload. It contains the headers, arguments, metadata, payload, etc.

    Is there a reason you need to overlay the VSE request object?  There seems to be a DPH filter that is taking care of the replace on the incoming request.  I thought your Response was the data you wanted to remove the CRLF and leave only a LF.

    Code Snippet: 

    req=testExec.getStateObject("lisa.vse.request");
    String payload = req.getBodyText();
    payload=payload.replace("\\x02","#");
    payload=payload.replace("\\x07","|");

    // This code snippet is overlaying the entire VSE request request object with a String object.

    // This is the cause of your index out-of-bounds as there is no formal VSE request object
    testExec.setStateValue("lisa.vse.request",payload);

    If you need to replace the CRLF with LF on the VSE Response selected by the VSI, then add a Scriptable DPH in the Responder step. Add code to replace the CRLF and put only the body back in the payload:

    lisa_vse_response.get(0).setBody( lisa_vse_response.get(0).getBodyAsString().replaceAll("\\r", "") );



  • 12.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 06, 2018 04:53 AM

    Hi Joel, 

    Thank you for your response. 

    Let me re-visit what we are trying to do. We started off trying to virtualize the below request and response. 

     

    Request :  005020 XYZ_COMM SPP_CO_STORE  Anotherfake (Pty) Ltd                                                                                                   1990010100                      74     900101                 Plot 12345                                                                                  Phase 0 Wonderland         ARandomtown                                     BW P O Box 12345                                                                                                          ARandomtown                                     BW 00000000                                                                                                                                       MadeUp                   False          Surnames                      0123454321    19900101Fake                     Surname                                      0101010101    19900101   

     

    Response:    CC658489711Record Stored successfully                                  |?     ?                                                                                                                                                                       

     

    Since we had the control symbols you had suggested we go with the custom DPH. 

     

    Alternatively, since we had no control symbols as part of the response what we tried to do was below. 

     

    1) In the Request -Replaced the control symbol with some other symbol which is supported                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

    Request : #005020|XYZ_COMM|SPP_CO_STORE||Anotherfake (Pty) Ltd                                                                                                   1990010100                      74     900101                 Plot 12345                                                                                  Phase 0 Wonderland         ARandomtown                                     BW P O Box 12345                                                                                                          ARandomtown                                     BW 00000000                                                                                                                                       MadeUp                   False          Surnames                      0123454321    19900101Fake                     Surname                                      0101010101    19900101   

     

    Response:    CC658489711Record Stored successfully                                  |?     ?      

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    2)  Created a VSM, VSI using the De-limited Data Protocol and deployed.                                                                                                                                                                                   

     

       3) Now at runtime, we are trying to replace the control symbols with the #, | so that Devtest can match the corresponding response above.

                                                                                                                                                                                                                                                                                                                     Please let us know if this is feasible.

     

    Thanks,

    Isaac                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             



  • 13.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 06, 2018 11:15 PM

    It should be possible, but I do not believe it solves all of your issues.  Let's not worry about the response until you get the request payload correct.

    Assume you have the following input request (# & | already replaced)

    #005020|XYZ_COMM|SPP_CO_STORE||Anotherfake (Pty) Ltd                                                                                                   1990010100                      74     900101                 Plot 12345                                                                                  Phase 0 Wonderland         ARandomtown                                     BW P O Box 12345                                                                                                          ARandomtown                                     BW 00000000                                                                                                                                       MadeUp                   False          Surnames                      0123454321    19900101Fake                     Surname                                      0101010101    19900101   

     

    In the Listen Step  -->  Filters,

     

    Add a Scriptable DPH (it is already in your code).  Remove the "\\x02" and replace it with "".  Replace "\\x07" with "|". Technically, we don't need to deal with the first byte (i.e., x02)

    %beanshell%

    String theBody = lisa_vse_request.getBodyAsString();
    // Change # back to x02

    // completely remove the first delimiter, it is not needed
    theBody=theBody.replaceAll("#","").replaceAll("\\x07","|");

    lisa_vse_request.setBody(theBody);

     

    Add a second DPH as a Delimited Text Data Protocol where the delimiter is the "|" symbol. 

    These two DPHs will parse a lisa_vse_request object having arguments val1 thru val5.

    Request {id=0, operation="POST /TCP", arguments=val1=005020&val2=XYZ_COMM&val3=SPP_CO_STORE&val4=&val5=Anotherfake (Pty) Ltd 1990010100 74 900101 Plot 12345 Phase 0 Wonderland ARandomtown BW P O Box 12345 ARandomtown BW 00000000 MadeUp False Surnames 0123454321 19900101Fake Surname 0101010101 19900101 , attributes=HTTP-Segment-Attr-0=TCP&delimiter=|&recorded_raw_request=005020|XYZ_COMM|SPP_CO_STORE||Anotherfake (Pty) Ltd 1990010100 74 900101 Plot 12345 Phase 0 Wonderland ARandomtown BW P O Box 12345 ARandomtown BW 00000000 MadeUp False Surnames 0123454321 19900101Fake Surname 0101010101 19900101 &recorded_xml_request=<payload delimiter="|">
    <val1>005020</val1>
    <val2>XYZ_COMM</val2>
    <val3>SPP_CO_STORE</val3>
    <val4></val4>
    <val5>Anotherfake (Pty) Ltd 1990010100 74 900101 Plot 12345 Phase 0 Wonderland ARandomtown BW P O Box 12345 ARandomtown BW 00000000 MadeUp False Surnames 0123454321 19900101Fake Surname 0101010101 19900101 </val5>
    </payload>&nameValueTokenDelimiter=&nameValueDelimiter=&listOfValuesDelimiter=|&pattern=&fieldNamesPath=, metaData=HTTP-Method=POST&HTTP-URI=/TCP&HTTP-Version=1.1&Accept-Encoding=gzip,deflate&Content-Type=application/text&frms_source=FFQ&frms_appid=FFQAPP&frms_tid=abc123&client_secret=2549110ff1434f4dBA8D4FEC5B3C51AA&client_id=ad453309a88c4eeeb2d2cfacf206180e&frms_region=TSJ1&Content-Length=883&Host=localhost:9021&Connection=Keep-Alive&User-Agent=Apache-HttpClient/4.1.1 (java 1.5)&lisa.vse.request.client.id=127.0.0.1:52157, matchTolerance=Exact, binary=false, body=(non-binary)[0 bytes] <null>}

     

    The issue lies in VAL5. There are no "|" delimiters but there appears to be many different fields.

    My assumption is that the VSM also needs to parse these values.

     

    Perhaps, an RDM DPH moves VAL5 to the Request Body. (The other arguments are not affected by this)

     

    Then, follow the RDM with a DPH to parse out the remaining part of the payload.

    Perhaps, a Copybook DPH could parse the entire payload or maybe you need some other code. There is not enough definition for us to be precise here.

     

    The code would parse the content of VAL5 (which was moved to the request body by the RDM) into meaningful arguments holding values such as:

    Val6 = Anotherfake (Pty) Ltd

    Val7 1990010100

    Val8 74

    Val9 900101

    Val10 Plot 12345

    Val11 Phase 0

    Val12 Wonderland

    Val13 = ARandomtown

    etc., etc., etc.

    Since we do not know the precise makeup of the incoming data payload, we cannot be specific about how this content needs to be parsed. This would be custom based on the variation that the consumer application can send to the service.



  • 14.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 14, 2018 05:40 AM

    Hi Joel,

     

    Thank you for your inputs. I did check out the Listen steps filters you had mentioned. So we do not need a JSR step explicitly.

     

    If we provide the request without the Control symbols we are able to get the corresponding response correctly in the existing setup.

     

     

     

    We need to change input (at runtime) from :     005020 XYZ_COMM MNM_CO_STORE  A Random Resturant Take Away(Pty)Ltd 

                                                                     to :    #005020|XYZ_COMM|MNM_CO_STORE||A Random Resturant Take Away(Pty)Ltd

     

    The existing code we have is not able to do it or we are missing something. Please advice.



  • 15.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 14, 2018 08:29 AM

    Is the graphic above a response as stored in the VSI? 

    If so, you can add the # and | in the VSI's response payload.  Adding these characters back into the response payload at runtime seems to be a challenge unless the content is fixed position. 



  • 16.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 14, 2018 08:59 AM

    Hi Joel,

    Thank you for your quick response. 

     

    the graphic is Postman actually ...sent the request and got the corresponding response.

     

    Actually We dont need to alter the response as they do not contain the control symbols. What we need is to alter the request @ runtime so that Lisa can fetch the corresponding response.

     

    Below is how things are in vsi.

     

     

    @ runtime we need to change the incoming request from:   005020 XYZ_COMM SPP_CO_STORE  Anotherfake (Pty) Ltd

     

      To:   #005020|XYZ_COMM|SPP_CO_STORE||Anotherfake (Pty) Ltd

     

      i.e.    Replace   with #,  Replace  with |

     

    We have the code in place in the Virtual Listener but it not working as expected.

     

    Thanks,

    Isaac 



  • 17.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 14, 2018 09:05 AM

    @ runtime we need to change the incoming request from: 

     

    To:  #005020|XYZ_COMM|SPP_CO_STORE||Anotherfake (Pty) Ltd

     

    PS: It is was not clear in the message so  putting in as a image.



  • 18.  Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Feb 19, 2018 03:19 AM

    Hi Joel,

     

    Please let me know if the above approach is possible.

     

    Regards,

    Isaac



  • 19.  RE: Re: Virtualize De-limited text with symbols ("control" characters)

    Posted Sep 15, 2020 04:12 PM
    ​I am having similar scenario. Can you let me know if the issue was resolved?