Service Virtualization

  • 1.  How to use predefined rules to record REST based traffic?

    Posted Jan 04, 2017 05:29 PM

    Hi All,

     

    We are going to record REST traffic everyday using CLI(ServiceImageManager). Kindly help me with below doubts:

     1. Are the rules going to be created automatically? If yes, are they going to be same every time?

     2. If I have a set of predefined rules in my VRS file, what will happen if a completely new operation/signature comes up?       Will it create any rule or not?



  • 2.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 01:33 PM

    I don't think you have any option to dynamically update the rules on vrs file but after the virtual services is created you can update the virtual services with new rules. 



  • 3.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 01:39 PM

    Thanks Vamsi for your reply!

    It will be a very tedious task to check the recordings everyday for the new operation/signature. I will be recording heavy traffic everyday. Need a work around for this atleast , to merge everyday's recordings in previous day's recording.



  • 4.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 02:26 PM

    Can you offer more background about the service you are trying to create and why you need so much data? 

     

    How many test scenarios are trying to support that require recording and merging so much data?  Are you trying to create some sort of 'super' service that has every possible combination of any type of request that can be sent to an endpoint?

     

    How many operations and data combinations (specific requests) within each operation are you anticipating supporting in your service?

     

    Given the limited amount of information given in the post, I am initially concerned about the following:

    - The resulting service is not "fit for purpose", rather trying to accommodate every possible combination of requests that can be sent to the live system

    - The resulting VSI will be so large that it will not open in Workstation

    - The SIManager will run out of memory trying to process the recording into a service.  

    - The server does not have enough horse power (CPUs) to process the data into a service image

    - The SIManager's magic string processing against such a large volume of data will cause your service creation to literally take hours to complete assuming you don't run out of heap first

    - The resulting VSI will be completely unmanageable when it comes to debugging why the consumer application got a "wrong" answer.



  • 5.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 06:23 PM

    Hi Joel,

     

    Actually we are trying to record a billing system, which is a REST backend. It renders account details, different taxes and amount details. We are recording everyday traffic to provide support in backend downtime. Everyday we will get around 200-300 mb traffic.

    To achieve this we merge each day's recording to a master file and hence keep the recording updated. There will be different number of URI combination depending on logic one is testing.

    To remove duplicate transactions and to make sure merge of two vsi is proper(i.e. same signature should be merged under one transaction) we have created some URI rules, according to which the VSI/VSM will be generated everyday.

    I just want to merge everyday's recording without the hassle to look into the vsi for a new signature and creating a new rule for it manually everytime.



  • 6.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 05:17 PM

    HI Ranjan, 

     

       Are you talking about match rules?

     

     

    Thanks,

    Vamsi



  • 7.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 05, 2017 06:07 PM

    URI Rules under REST Data Prtocol



  • 8.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 06, 2017 09:25 AM

    The merge process within DevTest is designed to merge VSIs not VSMs; hence, the issue with the URI rules. 

    Please consider the rest of this post as an idea not a recommendation.

     

    What if you could identify differences between the baseline VSM URI rules and the newly generated VSM rules.  The analysis would be the basis for adding the new rules to the REST protocol via Workstation's IDE. The VSM XML file contains the rules in the LISTEN Step.  If you open the baseline and the new VSM side-by-side in a text editor, you might identify the diffs.  An example of the rules stored in a VSM follows:

    <dataProtocol>com.itko.lisa.vse.stateful.protocol.rest.RestDataProtocol</dataProtocol>
    :
    <rules>
      <rule> <operation>GET /something/catalog/item/{URLPARAM0}/</operation> </rule>
      <rule> <operation>GET /something/catalog/department/</operation> </rule>
      <rule> <operation>GET /something/catalog/generic/{URLPARAM0}/</operation> </rule>
      <rule> <operation>GET /something/catalog/function/</operation> </rule>
      <rule> <operation>GET /somethingElse/catalog/function/{URLPARAM0}/itemGroup/</operation> </rule>
      <rule> <operation>GET /somethingElse/catalog/function/{URLPARAM0}/item/</operation>

     

     

    RISKS:

    I would recommend against updating the VSM manually for the following reasons.

    1) If you manipulate the VSM outside of DevTest's control (i.e., using a text editor) so you could inject unwanted errors that make the VSM unusable / unreadable.

    2) CA reserves the right to change the way we represent all components within the VSM when it is at rest without notification, so you are taking an inherent risk anytime you manipulate a DevTest asset outside of the product's editors (Portal, Workstation, etc.).

     

    Fringe options (risks still apply):

    1) Not certain this would help, but perhaps an XML Diff step that compares the baseline VSM to the newly generated VSM would help identify the URI rules changes / additions.  Have not attempted this.

     

    2) What if after the merge of the baseline and new VSI, you iterate over the Service and create a raw traffic file.  Then replay the raw traffic file through the recorder to reset all URI rules.  Problem with this approach is that an additional step is required to create the new baseline service.

     

    Lastly, my earlier concerns about SIManager memory, CPU utilization, and overall run time to create your service still applies -- even if the resulting VSI elminates duplicate transactions. 



  • 9.  Re: How to use predefined rules to record REST based traffic?

    Posted Jan 09, 2017 02:31 PM

    Thanks Joel for your suggestions.

     

    We have been storing all the transaction in rawTraffic file. We use raw traffic file to create the VSI/VSM. I tried the approach you suggested but ran into an issue.For example below are two transactions. If you see they both were recorded in 2 different raw traffic files. In first example Date wasn't parametrized whereas on the next day it was parameterized. Both the requests are same but the rules have now changed. I have 2nd example as well where a "fans" is parametrized one day and not on the second day but the on both days the request was same. 

     

    Now I have same transactions with two different rules created on 2 dates. How do we merge the VSI with manually making changes to transactions and verifying rules everyday. This activity is tedious and completely manual. 

     

    ***********************1st example******************************

    GET /restservices/csi-businessEBill/v1/wirelessTelephoneNumberBillDetails/masterAccounts/12345/accounts/12365498/invoices/2011-11-26/wirelessNumbers/1236665554477/planCharges
    GET /restservices/csi-businessEBill/v1/wirelessTelephoneNumberBillDetails/masterAccounts/{URLPARAM0}/accounts/{URLPARAM1}/invoices/2011-11-26/wirelessNumbers/{URLPARAM3}/{URLPARAM4}/

     

    GET /restservices/csi-businessEBill/v1/wirelessTelephoneNumberBillDetails/masterAccounts/12345/accounts/12365498/invoices/2011-11-26/wirelessNumbers/1236665554477/planCharges
    GET /restservices/csi-businessEBill/v1/wirelessTelephoneNumberBillDetails/masterAccounts/{URLPARAM0}/accounts/{URLPARAM1}/invoices/{URLPARAM2}/wirelessNumbers/{URLPARAM3}/{URLPARAM4}/

     

    ****************************2nd example********************************

    GET /restservices/csi-businessEBill/v1/billinghierarchy/users/*********/fans
    GET /restservices/csi-businessEBill/v1/{URLPARAM0}/users/{URLPARAM1}/{URLPARAM2}/

    GET /restservices/csi-businessEBill/v1/billinghierarchy/users/*********/fans
    GET /restservices/csi-businessEBill/v1/{URLPARAM0}/users/{URLPARAM1}/fans/



  • 10.  Re: How to use predefined rules to record REST based traffic?
    Best Answer

    Posted Jan 11, 2017 08:41 AM

    I see the frustration in the scenario above.  Perhaps, one of the other DevTest Community members has an idea on how to decrease the manual effort.

     

    You might raise an enhancement request for DevTest.  This is the only option I can think of at this point. 

    Unfortunately, short of developing some sort of programmatic merge process that accesses the 'raw' data, I do not have an idea of how the DevTest software would automate the scenario without a product enhancement.  The challenge for an enhancement would be to ensure that the software contains the ability to allow the developer to identify which pattern (for example, date vs parameterized date) is the valid pattern.  It would be inaccurate to have DevTest make that determination on its own since the tool does not understand business context.

     

    I am not at a place where I can view a Raw Traffic file.  Do the URL Params exist in the raw traffic or is the original URI stored in the request?  I thought the original URI was maintained in the raw traffic.  If that is true, one might build a custom process to merge two raw traffic files into a single file for replay.  If the rules are applied in the raw traffic, your scenario above comes back into play.