Service Virtualization

  • 1.  message queues vs stability

    Posted May 21, 2016 12:51 PM

    Typically what is the response time  of  MQ virtual service ? yes, off course i have not set any think time here ..

    currently we are hitting an issue during our performance testing of a soap service ..

     

    our soap service calls MQ VS in lisa.. and timeout of soap service is 7 seconds .. if no response received with in 7 sec from mq then our soap service gets time out..

    currently we have 5% calls failed during our 1 hour of load test and when we analyzed all calls faults at soap service says MQ 2033 error , which means timeout or no message available in  reply queues..

    as per MQ server concern , our MQ server and queues are configured properly and it is able to handle any load and which is proven in production.. as per the soap service concern it is also proven in production, so I would like to know how can we ensure LISA MQ VSMs are quick and also calls handled properly i.e., when multiple vusers hits very quickly there shouldn't be issues at VSM in identifying the message for right soap call and respond..



  • 2.  Re: message queues vs stability

    Posted May 21, 2016 02:09 PM

    There are a lot of variables involved with performance tuning IBM MQ.  What version of LISA are you on?  Are you using the new IBM MQ Native protocol, or the old  IBM MQ Series protocol?  Are there multiple request queues?  Are there multiple response messages/queues?  Is the response queue always the same, or are there temporary queues involved?  What's the average size of the request and response bodies?  What data protocols are you using and how many arguments are there?  Have you made any changes to the stock generated VSM? Have you looked at the 'share sessions' checkbox (IBM MQ Series) or Runtime Scope (IBM MQ Native) settings?  Have you gone through the general recommendations for performance tuning VSE?  At what capacity are you running the VS?  What does the CPU usage on your VSE server look like?  What is the actual load, in transactions per second or maybe concurrent users, that you're trying to push through?

     

    I've personally put upwards of a thousand transactions per second through an IBM MQ virtual service, but results vary depending on the exact scenario and circumstances.



  • 3.  Re: message queues vs stability

    Posted May 22, 2016 07:05 AM

    There are a lot of variables involved with performance tuning IBM MQ. 

     

    What version of LISA are you on?   DevTest 8.3 (devtest server is running on redhat and client is on windows 2008)

     

    Are you using the new IBM MQ Native protocol, or the old  IBM MQ Series protocol?   I am using 'Native Client'  i.e., 'IBM MQ Series (Deprecated) 'and VSM is created using both RR pairs & recording and then merged. mostly RR pair transaction are maximum. no think time set for any of the transaction. At this moment we dont have plan to change to new MQ series but yes eventually we do it.. and I hope using the protocol does not affect the performance..

     

    Are there multiple request queues? yes three request queues & only one response queue.. all the three requests , responses will be put it into same response queue. the soap service will internally make call to these three request queue one after the other based on condition and responses will be received on same response queue.

     

    Are there multiple response messages/queues?  answered above.. response queue is one.. but messages received every time is different.

     

    Is the response queue always the same, or are there temporary queues involved? same .. no temporary queues used in our case.. all queues are running on IBM MQ 7.1 (in our case only one response queue.. that we call as LISA.REPLY )

     

    What's the average size of the request and response bodies?  it is simple payload.. payload size is 200 to 300 lines for response.. 100 to 150 lines for request.

     

    What data protocols are you using and how many arguments are there?

        XML Data Protocal and Generic xml payload parser .. number of args in payload parser are around 30. But the reason it is 30 args because we have created individual mq call separately and merge the VSI and VSM args .. so it is growing in number of args.. but the request should always filters for arguments it needs and send the request to VSI image selection step ... in that  way we hardly use 8 to 12 args for request matching in VSI.

     

    Have you made any changes to the stock generated VSM? not much.. but yes there are two more steps added one is script step which is after 'MQ {vse.publisher} publish' step, this is very simple script to capture msg id & correlation id and followed by this step, there is another step which just writes these details file using 'write to delimited file'

    Have you looked at the 'share sessions' checkbox (IBM MQ Series) or Run time Scope (IBM MQ Native) settings?  I am not aware of this.. but I am using default as value for most of the properties in VSM & VSI step, well, there are some properties which needs to be changed as per our environment so those are changed and functionally it works fine when we use it for functional testing purpose.

     

     

    below is the property details from VSI response - metadata

    msg.applicationIdData

    msg.applicationOriginData

    msg.backoutCount 0

    msg.characterSet 819

    msg.encoding 546

    msg.expiry -1

    msg.feedback 0

    msg.format MQSTR  

    msg.getVersion 2

    msg.messageFlags 0

    msg.messageSequenceNumber 1

    msg.messageType 8

    msg.offset 0

    msg.originalLength -1

    msg.persistence 0

    msg.priority 0

    msg.putApplicationName TSYS.Req

    msg.putApplicationType 6

    msg.replyToQueueManagerName TSYS.ST1                                       

    msg.replyToQueueName

    msg.report 0

    msg.userId mqm        

    msg.accountingToken 0000000000000000000000000000000000000000000000000000000000000000

    msg.correlationId 414d51205379735465737431202020202aa09f5502a71820

    msg.groupId 000000000000000000000000000000000000000000000000

    msg.messageId 414d5120545359532e535431202020202ca09f55026f0d20

    msg.putDateTime 1437484367460

    recorded_raw_response_bytes PD94bWwgdmVyc2lvbj0iMS4wIj8+PHNvYXA6RW52ZWxvcGUgeG1sbnM6c29hcD0iaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvc29hcC9lbnZlbG9wZS8iIHhtbG5zOnhza

    dest.ccid

    dest.channel LISA.CLIENT.01.01

    dest.connectionMode Native Client

    dest.destination LISA.REPLY

    dest.destinationType Queue - ASYNC

    dest.host MQ Server host

    dest.overrideQueueManager

    dest.password

    dest.port 60100

    dest.queueManager SysTest1

    dest.receiveExit

    dest.securityExit

    dest.sendExit

    dest.tempQueueModel

    dest.isTemp false

    dest.user mqm

    dest.requestDestination false

    dest.useCorrelationID false

    dest.argumentsAll false

    dest.argumentsProperties false

    dest.argumentsCorrelationId false

    disregardReplyTo false

    mq.dest.env.Use Non-Destructive GETs false

    mq.dest.env.Local Address Property default

    mq.dest.env.connectOptions default

    mq.dest.env.ConnTag Property default

    mq.dest.env.Header Compression Property default

    mq.dest.env.Message Compression Property default

    mq.dest.env.SSL CertStores default

    mq.dest.env.SSL Cipher Suite SSL_RSA_WITH_RC4_128_MD5

    mq.dest.env.SSL Fips Required default

    mq.dest.env.SSL Peer Name default

    mq.dest.env.SSL Socket Factory default

    Have you gone through the general recommendations for performance tuning VSE?  Not sure about this.. if there is something like this , please share the wiki link, i ll have a look at it..

    At what capacity are you running the VS?  default.. 100%

    What does the CPU usage on your VSE server look like? usage is not bad.. normal load and enough space & mem available on linux box  which is hosting VSE server

      What is the actual load, in transactions per second or maybe concurrent users, that you're trying to push through?

    yeah, maximum is 5Vuser, laod is for 1 hour, tranx for a hour needed is 1250 .. every Vuser hits as soon as previous iteration ends..

    [ when I put one Vuser or two vusers calls seems to be fine, and 99% calls are success, when I increase to 3 or more , then I start hitting these issues]



  • 4.  Re: message queues vs stability
    Best Answer

    Posted May 22, 2016 02:03 PM

    > At this moment we dont have plan to change to new MQ series but yes eventually we do it.. and I hope using the protocol does not affect the performance..

     

    The new IBM MQ Native transport protocol significantly improves performance.  It's still possible to tune the old IBM MQ Series protocol for performance a little bit, but not to the extent that the new protocol supports.

     

    > yes three request queues

     

    Here I'm a little confused.  The old IBM MQ Series protocol doesn't support multiple request queues out of the box.  You would have modify the VSM in some sketchy and not at all obvious ways to make this happen.

     

    > yes there are two more steps added one is script step which is after 'MQ {vse.publisher} publish' step, this is very simple script to capture msg id & correlation id and followed by this step, there is another step which just writes these details file using 'write to delimited file'

     

    These will definitely affect performance, especially writing to a file, but I don't have any hard numbers for you.

     

    > Have you looked at the 'share sessions' checkbox (IBM MQ Series) or Run time Scope (IBM MQ Native) settings?  I am not aware of this..

     

    In your VSM, open the 'Respond' step and find the checkbox on the left side labelled 'Share Sessions'.  Make sure this box is *checked*.  Without this set, at playback it will create a whole new connection to send each response.  With this set, each "VU" will create a single connection to send responses and reuse it for the life of the VSE service.

     

    > At what capacity are you running the VS?  default.. 100%

     

    No, I'm talking about the "Capacity" field, which is basically the number of threads that VSE runs your service in.  If you are running five simultaneous users to drive your test then I would set this to at least five.

     

    > Have you gone through the general recommendations for performance tuning VSE?  Not sure about this.. if there is something like this , please share the wiki link, i ll have a look at it..

     

    Unfortunately the only links I can find are on an old wiki and really outdated.  I can give you the highlights I believe are still relevant:

     

    Open LISA_HOME/logging.properties and make sure the VSE logging settings are turned down.  In particular, change this:

    log4j.rootCategory=INFO,A1

     

    to this:

    log4j.rootCategory=WARN,A1

     

    and this:

    log4j.logger.VSE=INFO, VSEAPP

     

    to this:

    log4j.logger.VSE=WARN, VSEAPP

     

    Find or create LISA_HOME/local.properties, and add these lines:

    lisa.pathfinder.on=false

    lisa.eventPool.maxQueueSize=65535

    lisa.CycleExecHistory.buffer.size=2



  • 5.  Re: message queues vs stability

    Posted May 22, 2016 05:34 PM

    Thanks Kevin.. please find my updates below..

     

    > At this moment we dont have plan to change to new MQ series but yes eventually we do it.. and I hope using the protocol does not affect the performance..

     

    The new IBM MQ Native transport protocol significantly improves performance.  It's still possible to tune the old IBM MQ Series protocol for performance a little bit, but not to the extent that the new protocol supports.

     

    Thanks..do you have more details to share how this new mq helps in performance.. so we can analyze and start migrating our mq services..

     

    > yes three request queues

     

    Here I'm a little confused.  The old IBM MQ Series protocol doesn't support multiple request queues out of the box.  You would have modify the VSM in some sketchy and not at all obvious ways to make this happen.

     

    Sorry for the confusion.. when I say three request queue.. it is three separate MQ virtual service and but same response queue.. and all three service hosted on same VSE server..

     

    > yes there are two more steps added one is script step which is after 'MQ {vse.publisher} publish' step, this is very simple script to capture msg id & correlation id and followed by this step, there is another step which just writes these details file using 'write to delimited file'

     

    These will definitely affect performance, especially writing to a file, but I don't have any hard numbers for you.

    Yes, I agree. Even I had removed this and tested , seems improved in performance.. but I still need for some debugging purpose to capture message id and other details.. if it is https step we could have put this after responder step.. can we do this in MQ service as well ? or any alternative, please suggest..

     

    > Have you looked at the 'share sessions' checkbox (IBM MQ Series) or Run time Scope (IBM MQ Native) settings?  I am not aware of this..

     

    In your VSM, open the 'Respond' step and find the checkbox on the left side labelled 'Share Sessions'.  Make sure this box is *checked*.  Without this set, at playback it will create a whole new connection to send each response.  With this set, each "VU" will create a single connection to send responses and reuse it for the life of the VSE service.

    thanks.. got it.. I will check this and deploy..

     

    > At what capacity are you running the VS?  default.. 100%

     

    No, I'm talking about the "Capacity" field, which is basically the number of threads that VSE runs your service in.  If you are running five simultaneous users to drive your test then I would set this to at least five.

     

    Ok.. I think we don't have license for this so this is set to one and option is disabled.. with what type of license will this be enabled ? is this something really needed to make VS to respond for concurrent vusers ? how this make sense...

     

    I hope 100% to 0% think time scale also will help in improving performance..

     

    > Have you gone through the general recommendations for performance tuning VSE?  Not sure about this.. if there is something like this , please share the wiki link, i ll have a look at it..

     

    Unfortunately the only links I can find are on an old wiki and really outdated.  I can give you the highlights I believe are still relevant:

     

    thanks.. looks like I need to check this.. we have same server for both functional and performance testing purpose so I should make sure this change will not have any impact on functional testing aspects..

     

    Open LISA_HOME/logging.properties and make sure the VSE logging settings are turned down.  In particular, change this:

    log4j.rootCategory=INFO,A1

     

    to this:

    log4j.rootCategory=WARN,A1

     

    and this:

    log4j.logger.VSE=INFO, VSEAPP

     

    to this:

    log4j.logger.VSE=WARN, VSEAPP

     

    Find or create LISA_HOME/local.properties, and add these lines:

    lisa.pathfinder.on=false

    lisa.eventPool.maxQueueSize=65535

    lisa.CycleExecHistory.buffer.size=2



  • 6.  Re: message queues vs stability

    Posted May 23, 2016 01:01 AM

    > Thanks..do you have more details to share how this new mq helps in performance.. so we can analyze and start migrating our mq services..

     

    The new IBM MQ Native protocol is rewritten from the ground up.  The details are really too many to get into.  I will say that one of our people did a performance comparison recently and the basic takeaway was about a 20-40% increase in TPS, depending on the exact scenario.

     

    > if it is https step we could have put this after responder step.. can we do this in MQ service as well ?

     

    It doesn't matter whether it's before the Respond step, delaying the response, or after the Respond step, delaying the next request.  It delays transaction processing either way.

     

    > Ok.. I think we don't have license for this so this is set to one and option is disabled.. with what type of license will this be enabled ? is this something really needed to make VS to respond for concurrent vusers ? how this make sense...

     

    Unfortunately I'm not familiar with the licensing side of things.  However, if you are not licensed to increase the capacity of a VSE service past 1 then you will probably have a hard time getting serious performance out of it.