Service Virtualization

Expand all | Collapse all

Send EBCDIC byte stream as Response

  • 1.  Send EBCDIC byte stream as Response

    Posted Dec 06, 2017 05:22 AM

    Hi All,

     

    We are virtualizing the main frame transactions which are communicated over IBM MQ. Usually the consuming applications publish the request in the form of byte stream on the queues which will be picked up the down stream applications, Process them and publish the response again as a byte stream which will be consumed by the requesting applications.

     

    I don't have any issue for requests  which are of ASCII byte streams but when the request is an EBCDIC byte stream , I have the issue.

     

    When the consuming application publishes the EBCDIC byte stream , I use the below code to capture the byte request and convert that to a java string to compare against the virtual service transaction.

     

    import com.itko.lisa.test.TestExec;

    rawMessage = lisa_vse_request.getBodyBytes();
    String InputReq = new String(rawMessage, "Cp1047");
    lisa_vse_request.setBodyText(InputReq);

     

    This converted java string is used by the copy book to identify the matching transaction. once a positive match is identified, the response which is a java string need to be converted back to EBCDIC byte stream.

     

    I'm trying the below code but the issue is , the application is not able to consume it.

     

    String theBody = lisa_vse_response.getBodyText();
    byte[] bytes = theBody.getBytes("Cp1047");
    lisa_vse_response.setBodyBytes(bytes);
    return bytes;

     

    I believe the above code is converting the response java string to EBCDIC byte stream.The CCSID on the request side (Cp1047) converts the request as the correct java string which makes be believe that response should use the same to convert it back to EBCDIC byte stream.Please correct me if I'm missing any.

     

    Regards,

    Shivakumar.



  • 2.  Re: Send EBCDIC byte stream as Response

    Posted Dec 06, 2017 12:49 PM

    Which version of DevTest are you using?

    Are you using the newer IBM MQ Native steps in the VSM or the older steps?

    Check these links to see if there is any information that might help you solve this.

    Can EBCDIC format be converted to XML format  

    EBCDIC fomat to ASCII format in LISA 7.5 

    You also mentioned copybooks in the post. Are you using the Copybook DPH?

    Lastly, is the code snippet that sets the response body throwing an exception at run-time?  

    lisa_vse_response holds a list of responses. I would have expected to see something like:  lisa_vse_response.get( i ).setBodyBytes( ... ) where "i" is the occurrence of the response within the list.   



  • 3.  Re: Send EBCDIC byte stream as Response

    Posted Dec 06, 2017 04:06 PM

    Normally with IBM MQ, character set conversion is taken care of automatically.  It's driven by Coded Character Set ID (CCSID) specified in the MQ message headers.  You typically only run into this kind of issue when the CCSID specified by the message headers does not agree with the actual format of the message body.

     

    In 9.5.1 and later, the newer IBM MQ Native transport protocol has a built-in option to override the message's CCSID and use a different one for encoding/decoding the body.  You can do this for both receiving requests and sending responses.

     

    With the older IBM MQ Series transport protocol your only option is scriptable DPHs.  In that case, Joel is correct in that your response-side script looks like it will through an error.  lisa_vse_response is a list, so you need to be pulling the first element out of it or, if there are multiple responses, iterating over it.

     

    String theBody = lisa_vse_response.get(0).getBodyText();
    byte[] bytes = theBody.getBytes("Cp1047");
    lisa_vse_response.get(0).setBodyBytes(bytes);



  • 4.  Re: Send EBCDIC byte stream as Response

    Posted Dec 06, 2017 05:18 PM

    Hi Joel and Kevin,

     

    The DevTest version I'm using is 9.5 . We use the req/res to create the virtual services. These req and res files are fixed length java message strings which we generally get from the development team and we use copy book DPH to convert them into readable XMLs in VSI.

     

    But in the actual application implementation, the req is published as a byte stream into a queue from where the back end systems subscribes, process and publishes the response back in byte array. The kind of byte array which is getting published depends on the data type defined in the RIP/ROP files the application uses. The development team logs these byte arrays as java strings for us to use them to create the virtual services.

     

    So when the application publishes on to the virtual service queue, we also get the byte stream but I convert that to a java string using a scriptable DPH before handing it over the copy book filter to parse it and do the comparison with the specific transactions. This like I said, is working fine for the ASCII byte stream which is getting published but doesn't work for EBCDIC. 

     

    The code snippet I used is not throwing any exception. Not sure why though. What I'm suspecting is the code page I'm using might be wrong because when I googled, what I found is the EBCDIC code page for UK is Cp285. Not sure if this fix the issue.

     

    I will try what you and Kevin has suggested for setting up the response bytes from the list and will let know the outcome of it.

     

    @Kevin- In 9.5.1 and later, the newer IBM MQ Native transport protocol has a built-in option to override the message's CCSID and use a different one for encoding/decoding the body.  You can do this for both receiving requests and sending responses.. can you elaborate more on how can I do it at the VSI response level. What ever the code you have suggest, will that take care of it?

     

    Thanks again for both of you for the insights provided.

     

    Regards,

    Shivakumar.



  • 5.  Re: Send EBCDIC byte stream as Response
    Best Answer

    Posted Dec 06, 2017 06:25 PM

    With IBM MQ, the message payload is *always* a byte stream, but this does not mean you always have to deal with binary payloads in VSE.  The MQ CCSID header is there to allow the automatic translation from bytes to text and back, and in 9.5.1 the override CCSID field is there to allow you use a different CCSID for this conversion.  An MQ message payload is always ultimately sent and received as bytes, but in most cases you shouldn't have to deal with the bytes directly.  Things are designed to handle text <-> bytes conversions as automatically as possible.  If your application uses the CCSID header as it is intended then you don't even have to worry about character sets and byte streams.

     

    > The kind of byte array which is getting published depends on the data type defined in the RIP/ROP files the application uses.

     

    Does this mean that the same service can receive requests formatted either as ASCII or as EBCDIC?  Trying to support both character sets in the same VSE service, without being able to use the CCSID header to tell us which one to use, complicates things quite a bit.

     

    > What I'm suspecting is the code page I'm using might be wrong because when I googled, what I found is the EBCDIC code page for UK is Cp285. Not sure if this fix the issue.

     

    The most common code pages I see for EBCDIC are 37 (Cp037) and 500 (Cp500).  When I say "most common", I mean I cannot recall seeing anything else used until you mentioned 285 (a regional EBCDIC variant probably equivalent to 37/500 for your purposes) and 1047 (which I'm not sure is even EBCDIC). 

     

    > @Kevin- In 9.5.1 and later, the newer IBM MQ Native transport protocol has a built-in option to override the message's CCSID and use a different one for encoding/decoding the body. You can do this for both receiving requests and sending responses.. can you elaborate more on how can I do it at the VSI response level. 

     

    In 9.5.1+ with the IBM MQ Native transport protocol, there is a new meta-data property you can set on a VSI response: 'msg.overrideCharacterSet':

     

    Even without this property, you can manually set the regular 'msg.characterSet' property to 500 and the message will be encoded as EBCDIC automatically.  That works pre-9.5.1 and it works with the old IBM MQ Series transport protocol.  Changing the CCSID of an outgoing response is very easy.  You can use your scripted DPH to just set this meta-data property to whatever you want:

     

    lisa_vse_response.get(0).getMetaData().setParameterValue("msg.characterSet", "500");

     

    It's not as simple on the request side.  You can set the override CCSID in the 'Listen' VSM step:

     

    This will force all requests to be decoded as EBCDIC regardless of the CCSID given in their headers.  However, this is only available in the IBM MQ Native transport protocol, and it doesn't work if a single service can receive requests encoded in *either* ASCII or EBCDIC.



  • 6.  Re: Send EBCDIC byte stream as Response

    Posted Dec 07, 2017 04:54 AM

    Thanks Kevin. Let me try out these options suggested.

     

    Fortunately, one service will have only one specific data type. Either EBCDIC or ASCII.

     

    Regards,

    Shivakumar.



  • 7.  Re: Send EBCDIC byte stream as Response

    Posted Dec 07, 2017 11:54 AM

    Hi Kevin,

     

    Does the JMS implementation support setting the characterset property in the VSI meta data . I tried adding that property and consumed the virtual service from workstation and in the response, the custom properties in the header is still showing the JMS_IBM_Characterset as UTF-8. Not sure if it is the right way to check. 

     

    I tried adding a scriptable DPH after the copy book filter in the response step in vsi and added the below line which you have suggested, but still see  UTF-8  as the value to the JMS_IBM_Characterset in the workstation.

    lisa_vse_response.get(0).getMetaData().setParameterValue("msg.characterSet", "500");

     

    I tried in on VSM using IBM MQ Native and in the response in the workstation, JMS_IBM_Characterset is showing IBM500 when I set the characterset field in the meta data as 500. The response in the workstation test still shows as ASCII which I believe is expected.

     

    I cannot use the IBM MQ native implementation as we are running all our virtual services on single queue and use JMS Consumer in the listen step to make the right virtual service pick up the request.

     

    Not sure , it there is a way to implement this jms consumer functionality in the native MQ implementation.

     

    Could you please suggest how to get around with this.

     

    Regards,

    shivakumar.

     



  • 8.  Re: Send EBCDIC byte stream as Response

    Posted Dec 07, 2017 01:26 PM

    > I cannot use the IBM MQ native implementation as we are running all our virtual services on single queue and use JMS Consumer in the listen step to make the right virtual service pick up the request.

    >Not sure , it there is a way to implement this jms consumer functionality in the native MQ implementation.

     

    Huh, okay.  That's something that probably should have been mentioned earlier. 

     

    How does that work?  Does your application set some custom RFH2 properties and you're using a JMS selector query to separate out requests based on those properties?  Unfortunately there is no way to that in IBM MQ Native.  Even if we wanted to, the IBM MQ Native Java API doesn't appear to support it.

     

    With the JMS mode response you're going to want to set 'msg.props.JMS_IBM_Character_Set' to 'IBM500', and presumably the MQ client jar's JMS -> Native bridge will take care of the encoding.  You can do this directly in the VSI's response meta-data or with a response-side DPH:

     

    lisa_vse_response.get(0).getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "IBM500");

     

    You're out of luck on the request; there's no equivalent to 'Override Character Set' with the JMS VSE listener.  If an incoming request message's 'JMS_IBM_Character_Set' doesn't match the actual encoding of the message body then you'll have to use a script DPH and re-decode it manually.



  • 9.  Re: Send EBCDIC byte stream as Response

    Posted May 14, 2018 11:19 AM

    Hi Kevin,

     

    Reopening the post as I ran into an issue again for the same problem. The solution which you have provide worked fine when we did the PoC but now when we are trying to implement in the actual solution , it is not working. The only change I can see b/w then and now is earlier I used workstation version 9.5 and now 10.2

     

    Like I mentioned I was using JMS Direct connection factory for IBM MQ and the virtual response being published on to the response queue should be of EDCDIC byte stream.

     

    I used the lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "IBM500");

     

    The logging code at the application end has logic to log both ASCII and EBCDIC byte messages which are being subscribed from the response queue.

     

    The log used String temp= new String(bytesReply, "ASCII") for ASCII and String temp= new String(bytesReply, "Cp1047") for EDCDIC and the ASCII part is getting logged correctly and the EDCDIC part is logged as special characters. This makes be believe the message which the virtual service is publishing on the  response queue is not coming as EBCDIC byte array.

     

    Could you please advice how can I handle this.

     

    Regards,

    Shivakumar.



  • 10.  Re: Send EBCDIC byte stream as Response

    Posted May 14, 2018 01:13 PM

    Cp1047 is not EBCDIC.  It's something IBM called "Latin-1 / Open Systems".  It does not appear to be compatible with EBCDIC.

     

    Why don't you try this?

    lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "Cp1047");

    Or maybe this:

    lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "IBM1047");



  • 11.  Re: Send EBCDIC byte stream as Response

    Posted May 15, 2018 11:21 AM

    Hi Kevin,

     

    I tried what you have suggested but not luck. The application is still not able to consume the virtual response and in the logs I see the ASCII log part is logging correctly which means the response we are sending is going in ASCII instead of EBCDIC as per my understanding.

     

    what I additionally tried is below. In the palyback tab of the Scriptable DPH , I added this code.

     

    import com.itko.lisa.test.TestExec;
    rawMessage = lisa_vse_response.getBodyText();
    byte[] b = rawMessage.getBytes("Cp1047");
    testExec.log("ResponseBytes",new String(b));
    lisa_vse_response.setBodyBytes(b);

     

    lisa_vse_response.getMetaData().setParameterValue("msg.type", "javax.jms.BytesMessage");

     

    This is making the app to consume the response and the EDCDIC log part is logging the correct msg. But the only issue i'm seeing now is, the response msg is data is fine but there are additional spaces getting appended in the response. Could you please advice how can I get rid of them.

     

    Or is there anything that I'm missing.

     

    Regards,

    shivakumar.



  • 12.  Re: Send EBCDIC byte stream as Response

    Posted May 15, 2018 11:35 AM

    Did you try this?

    lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "IBM1047");

     

    Maybe just this?

    lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "1047");

     

    I'm just guessing, but it seems like one of those should have the same effect as the script you wrote.

     

    You said this same thing worked during the PoC with "IBM500".  What else is different about your current scenario?  Is it the same version of IBM MQ?  Are you using the same IBM MQ client jars?

     

    Remind me again why JMS is being used instead of IBM MQ Native?

     

    > But the only issue i'm seeing now is, the response msg is data is fine but there are additional spaces getting appended in the response.

     

    Check your VSI's response text for extra spaces at the end.



  • 13.  Re: Send EBCDIC byte stream as Response

    Posted May 16, 2018 10:20 AM

    Hi Kevin,

     

    I tried all the options which you have suggested,but no luck. The only difference between the POC and now is the MQ version and the Workstation version. Earlier MQ is 7.5 and now its 8 and workstation is 9.5 and now its 10.2. The jars I'm using are same.

     

    I could not able to use the IBM MQ native step because we want to use single queues for multiple service calls and in JMS we use the JMS consumer step to query on the transaction name coming in the inbound request for the respective VS to pick. I believe the MQ native step doesn't support this.

     

    I check the VSI response and they are fine. There are no additional spaces.

     

    Also in the code which I have mentioned in the previous post, If I don't use the below line, its not working for EBCDIC messages. For ASCII based messages, the msg.type as Text message is working fine.

    lisa_vse_response.getMetaData().setParameterValue("msg.type", "javax.jms.BytesMessage");

     

    The response which is coming using the above post code, there are additional spaces coming which are exactly the length of msg means if my resp msg is 10 char what I see in the log is 10 + 10 spaces as the response.

     

    I'm not sure where it is going wrong. Is it because of explicitly defining the msg.type as bytes? If I don't give, it fails.

     

    Could you please help me on this.

     

    Regards,

    Shivakumar.

     



  • 14.  Re: Send EBCDIC byte stream as Response

    Posted May 23, 2018 01:32 PM

    I think we're at the limit of the help I can give you over the forums.  The inter-operability between IBM MQ Native Mode and IBM MQ JMS Mode is not seamless, and changes from version to version.  We're at the point where I'd have to see it myself to go any further, and that means filing a support case and helping us replicate the issue ourselves.



  • 15.  Re: Send EBCDIC byte stream as Response

    Posted May 30, 2018 06:46 AM

    Thanks Kevin. I think there were some white spaces in the rr pairs which we extracted from the logs and used to create the virtual services. This seems to be causing the issue of additional spaces. I trimmed the virtual response before publishing the edcdic byte stream on the response queue which fixed the issue. The application is able to consume the message now. 

     

    Thanks for all the inputs!!

     

    Regards,

    shivakumar.



  • 16.  Re: Send EBCDIC byte stream as Response

    Posted Dec 08, 2017 12:39 PM

    Hi Kevin,

     

    Thanks a ton for the input. Adding the "msg.props.JMS_IBM_Character_Set" Property has fixed the issue. The only change I did is instead of get(0), I used the below in the Scriptable DPH in the response step in the VSI. I think the response is not coming as a list object.

     

    lisa_vse_response.getMetaData().setParameterValue("msg.props.JMS_IBM_Character_Set", "IBM500");

     

    How does that work?  Does your application set some custom RFH2 properties and you're using a JMS selector query to separate out requests based on those properties?  Unfortunately there is no way to that in IBM MQ Native.  Even if we wanted to, the IBM MQ Native Java API doesn't appear to support it.

    In the input request which is published, the message header has a property call transacationName which I use in the JMS consumer to route the requests to the right virtual service.

     

    The request side I'm not facing any issue , as the request byte stream is converting into UTF-8 by the copy book DPH encoding. The same is happening on the response I believe, which is the application is not able to consume as it is expecting the response in EBCDIC byte stream. Hence I added one more Scriptable filter after that where I added the above code which has fixed the issue.

     

    I believe, I can change the encoding in the response copybook to IBM500 which might fix the issue. Haven't tried though.

     

    Thanks a lot again !!

     

    Regards,

    Shivakumar.