CA Service Management

  • 1.  Maileater performance - taking hours to days to process

    Posted Sep 06, 2016 04:52 PM

    SDM 14.1.02

     

    We have three mailboxes.  Two have the same rules except they are set to create tickets in different Categories and Groups.  The third only allows tickets created using one specific subject line.  All have been working until I made this last change to the first two.

     

    I created a mailbox rule as a 'catch-all' to allow any subject line to create a Request.  This is the last rule in the list below:

     

    SequenceFilterFilter StringActionAction ObjectReply
    10Subject contains^UndeliverableIgnore email  
    20Subject contains^Out of OfficeIgnore email  
    100Subject contains^new incident[ \t\r\n]Create/Update ObjectIncident 
    200Subject contains^new request[ \t\r\n]Create/Update ObjectRequest 
    1000Subject contains^Re: Request {{object_id}}Update ObjectRequest 
    2000Subject contains^Re: Incident {{object_id}}Update ObjectIncident 
    3000Subject contains^Re: Change order {{object_id}}Update ObjectChange 
    4000Subject contains.[\n]Create/Update ObjectRequest 

    In DEV this had no issues but we couldn't put it under load.  Now in PROD, we are seeing severe delays delay between when a message is apparently picked up from the mailbox and when the action is actually taken. Early last week it was a few minutes, then hours, and now days.  There are no errors in stdlog and none of the messages are unknown.  The messages are processed correctly once they are actually processed. 

     

    The mailbox is Office 365 and there do not appear to be any connection issues and the messages are picked up within the polling interval - 60 seconds.

     

    I have an open issue with Support.  We have increased logging and see the message is being processed.  It is correctly parsing to the last rule but the log indicates it fails because it is trying to UPDATE rather than CREATE. It also appears that the message is then discarded.  However, it eventually is 're-processed' and the ticket is created.  I know the email is not being resent from the client side because I am sending test tickets with the time stamp in the subject.

     

    We reviewed the mailbox rule and it is correct as to basic fields - Ignore Case, <blank> on artifact, etc..  We have changed the rule to Update, saved, and then back to Create/Update just to see if there was something not being saved as this was the last rule created.  We had two previous rules (100 and 200 above) to look for specific subjects and then either create an Incident or Request, these were inactivated in favor the catch-all which only opens a Request. 

     

    The other rules are processing correctly, that is, a reply will update a ticket but even these rules are delayed by the same amount.  So I am not certain it is the rule itself that is affecting the performance but noted it was the last system change.

     

    The customer has been integrating new divisions from a merger/acquisition.  However, they do not see any changes in these mailboxes. 

     

    Any thoughts?

     

    Thanks in advance,

     

    J.W.



  • 2.  Re: Maileater performance - taking hours to days to process

    Posted Sep 07, 2016 11:20 AM

    UPDATE:

    The extended logging for a CREATE/UPDATE rule will always just show the action as UPDATE.  This is not an error.

     

    J.W.



  • 3.  Re: Maileater performance - taking hours to days to process

    Posted Sep 07, 2016 06:34 PM

    UPDATE:

    We restarted pdm_maileater_nxd with no change. 

    Restarted pdm_text_nxd and the inboxes were not being cleared so we restarted pdm_maileater_nxd again and the inboxes were cleared and new tickets were being created / updated per the rules.

     

    We assume all messages already cleared from the inboxes were lost as there is no indication of a massive backlog of tickets being created.

     

    So, no root cause yet and this procedure is a very limited workaround as messages are lost between maileater and text_api.



  • 4.  Re: Maileater performance - taking hours to days to process

    Posted Sep 09, 2016 01:49 PM

    UPDATE:

    We previously modified the text_api.pl to allow replies to generate the CUSTOM_LOG per:

    How to prevent an email notifications for a "Created Via Email" Activity. 

    This has been working for months without issue.  It's not something we can revert as the Update Status event depends on it.

     

    J.W.



  • 5.  Re: Maileater performance - taking hours to days to process

     
    Posted Sep 21, 2016 03:49 PM

    Hi J_W - Well you've had a lot of your own updates on this one but we're you ever able to actually find a solution? Thanks! Chris



  • 6.  Re: Maileater performance - taking hours to days to process

    Posted Sep 21, 2016 07:22 PM

    Chris,

    No solution  We are still working with support and SE with a test patch related to other text_api.pl issues.  However, since we don't know the root cause and can't reproduce it now that the services have been restarted, then we are patching in the dark. 

     

    Since no one has commented on this in two weeks then I'm pretty confident it is something in our environment but really the only thing is the modification of the text_api.pl for custom log for incoming email.

     

    J.W.



  • 7.  Re: Maileater performance - taking hours to days to process

    Broadcom Employee
    Posted Sep 22, 2016 03:33 AM

    HI J.W.

    Could you please try to create an input file with the maileater Attributes

    EX :  filename : inputfile.txt

     %PRIORITY=3
     %DESCRIPTION=Testing done on Priority 3
     %SUMMARY=Testing whether it works or not
     %LOG=Lets see if it works or not

     

    From the SDM command prompt run this command to check if pdm_text_cmd command process the request

    pdm_text_cmd  -u ServicePlus  -t Request  -f inputfile.txt

    # As you confirmed that there is no latency in mail reaching to maileater mailbox. And mails are been cleared.

    # Do you find the same behavior for all the mail box.

    # Do you have multiple mailservers, if so then is it possible to move one of the non-working mail account to another mail server. 

    # Also confirm the version of SDM , and details of patch you have applied.

     

    ~Vinod.



  • 8.  Re: Maileater performance - taking hours to days to process
    Best Answer

    Posted Sep 22, 2016 12:24 PM

    Thanks, Vinod.

     

    # Do you find the same behavior for all the mail box.

    A:  Yes

    # Do you have multiple mailservers, if so then is it possible to move one of the non-working mail account to another mail server. 

    A:  No

    # Also confirm the version of SDM , and details of patch you have applied.

    A:  14.1.02 applied with the .02 installer with one test patch for config_ui related to attachments paths reverted to OOTB when changed to UNC.

     

    It will be several days before I can test this.  I also have a new test patch from CA which will require a restart of services, so I may wait and test your ideas before and after.

     

    Regards,

     

    J.W.