Automic Workload Automation

First off, let me preface this by saying we're using AEv9 here...we're running way behind in our efforts to get upgraded to v12...so Automic Support hasn't been able to help me as v9 is no longer supported.  So, I'm reaching out to the community to see if this rings bells for anyone.

I'm doing a file transfer to transfer multiple files from Agent A1 to B1, where the filename filter contains multiple wildcards.

I've got logic in place to dynamically replace the wildcard characters in the source file specification with the &1, &2, etc placeholders in the destination file specification, like this:

File Name (S): \\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\*P1406V-FWKS01*

File Name (D): \\10.16.211.25\infraprod_moveit_int\data\pstln\pstln0002\outgoing\&1P1406V-FWKS01&2

When I go to run this file transfer, it correctly identifies the four files I want to transfer, but then it fails with some weird errors:

2018-11-01 21:38:08 - U2000063 Connection to Agent 'NXP01APPMVCV001' with connection data '10.16.211.12:2300' not possible. Error code '10060', error description: 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.'.
2018-11-01 21:38:08 - U0011124 Selection started with filter '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\*P1406V-FWKS01*' ...
2018-11-01 21:38:08 - U0011125 '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_accounts][P1406v-FWKS01].pgp'
2018-11-01 21:38:08 - U0011125 '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_cards][P1406v-FWKS01].pgp'
2018-11-01 21:38:08 - U0011125 '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_card_accounts][P1406v-FWKS01].pgp'
2018-11-01 21:38:08 - U0011125 '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_customers][P1406v-FWKS01].pgp'
2018-11-01 21:38:08 - U0011126 Files selected: '4'.
2018-11-01 21:38:08 - U0011134 ERROR '0' bytes, incorrect transfer of '0' records for file '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_accounts][P1406v-FWKS01].pgp'->'<name not yet determined>'. Duration '00:00:00'.
2018-11-01 21:38:08 - U0063009 FileTransfer '199421830' not found.
2018-11-01 21:38:08 - U0011134 ERROR '0' bytes, incorrect transfer of '0' records for file '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_cards][P1406v-FWKS01].pgp'->'<name not yet determined>'. Duration '00:00:00'.
2018-11-01 21:38:08 - U0063009 FileTransfer '199421830' not found.
2018-11-01 21:38:08 - U0011134 ERROR '0' bytes, incorrect transfer of '0' records for file '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_card_accounts][P1406v-FWKS01].pgp'->'<name not yet determined>'. Duration '00:00:00'.
2018-11-01 21:38:08 - U0063009 FileTransfer '199421830' not found.
2018-11-01 21:38:08 - U0011134 ERROR '0' bytes, incorrect transfer of '0' records for file '\\p2001-filr02a\data\p1406\p1406v-fwks01\batch\output\auditdatabulkextract\FL\[session-A][issuernr_3][ct_log_pc_customers][P1406v-FWKS01].pgp'->'<name not yet determined>'. Duration '00:00:00'.
2018-11-01 21:38:08 - U0063009 FileTransfer '199421830' not found.
2018-11-01 21:38:08 - U0011409 FT '199421830': FileTransfer ended abnormally.

Note: I'm aware of that 'Connection to agent [...] not possible' error, but that's normal for this particular agent due to our firewall.  The file transfer is able to make the transfer by instead doing a pull from that agent (B) instead of a push from the source agent (A1).

Here's what's killing me:

1. This same file transfer job is being used in an otherwise identical workflow to transfer files from a different source server to the same destination server (e.g., Agent A2 to B1), and it consistently works fine.  (The same is true with my UAT version of this workflow, so Agent A3 to B1 also works just fine.)
2. I have another workflow using this same file transfer job to transfer other files from Agent A1 to B1, and it also works fine.  The only difference in this other workflow with respect to this file transfer job is that the Source filename filter has only one wildcard in it, not two.
3. When I restart the file transfer and specify to use a different source filename filter -- e.g., *.pgp -- it works just fine.  (It still fails if I try to rerun it using two wildcards in the source filename filter, though.)
4. I've whipped up a test file transfer job to transfer test files from Agent A1 to B1 using multiple wildcards in the source filename specification, and that's worked fine as well.

Based on #1 and #2, I don't think the file transfer job itself is the problem, and based on #2 and #4, I don't think the problems are the particular agents, either.  The only conclusion I can draw here is that there's something weird that's going on when I'm trying to transfer multiple files using multiple wildcards in the specification...but #4 seems to refute that theory, too...

Has anyone else run into anything like this in the past?

Can anyone shed any light on what it means when UC4 is saying the destination filename is "<name not yet determined>"?  Or perhaps the "FileTransfer 'xxxxxxxx' not found." error message?  Does this provide any insight as to where the problem is arising?

Thanks in advance for any help!

EDIT:

I've removed my post.  After re-reading the original post, I now realize I don't have any insights of value to offer.

Hi.

I've seen "FileTransfer not found" at various points, but not due to wildcard selections, but more in case of file transfers that work most of the time, then mysteriously fail once. For almost all cases, the answer (from support) has been to try version 12, which we have not done on the affected servers (yet).

I just notice two things here, and in blatant disregard for your points 1 through 4, and thus probably not helping at all , but then some of your points 1-4 refute others, so here it goes anyway:

The file names that the wildcard seems to resolve into are quite long. This might be a problem if your platform is Windows, different API have problems with different lengths of path names. I once had a case where something worked on one agent, but not on the other, simply because one agent name was shorter, thus the full string not reaching the limit, while the other agent name was long enough to burst the overall limit. Especially on Windows (engine or agent), have you tried the same scenario with shorter file names?

Second, [angular brackets] are a POSIX wildcard. I can imagine that the agent or engine might have problems with it. Have you tried filenames without these characters?

Lastly, you could enable a FT or File Transfer trace flag in the agent settings in UC4, maybe set it to 2 or 3 initially and 9 if you didn't get anything useful with the smaller settings. The trace file might hold more clues. If your agent is on Linux or a Unix, you could also use strace (Linux) or truss (Solaris) or any similar tool and see if that gives any pointers to the root cause. There will be a lot of output woth trace files and strace though, so be warned

strace -f -p <pid_of_agent>

If you're on Windows, there's no real substitute. There's something by Mark Russinovich (sp?), I think it's Process Explorer, that can analyze some syscalls, but it only goes so far.

Hth,

Thanks, Carsten.

To address your questions:

• I'm dealing with all Windows agents here.  (Both source agents are running version 9.00A529-2D1, and the destination agent is running 9.00A431-4B1.  All of them are showing [Windows] SW version of 6.2.)  It's possible the destination agent version is an issue, except for the fact that the file transfers are working fine from the other source agents.
• The agent names are the same length; only the leading character is different.
• The path/filename lengths are the same from both agents.  (Technically, the souce path is the same -- the files are being copied from a shared location accessible from both of these agents.)
• I understand about the brackets in the filenames, but that's the way the extract utility creates and names them.  Again, I'm able to transfer them successfully if use other filename patterns, so I don't believe they're getting confused as wildcard characters.  I did run some manual tests without those characters (this was my fourth point in the original email) and they ran fine.  

So my takeaways for now:

1. See if the destination agent version is the issue
2. Do more testing between the two agents using different filenames
3. See what info I can glean from a trace.