Automic Workload Automation

We have a couple of apps that use the Automation Engine Java Application Interface to interact with our Automation Engine servers. We are interested in learning how best to manage the connections between the apps and the AE so that these connections are resilient and fault-tolerant. Specifically, we would like to achieve the following:

1. The app can get an up-to-date CP list from the AE server.
2. The app maintains a pool of connections with all CPs on all AE nodes of the AE system.
3. The app periodically checks the health of each connection, and is able to detect changes in the status of individual CP connections.
   
4. Upon loss of connectivity with a CP, the app automatically attempts to reconnect at regular intervals.
5. The app falls back gracefully to other CPs if some CPs are unavailable.
6. Individual AE nodes can be switched to maintenance mode in the app; the app will not attempt to establish connections to CPs on these nodes.

Right now, we use the Connection class to establish connections. We have built some rudimentary resilience into our apps, but have not yet met all of the requirements listed above. Based on the description, we see that the new (in v11) class AEConnectionPool might be useful for implementing the requirements above. None of the code samples I have seen use this class though.

Has anyone else faced similar challenges? If so, how have you addressed them? Does anyone use the AEConnectionPool class? If so, how?

Honestly from the looks of the Javadoc the AEConnectionPool doesn't appear all that useful for your purposes.  If you want to manage multiple Connection objects it may be better to write a custom class to collect and maintain them using threads to check each Connection and their collection of CPs at given intervals.  How long do you plan for your connections to occur?  Is this for some type of custom maintenance application?

Not sure how far along you are with your code--I tossed together a starting concept class for a generic Connection pool handler.  I haven't yet coded the CP checks and other status/recovery options, but I figured I'd share.  If you're interested I could probably structure something more complex.

import java.io.IOException; import java.net.InetSocketAddress; import java.net.Socket; import java.util.ArrayList; import java.util.Collections; import java.util.Iterator; import java.util.List; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; import com.uc4.api.UC4HostName; import com.uc4.api.systemoverview.AgentListItem; import com.uc4.communication.Connection; import com.uc4.communication.TimeoutException; import com.uc4.communication.requests.AgentList; import com.uc4.communication.requests.ClientList; import com.uc4.communication.requests.DisconnectHost; import com.uc4.communication.requests.StartHost; /** * ConnectionPool objects collect and maintain Connection UC4 objects for the purposes of monitoring */ public class ConnectionPool {      private List<ConnectionInfo> connections = Collections.synchronizedList(new ArrayList<ConnectionInfo>());      private ScheduledExecutorService exec;      boolean running = true;      boolean fullReport = false;      private Runnable checkConnections = new Runnable() {          public void run() {               if (running) {                    try {                          connectionCheck();                     } catch (TimeoutException | IOException e) {                          e.printStackTrace();                     }               }          }      };            /**       * Outputs all information or just inactive/missing Agents and CPs       * @param fullReport       */      public void setFullReporting(boolean fullReport) {           this.fullReport = fullReport;      }            /**       * Adds Connection object to the Connection pool using passed parameters       * @param client AE Client (Integer)       * @param level AE level (String)       */      public void addConnectionToPool(int client, String level) {           synchronized (connections) {                Connection newConnection;                try {                     newConnection = new ConnectionManager().authenticate(client, level);                     ConnectionInfo newConnectionInfo = new ConnectionInfo(newConnection,client,level);                     connections.add(newConnectionInfo);                } catch (IOException e) {                     e.printStackTrace();                }           }      }            /**       * Removes Connection object from Connection pool matching given parameters       * @param client AE Client (Integer)       * @param level AE level (String)       */      public void removeConnectionFromPool(int client, String level) {           synchronized (connections) {                for (Iterator<ConnectionInfo> iterator = connections.iterator(); iterator.hasNext();) {                     ConnectionInfo connectionInfo = iterator.next();                     if ((connectionInfo.getClient() == client) && (connectionInfo.getLevel().equals(level))) {                        iterator.remove();                    }                }           }      }            /**       * Starts Executor service for repeated status checks       */      public void healthCheck() {           exec = Executors.newScheduledThreadPool(1);           exec.scheduleAtFixedRate(checkConnections,0,10,TimeUnit.SECONDS);      }            /**       * Checks all ConnectionInfo objects in Connection pool       * @throws IOException       * @throws TimeoutException       */      private void connectionCheck() throws TimeoutException, IOException {           synchronized (connections) {                for (ConnectionInfo connectionInfo : connections) {                                          AgentList getAgents = new AgentList();                     connectionInfo.getConnection().sendRequestAndWait(getAgents);                     if (getAgents.getMessageBox() == null) {                          int totalAgents = 0;                          int agentsDown = 0;                          Iterator<AgentListItem> agentList = getAgents.iterator();                          while (agentList.hasNext()) {                               AgentListItem agent = agentList.next();                               totalAgents++;                               if (!agent.isActive()) {                                    agentsDown++;                               }                               if (fullReport) {                                    System.out.println(connectionInfo.getConnectionInfo() + "\tAgent:" + agent.getName().toString() + "\tActive: " + agent.isActive() + "\tAuthenticated: " + agent.isAuthenticated() + "\tLinked: " + agent.isLinked());                               } else if (!agent.isActive()){                                    System.out.println(connectionInfo.getConnectionInfo() + "\tAgent:" + agent.getName().toString() + "\tActive: " + agent.isActive() + "\tAuthenticated: " + agent.isAuthenticated() + "\tLinked: " + agent.isLinked());                               }                          }                          System.out.println("Total:" + totalAgents + "\tDown:" + agentsDown);                     } else {                          System.out.println("Error getting Agent list for client " + connectionInfo.getClient());                          System.out.println(getAgents.getMessageBox());                     }                                          //Checks CPs on the current Connection                     System.out.println("CPs for " + connectionInfo.getConnectionInfo());                     Iterator<InetSocketAddress> cps = connectionInfo.getConnection().cpIterator();                     while (cps.hasNext()) {                          InetSocketAddress cp = cps.next();                          System.out.println("\t\t" + cp.getAddress() + "\tPort:" + cp.getPort() + "\tActive:" + isPortAvailable(cp.getAddress().toString().split("/")[1],cp.getPort()));                     }                     ClientList checkClients = new ClientList();                     connectionInfo.getConnection().sendRequestAndWait(checkClients);                }           }      }            /**       * Starts a given Agent from specified Client/Level (requires fully qualified domain name)       * @param client Client (int)       * @param level Level (String)       * @param agent Agent fully qualified domain name (String)       * @throws IOException       * @throws TimeoutException       */      public void startAgent(int client, String level, String fqdn) throws TimeoutException, IOException {           synchronized (connections) {                for (Iterator<ConnectionInfo> iterator = connections.iterator(); iterator.hasNext();) {                     ConnectionInfo connectionInfo = iterator.next();                     if ((connectionInfo.getClient() == client) && (connectionInfo.getLevel().equals(level))) {                          UC4HostName host = new UC4HostName(fqdn);                          StartHost startAgent = new StartHost(host);                          connectionInfo.getConnection().sendRequestAndWait(startAgent);                          if (startAgent.getMessageBox() == null) {                               System.out.println(connectionInfo.getConnectionInfo() + " Agent " + host.getName() + " started successfully.");                          } else {                               System.out.println("Error starting Agent " + host.getName() + "!");                               System.out.println(startAgent.getMessageBox());                          }                    }                }           }      }            /**       * Disconnects a given Agent from specified Client/Level (requires fully qualified domain name)       * @param client Client (int)       * @param level Level (String)       * @param agent Agent fully qualified domain name (String)       * @throws IOException       * @throws TimeoutException       */      public void disconnectAgent(int client, String level, String fqdn) throws TimeoutException, IOException {           synchronized (connections) {                for (Iterator<ConnectionInfo> iterator = connections.iterator(); iterator.hasNext();) {                     ConnectionInfo connectionInfo = iterator.next();                     if ((connectionInfo.getClient() == client) && (connectionInfo.getLevel().equals(level))) {                          UC4HostName host = new UC4HostName(fqdn);                          AgentList clientAgents = new AgentList();                          connectionInfo.getConnection().sendRequestAndWait(clientAgents);                          AgentListItem agent = clientAgents.getAgentByName(host);                          DisconnectHost dropAgent = new DisconnectHost(agent);                          connectionInfo.getConnection().sendRequestAndWait(dropAgent);                          if (dropAgent.getMessageBox() == null) {                               System.out.println(connectionInfo.getConnectionInfo() + " Agent " + agent.getName().toString() + " disconnected successfully.");                          } else {                               System.out.println("Error disconnecting Agent " + agent.getName().toString() + "!");                               System.out.println(dropAgent.getMessageBox());                          }                    }                }           }      }            /**       * Kill Executor service, iterate all ConnectionInfo objects, close each connection, and finally kill thread       */      public void closeAllConnections() {           exec.shutdown();           synchronized (connections) {                for (ConnectionInfo connectionInfo : connections) {                     try {                          connectionInfo.getConnection().close();                     } catch (IOException e) {                          e.printStackTrace();                     }                }           }           running = false;      }            /**       * Checks specific host/port socket connection to see if it is available for use       * @param host (String)       * @param port (int)       * @return boolean       */      private boolean isPortAvailable(String host, int port) {             boolean result = false;             try {               (new Socket(host, port)).close();               result = true;             }             catch (IOException e) {             }             return result;      } }

ConnectionInfo class:

import com.uc4.communication.Connection; public class ConnectionInfo {      private Connection connection;      private int client;      private String level = "";            public ConnectionInfo(Connection connection, int client, String level) {           setConnection(connection);           setClient(client);           setLevel(level);      }      public Connection getConnection() {           return connection;      }      public void setConnection(Connection connection) {           this.connection = connection;      }      public int getClient() {           return client;      }      public void setClient(int client) {           this.client = client;      }      public String getLevel() {           return level;      }      public void setLevel(String level) {           this.level = level;      }      public String getConnectionInfo() {           return "Client:" + client + "\tLevel:" + level;      } }

I've updated the above class a bit to include the methods to start/stop Agents(Hosts) and report Agent/CP connectivity per client/level.