Automic Workload Automation

  • 1.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 09, 2017 10:20 AM
    I have made some progress running the AWI in a Cloud Foundry platform-as-a-service (PaaS) environment*  This is so much easier than setting up a dedicated machine or VM, installing & configuring Apache & tomcat, getting certificates, and so on. Once you’ve installed a web app this way, you will not want to do it the hard way ever again. Once the application directory has been prepared and customized, it takes literally 2 minutes to deploy a working AWI server.

    I prepared several different directories — one for each AWI environment I intend to deploy. These directories contain the contents of theawi.war(orecc.war) file.

    • awi-exp
      • config
        • uc4config.xml
        • logback.xml
        • framework-logging.xml
        • felix.properties
        • configuration.properties
        • theme
          • colors.properties
          • logo-exp.png
          • logo-dev.png
          • logo-ite.png
          • logo-prod.png
      • META-INF
      • WEB-INF
      • translations.msl
    • awi-dev
    • awi-ite
    • awi-prod

    The directories for the DEV, ITE, and PROD environments contain the same files. Only the files above in bold differ between the environments. I was able to avoid having to use a different uc4config.xml file for each environment by including all of the connections for all environments in the uc4config.xml file, and specifying the connection for each environment using the automationEngine.index property in configuration.properties. (Thanks to Markus_Holzer_493 for the [DEAD LINK https://community.automic.com/discussion/comment/24717/#Comment_24717]idea!)

    Theuc4config.xmlfile looks like this:

    <?xml version="1.0" encoding="ISO-8859-15"?>
    <configuration>
           <!-- 0: off, 1: send, 2: receive, >=3: all -->
           <trace count="10" xml="0"></trace>
           <connections>
                  <connection name="UC4_EXP" system="UC4_EXP">
                         <cp ip="uc4a-exp.mycompany.com" port="33330"/>
                         <cp ip="uc4b-exp.mycompany.com" port="33331"/>
                         <cp ip="uc4a-exp.mycompany.com" port="33332"/>
                         <cp ip="uc4b-exp.mycompany.com" port="33333"/>
                         <cp ip="uc4a-exp.mycompany.com" port="33334"/>
                  </connection>
                  <connection name="UC4_DEV" system="UC4_DEV">
                         <cp ip="uc4a-dev.mycompany.com" port="33330"/>
                         <cp ip="uc4b-dev.mycompany.com" port="33331"/>
                         <cp ip="uc4a-dev.mycompany.com" port="33332"/>
                         <cp ip="uc4b-dev.mycompany.com" port="33333"/>
                         <cp ip="uc4a-dev.mycompany.com" port="33334"/>
                  </connection>
                  <connection name="UC4_ITE" system="UC4_ITE">
                         <cp ip="uc4a-ite.mycompany.com" port="33330"/>
                         <cp ip="uc4b-ite.mycompany.com" port="33331"/>
                         <cp ip="uc4a-ite.mycompany.com" port="33332"/>
                         <cp ip="uc4b-ite.mycompany.com" port="33333"/>
                         <cp ip="uc4a-ite.mycompany.com" port="33334"/>
                  </connection>
                  <connection name="UC4_PROD" system="UC4_PROD">
                         <cp ip="uc4a.mycompany.com" port="33330"/>
                         <cp ip="uc4b.mycompany.com" port="33331"/>
                         <cp ip="uc4a.mycompany.com" port="33332"/>
                         <cp ip="uc4b.mycompany.com" port="33333"/>
                         <cp ip="uc4a.mycompany.com" port="33334"/>
                  </connection>
           </connections>
    </configuration>
     The configuration.properties file for EXP looks like this:
    # This is the configuration file for the AWI. Here you can customize the configuration according to the installation guide if needed.

    # Home Dashboards
    defaultHomeDashboard=MYCOMPANY_DASHBOARD
    customHomeDashboardsFolder=SYSTEM/AWI/DASHBOARDS/HOME

    # Cookies
    autofill.cookie.allowed=true

    # Login settings
    sso.enabled=true
    parameter_login.enabled=true

    # Automation Engine systems: 0=UC4_EXP, 1=UC4_DEV, 2=UC4_ITE, 3=UC4_PROD
    automationEngine.index=0

    # Network settings
    push=true

    # AWI Online Help
    #helpUrl.english=https://docs.automic.com/documentation/WEBHELP/English/all/components/DOCU/latest/AWA Guides/help.htm

    # AWI Object Validation
    #promptset.validation.mandatory=true

    To deploy to DEV, I simply changeautomationEngine.indexto 1; for ITE, 2; for PROD, 3.

    Tocustomize the look and feelof each environment, I also customized the main highlight color and the logo that appears in the header. Here is the colors.properties file for EXP:

    maincolor = #FFFF00;
    logo.filename = logo-exp.png

    There is acustom logo filefor each environment, color-coded to match the main highlight color.

    I push the app to the cloud foundry PaaS platform using thecf push command, with the standardjava_buildpack. It takes less than a minute to install, stage, and start up. So far it has worked just fine, with the exception of aproblem with single sign-on.

    * Cloud foundry providers include Pivotal, IBM, Cisco, Atos, SAP, and Swisscom.


  • 2.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 09, 2017 10:33 AM
    One snag that I have run into is that single sign-on does not work when the app is installed on the cloud platform. In fact, if SSO is enabled in the configuration.properties file, it prevents even normal (non-SSO) logins from working.

    Automic Support and Development investigated this problem and discovered that the likely reason for the problem was that the combined length of all of the HTTP cookies was exceeding tomcat’s maxHttpHeaderSize setting. The number and length of the HTTP headers can vary from user to user and context to context, so it was a tricky problem to debug.
    2017-02-09 18:37:18 [RTR/0] OUT awi-exp2.appcloud.mycompany.com - [09/02/2017:17:37:18.698 +0000] "GET /ssourl?v-uiId=0 HTTP/1.1" 401 0 56 "http://awi-exp2.appcloud.mycompany.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0" 10.56.123.98:52812 10.37.0.71:64558 x_forwarded_for:"10.56.123.98" x_forwarded_proto:"http" vcap_request_id:70bef69b-72ca-470c-7c36-65379183fea5 response_time:0.005794124 app_id:5a0bf242-2714-4167-ba13-90b0521216ad app_index:0
    2017-02-09 18:37:18 [APP/PROC/WEB/0] OUT [CONTAINER] org.apache.coyote.http11.Http11NioProcessor INFO Error parsing HTTP request header
    2017-02-09 18:37:18 [APP/PROC/WEB/0] OUT Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
    2017-02-09 18:37:18 [APP/PROC/WEB/0] OUT java.lang.IllegalArgumentException: Request header is too large
    I am currently trying to figure out the best way to override this setting when pushing, staging, or starting the app.


  • 3.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 09, 2017 12:26 PM

    Ideally, I would prefer to completely eliminate the file-level differences between the environments, deploy the same package to all environments, and specify any environment-specific configuration information at run time. Many web apps offer the ability to do just this, by allowing one to override any configuration setting via a corresponding environment variable.

    Automic Support confirmed that there is currently (in AWI v12) no way to override properties using environment variables, but I opened an enhancement request (PMPER-2002) for this.

    If all of these aspects of the AWI were customizable, and could be specified at run time using environment variables, then this is the sort of setup that would be possible:

                                         

    File

     
     

    Property

     
     

    Environment   variable

     
     

    Value in   EXP

     
     

    Value in DEV

     
     

    configuration.properties

     
     

    automationEngine.index

     
     

    AUTOMIC_AWI_AE_INDEX

     
     

    0

     
     

    1

     
     

    colors.properties

     
     

    maincolor

     
     

    AUTOMIC_AWI_AE_MAINCOLOR

     

    #FFFF00

     
     

    #2A97FF

     
     

    colors.properties

     
     

    logo.filename

     
     

    AUTOMIC_AWI_AE_LOGOFILE

     
     

    logo-exp.png

     
     

    logo-dev.png

    colors.properties
    favicon.filename*
    AUTOMIC_AWI_AE_FAVICONFILEfavicon-exp.png
    favicon-dev.png
    colors.properties
    title*
    AUTOMIC_AWI_AE_TITLEUC4_EXP
    UC4_DEV

    Just imagine:
    • A single package that contains all of the files needed for every AE environment;
    • A separate key/value-pair configuration service that sets the environment variables for each environment;
    • An AWI that is completely customized for each environment, and can be deployed in less than a minute.

    The HTML title and the favorite icon cannot actually be customized via properties today in v12. I requested the ability to customize these two things back in October 2016, in PMPER-1734. I include them here to show how it might work in the future.


  • 4.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 09, 2017 01:12 PM
    I received an answer from Automic Support. Today with v12, there are no such environment variables that can be used to specify/override AWI configuration details at run time. I opened PMPER-2002 for this.


  • 5.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 10, 2017 10:56 AM
    I fixed the problem with single sign-on. On a stand-alone server or VM, increasing the value of the maxHttpHeaderSize attribute in tomcat’s server.xml file would be simple. But since I’m running the app in a PasS app cloud server, most of these configuration settings are centrally controlled. Passing environment variables to the app is straightforward, so at first I tried to find a way to set this tomcat attribute via an environment variable. Not finding a way to do this, I next tried to find a Java property that would accomplish the same thing. Again, I found nothing.

    Eventually I found a way to convince tomcat use use an alternative server.xml file. This involved a few steps:
    1. I created an alternate version of the server.xml file,myserver.xml, containing the modified Connection element:
      <Connector port='${http.port}' bindOnInit="false" connectionTimeout="20000" maxHttpHeaderSize="32768"/>
    2. I placed this file in.java-buildpack/tomcat/confin the directory containing the AWI files that I would push to the app cloud server.
    3. I added a modifiedcommandline for the app in its manifest.yaml file:
      ---
      applications:
      - name: awi-exp2
        path: awi-exp2
        buildpack: java_buildpack
        command: CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-2.0.2_RELEASE -memorySizes=metaspace:64m..,stack:228k.. -memoryWeights=heap:65,metaspace:10,native:15,stack:10 -memoryInitials=heap:100%,metaspace:100% -stackThreads=300 -totMemory=$MEMORY_LIMIT) &&  JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR -Daccess.logging.enabled=false -Dhttp.port=$PORT" exec $PWD/.java-buildpack/tomcat/bin/catalina.sh run -config conf/myserver.xml
    4. I then pushed the app using this manifest.
    This worked fine at first. The AWI started up, and instead of hanging on authentication with the KDC, it finally presented me with the Use Kerberos login check-box. I enabled it and clicked Login. Unfortunately, I wasn’t out of the woods yet. As soon as the log-in completed, the app froze. Luckily, this problem was easy to diagnose. The application log revealed the reason:
    017-02-10 16:31:43 [APP/PROC/WEB/0] OUT Exit status 137
    2017-02-10 16:31:43 [CELL/0] OUT Exit status 0
    2017-02-10 16:31:43 [CELL/0] OUT Destroying container
    2017-02-10 16:31:43 [API/2] OUT Process has crashed with type: "web"
    2017-02-10 16:31:43 [API/2] OUT App instance exited with guid 12a39bb7-e3ff-4472-957a-ca0768af4c08 payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* 2 error(s) occurred:\n\n* Exited with status 137 (out of memory)\n* cancelled\n* cancelled", "crash_count"=>1, "crash_timestamp"=>1486740703221382261, "version"=>"3e049b26-4040-486b-9f21-3bd532f3e26e"}
    I increased the app’s memory from 683 to 2048 MB, started it up again, and it worked like a charm. I will now put it through its paces.


  • 6.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 13, 2017 09:57 AM
    Just a quick note: from the AWI system requirements, it is clear that the app needs much more memory and disk space than I was giving it. Fortunately, these are easy enough to change. I will do this before I put the AWI into production.


  • 7.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Feb 27, 2017 09:09 AM
    I recently stumbled across a new documentation page, Customizing the AWI Configuration Path. This page reveals the existence of two Java properties that can be used to change at start-up time the behavior the AWI. They may have existed in previous versions, but v12 is the first version in which they are documented. One of these, com.uc4.ecc.config.dir, can be used to specify the path to the AWI config files at run time.

    This means that it is now possible to prepare a single AWI package that includes all of the configuration files necessary for multiple different environments. For example, here is a directory structure containing a separate config subdirectory for each of four environments (EXP, DEV, ITE, and PROD).
    • awi
      • config-exp
        • uc4config.xml
        • logback.xml
        • framework-logging.xml
        • felix.properties
        • configuration.properties
        • theme
          • colors.properties
          • logo.png
      • config-dev
        • uc4config.xml
        • ...
      • config-ite
        • uc4config.xml
        • ...
      • config-prod
        • uc4config.xml
        • ...
      • META-INF
      • WEB-INF
      • translations.msl
    Each config directory can contain its own uc4config.xml, configuration.properties, colors.properties, and other files like icons. Because one can specify the location of the whole config directory for each instance, it is not necessary to specify the locations of individual config files.

    The AWI app must be started with different options in each environment. In our Cloud Foundry platform, the paths are under $PWD/app/.java-buildpack/tomcat/webapps/ROOT.

                               
    EnvironmentJava/tomcat option
    EXP   -Dcom.uc4.ecc.config.dir=$PWD/app/.java-buildpack/tomcat/webapps/ROOT/config-exp  
    DEV     -Dcom.uc4.ecc.config.dir=$PWD/app/.java-buildpack/tomcat/webapps/ROOT/config-dev
     
    ITE   -Dcom.uc4.ecc.config.dir=$PWD/app/.java-buildpack/tomcat/webapps/ROOT/config-ite
     
    PROD-Dcom.uc4.ecc.config.dir=$PWD/app/.java-buildpack/tomcat/webapps/ROOT/config-prod

    (The option should be added to JAVA_OPTS or CATALINA_OPTS.)

    Thanks for Automic for adding/documenting this very useful property.


  • 8.  Running the AWI in Cloud Foundry platform-as-a-service

    Posted Mar 07, 2017 10:18 AM
    Today I set things up so that AWI logs from the cloud platform are accessible via the web browser. I used the ELK stack (Elastisearch, Logstash & Kibana).

    ELK provides a convenient way to aggregate logs, and quickly find the information in them you seek.
    q9xabbg68dxs.pnghttps://us.v-cdn.net/5019921/uploads/editor/s4/q9xabbg68dxs.png" width="1091">
    It also has advanced features like saved filters, visualizations, and custom dashboards.

    Note: I modified the logging configuration files so that the logs are written to a directory monitored by the loggregator. In the logback.xml file, I changed the line:
    <file>${log_path:-}${HOSTNAME}_${app_name:-ECC}_LOG.00.txt</file>
    to:
    <file>${HOME}/../logs/${HOSTNAME}_${app_name:-ECC}_LOG.00.txt</file>
    I made a similar change to the framework-logging.xml file. In this context, $HOME is the root level of the cloud app, and the logs subdirectory is the standard location for application logs.