Layer7 API Management

  • 1.  GMU templating best practices

    Posted May 17, 2017 06:17 AM

    Please assume for a postulate that I am familiar with all current GMU docs online.

     

    I am wondering how people are managing an automated devops workflow properly with GMU ?

     

    The issues I see are many:

     

    - the template is tied to the specific export - it is not generic or reusable. Some quite advanced merging would be required to make it so.

     

    - though CWPs appear at the top and will be where some parameters which differ per environment go, many changes are not CWPs. These include backend trusted certs, IDP identities, LDAP configurations, RBAC roles and users, etc,etc. 

     

    Take for example, an HTTPS backend server you are using in an API:

     

    In your dev environment this will have a cert in your certificate store for SSL outgoing. In order to promote this API to your test environment, you export the API with GMU. If you are lucky, GMU picks up that it needs the trusted cert and exports that too.

     

    You template that. In order to create a TEST template to configure the API to work properly once GMU migrated into TEST, you will need to modify not only your CWP that points to the backend (since it's of course now a different machine in your test environment), but also add the relevant public cert. This is non trivial - you will need to find the cert in the template, and then someone get hold of the cert from your test environment and paste it in (perhaps you had to grab this via the test PM). And repeat for every similar backend. For LDAP settings, users in IDP, etc it's going to be similarly quite a pain and very very manual.

     

    I imagine what you could do, is pull out JUST those items from the template as the TRUE template for your test environment - because this is the stuff you are going to need for the next release...BUT:

     

    On your next release, you'll have to do this all over again - since the template will have all these scattered over the dev output again - so you'll have to say, have some sort of script that goes through replacing all those name/value pairs with the ones in your TRUE dev template. It's quite crufty. 

     

    In order to have a fully automated DEVOPS process I should be able to:

     

    - have a process I can run which completely sets up a new environment based on a common set of configuration (i.e. build) with whatever environment specific data is required.

    - have no need for any manual stages. no need to create any artefacts manually via policy manager, etc

    The above is a pretty basic definition of what automated deployment config management should be.

     

    It seems to me we are still a long way from that with the GW, without a significant amount of effort - and I'm wondering how many customer have actually carried out that work so far?

     

    Frankly, the only way I can really see this as working in a true production fashion is with manual configuration happening on PM on the target environment, and that defeats much of the benefit of devops automation.

     

    I'd like to see a non trivial worked example - end to end - which includes the resources above - which allows REPEATED delivery of builds from your dev environment to one or more higher environments completely handoff/automatically via scripts.



  • 2.  Re: GMU templating best practices

    Posted May 17, 2017 04:45 PM

    Hello,

     

    I think I understand where you are coming from...  Using automation to manage environment differences, dependencies, and deployment with GMU isn't as easy as it could be, but it can be done.  Here is my setup:

     

    • I tend to package stuff up into "folders" as much as possible.  I use folders as my "deployable" unit because it gives me a good hook to export from.  Downside is that if I want nested folder structures, I need to pre-create them before I deploy.  My folder structure is pretty stable once created though.
    • I use cluster-wide properties as much as possible.  I tried using the auto-assigned properties generated through templating, but it isn't stable.  Cluster-wide properties are stable whereas template generated property names are positional and subject to change as policy assertions are added/removed.
    • I create property files for each target environment.  I have a DEV-policy.properties, QA-policy.properties, etc...  These contain ONLY the property values that I wish to override.  For example, for a given policy, my DEV-policy.properties file would contain a property such as - template.cluster_property.company.some_api.routing.url.Value=https://foo.bar.com
    • I manage the exported policy bundle mapping actions and set ALL objects (certificates, policy, folders, cluster properties, etc...) to "Ignore" immediately after I export a folder's contents.
    • I re-enable ONLY the ones that I want to promote my hand.  I either set the object to "NewOrUpdate" or "NewOrExisting"
    • I've created a migrator account on the Gateway and exported the user's p12 keystore so that I can establish trust between my build automation server and the target gateway.

     

    That's my general setup, but I needed to invest a little time in building some scripting around GMU to smooth out some edges.  Here is what I ended up doing:

     

    For Migrate Out Activity:

    • I "wrapped" the GMU client into a Gradle build script.  This enables me to pass dynamic values to GMU so I can drive behavior.  For example, I need to declare the target gateway, the environment property config, location of the migrator's keystore, etc...  
    •  I use my script to tell GMU which "folder" to migrate out.  Once exported, my script immediately produces a template properties file.
    • I then set the properties appropriate for my target environments in their respective property files.
    • I then manage mappings to ignore all objects.
    • I then manually set the action on the objects that I want to move (could automate this, but it requires some work... maybe a UI).
    • Test...

     

    For Migrate In Activity:

    • I use my GMU wrapper to perform the migrateIn action telling it key values: gateway, environment property, target folder, etc...
    • My script "merges" properties from the generated template with my environment property file.  This is key to me as I want to use the property values for my target environment.
    • My migrateIn produces an output file so that I can see what was done.

     

    My source folder structure for a given policy:

     

    /myPolicy

    /myPolicy/build.gradle

    /myPolicy/bundle.xml

    /myPolicy/mappings.xml

    /myPolicy/src/main/conf/{environment}.properties

     

    I ended up identifying the GMU jars and pushed them to my private Maven repository so that I could relieve the client from having to know anything about GMU.  My script sets them as dependencies and pulls them down locally when needed.

     

    I also ended up hosting my actual build script on the Gateway so that my build server (or local) can just point to it as opposed to stashing copies of it all over the place for every artifact that I want to export.

     

    I use Team City to drive my policy migration automation.  I set all of my environment properties on the build server for each target environment (DEV, QA, PROD, etc..).  I also set my p12 key on the build server.  This could be done for any build server product.

     

    Hopefully, that gives you a sense of how my setup works.  If you are interested in the actual script, I may be able to push it up to a Github repository so you can see what its doing.  It isn't really anything fancy though...  It simply executes commands on the GMU client and does some file munging to handle properties.  Here are the Gradle tasks that I expose:

     

    Other tasks
    -----------
    browse - Displays the gateway artifacts for the given environment.
    buildTemplate - Generates a default template for the given bundle.xml file
    cleanup - Cleans up
    list - Displays the gateway artifacts, of a given type, for the given environment.
    manageMappings - Takes the bundle.xml file and creates a default mappings.xml file with 'reasonable' defaults for dependencies.
    migrateIn - Migrates an artifact into a destination folder on the target gateway.
    migrateOut - Migrates an artifact (AND ALL OF ITS DEPENDENCIES) out of the gateway and produces a bundle.xml file.
    wrapper

     

    I'm sure there are ways to improve what I've done... But it seems to work reasonably well.  The only part I hate is the need to manage mappings by hand.  If there were a UI of some sort that let me "select" the right action to perform on a given object, I'd be more comfortable.  I'm opened to any suggestions.

     

    Thanks,

     

    Alejandro



  • 3.  Re: GMU templating best practices

    Posted May 24, 2017 08:45 AM

    thanks for the reply Alejandro - I've been busy and not had a chance to give your reply the time it deserves. And still can't - but there's some good ideas there. When I get time I'll look into it more. There's still lots of manual stages unfortunately, but I think they're little that can be done about those just now I suppose. I like your property merging ideas.

     

    I still think getting at those properties is painful sometimes - ok for CWPs, but some stuff is a pain - i.e. client certs, private keys, etc. Once it's in your property file great, but getting it there without lots of manual hacking seems almost impossible.



  • 4.  Re: GMU templating best practices

    Posted Oct 23, 2017 06:48 AM

    Hi Both, Im interested in understanding this a bit more as Im currently working on a CI design for the office. I'm interested in your comment about "I create property files for each target environment.  I have a DEV-policy.properties, QA-policy.properties, etc...  These contain ONLY the property values that I wish to override.  For example, for a given policy, my DEV-policy.properties file would contain a property such as - template.cluster_property.company.some_api.routing.url.Value=https://foo.bar.com"

     

    As far as I can test, I cant see how you can create a per environment template properties file. As you have to run the template/detemplate script on every export , as far as i can understand, at least I think that's what Stuart is saying (Hi Stuart, I should have asked you this on our course).

     

    Like Acalbazana, I'm using root folders to manage individually deployable project units, and making an assumption that on environment promotion, I will move everything in that folder and using cluster wide properties to control environmental properties, so clearly want to protect these per environment.

     

    Therefore, my current thinking is (using Jenkins for the choreography) when running a migrateIn using a template'd properties file per environment, however like Stuart I'm under the impression that you can't create a generic properties file to run all migrateIn's per environment.

     

    Could you explain in some more detail how you manage to create this default file for all migrations? The only other way I can think of it to run a migrateout, manageMapping to set CLUSTER_PROPERTY's and possibly some other elements (such as JDBC_CONNECTION, SSG_* plus lots and lots more!) to NeworExisting for new additons then run a MigrateIn.

     

    many thanks in advance



  • 5.  Re: GMU templating best practices

    Posted Nov 02, 2017 09:55 AM

    Hello,

     

    I'm using the GMU command line utility to do this.  So, it may be a bit different if your using the restman API.  So far, I haven't found a way to "templatize" the bundle that you export using the API.  That would be nice....

     

    Using the GMU, I templatize the bundle:

    template --bundle bundle.xml --template template.properties

     

    On the import, I have a bit of script that merges template properties for my target environment as follows:

     

    1. Load the template.properties file

    2. Figure out which target environment properties I need to pull in

    3. Produce a temp file that merges the two together.  My environment properties win since they overwrite the base template property key and value.

    4. Migrate in uses the merges property file during the migration

     

    I do set all actions to "Ignore", so that I can select the actions that I want to have for my migrations.  I wish there was a better way to do this, but I currently do this by hand.  The good news is that once I do it, I don't have to touch it again during migration time.

     

    Hope that helps.

     

    Thanks,

     

    Alejandro



  • 6.  Re: GMU templating best practices

    Posted Jan 11, 2019 02:10 PM

    Hello,  

    When you use a fold to bundle your migration, how did you handle the global policy fragments and others like cert/ldap etc?  Still there are many manual steps in order to migrate things, any new progress that you can share?  We are trying to automate the deployment, still have not found a process can do that.



  • 7.  Re: GMU templating best practices

    Posted Jan 23, 2019 05:34 PM

    I package fragments in folders as well and migrate them separately. 

     

    To deal with shared components, I have migration script that defaults ALL objects to "Ignore".  Then, I set only the objects that  I want to have moved to "NewOrUpdate" based on type.  I use naming convention and object types to select which objects I want to change the mapping action for. 

     

    For example, I have my folders named as follows - "com.my_organization.service.foo.v1".  I name my service with the same name.  Additionally, I name all related cluster-wide properties with the same prefix.  My script looks for any object that contains the folder name prefix and sets the action accordingly.  Generally, I only have to deal with cluster-wide properties and services.  

     

    This isn't the most ideal solution, but it works.

     

    Alejandro