Automic Workload Automation

  • Private Community

  • 1.  RA Hadoop Agent and Oozie workflows?

    Posted Aug 14, 2015 01:31 PM
    This discussion was created from comments split from: Hadoop / Spark Integration. Hi there, I don't know much about Hadoop and I don't (yet) have any experience with the RA Hadoop Agent from Automic, but I have been asked whether the Hadoop agent can run Oozie Workflows. I dont know much about Oozie workflows, but hopefully someone in here can help me. Best regards Keld M.


  • 2.  RA Hadoop Agent and Oozie workflows?

    Posted Aug 14, 2015 01:38 PM
    The Hadoop Agent currently doesn't have an integration with Oozie, it communicates to Hadoop through the WebHCAT and WebHDFS REST Services.

    You could use the Web Service Agent to integrate with Oozie using their REST API.
    Oozie Web Services API Documentation


  • 3.  RA Hadoop Agent and Oozie workflows?

    Posted Aug 16, 2015 10:55 AM
    Hi Jennifer,

    Is it correct to say that Oozie is a Workflow Scheduler and an Oozie workflow is a collection of "jobs" such as Pig jobs, Hive Jobs and Mapreduce Jobs?
    The RA Hadoop Agent can run the same type of jobs, so why would I need to integrate with Oozie? What can Oozie do that the AE can not?
    Wouldn't it be much "prettier" to keep all scheduling in Automic?

    Regards
    Keld.


  • 4.  RA Hadoop Agent and Oozie workflows?

    Posted Aug 17, 2015 08:09 PM
    Yep, you're correct.

    Using Automic to schedule and monitor the Hadoop jobs would be the ideal scenario. ONE Automation definitely provides more functionality and integration that Oozie isn't design for.

    Oozie fills the basic scheduling needs for Hadoop jobs and many Hadoop engineers prefer using tools developed within the Apache community for other Apache projects.