api-examples/ambari-api-examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apache Ambari - API Fundamentals & Examples\n",
    "\n",
    "![Apache Ambari](https://ambari.apache.org/images/apache-ambari-project.png)\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "\n",
    "This living document will give an introduction & ongoing examples for integrating with Ambari's RESTful API.\n",
    "\n",
    "### Knowledge Requirements:\n",
    "\n",
    "- An understanding of RESTful APIs. This will help: [Learn REST: A RESTful Tutorial](http://www.restapitutorial.com/)\n",
    "- Basic programming experience. This document uses Python, but the methods can be translated to other languages.\n",
    "- Basic understanding of Ambari's functions.\n",
    "\n",
    "### How to use this notebook\n",
    "\n",
    "- a) Browse a read-only verison of the notebook [here](http://nbviewer.ipython.org/github/seanorama/ambari-bootstrap/blob/master/api-examples/ambari-api-examples.ipynb).\n",
    "- b) or, download the notebook and use [your own ipython Notebook](http://ipython.org/install.html).\n",
    "- c) or, have some inception by running ipython notebook from within Ambari!: https://github.com/randerzander/ipython-stack\n",
    "\n",
    "\n",
    "## Questions & Help\n",
    "\n",
    "* [Ambari Wiki](https://ambari.apache.org)\n",
    "* [Ambari Mailing Lists](https://ambari.apache.org/mail-lists.html)\n",
    "* [Github Issues for this Repo](https://github.com/seanorama/ambari-bootstrap/issues)\n",
    "* Me: http://twitter.com/seano\n",
    "\n",
    "\n",
    "## The Apache Ambari Ecosystem\n",
    "----\n",
    "\n",
    "* Interfaces:\n",
    "  * Ambari API\n",
    "  * Ambari Web UI (Web interface to the API)\n",
    "\n",
    "* Functions:\n",
    "  * Management, Metrics, Monitoring, Operations\n",
    "  * Blueprints\n",
    "  * Stacks\n",
    "  * Views\n",
    "  \n",
    "* Backend:\n",
    "  * Ambari Agent & Server\n",
    "\n",
    "## Ambari Web is an API Client\n",
    "\n",
    "Ambari Web is a graphical front-end to the API.\n",
    "\n",
    "Note the output on the right showing the requests which my browser is making:\n",
    "![Imgur](http://i.imgur.com/ejjvnq1l.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API Examples\n",
    "\n",
    "Below we will cover:\n",
    "\n",
    "* Authentication/Sessions\n",
    "* Change Password\n",
    "* List Clusters\n",
    "* List Hosts\n",
    "* Upload blueprint\n",
    "* Create cluster\n",
    "* Export of Cluster's blueprint\n",
    "* Show Cluster details\n",
    "* List Cluster's hosts\n",
    "* Change configuration _(example with `hive.execution.engine`)_\n",
    "\n",
    "Todo:\n",
    "\n",
    "* Restart service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Authentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "········\n",
      "You are authenticated to Ambari!\n"
     ]
    }
   ],
   "source": [
    "### Authenticate to Ambari\n",
    "\n",
    "#### Python requirements\n",
    "import difflib\n",
    "import getpass\n",
    "import json\n",
    "import requests\n",
    "import sys\n",
    "import time\n",
    "\n",
    "#### Change these to fit your Ambari configuration\n",
    "ambari_protocol = 'http'\n",
    "ambari_server = '172.28.128.3'\n",
    "ambari_port = 8080\n",
    "ambari_user = 'admin'\n",
    "#cluster = 'Sandbox'\n",
    "\n",
    "#### Above input gives us http://user:pass@hostname:port/api/v1/\n",
    "api_url = ambari_protocol + '://' + ambari_server + ':' + str(ambari_port)\n",
    "\n",
    "#### Prompt for password & build the HTTP session\n",
    "ambari_pass = getpass.getpass()\n",
    "s = requests.Session()\n",
    "s.auth = (ambari_user, ambari_pass)\n",
    "s.headers.update({'X-Requested-By':'seanorama'})\n",
    "\n",
    "#### Authenticate & verify authentication\n",
    "r = s.get(api_url + '/api/v1/clusters')\n",
    "assert r.status_code == 200\n",
    "print(\"You are authenticated to Ambari!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Change password of admin user\n",
    "\n",
    "old_pass = getpass.getpass()\n",
    "new_pass = getpass.getpass()\n",
    "\n",
    "body = {\n",
    "    \"Users\": {\n",
    "        \"user_name\": \"admin\",\n",
    "        \"old_password\": old_pass,\n",
    "        \"password\": new_pass\n",
    "}}\n",
    "\n",
    "r = s.put(api_url + '/api/v1/users/' + ambari_user, data=json.dumps(body))\n",
    "\n",
    "print(r.url)\n",
    "assert r.status_code == 200\n",
    "print(\"Password changed successfully!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Add a user\n",
    "new_pass = getpass.getpass()\n",
    "\n",
    "body = {\n",
    "    \"Users/user_name\": \"sean\",\n",
    "    \"Users/password\": new_pass,\n",
    "    \"Users/active\": 'true',\n",
    "    \"Users/admin\": 'false'\n",
    "}\n",
    "\n",
    "r = s.post(api_url + '/api/v1/users', data=json.dumps(body))\n",
    "print(r.url)\n",
    "assert r.status_code == 201\n",
    "print(\"User created successfully!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### Grant privileges\n",
    "\n",
    "#### allowed permission_names: CLUSTER.OPERATE, CLUSTER.READ\n",
    "\n",
    "body = [{\n",
    "        \"PrivilegeInfo\": {\n",
    "            \"permission_name\":\"CLUSTER.READ\",\n",
    "            \"principal_name\":\"sean\",\n",
    "            \"principal_type\":\"USER\"\n",
    "}}]\n",
    "\n",
    "r = s.post(api_url + '/api/v1/clusters/Sandbox/privileges', data=json.dumps(body))\n",
    "print(r.url)\n",
    "assert r.status_code == 201\n",
    "print(\"Privilege granted!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Clusters & Hosts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### List Clusters\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sandbox'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Set cluster based on existing cluster\n",
    "    \n",
    "cluster = r.json()['items'][0]['Clusters']['cluster_name']\n",
    "cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List registered hosts\n",
    "\n",
    "r = s.get(api_url + '/api/v1/hosts')\n",
    "print(r.url)\n",
    "#print(json.dumps(r.json(), indent=2))\n",
    "\n",
    "for host in [item[\"Hosts\"][\"host_name\"] for item in r.json()[\"items\"]]:\n",
    "    print(host)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### Cluster details\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster)\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List hosts in cluster\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster + '/hosts')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Ambari Stacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List Stacks\n",
    "\n",
    "r = s.get(api_url + '/api/v1/stacks')\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List Stack services\n",
    "        \n",
    "r = s.get(api_url + '/api/v1/stacks/HDP/versions/2.2')\n",
    "#print(json.dumps(r.json(), indent=2))\n",
    "\n",
    "print(\"\\nServices which Ambari 1.7.0 can deploy for HDP 2.2:\")\n",
    "#for services in r.json()['services']:\n",
    "#    print('\\t',services['StackServices']['service_name'])\n",
    "for service in [service['StackServices']['service_name'] for service in r.json()['services']]:\n",
    "    print('\\t',service)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Ambari Blueprints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List Blueprints\n",
    "\n",
    "r = s.get(api_url + '/api/v1/blueprints')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Blueprint: Load from file\n",
    "\n",
    "blueprint = json.loads(open('blueprint_hdp22-single-node-simple.json').read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Blueprint: Upload to Ambari\n",
    "\n",
    "body = blueprint\n",
    "r = s.post(api_url + '/api/v1/blueprints/myblueprint', data=json.dumps(body))\n",
    "assert r.status_code == 201\n",
    "print(\"Blueprint uploaded successfully!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 's' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-46e054ccfe93>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#### Show Blueprint\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mapi_url\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'/api/v1/blueprints/myblueprint'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdumps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 's' is not defined"
     ]
    }
   ],
   "source": [
    "#### Show Blueprint\n",
    "\n",
    "r = s.get(api_url + '/api/v1/blueprints/myblueprint')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Create Cluster from Blueprint\n",
    "\n",
    "body = {\n",
    "  \"blueprint\": \"myblueprint\",\n",
    "  \"default_password\": \"replacethiswithacomplexpass\",\n",
    "  \"configurations\": [\n",
    "      { \"hive-site\": { \"properties\": { \"javax.jdo.option.ConnectionPassword\": \"replacethiswithacomplexpass\" } } }\n",
    "  ],\n",
    "  \"host_groups\": [\n",
    "    {\n",
    "      \"hosts\": [ { \"fqdn\": \"hdp\" } ],\n",
    "      \"name\": \"host-group-1\"\n",
    "}]}\n",
    "\n",
    "r = s.post(api_url + '/api/v1/clusters/mycluster', data=json.dumps(body))\n",
    "print(r.url)\n",
    "print(r.status_code)\n",
    "print(json.dumps(r.json(), indent=2))\n",
    "status_url = r.json()['href']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Check Cluster creation status\n",
    "\n",
    "r = s.get(status_url)\n",
    "print(r.url)\n",
    "print(json.dumps(r.json()['Requests'], indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://172.28.128.3:8080/api/v1/clusters/Sandbox?format=blueprint\n",
      "{\n",
      "  \"Blueprints\": {\n",
      "    \"stack_name\": \"HDP\",\n",
      "    \"stack_version\": \"2.2\"\n",
      "  },\n",
      "  \"host_groups\": [\n",
      "    {\n",
      "      \"components\": [\n",
      "        {\n",
      "          \"name\": \"PIG\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HISTORYSERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"KAFKA_BROKER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"OOZIE_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HBASE_REGIONSERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HBASE_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"NAMENODE\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"SUPERVISOR\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"FALCON_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HCAT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"SLIDER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"AMBARI_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"APP_TIMELINE_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HDFS_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HIVE_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"NODEMANAGER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"FLUME_HANDLER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"DATANODE\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"WEBHCAT_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"RESOURCEMANAGER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"ZOOKEEPER_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"ZOOKEEPER_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"STORM_UI_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HBASE_MASTER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HIVE_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"OOZIE_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"FALCON_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"NAGIOS_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"SECONDARY_NAMENODE\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"TEZ_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"HIVE_METASTORE\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"GANGLIA_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"SQOOP\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"YARN_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"MAPREDUCE2_CLIENT\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"MYSQL_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"GANGLIA_MONITOR\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"DRPC_SERVER\"\n",
      "        },\n",
      "        {\n",
      "          \"name\": \"NIMBUS\"\n",
      "        }\n",
      "      ],\n",
      "      \"cardinality\": \"1\",\n",
      "      \"name\": \"host_group_1\",\n",
      "      \"configurations\": []\n",
      "    }\n",
      "  ],\n",
      "  \"configurations\": [\n",
      "    {\n",
      "      \"capacity-scheduler\": {\n",
      "        \"yarn.scheduler.capacity.root.default.maximum-capacity\": \"100\",\n",
      "        \"yarn.scheduler.capacity.root.accessible-node-labels.default.maximum-capacity\": \"-1\",\n",
      "        \"yarn.scheduler.capacity.resource-calculator\": \"org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator\",\n",
      "        \"yarn.scheduler.capacity.root.default.user-limit-factor\": \"1\",\n",
      "        \"yarn.scheduler.capacity.root.default.acl_administer_jobs\": \"*\",\n",
      "        \"yarn.scheduler.capacity.root.default-node-label-expression\": \" \",\n",
      "        \"yarn.scheduler.capacity.root.accessible-node-labels\": \"*\",\n",
      "        \"yarn.scheduler.capacity.maximum-applications\": \"10000\",\n",
      "        \"yarn.scheduler.capacity.root.default.maximum-am-resource-percent\": \"0.5\",\n",
      "        \"yarn.scheduler.capacity.root.capacity\": \"100\",\n",
      "        \"yarn.scheduler.capacity.maximum-am-resource-percent\": \"0.5\",\n",
      "        \"yarn.scheduler.capacity.node-locality-delay\": \"40\",\n",
      "        \"yarn.scheduler.capacity.root.acl_administer_queue\": \"*\",\n",
      "        \"yarn.scheduler.capacity.root.accessible-node-labels.default.capacity\": \"-1\",\n",
      "        \"yarn.scheduler.capacity.root.queues\": \"default\",\n",
      "        \"yarn.scheduler.capacity.root.default.state\": \"RUNNING\",\n",
      "        \"yarn.scheduler.capacity.root.default.acl_submit_applications\": \"*\",\n",
      "        \"yarn.scheduler.capacity.default.minimum-user-limit-percent\": \"100\",\n",
      "        \"yarn.scheduler.capacity.root.default.capacity\": \"100\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"cluster-env\": {\n",
      "        \"cluster_name\": \"sandbox\",\n",
      "        \"hadoop-streaming_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/\",\n",
      "        \"kerberos_domain\": \"EXAMPLE.COM\",\n",
      "        \"security_enabled\": \"false\",\n",
      "        \"mapreduce_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/\",\n",
      "        \"smokeuser_keytab\": \"/etc/security/keytabs/smokeuser.headless.keytab\",\n",
      "        \"smokeuser\": \"ambari-qa\",\n",
      "        \"hive_tar_source\": \"/usr/hdp/current/hive-client/hive.tar.gz\",\n",
      "        \"pig_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/pig/\",\n",
      "        \"ignore_groupsusers_create\": \"false\",\n",
      "        \"sqoop_tar_source\": \"/usr/hdp/current/sqoop-client/sqoop.tar.gz\",\n",
      "        \"user_group\": \"hadoop\",\n",
      "        \"mapreduce_tar_source\": \"/usr/hdp/current/hadoop-client/mapreduce.tar.gz\",\n",
      "        \"pig_tar_source\": \"/usr/hdp/current/pig-client/pig.tar.gz\",\n",
      "        \"sqoop_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/sqoop/\",\n",
      "        \"hive_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/hive/\",\n",
      "        \"hadoop-streaming_tar_source\": \"/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar\",\n",
      "        \"tez_tar_destination_folder\": \"hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/\",\n",
      "        \"tez_tar_source\": \"/usr/hdp/current/tez-client/lib/tez.tar.gz\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"core-site\": {\n",
      "        \"ipc.client.connection.maxidletime\": \"30000\",\n",
      "        \"hadoop.proxyuser.root.groups\": \"*\",\n",
      "        \"hadoop.security.authorization\": \"false\",\n",
      "        \"hadoop.security.auth_to_local\": \"\\n        RULE:[2:$1@$0]([rn]m@.*)s/.*/yarn/\\n        RULE:[2:$1@$0](jhs@.*)s/.*/mapred/\\n        RULE:[2:$1@$0]([nd]n@.*)s/.*/hdfs/\\n        RULE:[2:$1@$0](hm@.*)s/.*/hbase/\\n        RULE:[2:$1@$0](rs@.*)s/.*/hbase/\\n        DEFAULT\\n    \",\n",
      "        \"hadoop.proxyuser.hue.hosts\": \"*\",\n",
      "        \"hadoop.proxyuser.falcon.hosts\": \"*\",\n",
      "        \"fs.trash.interval\": \"360\",\n",
      "        \"hadoop.proxyuser.hbase.groups\": \"users\",\n",
      "        \"ipc.server.tcpnodelay\": \"true\",\n",
      "        \"ipc.client.idlethreshold\": \"8000\",\n",
      "        \"io.file.buffer.size\": \"131072\",\n",
      "        \"io.serializations\": \"org.apache.hadoop.io.serializer.WritableSerialization\",\n",
      "        \"fs.defaultFS\": \"hdfs://%HOSTGROUP::host_group_1%:8020\",\n",
      "        \"mapreduce.jobtracker.webinterface.trusted\": \"false\",\n",
      "        \"hadoop.proxyuser.root.hosts\": \"*\",\n",
      "        \"hadoop.proxyuser.hive.hosts\": \"*\",\n",
      "        \"hadoop.proxyuser.hive.groups\": \"users\",\n",
      "        \"io.compression.codecs\": \"org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec\",\n",
      "        \"hadoop.proxyuser.oozie.hosts\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"hadoop.proxyuser.hcat.hosts\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"hadoop.proxyuser.hcat.groups\": \"*\",\n",
      "        \"hadoop.proxyuser.hue.groups\": \"*\",\n",
      "        \"ipc.client.connect.max.retries\": \"50\",\n",
      "        \"hadoop.http.authentication.simple.anonymous.allowed\": \"true\",\n",
      "        \"hadoop.proxyuser.oozie.groups\": \"*\",\n",
      "        \"hadoop.proxyuser.falcon.groups\": \"users\",\n",
      "        \"hadoop.proxyuser.hbase.hosts\": \"*\",\n",
      "        \"hadoop.security.authentication\": \"simple\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"falcon-env\": {\n",
      "        \"falcon_local_dir\": \"/hadoop/falcon\",\n",
      "        \"falcon.embeddedmq.data\": \"/hadoop/falcon/embeddedmq/data\",\n",
      "        \"falcon.embeddedmq\": \"true\",\n",
      "        \"falcon_user\": \"falcon\",\n",
      "        \"falcon.emeddedmq.port\": \"61616\",\n",
      "        \"falcon_pid_dir\": \"/var/run/falcon\",\n",
      "        \"content\": \"\\n# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path\\nexport JAVA_HOME={{java_home}}\\n\\n# any additional java opts you want to set. This will apply to both client and server operations\\n#export FALCON_OPTS=\\n\\n# any additional java opts that you want to set for client only\\n#export FALCON_CLIENT_OPTS=\\n\\n# java heap size we want to set for the client. Default is 1024MB\\n#export FALCON_CLIENT_HEAP=\\n\\n# any additional opts you want to set for prisim service.\\n#export FALCON_PRISM_OPTS=\\n\\n# java heap size we want to set for the prisim service. Default is 1024MB\\n#export FALCON_PRISM_HEAP=\\n\\n# any additional opts you want to set for falcon service.\\nexport FALCON_SERVER_OPTS=\\\"-Dfalcon.embeddedmq={{falcon_embeddedmq_enabled}} -Dfalcon.emeddedmq.port={{falcon_emeddedmq_port}}\\\"\\n\\n# java heap size we want to set for the falcon server. Default is 1024MB\\n#export FALCON_SERVER_HEAP=\\n\\n# What is is considered as falcon home dir. Default is the base location of the installed software\\n#export FALCON_HOME_DIR=\\n\\n# Where log files are stored. Defatult is logs directory under the base install location\\nexport FALCON_LOG_DIR={{falcon_log_dir}}\\n\\n# Where pid files are stored. Defatult is logs directory under the base install location\\nexport FALCON_PID_DIR={{falcon_pid_dir}}\\n\\n# where the falcon active mq data is stored. Defatult is logs/data directory under the base install location\\nexport FALCON_DATA_DIR={{falcon_embeddedmq_data}}\\n\\n# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.\\n#export FALCON_EXPANDED_WEBAPP_DIR=\\n    \",\n",
      "        \"falcon_port\": \"15000\",\n",
      "        \"falcon_log_dir\": \"/var/log/falcon\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"falcon-runtime.properties\": {\n",
      "        \"*.log.cleanup.frequency.hours.retention\": \"minutes(1)\",\n",
      "        \"*.log.cleanup.frequency.months.retention\": \"months(3)\",\n",
      "        \"*.log.cleanup.frequency.days.retention\": \"days(7)\",\n",
      "        \"*.log.cleanup.frequency.minutes.retention\": \"hours(6)\",\n",
      "        \"*.domain\": \"${falcon.app.type}\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"falcon-startup.properties\": {\n",
      "        \"prism.application.services\": \"org.apache.falcon.entity.store.ConfigurationStore\",\n",
      "        \"*.SchedulableEntityManager.impl\": \"org.apache.falcon.resource.SchedulableEntityManager\",\n",
      "        \"*.domain\": \"${falcon.app.type}\",\n",
      "        \"*.falcon.cleanup.service.frequency\": \"days(1)\",\n",
      "        \"*.application.services\": \"org.apache.falcon.security.AuthenticationInitializationService,\\\\\\n      org.apache.falcon.workflow.WorkflowJobEndNotificationService, \\\\\\n      org.apache.falcon.service.ProcessSubscriberService,\\\\\\n      org.apache.falcon.entity.store.ConfigurationStore,\\\\\\n      org.apache.falcon.rerun.service.RetryService,\\\\\\n      org.apache.falcon.rerun.service.LateRunService,\\\\\\n      org.apache.falcon.service.LogCleanupService,\\\\\\n      org.apache.falcon.metadata.MetadataMappingService\\n    \",\n",
      "        \"*.journal.impl\": \"org.apache.falcon.transaction.SharedFileSystemJournal\",\n",
      "        \"prism.configstore.listeners\": \"org.apache.falcon.entity.v0.EntityGraph,\\\\\\n      org.apache.falcon.entity.ColoClusterRelation,\\\\\\n      org.apache.falcon.group.FeedGroupMap\\n    \",\n",
      "        \"*.broker.ttlInMins\": \"4320\",\n",
      "        \"*.falcon.security.authorization.admin.users\": \"falcon,ambari-qa\",\n",
      "        \"*.retry.recorder.path\": \"${falcon.log.dir}/retry\",\n",
      "        \"*.ConfigSyncService.impl\": \"org.apache.falcon.resource.ConfigSyncService\",\n",
      "        \"*.ProcessInstanceManager.impl\": \"org.apache.falcon.resource.InstanceManager\",\n",
      "        \"*.falcon.graph.preserve.history\": \"false\",\n",
      "        \"*.internal.queue.size\": \"1000\",\n",
      "        \"*.system.lib.location\": \"${falcon.home}/server/webapp/${falcon.app.type}/WEB-INF/lib\",\n",
      "        \"*.configstore.listeners\": \"org.apache.falcon.entity.v0.EntityGraph,\\\\\\n      org.apache.falcon.entity.ColoClusterRelation,\\\\\\n      org.apache.falcon.group.FeedGroupMap,\\\\\\n      org.apache.falcon.service.SharedLibraryHostingService\\n    \",\n",
      "        \"*.falcon.http.authentication.cookie.domain\": \"EXAMPLE.COM\",\n",
      "        \"*.falcon.enableTLS\": \"false\",\n",
      "        \"*.broker.url\": \"tcp://%HOSTGROUP::host_group_1%:61616\",\n",
      "        \"*.falcon.http.authentication.kerberos.name.rules\": \"DEFAULT\",\n",
      "        \"*.falcon.graph.serialize.path\": \"/hadoop/falcon/data/lineage\",\n",
      "        \"*.falcon.http.authentication.token.validity\": \"36000\",\n",
      "        \"*.falcon.http.authentication.simple.anonymous.allowed\": \"true\",\n",
      "        \"*.max.retry.failure.count\": \"1\",\n",
      "        \"*.falcon.security.authorization.enabled\": \"false\",\n",
      "        \"*.broker.impl.class\": \"org.apache.activemq.ActiveMQConnectionFactory\",\n",
      "        \"*.falcon.security.authorization.admin.groups\": \"falcon\",\n",
      "        \"*.falcon.service.authentication.kerberos.keytab\": \"/etc/security/keytabs/falcon.service.keytab\",\n",
      "        \"*.falcon.graph.storage.backend\": \"berkeleyje\",\n",
      "        \"*.falcon.http.authentication.blacklisted.users\": \"\",\n",
      "        \"*.oozie.process.workflow.builder\": \"org.apache.falcon.workflow.OozieProcessWorkflowBuilder\",\n",
      "        \"*.falcon.security.authorization.superusergroup\": \"falcon\",\n",
      "        \"*.falcon.authentication.type\": \"simple\",\n",
      "        \"*.falcon.graph.blueprints.graph\": \"com.thinkaurelius.titan.core.TitanFactory\",\n",
      "        \"*.falcon.http.authentication.type\": \"simple\",\n",
      "        \"*.falcon.http.authentication.signature.secret\": \"falcon\",\n",
      "        \"*.falcon.http.authentication.kerberos.keytab\": \"/etc/security/keytabs/spnego.service.keytab\",\n",
      "        \"*.falcon.security.authorization.provider\": \"org.apache.falcon.security.DefaultAuthorizationProvider\",\n",
      "        \"*.catalog.service.impl\": \"org.apache.falcon.catalog.HiveCatalogService\",\n",
      "        \"*.entity.topic\": \"FALCON.ENTITY.TOPIC\",\n",
      "        \"*.oozie.feed.workflow.builder\": \"org.apache.falcon.workflow.OozieFeedWorkflowBuilder\",\n",
      "        \"*.falcon.graph.storage.directory\": \"/hadoop/falcon/data/lineage/graphdb\",\n",
      "        \"*.config.store.uri\": \"file:///hadoop/falcon/store\",\n",
      "        \"*.dfs.namenode.kerberos.principal\": \"nn/_HOST@EXAMPLE.COM\",\n",
      "        \"*.shared.libs\": \"activemq-core,ant,geronimo-j2ee-management,hadoop-distcp,jms,json-simple,oozie-client,spring-jms\",\n",
      "        \"*.workflow.engine.impl\": \"org.apache.falcon.workflow.engine.OozieWorkflowEngine\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"flume-conf\": {\n",
      "        \"content\": \"\\n# Flume agent config\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"flume-env\": {\n",
      "        \"flume_conf_dir\": \"/etc/flume/conf\",\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one or more\\n# contributor license agreements.  See the NOTICE file distributed with\\n# this work for additional information regarding copyright ownership.\\n# The ASF licenses this file to You under the Apache License, Version 2.0\\n# (the \\\"License\\\"); you may not use this file except in compliance with\\n# the License.  You may obtain a copy of the License at\\n#\\n#     http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n\\n# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced\\n# during Flume startup.\\n\\n# Enviroment variables can be set here.\\n\\nexport JAVA_HOME={{java_home}}\\n\\n# Give Flume more memory and pre-allocate, enable remote monitoring via JMX\\n# export JAVA_OPTS=\\\"-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote\\\"\\n\\n# Note that the Flume conf directory is always included in the classpath.\\n#FLUME_CLASSPATH=\\\"\\\"\\n\\nexport HIVE_HOME={{flume_hive_home}}\\nexport HCAT_HOME={{flume_hcat_home}}\\n    \",\n",
      "        \"flume_user\": \"flume\",\n",
      "        \"flume_log_dir\": \"/var/log/flume\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"ganglia-env\": {\n",
      "        \"rrdcached_flush_timeout\": \"7200\",\n",
      "        \"rrdcached_timeout\": \"3600\",\n",
      "        \"rrdcached_base_dir\": \"/var/lib/ganglia/rrds\",\n",
      "        \"ganglia_conf_dir\": \"/etc/ganglia/hdp\",\n",
      "        \"rrdcached_write_threads\": \"4\",\n",
      "        \"gmond_user\": \"nobody\",\n",
      "        \"additional_clusters\": \" \",\n",
      "        \"rrdcached_delay\": \"1800\",\n",
      "        \"gmetad_user\": \"nobody\",\n",
      "        \"ganglia_runtime_dir\": \"/var/run/ganglia/hdp\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hadoop-env\": {\n",
      "        \"hadoop_heapsize\": \"250\",\n",
      "        \"dtnode_heapsize\": \"250m\",\n",
      "        \"proxyuser_group\": \"users\",\n",
      "        \"dfs.datanode.data.dir.mount.file\": \"/etc/hadoop/conf/dfs_data_dir_mount.hist\",\n",
      "        \"hdfs_log_dir_prefix\": \"/var/log/hadoop\",\n",
      "        \"namenode_opt_maxnewsize\": \"100m\",\n",
      "        \"namenode_opt_maxpermsize\": \"256m\",\n",
      "        \"hadoop_pid_dir_prefix\": \"/var/run/hadoop\",\n",
      "        \"hdfs_user\": \"hdfs\",\n",
      "        \"namenode_opt_newsize\": \"50m\",\n",
      "        \"namenode_opt_permsize\": \"128m\",\n",
      "        \"content\": \"\\n# Set Hadoop-specific environment variables here.\\n\\n# The only required environment variable is JAVA_HOME.  All others are\\n# optional.  When running a distributed configuration it is best to\\n# set JAVA_HOME in this file, so that it is correctly defined on\\n# remote nodes.\\n\\n# The java implementation to use.  Required.\\nexport JAVA_HOME={{java_home}}\\nexport HADOOP_HOME_WARN_SUPPRESS=1\\n\\n# Hadoop home directory\\nexport HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n\\n# Hadoop Configuration Directory\\n\\n{# this is different for HDP1 #}\\n# Path to jsvc required by secure HDP 2.0 datanode\\nexport JSVC_HOME={{jsvc_path}}\\n\\n\\n# The maximum amount of heap to use, in MB. Default is 1000.\\nexport HADOOP_HEAPSIZE=\\\"{{hadoop_heapsize}}\\\"\\n\\nexport HADOOP_NAMENODE_INIT_HEAPSIZE=\\\"-Xms{{namenode_heapsize}}\\\"\\n\\n# Extra Java runtime options.  Empty by default.\\nexport HADOOP_OPTS=\\\"-Djava.net.preferIPv4Stack=true ${HADOOP_OPTS}\\\"\\n\\n# Command specific options appended to HADOOP_OPTS when specified\\nexport HADOOP_NAMENODE_OPTS=\\\"-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{namenode_opt_newsize}} -XX:MaxNewSize={{namenode_opt_maxnewsize}} -XX:PermSize={{namenode_opt_permsize}} -XX:MaxPermSize={{namenode_opt_maxpermsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms{{namenode_heapsize}} -Xmx{{namenode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_NAMENODE_OPTS}\\\"\\nHADOOP_JOBTRACKER_OPTS=\\\"-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{jtnode_opt_newsize}} -XX:MaxNewSize={{jtnode_opt_maxnewsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xmx{{jtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA ${HADOOP_JOBTRACKER_OPTS}\\\"\\n\\nHADOOP_TASKTRACKER_OPTS=\\\"-server -Xmx{{ttnode_heapsize}} -Dhadoop.security.logger=ERROR,console -Dmapred.audit.logger=ERROR,console ${HADOOP_TASKTRACKER_OPTS}\\\"\\nexport HADOOP_DATANODE_OPTS=\\\"-server -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -XX:PermSize=128m -XX:MaxPermSize=256m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms{{dtnode_heapsize}} -Xmx{{dtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_DATANODE_OPTS}\\\"\\nHADOOP_BALANCER_OPTS=\\\"-server -Xmx{{hadoop_heapsize}}m ${HADOOP_BALANCER_OPTS}\\\"\\n\\nexport HADOOP_SECONDARYNAMENODE_OPTS=$HADOOP_NAMENODE_OPTS\\n\\n# The following applies to multiple commands (fs, dfs, fsck, distcp etc)\\nexport HADOOP_CLIENT_OPTS=\\\"-Xmx${HADOOP_HEAPSIZE}m -XX:MaxPermSize=512m $HADOOP_CLIENT_OPTS\\\"\\n\\n# On secure datanodes, user to run the datanode as after dropping privileges\\nexport HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER:-{{hadoop_secure_dn_user}}}\\n\\n# Extra ssh options.  Empty by default.\\nexport HADOOP_SSH_OPTS=\\\"-o ConnectTimeout=5 -o SendEnv=HADOOP_CONF_DIR\\\"\\n\\n# Where log files are stored.  $HADOOP_HOME/logs by default.\\nexport HADOOP_LOG_DIR={{hdfs_log_dir_prefix}}/$USER\\n\\n# History server logs\\nexport HADOOP_MAPRED_LOG_DIR={{mapred_log_dir_prefix}}/$USER\\n\\n# Where log files are stored in the secure data environment.\\nexport HADOOP_SECURE_DN_LOG_DIR={{hdfs_log_dir_prefix}}/$HADOOP_SECURE_DN_USER\\n\\n# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.\\n# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves\\n\\n# host:path where hadoop code should be rsync'd from.  Unset by default.\\n# export HADOOP_MASTER=master:/home/$USER/src/hadoop\\n\\n# Seconds to sleep between slave commands.  Unset by default.  This\\n# can be useful in large clusters, where, e.g., slave rsyncs can\\n# otherwise arrive faster than the master can service them.\\n# export HADOOP_SLAVE_SLEEP=0.1\\n\\n# The directory where pid files are stored. /tmp by default.\\nexport HADOOP_PID_DIR={{hadoop_pid_dir_prefix}}/$USER\\nexport HADOOP_SECURE_DN_PID_DIR={{hadoop_pid_dir_prefix}}/$HADOOP_SECURE_DN_USER\\n\\n# History server pid\\nexport HADOOP_MAPRED_PID_DIR={{mapred_pid_dir_prefix}}/$USER\\n\\nYARN_RESOURCEMANAGER_OPTS=\\\"-Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY\\\"\\n\\n# A string representing this instance of hadoop. $USER by default.\\nexport HADOOP_IDENT_STRING=$USER\\n\\n# The scheduling priority for daemon processes.  See 'man nice'.\\n\\n# export HADOOP_NICENESS=10\\n\\n# Use libraries from standard classpath\\nJAVA_JDBC_LIBS=\\\"\\\"\\n#Add libraries required by mysql connector\\nfor jarFile in `ls /usr/share/java/*mysql* 2>/dev/null`\\ndo\\n  JAVA_JDBC_LIBS=${JAVA_JDBC_LIBS}:$jarFile\\ndone\\n# Add libraries required by oracle connector\\nfor jarFile in `ls /usr/share/java/*ojdbc* 2>/dev/null`\\ndo\\n  JAVA_JDBC_LIBS=${JAVA_JDBC_LIBS}:$jarFile\\ndone\\n# Add libraries required by nodemanager\\nMAPREDUCE_LIBS={{mapreduce_libs_path}}\\nexport HADOOP_CLASSPATH=${HADOOP_CLASSPATH}${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}\\n\\n# added to the HADOOP_CLASSPATH\\nif [ -d \\\"/usr/hdp/current/tez-client\\\" ]; then\\n  if [ -d \\\"/etc/tez/conf/\\\" ]; then\\n    # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.\\n    export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/\\n  fi\\nfi\\n\\n# Setting path to hdfs command line\\nexport HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}\\n\\n# Mostly required for hadoop 2.0\\nexport JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}\\n\\nexport HADOOP_OPTS=\\\"-Dhdp.version=$HDP_VERSION $HADOOP_OPTS\\\"\\n    \",\n",
      "        \"hadoop_root_logger\": \"INFO,RFA\",\n",
      "        \"namenode_heapsize\": \"250m\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hadoop-policy\": {\n",
      "        \"security.client.datanode.protocol.acl\": \"*\",\n",
      "        \"security.refresh.policy.protocol.acl\": \"hadoop\",\n",
      "        \"security.job.task.protocol.acl\": \"*\",\n",
      "        \"security.admin.operations.protocol.acl\": \"hadoop\",\n",
      "        \"security.client.protocol.acl\": \"*\",\n",
      "        \"security.inter.tracker.protocol.acl\": \"*\",\n",
      "        \"security.refresh.usertogroups.mappings.protocol.acl\": \"hadoop\",\n",
      "        \"security.inter.datanode.protocol.acl\": \"*\",\n",
      "        \"security.datanode.protocol.acl\": \"*\",\n",
      "        \"security.namenode.protocol.acl\": \"*\",\n",
      "        \"security.job.client.protocol.acl\": \"*\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hbase-env\": {\n",
      "        \"hbase_log_dir\": \"/var/log/hbase\",\n",
      "        \"hbase_pid_dir\": \"/var/run/hbase\",\n",
      "        \"hbase_regionserver_xmn_max\": \"512\",\n",
      "        \"hbase_user\": \"hbase\",\n",
      "        \"content\": \"\\n# Set environment variables here.\\n\\n# The java implementation to use. Java 1.6 required.\\nexport JAVA_HOME={{java64_home}}\\n\\n# HBase Configuration directory\\nexport HBASE_CONF_DIR=${HBASE_CONF_DIR:-{{hbase_conf_dir}}}\\n\\n# Extra Java CLASSPATH elements. Optional.\\nexport HBASE_CLASSPATH=${HBASE_CLASSPATH}\\n\\n# The maximum amount of heap to use, in MB. Default is 1000.\\n# export HBASE_HEAPSIZE=1000\\n\\n# Extra Java runtime options.\\n# Below are what we set by default. May only work with SUN JVM.\\n# For more on why as well as other possible settings,\\n# see http://wiki.apache.org/hadoop/PerformanceTuning\\nexport SERVER_GC_OPTS=\\\"-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:{{log_dir}}/gc.log-`date +'%Y%m%d%H%M'`\\\"\\n# Uncomment below to enable java garbage collection logging.\\n# export HBASE_OPTS=\\\"$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log\\\"\\n\\n# Uncomment and adjust to enable JMX exporting\\n# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.\\n# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html\\n#\\n# export HBASE_JMX_BASE=\\\"-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false\\\"\\n# If you want to configure BucketCache, specify '-XX: MaxDirectMemorySize=' with proper direct memory size\\n# export HBASE_THRIFT_OPTS=\\\"$HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103\\\"\\n# export HBASE_ZOOKEEPER_OPTS=\\\"$HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104\\\"\\n\\n# File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default.\\nexport HBASE_REGIONSERVERS=${HBASE_CONF_DIR}/regionservers\\n\\n# Extra ssh options. Empty by default.\\n# export HBASE_SSH_OPTS=\\\"-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR\\\"\\n\\n# Where log files are stored. $HBASE_HOME/logs by default.\\nexport HBASE_LOG_DIR={{log_dir}}\\n\\n# A string representing this instance of hbase. $USER by default.\\n# export HBASE_IDENT_STRING=$USER\\n\\n# The scheduling priority for daemon processes. See 'man nice'.\\n# export HBASE_NICENESS=10\\n\\n# The directory where pid files are stored. /tmp by default.\\nexport HBASE_PID_DIR={{pid_dir}}\\n\\n# Seconds to sleep between slave commands. Unset by default. This\\n# can be useful in large clusters, where, e.g., slave rsyncs can\\n# otherwise arrive faster than the master can service them.\\n# export HBASE_SLAVE_SLEEP=0.1\\n\\n# Tell HBase whether it should manage it's own instance of Zookeeper or not.\\nexport HBASE_MANAGES_ZK=false\\n\\n{% if security_enabled %}\\nexport HBASE_OPTS=\\\"$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile={{log_dir}}/hs_err_pid%p.log -Djava.security.auth.login.config={{client_jaas_config_file}}\\\"\\nexport HBASE_MASTER_OPTS=\\\"$HBASE_MASTER_OPTS -Xmx{{master_heapsize}} -Djava.security.auth.login.config={{master_jaas_config_file}}\\\"\\nexport HBASE_REGIONSERVER_OPTS=\\\"$HBASE_REGIONSERVER_OPTS -Xmn{{regionserver_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70  -Xms{{regionserver_heapsize}} -Xmx{{regionserver_heapsize}} -Djava.security.auth.login.config={{regionserver_jaas_config_file}}\\\"\\n{% else %}\\nexport HBASE_OPTS=\\\"$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile={{log_dir}}/hs_err_pid%p.log\\\"\\nexport HBASE_MASTER_OPTS=\\\"$HBASE_MASTER_OPTS -Xmx{{master_heapsize}}\\\"\\nexport HBASE_REGIONSERVER_OPTS=\\\"$HBASE_REGIONSERVER_OPTS -Xmn{{regionserver_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70  -Xms{{regionserver_heapsize}} -Xmx{{regionserver_heapsize}}\\\"\\n{% endif %}\\n    \",\n",
      "        \"hbase_master_heapsize\": \"1024m\",\n",
      "        \"hbase_regionserver_heapsize\": \"1024m\",\n",
      "        \"hbase_regionserver_xmn_ratio\": \"0.2\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hbase-log4j\": {\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#     http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n\\n\\n# Define some default values that can be overridden by system properties\\nhbase.root.logger=INFO,console\\nhbase.security.logger=INFO,console\\nhbase.log.dir=.\\nhbase.log.file=hbase.log\\n\\n# Define the root logger to the system property \\\"hbase.root.logger\\\".\\nlog4j.rootLogger=${hbase.root.logger}\\n\\n# Logging Threshold\\nlog4j.threshold=ALL\\n\\n#\\n# Daily Rolling File Appender\\n#\\nlog4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.DRFA.File=${hbase.log.dir}/${hbase.log.file}\\n\\n# Rollver at midnight\\nlog4j.appender.DRFA.DatePattern=.yyyy-MM-dd\\n\\n# 30-day backup\\n#log4j.appender.DRFA.MaxBackupIndex=30\\nlog4j.appender.DRFA.layout=org.apache.log4j.PatternLayout\\n\\n# Pattern format: Date LogLevel LoggerName LogMessage\\nlog4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n\\n\\n# Rolling File Appender properties\\nhbase.log.maxfilesize=256MB\\nhbase.log.maxbackupindex=20\\n\\n# Rolling File Appender\\nlog4j.appender.RFA=org.apache.log4j.RollingFileAppender\\nlog4j.appender.RFA.File=${hbase.log.dir}/${hbase.log.file}\\n\\nlog4j.appender.RFA.MaxFileSize=${hbase.log.maxfilesize}\\nlog4j.appender.RFA.MaxBackupIndex=${hbase.log.maxbackupindex}\\n\\nlog4j.appender.RFA.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n\\n\\n#\\n# Security audit appender\\n#\\nhbase.security.log.file=SecurityAuth.audit\\nhbase.security.log.maxfilesize=256MB\\nhbase.security.log.maxbackupindex=20\\nlog4j.appender.RFAS=org.apache.log4j.RollingFileAppender\\nlog4j.appender.RFAS.File=${hbase.log.dir}/${hbase.security.log.file}\\nlog4j.appender.RFAS.MaxFileSize=${hbase.security.log.maxfilesize}\\nlog4j.appender.RFAS.MaxBackupIndex=${hbase.security.log.maxbackupindex}\\nlog4j.appender.RFAS.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\nlog4j.category.SecurityLogger=${hbase.security.logger}\\nlog4j.additivity.SecurityLogger=false\\n#log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE\\n\\n#\\n# Null Appender\\n#\\nlog4j.appender.NullAppender=org.apache.log4j.varia.NullAppender\\n\\n#\\n# console\\n# Add \\\"console\\\" to rootlogger above if you want to use this\\n#\\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\\nlog4j.appender.console.target=System.err\\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n\\n\\n# Custom Logging levels\\n\\nlog4j.logger.org.apache.zookeeper=INFO\\n#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG\\nlog4j.logger.org.apache.hadoop.hbase=DEBUG\\n# Make these two classes INFO-level. Make them DEBUG to see more zk debug.\\nlog4j.logger.org.apache.hadoop.hbase.zookeeper.ZKUtil=INFO\\nlog4j.logger.org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher=INFO\\n#log4j.logger.org.apache.hadoop.dfs=DEBUG\\n# Set this class to log INFO only otherwise its OTT\\n# Enable this to get detailed connection error/retry logging.\\n# log4j.logger.org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation=TRACE\\n\\n\\n# Uncomment this line to enable tracing on _every_ RPC call (this can be a lot of output)\\n#log4j.logger.org.apache.hadoop.ipc.HBaseServer.trace=DEBUG\\n\\n# Uncomment the below if you want to remove logging of client region caching'\\n# and scan of .META. messages\\n# log4j.logger.org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation=INFO\\n# log4j.logger.org.apache.hadoop.hbase.client.MetaScanner=INFO\\n\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hbase-policy\": {\n",
      "        \"security.admin.protocol.acl\": \"*\",\n",
      "        \"security.client.protocol.acl\": \"*\",\n",
      "        \"security.masterregion.protocol.acl\": \"*\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hcat-env\": {\n",
      "        \"content\": \"\\n      # Licensed to the Apache Software Foundation (ASF) under one\\n      # or more contributor license agreements. See the NOTICE file\\n      # distributed with this work for additional information\\n      # regarding copyright ownership. The ASF licenses this file\\n      # to you under the Apache License, Version 2.0 (the\\n      # \\\"License\\\"); you may not use this file except in compliance\\n      # with the License. You may obtain a copy of the License at\\n      #\\n      # http://www.apache.org/licenses/LICENSE-2.0\\n      #\\n      # Unless required by applicable law or agreed to in writing, software\\n      # distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n      # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n      # See the License for the specific language governing permissions and\\n      # limitations under the License.\\n\\n      JAVA_HOME={{java64_home}}\\n      HCAT_PID_DIR={{hcat_pid_dir}}/\\n      HCAT_LOG_DIR={{hcat_log_dir}}/\\n      HCAT_CONF_DIR={{hcat_conf_dir}}\\n      HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n      #DBROOT is the path where the connector jars are downloaded\\n      DBROOT={{hcat_dbroot}}\\n      USER={{hcat_user}}\\n      METASTORE_PORT={{hive_metastore_port}}\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hdfs-log4j\": {\n",
      "        \"content\": \"\\n#\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#  http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing,\\n# software distributed under the License is distributed on an\\n# \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\\n# KIND, either express or implied.  See the License for the\\n# specific language governing permissions and limitations\\n# under the License.\\n#\\n\\n\\n# Define some default values that can be overridden by system properties\\n# To change daemon root logger use hadoop_root_logger in hadoop-env\\nhadoop.root.logger=INFO,console\\nhadoop.log.dir=.\\nhadoop.log.file=hadoop.log\\n\\n\\n# Define the root logger to the system property \\\"hadoop.root.logger\\\".\\nlog4j.rootLogger=${hadoop.root.logger}, EventCounter\\n\\n# Logging Threshold\\nlog4j.threshhold=ALL\\n\\n#\\n# Daily Rolling File Appender\\n#\\n\\nlog4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}\\n\\n# Rollver at midnight\\nlog4j.appender.DRFA.DatePattern=.yyyy-MM-dd\\n\\n# 30-day backup\\n#log4j.appender.DRFA.MaxBackupIndex=30\\nlog4j.appender.DRFA.layout=org.apache.log4j.PatternLayout\\n\\n# Pattern format: Date LogLevel LoggerName LogMessage\\nlog4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\n# Debugging Pattern format\\n#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n\\n\\n\\n#\\n# console\\n# Add \\\"console\\\" to rootlogger above if you want to use this\\n#\\n\\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\\nlog4j.appender.console.target=System.err\\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n\\n\\n#\\n# TaskLog Appender\\n#\\n\\n#Default values\\nhadoop.tasklog.taskid=null\\nhadoop.tasklog.iscleanup=false\\nhadoop.tasklog.noKeepSplits=4\\nhadoop.tasklog.totalLogFileSize=100\\nhadoop.tasklog.purgeLogSplits=true\\nhadoop.tasklog.logsRetainHours=12\\n\\nlog4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender\\nlog4j.appender.TLA.taskId=${hadoop.tasklog.taskid}\\nlog4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}\\nlog4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}\\n\\nlog4j.appender.TLA.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\n\\n#\\n#Security audit appender\\n#\\nhadoop.security.logger=INFO,console\\nhadoop.security.log.maxfilesize=256MB\\nhadoop.security.log.maxbackupindex=20\\nlog4j.category.SecurityLogger=${hadoop.security.logger}\\nhadoop.security.log.file=SecurityAuth.audit\\nlog4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}\\nlog4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\nlog4j.appender.DRFAS.DatePattern=.yyyy-MM-dd\\n\\nlog4j.appender.RFAS=org.apache.log4j.RollingFileAppender\\nlog4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}\\nlog4j.appender.RFAS.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\nlog4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}\\nlog4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}\\n\\n#\\n# hdfs audit logging\\n#\\nhdfs.audit.logger=INFO,console\\nlog4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}\\nlog4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false\\nlog4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log\\nlog4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n\\nlog4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd\\n\\n#\\n# mapred audit logging\\n#\\nmapred.audit.logger=INFO,console\\nlog4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}\\nlog4j.additivity.org.apache.hadoop.mapred.AuditLogger=false\\nlog4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log\\nlog4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n\\nlog4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd\\n\\n#\\n# Rolling File Appender\\n#\\n\\nlog4j.appender.RFA=org.apache.log4j.RollingFileAppender\\nlog4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}\\n\\n# Logfile size and and 30-day backups\\nlog4j.appender.RFA.MaxFileSize=256MB\\nlog4j.appender.RFA.MaxBackupIndex=10\\n\\nlog4j.appender.RFA.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n\\nlog4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n\\n\\n\\n# Custom Logging levels\\n\\nhadoop.metrics.log.level=INFO\\n#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG\\n#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG\\n#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG\\nlog4j.logger.org.apache.hadoop.metrics2=${hadoop.metrics.log.level}\\n\\n# Jets3t library\\nlog4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR\\n\\n#\\n# Null Appender\\n# Trap security logger on the hadoop client side\\n#\\nlog4j.appender.NullAppender=org.apache.log4j.varia.NullAppender\\n\\n#\\n# Event Counter Appender\\n# Sends counts of logging messages at different severity levels to Hadoop Metrics.\\n#\\nlog4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter\\n\\n# Removes \\\"deprecated\\\" messages\\nlog4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN\\n\\n#\\n# HDFS block state change log from block manager\\n#\\n# Uncomment the following to suppress normal block state change\\n# messages from BlockManager in NameNode.\\n#log4j.logger.BlockStateChange=WARN\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hdfs-site\": {\n",
      "        \"dfs.replication.max\": \"50\",\n",
      "        \"dfs.datanode.data.dir.perm\": \"750\",\n",
      "        \"dfs.nfs3.dump.dir\": \"/tmp/.hdfs-nfs\",\n",
      "        \"dfs.datanode.max.transfer.threads\": \"1024\",\n",
      "        \"dfs.datanode.address\": \"0.0.0.0:50010\",\n",
      "        \"dfs.heartbeat.interval\": \"3\",\n",
      "        \"dfs.https.port\": \"50470\",\n",
      "        \"dfs.hosts.exclude\": \"/etc/hadoop/conf/dfs.exclude\",\n",
      "        \"dfs.namenode.checkpoint.period\": \"21600\",\n",
      "        \"dfs.datanode.max.xcievers\": \"1024\",\n",
      "        \"dfs.datanode.balance.bandwidthPerSec\": \"6250000\",\n",
      "        \"dfs.namenode.http-address\": \"%HOSTGROUP::host_group_1%:50070\",\n",
      "        \"dfs.blocksize\": \"134217728\",\n",
      "        \"dfs.datanode.ipc.address\": \"0.0.0.0:8010\",\n",
      "        \"dfs.datanode.data.dir\": \"/hadoop/hdfs/data\",\n",
      "        \"dfs.namenode.safemode.threshold-pct\": \"1.0f\",\n",
      "        \"dfs.journalnode.http-address\": \"0.0.0.0:8480\",\n",
      "        \"dfs.permissions.enabled\": \"true\",\n",
      "        \"dfs.namenode.https-address\": \"%HOSTGROUP::host_group_1%:50470\",\n",
      "        \"dfs.blockreport.initialDelay\": \"120\",\n",
      "        \"dfs.replication\": \"1\",\n",
      "        \"dfs.permissions.superusergroup\": \"hdfs\",\n",
      "        \"dfs.datanode.http.address\": \"0.0.0.0:50075\",\n",
      "        \"dfs.domain.socket.path\": \"/var/lib/hadoop-hdfs/dn_socket\",\n",
      "        \"dfs.webhdfs.enabled\": \"true\",\n",
      "        \"dfs.namenode.startup.delay.block.deletion.sec\": \"3600\",\n",
      "        \"dfs.datanode.https.address\": \"0.0.0.0:50475\",\n",
      "        \"dfs.cluster.administrators\": \" hdfs\",\n",
      "        \"dfs.namenode.stale.datanode.interval\": \"30000\",\n",
      "        \"dfs.namenode.avoid.write.stale.datanode\": \"true\",\n",
      "        \"dfs.namenode.handler.count\": \"100\",\n",
      "        \"dfs.datanode.du.reserved\": \"1073741824\",\n",
      "        \"dfs.http.policy\": \"HTTP_ONLY\",\n",
      "        \"dfs.namenode.checkpoint.txns\": \"1000000\",\n",
      "        \"fs.permissions.umask-mode\": \"022\",\n",
      "        \"dfs.block.size\": \"34217472\",\n",
      "        \"dfs.journalnode.edits.dir\": \"/hadoop/hdfs/journalnode\",\n",
      "        \"dfs.namenode.secondary.http-address\": \"%HOSTGROUP::host_group_1%:50090\",\n",
      "        \"dfs.datanode.failed.volumes.tolerated\": \"0\",\n",
      "        \"dfs.namenode.name.dir.restore\": \"true\",\n",
      "        \"dfs.client.read.shortcircuit.streams.cache.size\": \"4096\",\n",
      "        \"dfs.namenode.checkpoint.edits.dir\": \"${dfs.namenode.checkpoint.dir}\",\n",
      "        \"dfs.namenode.checkpoint.dir\": \"/hadoop/hdfs/namesecondary\",\n",
      "        \"dfs.nfs.exports.allowed.hosts\": \"* rw\",\n",
      "        \"dfs.support.append\": \"true\",\n",
      "        \"dfs.namenode.write.stale.datanode.ratio\": \"1.0f\",\n",
      "        \"dfs.namenode.name.dir\": \"/hadoop/hdfs/namenode\",\n",
      "        \"dfs.block.access.token.enable\": \"false\",\n",
      "        \"dfs.client.read.shortcircuit\": \"true\",\n",
      "        \"dfs.namenode.avoid.read.stale.datanode\": \"true\",\n",
      "        \"dfs.namenode.accesstime.precision\": \"3600000\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hive-env\": {\n",
      "        \"hive_ambari_database\": \"MySQL\",\n",
      "        \"hcat_user\": \"hcat\",\n",
      "        \"hive_pid_dir\": \"/var/run/hive\",\n",
      "        \"hcat_pid_dir\": \"/var/run/webhcat\",\n",
      "        \"hive_user\": \"hive\",\n",
      "        \"hive_database\": \"New MySQL Database\",\n",
      "        \"hcat_log_dir\": \"/var/log/webhcat\",\n",
      "        \"hive_database_type\": \"mysql\",\n",
      "        \"content\": \"\\n if [ \\\"$SERVICE\\\" = \\\"cli\\\" ]; then\\n   if [ -z \\\"$DEBUG\\\" ]; then\\n     export HADOOP_OPTS=\\\"$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit\\\"\\n   else\\n     export HADOOP_OPTS=\\\"$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit\\\"\\n   fi\\n fi\\n\\n# The heap size of the jvm stared by hive shell script can be controlled via:\\n\\nexport HADOOP_HEAPSIZE=\\\"{{hive_heapsize}}\\\"\\nexport HADOOP_CLIENT_OPTS=\\\"-Xmx${HADOOP_HEAPSIZE}m $HADOOP_CLIENT_OPTS\\\"\\n\\n# Larger heap size may be required when running queries over large number of files or partitions.\\n# By default hive shell scripts use a heap size of 256 (MB).  Larger heap size would also be\\n# appropriate for hive server (hwi etc).\\n\\n\\n# Set HADOOP_HOME to point to a specific hadoop install directory\\nHADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n\\n# Hive Configuration Directory can be controlled by:\\nexport HIVE_CONF_DIR={{hive_config_dir}}\\n\\n# Folder containing extra libraries required for hive compilation/execution can be controlled by:\\nif [ \\\"${HIVE_AUX_JARS_PATH}\\\" != \\\"\\\" ]; then\\n  export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}\\nelif [ -d \\\"/usr/hdp/current/hive-webhcat/share/hcatalog\\\" ]; then\\n  export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog\\nfi\\n\\nexport METASTORE_PORT={{hive_metastore_port}}\\n    \",\n",
      "        \"webhcat_user\": \"hcat\",\n",
      "        \"hive_dbroot\": \"/usr/lib/hive/lib/\",\n",
      "        \"hive_database_name\": \"hive\",\n",
      "        \"hive_log_dir\": \"/var/log/hive\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hive-exec-log4j\": {\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#     http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n\\n# Define some default values that can be overridden by system properties\\n\\nhive.log.threshold=ALL\\nhive.root.logger=INFO,FA\\nhive.log.dir=${java.io.tmpdir}/${user.name}\\nhive.query.id=hadoop\\nhive.log.file=${hive.query.id}.log\\n\\n# Define the root logger to the system property \\\"hadoop.root.logger\\\".\\nlog4j.rootLogger=${hive.root.logger}, EventCounter\\n\\n# Logging Threshold\\nlog4j.threshhold=${hive.log.threshold}\\n\\n#\\n# File Appender\\n#\\n\\nlog4j.appender.FA=org.apache.log4j.FileAppender\\nlog4j.appender.FA.File=${hive.log.dir}/${hive.log.file}\\nlog4j.appender.FA.layout=org.apache.log4j.PatternLayout\\n\\n# Pattern format: Date LogLevel LoggerName LogMessage\\n#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\n# Debugging Pattern format\\nlog4j.appender.FA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n\\n\\n\\n#\\n# console\\n# Add \\\"console\\\" to rootlogger above if you want to use this\\n#\\n\\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\\nlog4j.appender.console.target=System.err\\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n\\n\\n#custom logging levels\\n#log4j.logger.xxx=DEBUG\\n\\n#\\n# Event Counter Appender\\n# Sends counts of logging messages at different severity levels to Hadoop Metrics.\\n#\\nlog4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter\\n\\n\\nlog4j.category.DataNucleus=ERROR,FA\\nlog4j.category.Datastore=ERROR,FA\\nlog4j.category.Datastore.Schema=ERROR,FA\\nlog4j.category.JPOX.Datastore=ERROR,FA\\nlog4j.category.JPOX.Plugin=ERROR,FA\\nlog4j.category.JPOX.MetaData=ERROR,FA\\nlog4j.category.JPOX.Query=ERROR,FA\\nlog4j.category.JPOX.General=ERROR,FA\\nlog4j.category.JPOX.Enhancer=ERROR,FA\\n\\n\\n# Silence useless ZK logs\\nlog4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,FA\\nlog4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,FA\\n\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hive-log4j\": {\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#     http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n\\n# Define some default values that can be overridden by system properties\\nhive.log.threshold=ALL\\nhive.root.logger=INFO,DRFA\\nhive.log.dir=${java.io.tmpdir}/${user.name}\\nhive.log.file=hive.log\\n\\n# Define the root logger to the system property \\\"hadoop.root.logger\\\".\\nlog4j.rootLogger=${hive.root.logger}, EventCounter\\n\\n# Logging Threshold\\nlog4j.threshold=${hive.log.threshold}\\n\\n#\\n# Daily Rolling File Appender\\n#\\n# Use the PidDailyerRollingFileAppend class instead if you want to use separate log files\\n# for different CLI session.\\n#\\n# log4j.appender.DRFA=org.apache.hadoop.hive.ql.log.PidDailyRollingFileAppender\\n\\nlog4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender\\n\\nlog4j.appender.DRFA.File=${hive.log.dir}/${hive.log.file}\\n\\n# Rollver at midnight\\nlog4j.appender.DRFA.DatePattern=.yyyy-MM-dd\\n\\n# 30-day backup\\n#log4j.appender.DRFA.MaxBackupIndex=30\\nlog4j.appender.DRFA.layout=org.apache.log4j.PatternLayout\\n\\n# Pattern format: Date LogLevel LoggerName LogMessage\\n#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n\\n# Debugging Pattern format\\nlog4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n\\n\\n\\n#\\n# console\\n# Add \\\"console\\\" to rootlogger above if you want to use this\\n#\\n\\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\\nlog4j.appender.console.target=System.err\\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n\\nlog4j.appender.console.encoding=UTF-8\\n\\n#custom logging levels\\n#log4j.logger.xxx=DEBUG\\n\\n#\\n# Event Counter Appender\\n# Sends counts of logging messages at different severity levels to Hadoop Metrics.\\n#\\nlog4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter\\n\\n\\nlog4j.category.DataNucleus=ERROR,DRFA\\nlog4j.category.Datastore=ERROR,DRFA\\nlog4j.category.Datastore.Schema=ERROR,DRFA\\nlog4j.category.JPOX.Datastore=ERROR,DRFA\\nlog4j.category.JPOX.Plugin=ERROR,DRFA\\nlog4j.category.JPOX.MetaData=ERROR,DRFA\\nlog4j.category.JPOX.Query=ERROR,DRFA\\nlog4j.category.JPOX.General=ERROR,DRFA\\nlog4j.category.JPOX.Enhancer=ERROR,DRFA\\n\\n\\n# Silence useless ZK logs\\nlog4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,DRFA\\nlog4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,DRFA\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"kafka-broker\": {\n",
      "        \"kafka.ganglia.metrics.group\": \"kafka\",\n",
      "        \"controlled.shutdown.enable\": \"false\",\n",
      "        \"num.replica.fetchers\": \"1\",\n",
      "        \"log.retention.bytes\": \"-1\",\n",
      "        \"replica.lag.time.max.ms\": \"10000\",\n",
      "        \"controlled.shutdown.max.retries\": \"3\",\n",
      "        \"kafka.ganglia.metrics.port\": \"8671\",\n",
      "        \"kafka.metrics.reporters\": \"kafka.ganglia.KafkaGangliaMetricsReporter\",\n",
      "        \"zookeeper.sync.time.ms\": \"2000\",\n",
      "        \"num.network.threads\": \"3\",\n",
      "        \"replica.socket.receive.buffer.bytes\": \"65536\",\n",
      "        \"zookeeper.connection.timeout.ms\": \"6000\",\n",
      "        \"log.flush.scheduler.interval.ms\": \"3000\",\n",
      "        \"zookeeper.session.timeout.ms\": \"6000\",\n",
      "        \"replica.lag.max.messages\": \"4000\",\n",
      "        \"port\": \"6667\",\n",
      "        \"log.index.interval.bytes\": \"4096\",\n",
      "        \"num.io.threads\": \"8\",\n",
      "        \"socket.send.buffer.bytes\": \"102400\",\n",
      "        \"num.partitions\": \"1\",\n",
      "        \"log.roll.hours\": \"168\",\n",
      "        \"replica.fetch.wait.max.ms\": \"500\",\n",
      "        \"controller.message.queue.size\": \"10\",\n",
      "        \"socket.receive.buffer.bytes\": \"102400\",\n",
      "        \"replica.socket.timeout.ms\": \"30000\",\n",
      "        \"default.replication.factor\": \"1\",\n",
      "        \"kafka.ganglia.metrics.host\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"log.flush.interval.ms\": \"3000\",\n",
      "        \"log.dirs\": \"/kafka-logs\",\n",
      "        \"log.flush.interval.messages\": \"10000\",\n",
      "        \"replica.fetch.min.bytes\": \"1\",\n",
      "        \"replica.high.watermark.checkpoint.interval.ms\": \"5000\",\n",
      "        \"queued.max.requests\": \"500\",\n",
      "        \"log.retention.hours\": \"168\",\n",
      "        \"fetch.purgatory.purge.interval.requests\": \"10000\",\n",
      "        \"controlled.shutdown.retry.backoff.ms\": \"5000\",\n",
      "        \"log.index.size.max.bytes\": \"10485760\",\n",
      "        \"controller.socket.timeout.ms\": \"30000\",\n",
      "        \"replica.fetch.max.bytes\": \"1048576\",\n",
      "        \"log.cleanup.interval.mins\": \"10\",\n",
      "        \"zookeeper.connect\": \"%HOSTGROUP::host_group_1%:2181\",\n",
      "        \"kafka.ganglia.metrics.reporter.enabled\": \"true\",\n",
      "        \"message.max.bytes\": \"1000000\",\n",
      "        \"auto.create.topics.enable\": \"true\",\n",
      "        \"socket.request.max.bytes\": \"104857600\",\n",
      "        \"producer.purgatory.purge.interval.requests\": \"10000\",\n",
      "        \"log.segment.bytes\": \"1073741824\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"kafka-env\": {\n",
      "        \"kafka_log_dir\": \"/var/log/kafka\",\n",
      "        \"kafka_pid_dir\": \"/var/run/kafka\",\n",
      "        \"content\": \"\\n#!/bin/bash\\n\\n# Set KAFKA specific environment variables here.\\n\\n# The java implementation to use.\\nexport JAVA_HOME={{java64_home}}\\nexport PATH=$PATH:$JAVA_HOME/bin\\n    \",\n",
      "        \"kafka_user\": \"kafka\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"kafka-log4j\": {\n",
      "        \"content\": \"\\n#\\n#\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#   http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing,\\n# software distributed under the License is distributed on an\\n# \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\\n# KIND, either express or implied.  See the License for the\\n# specific language governing permissions and limitations\\n# under the License.\\n#\\n#\\n#\\nkafka.logs.dir=logs\\n\\nlog4j.rootLogger=INFO, stdout\\n\\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\nlog4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log\\nlog4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\nlog4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log\\nlog4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\nlog4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log\\nlog4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\nlog4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log\\nlog4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\nlog4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log\\nlog4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n\\n\\n# Turn on all our debugging info\\n#log4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG, kafkaAppender\\n#log4j.logger.kafka.client.ClientUtils=DEBUG, kafkaAppender\\n#log4j.logger.kafka.perf=DEBUG, kafkaAppender\\n#log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG, kafkaAppender\\n#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG\\nlog4j.logger.kafka=INFO, kafkaAppender\\nlog4j.logger.kafka.network.RequestChannel$=WARN, requestAppender\\nlog4j.additivity.kafka.network.RequestChannel$=false\\n\\n#log4j.logger.kafka.network.Processor=TRACE, requestAppender\\n#log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender\\n#log4j.additivity.kafka.server.KafkaApis=false\\nlog4j.logger.kafka.request.logger=WARN, requestAppender\\nlog4j.additivity.kafka.request.logger=false\\n\\nlog4j.logger.kafka.controller=TRACE, controllerAppender\\nlog4j.additivity.kafka.controller=false\\n\\nlog4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender\\nlog4j.additivity.kafka.log.LogCleaner=false\\n\\nlog4j.logger.state.change.logger=TRACE, stateChangeAppender\\nlog4j.additivity.state.change.logger=false\\n\\n   \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"mapred-env\": {\n",
      "        \"jobhistory_heapsize\": \"250\",\n",
      "        \"content\": \"\\n# export JAVA_HOME=/home/y/libexec/jdk1.6.0/\\n\\nexport HADOOP_JOB_HISTORYSERVER_HEAPSIZE={{jobhistory_heapsize}}\\n\\nexport HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA\\n\\n#export HADOOP_JOB_HISTORYSERVER_OPTS=\\n#export HADOOP_MAPRED_LOG_DIR=\\\"\\\" # Where log files are stored.  $HADOOP_MAPRED_HOME/logs by default.\\n#export HADOOP_JHS_LOGGER=INFO,RFA # Hadoop JobSummary logger.\\n#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.\\n#export HADOOP_MAPRED_IDENT_STRING= #A string representing this instance of hadoop. $USER by default\\n#export HADOOP_MAPRED_NICENESS= #The scheduling priority for daemons. Defaults to 0.\\nexport HADOOP_OPTS=\\\"-Dhdp.version=$HDP_VERSION $HADOOP_OPTS\\\"\\n    \",\n",
      "        \"mapred_log_dir_prefix\": \"/var/log/hadoop-mapreduce\",\n",
      "        \"mapred_user\": \"mapred\",\n",
      "        \"mapred_pid_dir_prefix\": \"/var/run/hadoop-mapreduce\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"mapred-site\": {\n",
      "        \"mapreduce.jobhistory.intermediate-done-dir\": \"/mr-history/tmp\",\n",
      "        \"mapreduce.cluster.administrators\": \" hadoop\",\n",
      "        \"mapreduce.map.log.level\": \"INFO\",\n",
      "        \"mapreduce.job.reduce.slowstart.completedmaps\": \"0.05\",\n",
      "        \"mapreduce.reduce.shuffle.input.buffer.percent\": \"0.7\",\n",
      "        \"mapreduce.map.sort.spill.percent\": \"0.7\",\n",
      "        \"mapreduce.admin.reduce.child.java.opts\": \"-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}\",\n",
      "        \"yarn.app.mapreduce.am.resource.mb\": \"250\",\n",
      "        \"mapreduce.jobhistory.address\": \"%HOSTGROUP::host_group_1%:10020\",\n",
      "        \"mapred.job.reduce.memory.mb\": \"250\",\n",
      "        \"mapreduce.admin.user.env\": \"LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64\",\n",
      "        \"mapreduce.reduce.memory.mb\": \"250\",\n",
      "        \"yarn.app.mapreduce.am.command-opts\": \"-Xmx200m\",\n",
      "        \"mapreduce.map.memory.mb\": \"250\",\n",
      "        \"mapreduce.shuffle.port\": \"13562\",\n",
      "        \"mapreduce.reduce.input.buffer.percent\": \"0.0\",\n",
      "        \"mapreduce.reduce.shuffle.parallelcopies\": \"30\",\n",
      "        \"mapreduce.jobhistory.done-dir\": \"/mr-history/done\",\n",
      "        \"mapreduce.reduce.shuffle.fetch.retry.timeout-ms\": \"30000\",\n",
      "        \"mapreduce.reduce.shuffle.merge.percent\": \"0.66\",\n",
      "        \"mapred.child.java.opts\": \"-Xmx200m\",\n",
      "        \"mapreduce.output.fileoutputformat.compress.type\": \"BLOCK\",\n",
      "        \"yarn.app.mapreduce.am.admin-command-opts\": \"-Dhdp.version=${hdp.version}\",\n",
      "        \"mapreduce.output.fileoutputformat.compress\": \"false\",\n",
      "        \"mapreduce.application.classpath\": \"$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure\",\n",
      "        \"mapreduce.reduce.speculative\": \"false\",\n",
      "        \"mapreduce.application.framework.path\": \"/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework\",\n",
      "        \"mapreduce.map.output.compress\": \"false\",\n",
      "        \"mapreduce.task.io.sort.mb\": \"64\",\n",
      "        \"mapreduce.job.emit-timeline-data\": \"false\",\n",
      "        \"mapreduce.map.java.opts\": \"-Xmx200m\",\n",
      "        \"mapreduce.framework.name\": \"yarn\",\n",
      "        \"mapreduce.reduce.shuffle.fetch.retry.interval-ms\": \"1000\",\n",
      "        \"mapreduce.reduce.java.opts\": \"-Xmx200m\",\n",
      "        \"mapreduce.am.max-attempts\": \"2\",\n",
      "        \"mapreduce.reduce.log.level\": \"INFO\",\n",
      "        \"yarn.app.mapreduce.am.log.level\": \"INFO\",\n",
      "        \"yarn.app.mapreduce.am.staging-dir\": \"/user\",\n",
      "        \"io.sort.mb\": \"64\",\n",
      "        \"mapreduce.map.speculative\": \"false\",\n",
      "        \"mapreduce.admin.map.child.java.opts\": \"-server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}\",\n",
      "        \"mapred.job.map.memory.mb\": \"250\",\n",
      "        \"mapreduce.reduce.shuffle.fetch.retry.enabled\": \"1\",\n",
      "        \"mapreduce.jobhistory.webapp.address\": \"%HOSTGROUP::host_group_1%:19888\",\n",
      "        \"mapreduce.jobhistory.bind-host\": \"0.0.0.0\",\n",
      "        \"mapreduce.task.io.sort.factor\": \"100\",\n",
      "        \"mapreduce.task.timeout\": \"300000\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"nagios-env\": {\n",
      "        \"nagios_web_login\": \"nagiosadmin\",\n",
      "        \"nagios_user\": \"nagios\",\n",
      "        \"nagios_group\": \"nagios\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"oozie-env\": {\n",
      "        \"oozie_heapsize\": \"2048\",\n",
      "        \"oozie_pid_dir\": \"/var/run/oozie\",\n",
      "        \"oozie_user\": \"oozie\",\n",
      "        \"oozie_log_dir\": \"/var/log/oozie\",\n",
      "        \"oozie_derby_database\": \"Derby\",\n",
      "        \"oozie_admin_port\": \"11001\",\n",
      "        \"content\": \"\\n#!/bin/bash\\n\\nif [ -d \\\"/usr/lib/bigtop-tomcat\\\" ]; then\\n  export OOZIE_CONFIG=${OOZIE_CONFIG:-/etc/oozie/conf}\\n  export CATALINA_BASE=${CATALINA_BASE:-{{oozie_server_dir}}}\\n  export CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}\\n  export OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat\\nfi\\n\\n#Set JAVA HOME\\nexport JAVA_HOME={{java_home}}\\n\\nexport JRE_HOME=${JAVA_HOME}\\n\\n# Set Oozie specific environment variables here.\\n\\n# Settings for the Embedded Tomcat that runs Oozie\\n# Java System properties for Oozie should be specified in this variable\\n#\\n# export CATALINA_OPTS=\\n\\n# Oozie configuration file to load from Oozie configuration directory\\n#\\n# export OOZIE_CONFIG_FILE=oozie-site.xml\\n\\n# Oozie logs directory\\n#\\nexport OOZIE_LOG={{oozie_log_dir}}\\n\\n# Oozie pid directory\\n#\\nexport CATALINA_PID={{pid_file}}\\n\\n#Location of the data for oozie\\nexport OOZIE_DATA={{oozie_data_dir}}\\n\\n# Oozie Log4J configuration file to load from Oozie configuration directory\\n#\\n# export OOZIE_LOG4J_FILE=oozie-log4j.properties\\n\\n# Reload interval of the Log4J configuration file, in seconds\\n#\\n# export OOZIE_LOG4J_RELOAD=10\\n\\n# The port Oozie server runs\\n#\\nexport OOZIE_HTTP_PORT={{oozie_server_port}}\\n\\n# The admin port Oozie server runs\\n#\\nexport OOZIE_ADMIN_PORT={{oozie_server_admin_port}}\\n\\n# The host name Oozie server runs on\\n#\\n# export OOZIE_HTTP_HOSTNAME=`hostname -f`\\n\\n# The base URL for callback URLs to Oozie\\n#\\n# export OOZIE_BASE_URL=\\\"http://${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie\\\"\\nexport JAVA_LIBRARY_PATH={{hadoop_lib_home}}/native/Linux-amd64-64\\n\\n# At least 1 minute of retry time to account for server downtime during\\n# upgrade/downgrade\\nexport OOZIE_CLIENT_OPTS=\\\"${OOZIE_CLIENT_OPTS} -Doozie.connection.retry.count=5 \\\"\\n\\n# This is needed so that Oozie does not run into OOM or GC Overhead limit\\n# exceeded exceptions. If the oozie server is handling large number of\\n# workflows/coordinator jobs, the memory settings may need to be revised\\nexport CATALINA_OPTS=\\\"${CATALINA_OPTS} -Xmx2048m -XX:MaxPermSize=256m \\\"\\n    \",\n",
      "        \"oozie_data_dir\": \"/hadoop/oozie/data\",\n",
      "        \"oozie_database\": \"New Derby Database\",\n",
      "        \"oozie_permsize\": \"256\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"oozie-log4j\": {\n",
      "        \"content\": \"\\n#\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#    http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License. See accompanying LICENSE file.\\n#\\n\\n# If the Java System property 'oozie.log.dir' is not defined at Oozie start up time\\n# XLogService sets its value to '${oozie.home}/logs'\\n\\nlog4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH\\nlog4j.appender.oozie.File=${oozie.log.dir}/oozie.log\\nlog4j.appender.oozie.Append=true\\nlog4j.appender.oozie.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - SERVER[${oozie.instance.id}] %m%n\\n\\nlog4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.oozieops.DatePattern='.'yyyy-MM-dd\\nlog4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log\\nlog4j.appender.oozieops.Append=true\\nlog4j.appender.oozieops.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n\\n\\nlog4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd\\nlog4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log\\nlog4j.appender.oozieinstrumentation.Append=true\\nlog4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n\\n\\nlog4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd\\nlog4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log\\nlog4j.appender.oozieaudit.Append=true\\nlog4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n\\n\\nlog4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender\\nlog4j.appender.openjpa.DatePattern='.'yyyy-MM-dd\\nlog4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log\\nlog4j.appender.openjpa.Append=true\\nlog4j.appender.openjpa.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n\\n\\nlog4j.logger.openjpa=INFO, openjpa\\nlog4j.logger.oozieops=INFO, oozieops\\nlog4j.logger.oozieinstrumentation=ALL, oozieinstrumentation\\nlog4j.logger.oozieaudit=ALL, oozieaudit\\nlog4j.logger.org.apache.oozie=INFO, oozie\\nlog4j.logger.org.apache.hadoop=WARN, oozie\\nlog4j.logger.org.mortbay=WARN, oozie\\nlog4j.logger.org.hsqldb=WARN, oozie\\nlog4j.logger.org.apache.hadoop.security.authentication.server=INFO, oozie\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"oozie-site\": {\n",
      "        \"oozie.service.ActionService.executor.ext.classes\": \"\\n      org.apache.oozie.action.email.EmailActionExecutor,\\n      org.apache.oozie.action.hadoop.HiveActionExecutor,\\n      org.apache.oozie.action.hadoop.ShellActionExecutor,\\n      org.apache.oozie.action.hadoop.SqoopActionExecutor,\\n      org.apache.oozie.action.hadoop.DistcpActionExecutor\\n    \",\n",
      "        \"oozie.service.ELService.ext.functions.coord-sla-submit\": \"\\n      instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.service.AuthorizationService.security.enabled\": \"true\",\n",
      "        \"oozie.service.ProxyUserService.proxyuser.hue.groups\": \"*\",\n",
      "        \"oozie.services\": \"\\n      org.apache.oozie.service.SchedulerService,\\n      org.apache.oozie.service.InstrumentationService,\\n      org.apache.oozie.service.MemoryLocksService,\\n      org.apache.oozie.service.UUIDService,\\n      org.apache.oozie.service.ELService,\\n      org.apache.oozie.service.AuthorizationService,\\n      org.apache.oozie.service.UserGroupInformationService,\\n      org.apache.oozie.service.HadoopAccessorService,\\n      org.apache.oozie.service.JobsConcurrencyService,\\n      org.apache.oozie.service.URIHandlerService,\\n      org.apache.oozie.service.DagXLogInfoService,\\n      org.apache.oozie.service.SchemaService,\\n      org.apache.oozie.service.LiteWorkflowAppService,\\n      org.apache.oozie.service.JPAService,\\n      org.apache.oozie.service.StoreService,\\n      org.apache.oozie.service.CoordinatorStoreService,\\n      org.apache.oozie.service.SLAStoreService,\\n      org.apache.oozie.service.DBLiteWorkflowStoreService,\\n      org.apache.oozie.service.CallbackService,\\n      org.apache.oozie.service.ShareLibService,\\n      org.apache.oozie.service.CallableQueueService,\\n      org.apache.oozie.service.ActionService,\\n      org.apache.oozie.service.ActionCheckerService,\\n      org.apache.oozie.service.RecoveryService,\\n      org.apache.oozie.service.PurgeService,\\n      org.apache.oozie.service.CoordinatorEngineService,\\n      org.apache.oozie.service.BundleEngineService,\\n      org.apache.oozie.service.DagEngineService,\\n      org.apache.oozie.service.CoordMaterializeTriggerService,\\n      org.apache.oozie.service.StatusTransitService,\\n      org.apache.oozie.service.PauseTransitService,\\n      org.apache.oozie.service.GroupsService,\\n      org.apache.oozie.service.ProxyUserService,\\n      org.apache.oozie.service.XLogStreamingService,\\n      org.apache.oozie.service.JvmPauseMonitorService\\n    \",\n",
      "        \"oozie.service.ELService.ext.functions.coord-job-submit-instances\": \"\\n      now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo,\\n      today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo,\\n      yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo,\\n      currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo,\\n      lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo,\\n      currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo,\\n      lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo,\\n      formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,\\n      latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,\\n      future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo\\n    \",\n",
      "        \"oozie.service.ELService.ext.functions.coord-action-create\": \"\\n      now=org.apache.oozie.extensions.OozieELExtensions#ph2_now,\\n      today=org.apache.oozie.extensions.OozieELExtensions#ph2_today,\\n      yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday,\\n      currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth,\\n      lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth,\\n      currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear,\\n      lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear,\\n      latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,\\n      future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,\\n      formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.systemmode\": \"NORMAL\",\n",
      "        \"oozie.service.CallableQueueService.threads\": \"10\",\n",
      "        \"oozie.service.JPAService.create.db.schema\": \"false\",\n",
      "        \"oozie.service.JPAService.jdbc.driver\": \"org.apache.derby.jdbc.EmbeddedDriver\",\n",
      "        \"oozie.service.ELService.ext.functions.coord-sla-create\": \"\\n      instanceTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.db.schema.name\": \"oozie\",\n",
      "        \"oozie.service.PurgeService.older.than\": \"30\",\n",
      "        \"oozie.credentials.credentialclasses\": \"hcat=org.apache.oozie.action.hadoop.HCatCredentials\",\n",
      "        \"oozie.service.ProxyUserService.proxyuser.falcon.hosts\": \"*\",\n",
      "        \"oozie.service.CallableQueueService.callable.concurrency\": \"3\",\n",
      "        \"oozie.service.coord.check.maximum.frequency\": \"false\",\n",
      "        \"oozie.service.WorkflowAppService.system.libpath\": \"/user/${user.name}/share/lib\",\n",
      "        \"oozie.service.SchemaService.wf.ext.schemas\": \"shell-action-0.1.xsd,shell-action-0.2.xsd,shell-action-0.3.xsd,email-action-0.1.xsd,email-action-0.2.xsd,hive-action-0.2.xsd,hive-action-0.3.xsd,hive-action-0.4.xsd,hive-action-0.5.xsd,sqoop-action-0.2.xsd,sqoop-action-0.3.xsd,sqoop-action-0.4.xsd,ssh-action-0.1.xsd,ssh-action-0.2.xsd,distcp-action-0.1.xsd,distcp-action-0.2.xsd,oozie-sla-0.1.xsd,oozie-sla-0.2.xsd\",\n",
      "        \"use.system.libpath.for.mapreduce.and.pig.jobs\": \"false\",\n",
      "        \"oozie.service.coord.push.check.requeue.interval\": \"30000\",\n",
      "        \"oozie.service.ProxyUserService.proxyuser.hue.hosts\": \"*\",\n",
      "        \"oozie.authentication.simple.anonymous.allowed\": \"true\",\n",
      "        \"oozie.authentication.type\": \"simple\",\n",
      "        \"oozie.service.PurgeService.purge.interval\": \"3600\",\n",
      "        \"oozie.authentication.kerberos.name.rules\": \"\\n      RULE:[2:$1@$0]([jt]t@.*TODO-KERBEROS-DOMAIN)s/.*/TODO-MAPREDUSER/\\n      RULE:[2:$1@$0]([nd]n@.*TODO-KERBEROS-DOMAIN)s/.*/TODO-HDFSUSER/\\n      RULE:[2:$1@$0](hm@.*TODO-KERBEROS-DOMAIN)s/.*/TODO-HBASE-USER/\\n      RULE:[2:$1@$0](rs@.*TODO-KERBEROS-DOMAIN)s/.*/TODO-HBASE-USER/\\n      DEFAULT\\n    \",\n",
      "        \"oozie.service.HadoopAccessorService.hadoop.configurations\": \"*=/etc/hadoop/conf\",\n",
      "        \"oozie.service.HadoopAccessorService.supported.filesystems\": \"*\",\n",
      "        \"oozie.service.HadoopAccessorService.kerberos.enabled\": \"false\",\n",
      "        \"oozie.service.JPAService.jdbc.username\": \"oozie\",\n",
      "        \"oozie.service.ELService.ext.functions.coord-action-create-inst\": \"\\n      now=org.apache.oozie.extensions.OozieELExtensions#ph2_now_inst,\\n      today=org.apache.oozie.extensions.OozieELExtensions#ph2_today_inst,\\n      yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday_inst,\\n      currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth_inst,\\n      lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth_inst,\\n      currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear_inst,\\n      lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear_inst,\\n      latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,\\n      future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,\\n      formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.system.id\": \"oozie-${user.name}\",\n",
      "        \"oozie.service.URIHandlerService.uri.handlers\": \"org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler\",\n",
      "        \"oozie.services.ext\": \"org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService\\n    \",\n",
      "        \"oozie.base.url\": \"http://%HOSTGROUP::host_group_1%:11000/oozie\",\n",
      "        \"oozie.service.ProxyUserService.proxyuser.falcon.groups\": \"*\",\n",
      "        \"oozie.service.ELService.ext.functions.coord-action-start\": \"\\n      now=org.apache.oozie.extensions.OozieELExtensions#ph2_now,\\n      today=org.apache.oozie.extensions.OozieELExtensions#ph2_today,\\n      yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday,\\n      currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth,\\n      lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth,\\n      currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear,\\n      lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear,\\n      latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest,\\n      future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future,\\n      dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn,\\n      instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime,\\n      dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset,\\n      formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.service.coord.normal.default.timeout\": \"120\",\n",
      "        \"oozie.service.CallableQueueService.queue.size\": \"1000\",\n",
      "        \"oozie.service.ELService.ext.functions.coord-job-submit-data\": \"\\n      now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo,\\n      today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo,\\n      yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo,\\n      currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo,\\n      lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo,\\n      currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo,\\n      lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo,\\n      dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo,\\n      instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_wrap,\\n      formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,\\n      dateOffset=org.apache.oozie.coord.CoordELFunctions#ph1_coord_dateOffset_echo,\\n      user=org.apache.oozie.coord.CoordELFunctions#coord_user\\n    \",\n",
      "        \"oozie.service.JPAService.pool.max.active.conn\": \"10\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"pig-env\": {\n",
      "        \"content\": \"\\nJAVA_HOME={{java64_home}}\\nHADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n\\nif [ -d \\\"/usr/lib/tez\\\" ]; then\\n  PIG_OPTS=\\\"$PIG_OPTS -Dmapreduce.framework.name=yarn\\\"\\nfi\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"pig-log4j\": {\n",
      "        \"content\": \"\\n#\\n#\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#   http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing,\\n# software distributed under the License is distributed on an\\n# \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\\n# KIND, either express or implied.  See the License for the\\n# specific language governing permissions and limitations\\n# under the License.\\n#\\n#\\n#\\n\\n# ***** Set root logger level to DEBUG and its only appender to A.\\nlog4j.logger.org.apache.pig=info, A\\n\\n# ***** A is set to be a ConsoleAppender.\\nlog4j.appender.A=org.apache.log4j.ConsoleAppender\\n# ***** A uses PatternLayout.\\nlog4j.appender.A.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.A.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"pig-properties\": {\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#   http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing,\\n# software distributed under the License is distributed on an\\n# \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\\n# KIND, either express or implied.  See the License for the\\n# specific language governing permissions and limitations\\n# under the License.\\n\\n# Pig default configuration file. All values can be overwritten by pig.properties and command line arguments.\\n# see bin/pig -help\\n\\n# brief logging (no timestamps)\\nbrief=false\\n\\n# debug level, INFO is default\\ndebug=INFO\\n\\n# verbose print all log messages to screen (default to print only INFO and above to screen)\\nverbose=false\\n\\n# exectype local|mapreduce, mapreduce is default\\nexectype=mapreduce\\n\\n# Enable insertion of information about script into hadoop job conf \\npig.script.info.enabled=true\\n\\n# Do not spill temp files smaller than this size (bytes)\\npig.spill.size.threshold=5000000\\n\\n# EXPERIMENT: Activate garbage collection when spilling a file bigger than this size (bytes)\\n# This should help reduce the number of files being spilled.\\npig.spill.gc.activation.size=40000000\\n\\n# the following two parameters are to help estimate the reducer number\\npig.exec.reducers.bytes.per.reducer=1000000000\\npig.exec.reducers.max=999\\n\\n# Temporary location to store the intermediate data.\\npig.temp.dir=/tmp/\\n\\n# Threshold for merging FRJoin fragment files\\npig.files.concatenation.threshold=100\\npig.optimistic.files.concatenation=false;\\n\\npig.disable.counter=false\\n\\n# Avoid pig failures when multiple jobs write to the same location\\npig.location.check.strict=false\\n\\nhcat.bin=/usr/bin/hcat\\n\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"slider-env\": {\n",
      "        \"content\": \"\\n# Set Slider-specific environment variables here.\\n\\n# The only required environment variable is JAVA_HOME.  All others are\\n# optional.  When running a distributed configuration it is best to\\n# set JAVA_HOME in this file, so that it is correctly defined on\\n# remote nodes.\\n\\n# The java implementation to use.  Required.\\nexport JAVA_HOME={{java64_home}}\\n# The hadoop conf directory.  Optional as slider-client.xml can be edited to add properties.\\nexport HADOOP_CONF_DIR={{hadoop_conf_dir}}\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"slider-log4j\": {\n",
      "        \"content\": \"\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#     http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing, software\\n# distributed under the License is distributed on an \\\"AS IS\\\" BASIS,\\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\\n# See the License for the specific language governing permissions and\\n# limitations under the License.\\n\\n\\n# Define some default values that can be overridden by system properties\\nlog4j.rootLogger=INFO,stdout\\nlog4j.threshhold=ALL\\nlog4j.appender.stdout=org.apache.log4j.ConsoleAppender\\nlog4j.appender.stdout.layout=org.apache.log4j.PatternLayout\\n\\n# log layout skips stack-trace creation operations by avoiding line numbers and method\\nlog4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} - %m%n\\n\\n# debug edition is much more expensive\\n#log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} (%F:%M(%L)) - %m%n\\n\\n\\nlog4j.appender.subprocess=org.apache.log4j.ConsoleAppender\\nlog4j.appender.subprocess.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.subprocess.layout.ConversionPattern=[%c{1}]: %m%n\\n#log4j.logger.org.apache.slider.yarn.appmaster.SliderAppMasterer.master=INFO,subprocess\\n\\n# for debugging Slider\\n#log4j.logger.org.apache.slider=DEBUG\\n#log4j.logger.org.apache.slider=DEBUG\\n\\n# uncomment to debug service lifecycle issues\\n#log4j.logger.org.apache.hadoop.yarn.service.launcher=DEBUG\\n#log4j.logger.org.apache.hadoop.yarn.service=DEBUG\\n\\n# uncomment for YARN operations\\n#log4j.logger.org.apache.hadoop.yarn.client=DEBUG\\n\\n# uncomment this to debug security problems\\n#log4j.logger.org.apache.hadoop.security=DEBUG\\n\\n#crank back on some noise\\nlog4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR\\nlog4j.logger.org.apache.hadoop.hdfs=WARN\\n\\n\\nlog4j.logger.org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor=WARN\\nlog4j.logger.org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl=WARN\\nlog4j.logger.org.apache.zookeeper=WARN\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"sqoop-env\": {\n",
      "        \"content\": \"\\n# Set Hadoop-specific environment variables here.\\n\\n#Set path to where bin/hadoop is available\\n#Set path to where bin/hadoop is available\\nexport HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n\\n#set the path to where bin/hbase is available\\nexport HBASE_HOME=${HBASE_HOME:-{{hbase_home}}}\\n\\n#Set the path to where bin/hive is available\\nexport HIVE_HOME=${HIVE_HOME:-{{hive_home}}}\\n\\n#Set the path for where zookeper config dir is\\nexport ZOOCFGDIR=${ZOOCFGDIR:-/etc/zookeeper/conf}\\n\\n# add libthrift in hive to sqoop class path first so hive imports work\\nexport SQOOP_USER_CLASSPATH=\\\"`ls ${HIVE_HOME}/lib/libthrift-*.jar 2> /dev/null`:${SQOOP_USER_CLASSPATH}\\\"\\n    \",\n",
      "        \"sqoop_user\": \"sqoop\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"storm-env\": {\n",
      "        \"storm_log_dir\": \"/var/log/storm\",\n",
      "        \"storm_pid_dir\": \"/var/run/storm\",\n",
      "        \"content\": \"\\n#!/bin/bash\\n\\n# Set Storm specific environment variables here.\\n\\n# The java implementation to use.\\nexport JAVA_HOME={{java64_home}}\\n\\n# export STORM_CONF_DIR=\\\"\\\"\\nSTORM_HOME=/usr/hdp/current/storm-client\\n    \",\n",
      "        \"storm_user\": \"storm\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"storm-site\": {\n",
      "        \"logviewer.port\": \"8005\",\n",
      "        \"storm.messaging.netty.max_retries\": \"30\",\n",
      "        \"drpc.port\": \"3772\",\n",
      "        \"storm.local.mode.zmq\": \"false\",\n",
      "        \"storm.zookeeper.servers\": \"['%HOSTGROUP::host_group_1%']\",\n",
      "        \"topology.executor.send.buffer.size\": \"1024\",\n",
      "        \"nimbus.inbox.jar.expiration.secs\": \"3600\",\n",
      "        \"topology.transfer.buffer.size\": \"1024\",\n",
      "        \"logviewer.appender.name\": \"A1\",\n",
      "        \"topology.executor.receive.buffer.size\": \"1024\",\n",
      "        \"nimbus.task.launch.secs\": \"120\",\n",
      "        \"nimbus.file.copy.expiration.secs\": \"600\",\n",
      "        \"nimbus.task.timeout.secs\": \"30\",\n",
      "        \"topology.tick.tuple.freq.secs\": \"null\",\n",
      "        \"drpc.queue.size\": \"128\",\n",
      "        \"nimbus.thrift.max_buffer_size\": \"1048576\",\n",
      "        \"topology.disruptor.wait.strategy\": \"com.lmax.disruptor.BlockingWaitStrategy\",\n",
      "        \"storm.local.dir\": \"/hadoop/storm\",\n",
      "        \"topology.receiver.buffer.size\": \"8\",\n",
      "        \"nimbus.thrift.port\": \"6627\",\n",
      "        \"supervisor.monitor.frequency.secs\": \"3\",\n",
      "        \"nimbus.reassign\": \"true\",\n",
      "        \"storm.messaging.netty.max_wait_ms\": \"1000\",\n",
      "        \"supervisor.heartbeat.frequency.secs\": \"5\",\n",
      "        \"topology.skip.missing.kryo.registrations\": \"false\",\n",
      "        \"drpc.worker.threads\": \"64\",\n",
      "        \"drpc.request.timeout.secs\": \"600\",\n",
      "        \"topology.max.error.report.per.interval\": \"5\",\n",
      "        \"worker.childopts\": \"-Xmx768m _JAAS_PLACEHOLDER -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=%HOSTGROUP::host_group_1%,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_%ID%_JVM\",\n",
      "        \"topology.enable.message.timeouts\": \"true\",\n",
      "        \"ui.childopts\": \"-Xmx220m\",\n",
      "        \"supervisor.worker.start.timeout.secs\": \"120\",\n",
      "        \"task.heartbeat.frequency.secs\": \"3\",\n",
      "        \"storm.zookeeper.retry.intervalceiling.millis\": \"30000\",\n",
      "        \"drpc.invocations.port\": \"3773\",\n",
      "        \"storm.zookeeper.port\": \"2181\",\n",
      "        \"topology.error.throttle.interval.secs\": \"10\",\n",
      "        \"topology.message.timeout.secs\": \"30\",\n",
      "        \"storm.messaging.netty.server_worker_threads\": \"1\",\n",
      "        \"transactional.zookeeper.root\": \"/transactional\",\n",
      "        \"task.refresh.poll.secs\": \"10\",\n",
      "        \"supervisor.childopts\": \"-Xmx256m _JAAS_PLACEHOLDER -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=56431 -javaagent:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=%HOSTGROUP::host_group_1%,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Supervisor_JVM\",\n",
      "        \"topology.fall.back.on.java.serialization\": \"true\",\n",
      "        \"topology.state.synchronization.timeout.secs\": \"60\",\n",
      "        \"topology.spout.wait.strategy\": \"backtype.storm.spout.SleepSpoutWaitStrategy\",\n",
      "        \"topology.sleep.spout.wait.strategy.time.ms\": \"1\",\n",
      "        \"storm.zookeeper.retry.times\": \"5\",\n",
      "        \"java.library.path\": \"/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib\",\n",
      "        \"nimbus.supervisor.timeout.secs\": \"60\",\n",
      "        \"nimbus.host\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"supervisor.slots.ports\": \"[6700, 6701]\",\n",
      "        \"ui.port\": \"8744\",\n",
      "        \"topology.optimize\": \"true\",\n",
      "        \"nimbus.cleanup.inbox.freq.secs\": \"600\",\n",
      "        \"topology.acker.executors\": \"null\",\n",
      "        \"storm.zookeeper.retry.interval\": \"1000\",\n",
      "        \"topology.workers\": \"1\",\n",
      "        \"storm.cluster.mode\": \"distributed\",\n",
      "        \"topology.max.spout.pending\": \"null\",\n",
      "        \"storm.zookeeper.root\": \"/storm\",\n",
      "        \"topology.debug\": \"false\",\n",
      "        \"nimbus.childopts\": \"-Xmx220m -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=%HOSTGROUP::host_group_1%,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM\",\n",
      "        \"nimbus.monitor.freq.secs\": \"10\",\n",
      "        \"topology.builtin.metrics.bucket.size.secs\": \"60\",\n",
      "        \"topology.trident.batch.emit.interval.millis\": \"500\",\n",
      "        \"supervisor.worker.timeout.secs\": \"30\",\n",
      "        \"storm.messaging.netty.client_worker_threads\": \"1\",\n",
      "        \"zmq.linger.millis\": \"5000\",\n",
      "        \"dev.zookeeper.path\": \"/tmp/dev-storm-zookeeper\",\n",
      "        \"_storm.thrift.secure.transport\": \"backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin\",\n",
      "        \"logviewer.childopts\": \"-Xmx128m _JAAS_PLACEHOLDER\",\n",
      "        \"transactional.zookeeper.servers\": \"null\",\n",
      "        \"_storm.thrift.nonsecure.transport\": \"backtype.storm.security.auth.SimpleTransportPlugin\",\n",
      "        \"nimbus.topology.validator\": \"backtype.storm.nimbus.DefaultTopologyValidator\",\n",
      "        \"storm.messaging.netty.buffer_size\": \"5242880\",\n",
      "        \"topology.tuple.serializer\": \"backtype.storm.serialization.types.ListDelegateSerializer\",\n",
      "        \"storm.messaging.transport\": \"backtype.storm.messaging.netty.Context\",\n",
      "        \"drpc.childopts\": \"-Xmx220m\",\n",
      "        \"_storm.min.ruid\": \"null\",\n",
      "        \"topology.worker.childopts\": \"null\",\n",
      "        \"worker.heartbeat.frequency.secs\": \"1\",\n",
      "        \"storm.messaging.netty.min_wait_ms\": \"100\",\n",
      "        \"topology.kryo.factory\": \"backtype.storm.serialization.DefaultKryoFactory\",\n",
      "        \"topology.worker.shared.thread.pool.size\": \"4\",\n",
      "        \"zmq.threads\": \"1\",\n",
      "        \"zmq.hwm\": \"0\",\n",
      "        \"topology.max.task.parallelism\": \"null\",\n",
      "        \"transactional.zookeeper.port\": \"null\",\n",
      "        \"storm.zookeeper.connection.timeout\": \"15000\",\n",
      "        \"storm.zookeeper.session.timeout\": \"20000\",\n",
      "        \"topology.stats.sample.rate\": \"0.05\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"tez-env\": {\n",
      "        \"content\": \"\\n# Tez specific configuration\\nexport TEZ_CONF_DIR={{config_dir}}\\n\\n# Set HADOOP_HOME to point to a specific hadoop install directory\\nexport HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}\\n\\n# The java implementation to use.\\nexport JAVA_HOME={{java64_home}}\\n    \",\n",
      "        \"tez_user\": \"tez\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"tez-site\": {\n",
      "        \"tez.task.max-events-per-heartbeat\": \"500\",\n",
      "        \"tez.task.resource.memory.mb\": \"1536\",\n",
      "        \"tez.runtime.compress\": \"true\",\n",
      "        \"tez.runtime.io.sort.mb\": \"512\",\n",
      "        \"tez.history.logging.service.class\": \"org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService\",\n",
      "        \"tez.am.resource.memory.mb\": \"250\",\n",
      "        \"tez.am.log.level\": \"INFO\",\n",
      "        \"tez.shuffle-vertex-manager.min-src-fraction\": \"0.2\",\n",
      "        \"tez.am.container.idle.release-timeout-min.millis\": \"10000\",\n",
      "        \"tez.session.am.dag.submit.timeout.secs\": \"300\",\n",
      "        \"tez.task.launch.cluster-default.cmd-opts\": \"-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}\",\n",
      "        \"tez.am.container.idle.release-timeout-max.millis\": \"20000\",\n",
      "        \"tez.am.container.reuse.rack-fallback.enabled\": \"true\",\n",
      "        \"tez.task.am.heartbeat.counter.interval-ms.max\": \"4000\",\n",
      "        \"tez.am.container.reuse.enabled\": \"true\",\n",
      "        \"tez.counters.max\": \"2000\",\n",
      "        \"tez.runtime.unordered.output.buffer.size-mb\": \"100\",\n",
      "        \"tez.shuffle-vertex-manager.max-src-fraction\": \"0.4\",\n",
      "        \"tez.grouping.split-waves\": \"1.7\",\n",
      "        \"tez.runtime.compress.codec\": \"org.apache.hadoop.io.compress.SnappyCodec\",\n",
      "        \"tez.grouping.min-size\": \"16777216\",\n",
      "        \"tez.am.launch.cluster-default.cmd-opts\": \"-server -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version}\",\n",
      "        \"tez.am.am-rm.heartbeat.interval-ms.max\": \"250\",\n",
      "        \"yarn.app.mapreduce.am.command-opts\": \"-Xmx200m\",\n",
      "        \"tez.am.maxtaskfailures.per.node\": \"10\",\n",
      "        \"tez.dag.am.resource.memory.mb\": \"250\",\n",
      "        \"tez.staging-dir\": \"/tmp/${user.name}/staging\",\n",
      "        \"tez.am.launch.env\": \"LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64\",\n",
      "        \"tez.am.java.opts\": \"-server -Xmx200m -Djava.net.preferIPv4Stack=true -XX:+UseNUMA -XX:+UseParallelGC\",\n",
      "        \"tez.lib.uris\": \"/hdp/apps/${hdp.version}/tez/tez.tar.gz\",\n",
      "        \"tez.cluster.additional.classpath.prefix\": \"/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure\",\n",
      "        \"tez.task.get-task.sleep.interval-ms.max\": \"200\",\n",
      "        \"tez.task.launch.env\": \"LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64\",\n",
      "        \"tez.am.container.reuse.non-local-fallback.enabled\": \"false\",\n",
      "        \"tez.am.container.reuse.locality.delay-allocation-millis\": \"250\",\n",
      "        \"tez.task.launch.cmd-opts\": \"-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC\",\n",
      "        \"tez.am.launch.cmd-opts\": \"-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC\",\n",
      "        \"tez.generate.debug.artifacts\": \"false\",\n",
      "        \"tez.counters.max.groups\": \"1000\",\n",
      "        \"tez.grouping.max-size\": \"1073741824\",\n",
      "        \"tez.am.max.app.attempts\": \"2\",\n",
      "        \"tez.session.client.timeout.secs\": \"-1\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"webhcat-env\": {\n",
      "        \"content\": \"\\n# The file containing the running pid\\nPID_FILE={{webhcat_pid_file}}\\n\\nTEMPLETON_LOG_DIR={{templeton_log_dir}}/\\n\\n\\nWEBHCAT_LOG_DIR={{templeton_log_dir}}/\\n\\n# The console error log\\nERROR_LOG={{templeton_log_dir}}/webhcat-console-error.log\\n\\n# The console log\\nCONSOLE_LOG={{templeton_log_dir}}/webhcat-console.log\\n\\n#TEMPLETON_JAR=templeton_jar_name\\n\\n#HADOOP_PREFIX=hadoop_prefix\\n\\n#HCAT_PREFIX=hive_prefix\\n\\n# Set HADOOP_HOME to point to a specific hadoop install directory\\nexport HADOOP_HOME={{hadoop_home}}\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"webhcat-site\": {\n",
      "        \"templeton.hcat\": \"/usr/hdp/current/hive-client/bin/hcat\",\n",
      "        \"templeton.hadoop\": \"/usr/hdp/current/hadoop-client/bin/hadoop\",\n",
      "        \"webhcat.proxyuser.hcat.hosts\": \"*\",\n",
      "        \"templeton.zookeeper.hosts\": \"%HOSTGROUP::host_group_1%:2181\",\n",
      "        \"templeton.streaming.jar\": \"hdfs:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar\",\n",
      "        \"templeton.override.enabled\": \"false\",\n",
      "        \"templeton.hcat.home\": \"hive.tar.gz/hive/hcatalog\",\n",
      "        \"templeton.storage.class\": \"org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage\",\n",
      "        \"templeton.pig.archive\": \"hdfs:///hdp/apps/${hdp.version}/pig/pig.tar.gz\",\n",
      "        \"templeton.pig.path\": \"pig.tar.gz/pig/bin/pig\",\n",
      "        \"webhcat.proxyuser.hcat.groups\": \"*\",\n",
      "        \"templeton.jar\": \"/usr/hdp/current/hive-webhcat/share/webhcat/svr/lib/hive-webhcat-*.jar\",\n",
      "        \"templeton.sqoop.path\": \"sqoop.tar.gz/sqoop/bin/sqoop\",\n",
      "        \"templeton.hive.path\": \"hive.tar.gz/hive/bin/hive\",\n",
      "        \"templeton.hive.archive\": \"hdfs:///hdp/apps/${hdp.version}/hive/hive.tar.gz\",\n",
      "        \"templeton.sqoop.archive\": \"hdfs:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz\",\n",
      "        \"templeton.exec.timeout\": \"60000\",\n",
      "        \"webhcat.proxyuser.hue.groups\": \"*\",\n",
      "        \"templeton.hadoop.conf.dir\": \"/etc/hadoop/conf\",\n",
      "        \"webhcat.proxyuser.hue.hosts\": \"*\",\n",
      "        \"templeton.hive.home\": \"hive.tar.gz/hive\",\n",
      "        \"templeton.port\": \"50111\",\n",
      "        \"templeton.sqoop.home\": \"sqoop.tar.gz/sqoop\",\n",
      "        \"templeton.libjars\": \"/usr/hdp/current/zookeeper-client/zookeeper.jar\",\n",
      "        \"templeton.hive.properties\": \"%HOSTGROUP::host_group_1%://%HOSTGROUP::host_group_1%:9083,hive.metastore.sasl.enabled=false,hive.metastore.execute.setugi=true,hive.metastore.warehouse.dir=/apps/hive/warehouse\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"yarn-env\": {\n",
      "        \"apptimelineserver_heapsize\": \"250\",\n",
      "        \"nodemanager_heapsize\": \"250\",\n",
      "        \"resourcemanager_heapsize\": \"250\",\n",
      "        \"yarn_heapsize\": \"250\",\n",
      "        \"min_user_id\": \"1000\",\n",
      "        \"yarn_user\": \"yarn\",\n",
      "        \"yarn_log_dir_prefix\": \"/var/log/hadoop-yarn\",\n",
      "        \"yarn_pid_dir_prefix\": \"/var/run/hadoop-yarn\",\n",
      "        \"content\": \"\\nexport HADOOP_YARN_HOME={{hadoop_yarn_home}}\\nexport YARN_LOG_DIR={{yarn_log_dir_prefix}}/$USER\\nexport YARN_PID_DIR={{yarn_pid_dir_prefix}}/$USER\\nexport HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}\\nexport JAVA_HOME={{java64_home}}\\n\\n# User for YARN daemons\\nexport HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}\\n\\n# resolve links - $0 may be a softlink\\nexport YARN_CONF_DIR=\\\"${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}\\\"\\n\\n# some Java parameters\\n# export JAVA_HOME=/home/y/libexec/jdk1.6.0/\\nif [ \\\"$JAVA_HOME\\\" != \\\"\\\" ]; then\\n  #echo \\\"run java in $JAVA_HOME\\\"\\n  JAVA_HOME=$JAVA_HOME\\nfi\\n\\nif [ \\\"$JAVA_HOME\\\" = \\\"\\\" ]; then\\n  echo \\\"Error: JAVA_HOME is not set.\\\"\\n  exit 1\\nfi\\n\\nJAVA=$JAVA_HOME/bin/java\\nJAVA_HEAP_MAX=-Xmx1000m\\n\\n# For setting YARN specific HEAP sizes please use this\\n# Parameter and set appropriately\\nYARN_HEAPSIZE={{yarn_heapsize}}\\n\\n# check envvars which might override default args\\nif [ \\\"$YARN_HEAPSIZE\\\" != \\\"\\\" ]; then\\n  JAVA_HEAP_MAX=\\\"-Xmx\\\"\\\"$YARN_HEAPSIZE\\\"\\\"m\\\"\\nfi\\n\\n# Resource Manager specific parameters\\n\\n# Specify the max Heapsize for the ResourceManager using a numerical value\\n# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set\\n# the value to 1000.\\n# This value will be overridden by an Xmx setting specified in either YARN_OPTS\\n# and/or YARN_RESOURCEMANAGER_OPTS.\\n# If not specified, the default value will be picked from either YARN_HEAPMAX\\n# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.\\nexport YARN_RESOURCEMANAGER_HEAPSIZE={{resourcemanager_heapsize}}\\n\\n# Specify the JVM options to be used when starting the ResourceManager.\\n# These options will be appended to the options specified as YARN_OPTS\\n# and therefore may override any similar flags set in YARN_OPTS\\n#export YARN_RESOURCEMANAGER_OPTS=\\n\\n# Node Manager specific parameters\\n\\n# Specify the max Heapsize for the NodeManager using a numerical value\\n# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set\\n# the value to 1000.\\n# This value will be overridden by an Xmx setting specified in either YARN_OPTS\\n# and/or YARN_NODEMANAGER_OPTS.\\n# If not specified, the default value will be picked from either YARN_HEAPMAX\\n# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.\\nexport YARN_NODEMANAGER_HEAPSIZE={{nodemanager_heapsize}}\\n\\n# Specify the max Heapsize for the HistoryManager using a numerical value\\n# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set\\n# the value to 1024.\\n# This value will be overridden by an Xmx setting specified in either YARN_OPTS\\n# and/or YARN_HISTORYSERVER_OPTS.\\n# If not specified, the default value will be picked from either YARN_HEAPMAX\\n# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.\\nexport YARN_HISTORYSERVER_HEAPSIZE={{apptimelineserver_heapsize}}\\n\\n# Specify the JVM options to be used when starting the NodeManager.\\n# These options will be appended to the options specified as YARN_OPTS\\n# and therefore may override any similar flags set in YARN_OPTS\\n#export YARN_NODEMANAGER_OPTS=\\n\\n# so that filenames w/ spaces are handled correctly in loops below\\nIFS=\\n\\n\\n# default log directory and file\\nif [ \\\"$YARN_LOG_DIR\\\" = \\\"\\\" ]; then\\n  YARN_LOG_DIR=\\\"$HADOOP_YARN_HOME/logs\\\"\\nfi\\nif [ \\\"$YARN_LOGFILE\\\" = \\\"\\\" ]; then\\n  YARN_LOGFILE='yarn.log'\\nfi\\n\\n# default policy file for service-level authorization\\nif [ \\\"$YARN_POLICYFILE\\\" = \\\"\\\" ]; then\\n  YARN_POLICYFILE=\\\"hadoop-policy.xml\\\"\\nfi\\n\\n# restore ordinary behaviour\\nunset IFS\\n\\n\\nYARN_OPTS=\\\"$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}\\\"\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}\\\"\\nif [ \\\"x$JAVA_LIBRARY_PATH\\\" != \\\"x\\\" ]; then\\n  YARN_OPTS=\\\"$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH\\\"\\nfi\\nYARN_OPTS=\\\"$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE\\\"\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"yarn-log4j\": {\n",
      "        \"content\": \"\\n#Relative to Yarn Log Dir Prefix\\nyarn.log.dir=.\\n#\\n# Job Summary Appender\\n#\\n# Use following logger to send summary to separate file defined by\\n# hadoop.mapreduce.jobsummary.log.file rolled daily:\\n# hadoop.mapreduce.jobsummary.logger=INFO,JSA\\n#\\nhadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}\\nhadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log\\nlog4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender\\n# Set the ResourceManager summary log filename\\nyarn.server.resourcemanager.appsummary.log.file=hadoop-mapreduce.jobsummary.log\\n# Set the ResourceManager summary log level and appender\\nyarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger}\\n#yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY\\n\\n# To enable AppSummaryLogging for the RM,\\n# set yarn.server.resourcemanager.appsummary.logger to\\n# LEVEL,RMSUMMARY in hadoop-env.sh\\n\\n# Appender for ResourceManager Application Summary Log\\n# Requires the following properties to be set\\n#    - hadoop.log.dir (Hadoop Log directory)\\n#    - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)\\n#    - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)\\nlog4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender\\nlog4j.appender.RMSUMMARY.File=${yarn.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}\\nlog4j.appender.RMSUMMARY.MaxFileSize=256MB\\nlog4j.appender.RMSUMMARY.MaxBackupIndex=20\\nlog4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n\\nlog4j.appender.JSA.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n\\nlog4j.appender.JSA.DatePattern=.yyyy-MM-dd\\nlog4j.appender.JSA.layout=org.apache.log4j.PatternLayout\\nlog4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}\\nlog4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"yarn-site\": {\n",
      "        \"yarn.nodemanager.container-executor.class\": \"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor\",\n",
      "        \"yarn.resourcemanager.webapp.address\": \"%HOSTGROUP::host_group_1%:8088\",\n",
      "        \"yarn.nodemanager.aux-services.mapreduce_shuffle.class\": \"org.apache.hadoop.mapred.ShuffleHandler\",\n",
      "        \"yarn.nodemanager.address\": \"0.0.0.0:45454\",\n",
      "        \"yarn.nodemanager.remote-app-log-dir\": \"/app-logs\",\n",
      "        \"yarn.resourcemanager.zk-timeout-ms\": \"10000\",\n",
      "        \"yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms\": \"300000\",\n",
      "        \"yarn.timeline-service.leveldb-timeline-store.path\": \"/hadoop/yarn/timeline\",\n",
      "        \"yarn.timeline-service.bind-host\": \"0.0.0.0\",\n",
      "        \"yarn.resourcemanager.webapp.proxyuser.oozie.groups\": \"*\",\n",
      "        \"yarn.nodemanager.resource.percentage-physical-cpu-limit\": \"100\",\n",
      "        \"yarn.nodemanager.linux-container-executor.cgroups.hierarchy\": \"hadoop-yarn\",\n",
      "        \"yarn.nodemanager.recovery.enabled\": \"false\",\n",
      "        \"yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size\": \"10000\",\n",
      "        \"yarn.resourcemanager.recovery.enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.resource-tracker.address\": \"%HOSTGROUP::host_group_1%:8025\",\n",
      "        \"yarn.nodemanager.vmem-check-enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.admin.address\": \"%HOSTGROUP::host_group_1%:8141\",\n",
      "        \"yarn.nodemanager.delete.debug-delay-sec\": \"0\",\n",
      "        \"yarn.nodemanager.admin-env\": \"MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX\",\n",
      "        \"yarn.timeline-service.client.max-retries\": \"30\",\n",
      "        \"yarn.nodemanager.log-aggregation.compression-type\": \"gz\",\n",
      "        \"yarn.resourcemanager.zk-retry-interval-ms\": \"1000\",\n",
      "        \"yarn.nodemanager.recovery.dir\": \"/var/log/hadoop-yarn/nodemanager/recovery-state\",\n",
      "        \"yarn.timeline-service.webapp.https.address\": \"%HOSTGROUP::host_group_1%:8190\",\n",
      "        \"yarn.resourcemanager.system-metrics-publisher.enabled\": \"true\",\n",
      "        \"yarn.nodemanager.log-dirs\": \"/hadoop/yarn/log\",\n",
      "        \"yarn.resourcemanager.fs.state-store.retry-policy-spec\": \"2000, 500\",\n",
      "        \"yarn.log-aggregation.retain-seconds\": \"2592000\",\n",
      "        \"yarn.scheduler.maximum-allocation-mb\": \"2250\",\n",
      "        \"yarn.nodemanager.bind-host\": \"0.0.0.0\",\n",
      "        \"yarn.acl.enable\": \"false\",\n",
      "        \"yarn.resourcemanager.fs.state-store.uri\": \" \",\n",
      "        \"yarn.resourcemanager.webapp.proxyuser.oozie.hosts\": \"*\",\n",
      "        \"yarn.nodemanager.aux-services\": \"mapreduce_shuffle\",\n",
      "        \"yarn.timeline-service.store-class\": \"org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore\",\n",
      "        \"yarn.nodemanager.log-aggregation.debug-enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.nodes.exclude-path\": \"/etc/hadoop/conf/yarn.exclude\",\n",
      "        \"yarn.resourcemanager.address\": \"%HOSTGROUP::host_group_1%:8050\",\n",
      "        \"yarn.nodemanager.log.retain-second\": \"604800\",\n",
      "        \"yarn.log.server.url\": \"http://%HOSTGROUP::host_group_1%:19888/jobhistory/logs\",\n",
      "        \"yarn.resourcemanager.hostname\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"yarn.resourcemanager.zk-acl\": \"world:anyone:rwcda \",\n",
      "        \"yarn.nodemanager.container-monitor.interval-ms\": \"3000\",\n",
      "        \"yarn.nodemanager.linux-container-executor.cgroups.mount\": \"false\",\n",
      "        \"yarn.nodemanager.linux-container-executor.resources-handler.class\": \"org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler\",\n",
      "        \"yarn.resourcemanager.webapp.proxyuser.hcat.hosts\": \"*\",\n",
      "        \"yarn.resourcemanager.zk-state-store.parent-path\": \"/rmstore\",\n",
      "        \"yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size\": \"10000\",\n",
      "        \"yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage\": \"false\",\n",
      "        \"yarn.admin.acl\": \"\",\n",
      "        \"yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage\": \"90\",\n",
      "        \"yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms\": \"10000\",\n",
      "        \"yarn.nodemanager.linux-container-executor.group\": \"hadoop\",\n",
      "        \"yarn.timeline-service.http-authentication.simple.anonymous.allowed\": \"true\",\n",
      "        \"yarn.timeline-service.http-authentication.type\": \"simple\",\n",
      "        \"yarn.node-labels.fs-store.root-dir\": \"/system/yarn/node-labels\",\n",
      "        \"yarn.timeline-service.leveldb-timeline-store.read-cache-size\": \"104857600\",\n",
      "        \"yarn.log-aggregation-enable\": \"true\",\n",
      "        \"yarn.resourcemanager.connect.retry-interval.ms\": \"30000\",\n",
      "        \"yarn.application.classpath\": \"$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*\",\n",
      "        \"yarn.timeline-service.address\": \"%HOSTGROUP::host_group_1%:10200\",\n",
      "        \"yarn.timeline-service.webapp.address\": \"%HOSTGROUP::host_group_1%:8188\",\n",
      "        \"yarn.nodemanager.local-dirs\": \"/hadoop/yarn/local\",\n",
      "        \"yarn.nodemanager.log-aggregation.num-log-files-per-app\": \"30\",\n",
      "        \"yarn.resourcemanager.ha.enabled\": \"false\",\n",
      "        \"yarn.client.nodemanager-connect.max-wait-ms\": \"900000\",\n",
      "        \"yarn.resourcemanager.scheduler.class\": \"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler\",\n",
      "        \"yarn.nodemanager.health-checker.interval-ms\": \"135000\",\n",
      "        \"yarn.node-labels.fs-store.retry-policy-spec\": \"2000, 500\",\n",
      "        \"yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds\": \"-1\",\n",
      "        \"yarn.nodemanager.health-checker.script.timeout-ms\": \"60000\",\n",
      "        \"yarn.nodemanager.vmem-pmem-ratio\": \"10\",\n",
      "        \"yarn.timeline-service.ttl-enable\": \"true\",\n",
      "        \"yarn.nodemanager.pmem-check-enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.work-preserving-recovery.enabled\": \"false\",\n",
      "        \"yarn.nodemanager.remote-app-log-dir-suffix\": \"logs\",\n",
      "        \"yarn.resourcemanager.am.max-attempts\": \"2\",\n",
      "        \"yarn.timeline-service.enabled\": \"true\",\n",
      "        \"yarn.resourcemanager.scheduler.address\": \"%HOSTGROUP::host_group_1%:8030\",\n",
      "        \"yarn.scheduler.minimum-allocation-mb\": \"250\",\n",
      "        \"hadoop.registry.zk.quorum\": \"%HOSTGROUP::host_group_1%:2181\",\n",
      "        \"yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb\": \"1000\",\n",
      "        \"yarn.node-labels.manager-class\": \"org.apache.hadoop.yarn.server.resourcemanager.nodelabels.MemoryRMNodeLabelsManager\",\n",
      "        \"yarn.resourcemanager.connect.max-wait.ms\": \"900000\",\n",
      "        \"yarn.timeline-service.ttl-ms\": \"2678400000\",\n",
      "        \"yarn.resourcemanager.state-store.max-completed-applications\": \"${yarn.resourcemanager.max-completed-applications}\",\n",
      "        \"yarn.timeline-service.client.retry-interval-ms\": \"1000\",\n",
      "        \"yarn.resourcemanager.zk-num-retries\": \"1000\",\n",
      "        \"yarn.timeline-service.generic-application-history.store-class\": \"org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore\",\n",
      "        \"yarn.nodemanager.disk-health-checker.min-healthy-disks\": \"0.25\",\n",
      "        \"hadoop.registry.rm.enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.store.class\": \"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore\",\n",
      "        \"yarn.resourcemanager.bind-host\": \"0.0.0.0\",\n",
      "        \"yarn.nodemanager.resource.memory-mb\": \"2250\",\n",
      "        \"yarn.nodemanager.resource.cpu-vcores\": \"8\",\n",
      "        \"yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled\": \"false\",\n",
      "        \"yarn.resourcemanager.webapp.proxyuser.hcat.groups\": \"*\",\n",
      "        \"yarn.resourcemanager.zk-address\": \"localhost:2181\",\n",
      "        \"yarn.client.nodemanager-connect.retry-interval-ms\": \"10000\",\n",
      "        \"yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size\": \"10\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"zoo.cfg\": {\n",
      "        \"clientPort\": \"2181\",\n",
      "        \"dataDir\": \"/hadoop/zookeeper\",\n",
      "        \"tickTime\": \"2000\",\n",
      "        \"syncLimit\": \"5\",\n",
      "        \"initLimit\": \"10\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"zookeeper-env\": {\n",
      "        \"zk_log_dir\": \"/var/log/zookeeper\",\n",
      "        \"content\": \"\\nexport JAVA_HOME={{java64_home}}\\nexport ZOOKEEPER_HOME={{zk_home}}\\nexport ZOO_LOG_DIR={{zk_log_dir}}\\nexport ZOOPIDFILE={{zk_pid_file}}\\nexport SERVER_JVMFLAGS={{zk_server_heapsize}}\\nexport JAVA=$JAVA_HOME/bin/java\\nexport CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*\\n\\n{% if security_enabled %}\\nexport SERVER_JVMFLAGS=\\\"$SERVER_JVMFLAGS -Djava.security.auth.login.config={{zk_server_jaas_file}}\\\"\\nexport CLIENT_JVMFLAGS=\\\"$CLIENT_JVMFLAGS -Djava.security.auth.login.config={{zk_client_jaas_file}}\\\"\\n{% endif %}\\n    \",\n",
      "        \"zk_user\": \"zookeeper\",\n",
      "        \"zk_pid_dir\": \"/var/run/zookeeper\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"zookeeper-log4j\": {\n",
      "        \"content\": \"\\n#\\n#\\n# Licensed to the Apache Software Foundation (ASF) under one\\n# or more contributor license agreements.  See the NOTICE file\\n# distributed with this work for additional information\\n# regarding copyright ownership.  The ASF licenses this file\\n# to you under the Apache License, Version 2.0 (the\\n# \\\"License\\\"); you may not use this file except in compliance\\n# with the License.  You may obtain a copy of the License at\\n#\\n#   http://www.apache.org/licenses/LICENSE-2.0\\n#\\n# Unless required by applicable law or agreed to in writing,\\n# software distributed under the License is distributed on an\\n# \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\\n# KIND, either express or implied.  See the License for the\\n# specific language governing permissions and limitations\\n# under the License.\\n#\\n#\\n#\\n\\n#\\n# ZooKeeper Logging Configuration\\n#\\n\\n# DEFAULT: console appender only\\nlog4j.rootLogger=INFO, CONSOLE\\n\\n# Example with rolling log file\\n#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE\\n\\n# Example with rolling log file and tracing\\n#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE\\n\\n#\\n# Log INFO level and above messages to the console\\n#\\nlog4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender\\nlog4j.appender.CONSOLE.Threshold=INFO\\nlog4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n\\n\\n#\\n# Add ROLLINGFILE to rootLogger to get log file output\\n#    Log DEBUG level and above messages to a log file\\nlog4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender\\nlog4j.appender.ROLLINGFILE.Threshold=DEBUG\\nlog4j.appender.ROLLINGFILE.File=zookeeper.log\\n\\n# Max log file size of 10MB\\nlog4j.appender.ROLLINGFILE.MaxFileSize=10MB\\n# uncomment the next line to limit number of backup files\\n#log4j.appender.ROLLINGFILE.MaxBackupIndex=10\\n\\nlog4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout\\nlog4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n\\n\\n\\n#\\n# Add TRACEFILE to rootLogger to get log file output\\n#    Log DEBUG level and above messages to a log file\\nlog4j.appender.TRACEFILE=org.apache.log4j.FileAppender\\nlog4j.appender.TRACEFILE.Threshold=TRACE\\nlog4j.appender.TRACEFILE.File=zookeeper_trace.log\\n\\nlog4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout\\n### Notice we are including log4j's NDC here (%x)\\nlog4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n\\n    \"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hive-site\": {\n",
      "        \"hive.fetch.task.conversion\": \"more\",\n",
      "        \"hive.map.aggr.hash.percentmemory\": \"0.5\",\n",
      "        \"hive.compactor.initiator.on\": \"true\",\n",
      "        \"hive.server2.authentication\": \"NONE\",\n",
      "        \"hive.tez.log.level\": \"INFO\",\n",
      "        \"hive.exec.dynamic.partition\": \"true\",\n",
      "        \"hive.tez.smb.number.waves\": \"0.5\",\n",
      "        \"hive.user.install.directory\": \"/user/\",\n",
      "        \"hive.zookeeper.namespace\": \"hive_zookeeper_namespace\",\n",
      "        \"hive.fetch.task.conversion.threshold\": \"1073741824\",\n",
      "        \"hive.metastore.sasl.enabled\": \"false\",\n",
      "        \"hive.exec.parallel.thread.number\": \"8\",\n",
      "        \"hive.fetch.task.aggr\": \"false\",\n",
      "        \"hive.merge.smallfiles.avgsize\": \"16000000\",\n",
      "        \"hive.heapsize\": \"250\",\n",
      "        \"hive.exec.submitviachild\": \"false\",\n",
      "        \"hive.stats.autogather\": \"true\",\n",
      "        \"hive.auto.convert.sortmerge.join.noconditionaltask\": \"true\",\n",
      "        \"hive.zookeeper.quorum\": \"%HOSTGROUP::host_group_1%:2181\",\n",
      "        \"hive.optimize.bucketmapjoin\": \"true\",\n",
      "        \"datanucleus.cache.level2.type\": \"none\",\n",
      "        \"hive.exec.max.created.files\": \"100000\",\n",
      "        \"hive.compute.query.using.stats\": \"true\",\n",
      "        \"hive.limit.optimize.enable\": \"true\",\n",
      "        \"hive.optimize.mapjoin.mapreduce\": \"true\",\n",
      "        \"hive.cluster.delegation.token.store.zookeeper.znode\": \"/hive/cluster/delegation\",\n",
      "        \"hive.smbjoin.cache.rows\": \"10000\",\n",
      "        \"hive.metastore.connect.retries\": \"24\",\n",
      "        \"hive.metastore.pre.event.listeners\": \"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener\",\n",
      "        \"hive.auto.convert.sortmerge.join.to.mapjoin\": \"false\",\n",
      "        \"hive.cluster.delegation.token.store.class\": \"org.apache.hadoop.hive.thrift.ZooKeeperTokenStore\",\n",
      "        \"hive.exec.orc.compression.strategy\": \"SPEED\",\n",
      "        \"hive.cli.print.header\": \"false\",\n",
      "        \"hive.merge.mapredfiles\": \"false\",\n",
      "        \"hive.metastore.cache.pinobjtypes\": \"Table,Database,Type,FieldSchema,Order\",\n",
      "        \"hive.tez.container.size\": \"250\",\n",
      "        \"hive.tez.dynamic.partition.pruning.max.event.size\": \"1048576\",\n",
      "        \"hive.compactor.delta.num.threshold\": \"10\",\n",
      "        \"hive.enforce.sortmergebucketmapjoin\": \"true\",\n",
      "        \"hive.auto.convert.join\": \"true\",\n",
      "        \"hive.map.aggr.hash.min.reduction\": \"0.5\",\n",
      "        \"hive.security.authorization.manager\": \"org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory\",\n",
      "        \"hive.server2.allow.user.substitution\": \"true\",\n",
      "        \"hive.server2.enable.doAs\": \"false\",\n",
      "        \"hive.optimize.reducededuplication.min.reducer\": \"4\",\n",
      "        \"hive.compactor.worker.threads\": \"0\",\n",
      "        \"hive.server2.use.SSL\": \"false\",\n",
      "        \"hive.security.metastore.authorization.auth.reads\": \"true\",\n",
      "        \"hive.metastore.server.max.threads\": \"100000\",\n",
      "        \"hive.server2.zookeeper.namespace\": \"hiveserver2\",\n",
      "        \"hive.exec.failure.hooks\": \"org.apache.hadoop.hive.ql.hooks.ATSHook\",\n",
      "        \"hive.optimize.null.scan\": \"true\",\n",
      "        \"hive.security.metastore.authenticator.manager\": \"org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator\",\n",
      "        \"hive.exec.pre.hooks\": \"org.apache.hadoop.hive.ql.hooks.ATSHook\",\n",
      "        \"hive.support.concurrency\": \"true\",\n",
      "        \"hive.tez.java.opts\": \"-server -Xmx200m -Djava.net.preferIPv4Stack=true\",\n",
      "        \"hive.conf.restricted.list\": \"hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role\",\n",
      "        \"hive.server2.logging.operation.enabled\": \"true\",\n",
      "        \"hive.security.authenticator.manager\": \"org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator\",\n",
      "        \"hive.cbo.enable\": \"true\",\n",
      "        \"hive.metastore.kerberos.keytab.file\": \"/etc/security/keytabs/hive.service.keytab\",\n",
      "        \"hive.metastore.execute.setugi\": \"true\",\n",
      "        \"hive.users.in.admin.role\": \"hue,hive\",\n",
      "        \"hive.limit.pushdown.memory.usage\": \"0.04\",\n",
      "        \"hive.security.authorization.enabled\": \"false\",\n",
      "        \"hive.merge.mapfiles\": \"true\",\n",
      "        \"javax.jdo.option.ConnectionUserName\": \"hive\",\n",
      "        \"hive.metastore.uris\": \"thrift://%HOSTGROUP::host_group_1%:9083\",\n",
      "        \"hive.merge.tezfiles\": \"false\",\n",
      "        \"hive.server2.thrift.port\": \"10000\",\n",
      "        \"hive.map.aggr.hash.force.flush.memory.threshold\": \"0.9\",\n",
      "        \"hive.server2.authentication.spnego.principal\": \"/etc/security/keytabs/spnego.service.keytab\",\n",
      "        \"hive.execution.engine\": \"mr\",\n",
      "        \"hive.compactor.abortedtxn.threshold\": \"1000\",\n",
      "        \"hive.exec.post.hooks\": \"org.apache.hadoop.hive.ql.hooks.ATSHook\",\n",
      "        \"hive.vectorized.groupby.maxentries\": \"100000\",\n",
      "        \"hive.stats.fetch.partition.stats\": \"true\",\n",
      "        \"hive.exec.max.dynamic.partitions\": \"5000\",\n",
      "        \"hive.server2.tez.default.queues\": \"default\",\n",
      "        \"hive.exec.max.dynamic.partitions.pernode\": \"2000\",\n",
      "        \"hive.exec.reducers.bytes.per.reducer\": \"67108864\",\n",
      "        \"hive_metastore_user_passwd\": \"hive\",\n",
      "        \"ambari.hive.db.schema.name\": \"hive\",\n",
      "        \"hive.tez.input.format\": \"org.apache.hadoop.hive.ql.io.HiveInputFormat\",\n",
      "        \"hive.server2.tez.initialize.default.sessions\": \"false\",\n",
      "        \"hive.security.metastore.authorization.manager\": \"org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider,org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly\",\n",
      "        \"hive.tez.max.partition.factor\": \"2.0\",\n",
      "        \"hive.server2.thrift.max.worker.threads\": \"500\",\n",
      "        \"hive.metastore.authorization.storage.checks\": \"false\",\n",
      "        \"hive.vectorized.groupby.flush.percent\": \"0.1\",\n",
      "        \"hive.exec.submit.local.task.via.child\": \"true\",\n",
      "        \"hive.auto.convert.join.noconditionaltask\": \"true\",\n",
      "        \"hive.metastore.failure.retries\": \"24\",\n",
      "        \"hive.tez.min.partition.factor\": \"0.25\",\n",
      "        \"javax.jdo.option.ConnectionPassword\": \"hive\",\n",
      "        \"hive.mapjoin.optimized.hashtable\": \"true\",\n",
      "        \"hive.exec.parallel\": \"false\",\n",
      "        \"hive.exec.compress.intermediate\": \"false\",\n",
      "        \"hive.cluster.delegation.token.store.zookeeper.connectString\": \"%HOSTGROUP::host_group_1%:2181\",\n",
      "        \"hive.exec.orc.default.stripe.size\": \"67108864\",\n",
      "        \"hive.stats.dbclass\": \"fs\",\n",
      "        \"hive.server2.authentication.spnego.keytab\": \"HTTP/_HOST@EXAMPLE.COM\",\n",
      "        \"hive.compactor.worker.timeout\": \"86400s\",\n",
      "        \"hive.tez.cpu.vcores\": \"-1\",\n",
      "        \"hive.prewarm.enabled\": \"false\",\n",
      "        \"hive.tez.dynamic.partition.pruning.max.data.size\": \"104857600\",\n",
      "        \"hive.vectorized.execution.reduce.enabled\": \"false\",\n",
      "        \"hive.exec.compress.output\": \"false\",\n",
      "        \"hive.map.aggr\": \"true\",\n",
      "        \"hive.server2.thrift.sasl.qop\": \"auth\",\n",
      "        \"hive.optimize.reducededuplication\": \"true\",\n",
      "        \"hive.metastore.kerberos.principal\": \"hive/_HOST@EXAMPLE.COM\",\n",
      "        \"hive.txn.max.open.batch\": \"1000\",\n",
      "        \"hive.server2.thrift.http.path\": \"cliservice\",\n",
      "        \"hive.vectorized.execution.enabled\": \"true\",\n",
      "        \"hive.merge.size.per.task\": \"256000000\",\n",
      "        \"hive.compactor.delta.pct.threshold\": \"0.1f\",\n",
      "        \"hive.server2.support.dynamic.service.discovery\": \"true\",\n",
      "        \"hive.exec.scratchdir\": \"/tmp/hive\",\n",
      "        \"javax.jdo.option.ConnectionURL\": \"jdbc:mysql://%HOSTGROUP::host_group_1%/hive?createDatabaseIfNotExist=true\",\n",
      "        \"hive.enforce.sorting\": \"true\",\n",
      "        \"hive.exec.reducers.max\": \"1009\",\n",
      "        \"hive.txn.manager\": \"org.apache.hadoop.hive.ql.lockmgr.DbTxnManager\",\n",
      "        \"hive.server2.table.type.mapping\": \"CLASSIC\",\n",
      "        \"hive.compactor.check.interval\": \"300s\",\n",
      "        \"hive.optimize.index.filter\": \"true\",\n",
      "        \"hive.optimize.bucketmapjoin.sortedmerge\": \"false\",\n",
      "        \"hive.zookeeper.client.port\": \"2181\",\n",
      "        \"hive.server2.tez.sessions.per.default.queue\": \"1\",\n",
      "        \"hive.auto.convert.sortmerge.join\": \"true\",\n",
      "        \"hive.exec.orc.default.compress\": \"ZLIB\",\n",
      "        \"hive.vectorized.groupby.checkinterval\": \"4096\",\n",
      "        \"hive.merge.rcfile.block.level\": \"true\",\n",
      "        \"hive.metastore.client.connect.retry.delay\": \"5s\",\n",
      "        \"hive.metastore.client.socket.timeout\": \"1800s\",\n",
      "        \"hive.optimize.sort.dynamic.partition\": \"false\",\n",
      "        \"hive.optimize.metadataonly\": \"true\",\n",
      "        \"hive.orc.compute.splits.num.threads\": \"10\",\n",
      "        \"javax.jdo.option.ConnectionDriverName\": \"com.mysql.jdbc.Driver\",\n",
      "        \"hive.mapred.reduce.tasks.speculative.execution\": \"false\",\n",
      "        \"hive.server2.logging.operation.log.location\": \"${system:java.io.tmpdir}/${system:user.name}/operation_logs\",\n",
      "        \"hive.prewarm.numcontainers\": \"10\",\n",
      "        \"hive.txn.timeout\": \"300\",\n",
      "        \"hive.server2.enable.impersonation\": \"true\",\n",
      "        \"hive.merge.orcfile.stripe.level\": \"true\",\n",
      "        \"hive.orc.splits.include.file.footer\": \"false\",\n",
      "        \"hive.server2.thrift.http.port\": \"10001\",\n",
      "        \"hive.mapjoin.bucket.cache.size\": \"10000\",\n",
      "        \"hive.metastore.warehouse.dir\": \"/apps/hive/warehouse\",\n",
      "        \"hive.optimize.constant.propagation\": \"true\",\n",
      "        \"hive.tez.dynamic.partition.pruning\": \"true\",\n",
      "        \"hive.auto.convert.join.noconditionaltask.size\": \"1000000000\",\n",
      "        \"hive.tez.auto.reducer.parallelism\": \"false\",\n",
      "        \"hive.enforce.bucketing\": \"true\",\n",
      "        \"hive.exec.dynamic.partition.mode\": \"nonstrict\",\n",
      "        \"hive.stats.fetch.column.stats\": \"false\",\n",
      "        \"hive.server2.transport.mode\": \"binary\",\n",
      "        \"hive.convert.join.bucket.mapjoin.tez\": \"false\"\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"hbase-site\": {\n",
      "        \"hbase.client.keyvalue.maxsize\": \"10485760\",\n",
      "        \"hbase.coprocessor.region.classes\": \"com.xasecure.authorization.hbase.XaSecureAuthorizationCoprocessor\",\n",
      "        \"hbase.regionserver.handler.count\": \"60\",\n",
      "        \"hbase.regionserver.info.port\": \"60030\",\n",
      "        \"hbase.security.authentication\": \"simple\",\n",
      "        \"dfs.domain.socket.path\": \"/var/lib/hadoop-hdfs/dn_socket\",\n",
      "        \"hbase.master.info.port\": \"60010\",\n",
      "        \"hbase.superuser\": \"hbase\",\n",
      "        \"hbase.cluster.distributed\": \"true\",\n",
      "        \"hbase.regionserver.global.memstore.lowerLimit\": \"0.38\",\n",
      "        \"hbase_regionserver_heapsize\": \"250\",\n",
      "        \"hbase.defaults.for.version.skip\": \"true\",\n",
      "        \"hbase.security.authorization\": \"true\",\n",
      "        \"hbase.hregion.memstore.mslab.enabled\": \"true\",\n",
      "        \"zookeeper.znode.parent\": \"/hbase-unsecure\",\n",
      "        \"zookeeper.session.timeout\": \"30000\",\n",
      "        \"hbase.master.info.bindAddress\": \"0.0.0.0\",\n",
      "        \"hbase.tmp.dir\": \"/hadoop/hbase\",\n",
      "        \"hbase.hstore.blockingStoreFiles\": \"10\",\n",
      "        \"hbase.rpc.engine\": \"org.apache.hadoop.hbase.ipc.SecureRpcEngine\",\n",
      "        \"hfile.block.cache.size\": \"0.40\",\n",
      "        \"hbase.client.scanner.caching\": \"100\",\n",
      "        \"hbase.local.dir\": \"${hbase.tmp.dir}/local\",\n",
      "        \"hbase.hstore.compactionThreshold\": \"3\",\n",
      "        \"hbase.hregion.majorcompaction\": \"604800000\",\n",
      "        \"hbase.hregion.majorcompaction.jitter\": \"0.50\",\n",
      "        \"hbase_master_heapsize\": \"250\",\n",
      "        \"hbase.master.port\": \"60000\",\n",
      "        \"hbase.hregion.memstore.block.multiplier\": \"4\",\n",
      "        \"hbase.zookeeper.quorum\": \"%HOSTGROUP::host_group_1%\",\n",
      "        \"hbase.rootdir\": \"hdfs://%HOSTGROUP::host_group_1%:8020/apps/hbase/data\",\n",
      "        \"hbase.regionserver.global.memstore.upperLimit\": \"0.4\",\n",
      "        \"hbase.rpc.protection\": \"PRIVACY\",\n",
      "        \"hbase.hregion.max.filesize\": \"10737418240\",\n",
      "        \"hbase.zookeeper.property.clientPort\": \"2181\",\n",
      "        \"hbase.coprocessor.master.classes\": \"com.xasecure.authorization.hbase.XaSecureAuthorizationCoprocessor\",\n",
      "        \"hbase.hregion.memstore.flush.size\": \"134217728\",\n",
      "        \"hbase.zookeeper.useMulti\": \"true\"\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "#### Blueprint: Export Blueprint from existing Cluster\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster + '?format=blueprint')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://172.28.128.3:8080/api/v1/clusters/Sandbox?format=blueprint\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "list indices must be integers, not str",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-1e7ec3fa3958>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m#print(json.dumps(r.json(), indent=2))\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'configurations'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'hdfs-site'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: list indices must be integers, not str"
     ]
    }
   ],
   "source": [
    "#### Blueprint: Export Blueprint from existing Cluster\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster + '?format=blueprint')\n",
    "print(r.url)\n",
    "#print(json.dumps(r.json(), indent=2))\n",
    "\n",
    "print(r.json()['configurations']['hdfs-site'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Ambari Views"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### List Views\n",
    "\n",
    "r = s.get(api_url + '/api/v1/views')\n",
    "#print(json.dumps(r.json(), indent=2))\n",
    "for view in [item['ViewInfo']['view_name'] for item in r.json()['items']]:\n",
    "    print(view)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Change Configuration\n",
    "\n",
    "As part of the Stinger project, Tez brings many performance improvements to Hive. But as of HDP 2.2.0 they are not turned on my default.\n",
    "\n",
    "The below will make the required changes using the API.\n",
    "\n",
    "See the blog for more details: http://hortonworks.com/hadoop-tutorial/supercharging-interactive-queries-hive-tez/\n",
    "\n",
    "#### Process to change configuration from API:\n",
    "\n",
    "1. Get current configuration tag\n",
    "2. Get current configuration as JSON\n",
    "3. Update the configuration JSON with your changes\n",
    "4. Modify the configuration JSON to Ambari's required format, including setting new tag\n",
    "5. Submit the new configuration JSON to Ambari\n",
    "6. Restart services\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://172.28.128.3:8080/api/v1/clusters/Sandbox?fields=Clusters/desired_configs/hdfs-site\n",
      "{\n",
      "  \"Clusters\": {\n",
      "    \"version\": \"HDP-2.2\",\n",
      "    \"cluster_name\": \"Sandbox\",\n",
      "    \"desired_configs\": {\n",
      "      \"hdfs-site\": {\n",
      "        \"tag\": \"1\",\n",
      "        \"version\": 1,\n",
      "        \"user\": \"admin\"\n",
      "      }\n",
      "    }\n",
      "  },\n",
      "  \"href\": \"http://172.28.128.3:8080/api/v1/clusters/Sandbox?fields=Clusters/desired_configs/hdfs-site\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "#### Get current configuration tag\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster + '?fields=Clusters/desired_configs/hive-site')\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))\n",
    "\n",
    "tag = r.json()['Clusters']['desired_configs']['hive-site']['tag']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://172.28.128.3:8080/api/v1/clusters/Sandbox/configurations?type=hdfs-site&tag=1\n",
      "{\n",
      "  \"items\": [\n",
      "    {\n",
      "      \"tag\": \"1\",\n",
      "      \"Config\": {\n",
      "        \"cluster_name\": \"Sandbox\"\n",
      "      },\n",
      "      \"href\": \"http://172.28.128.3:8080/api/v1/clusters/Sandbox/configurations?type=hdfs-site&tag=1\",\n",
      "      \"properties\": {\n",
      "        \"dfs.replication.max\": \"50\",\n",
      "        \"dfs.datanode.data.dir.perm\": \"750\",\n",
      "        \"dfs.nfs3.dump.dir\": \"/tmp/.hdfs-nfs\",\n",
      "        \"dfs.datanode.max.transfer.threads\": \"1024\",\n",
      "        \"dfs.datanode.address\": \"0.0.0.0:50010\",\n",
      "        \"dfs.heartbeat.interval\": \"3\",\n",
      "        \"dfs.https.port\": \"50470\",\n",
      "        \"dfs.hosts.exclude\": \"/etc/hadoop/conf/dfs.exclude\",\n",
      "        \"dfs.namenode.checkpoint.period\": \"21600\",\n",
      "        \"dfs.datanode.max.xcievers\": \"1024\",\n",
      "        \"dfs.datanode.balance.bandwidthPerSec\": \"6250000\",\n",
      "        \"dfs.namenode.http-address\": \"sandbox.hortonworks.com:50070\",\n",
      "        \"dfs.blocksize\": \"134217728\",\n",
      "        \"dfs.datanode.ipc.address\": \"0.0.0.0:8010\",\n",
      "        \"dfs.datanode.data.dir\": \"/hadoop/hdfs/data\",\n",
      "        \"dfs.namenode.safemode.threshold-pct\": \"1.0f\",\n",
      "        \"dfs.journalnode.http-address\": \"0.0.0.0:8480\",\n",
      "        \"dfs.permissions.enabled\": \"true\",\n",
      "        \"dfs.namenode.https-address\": \"sandbox.hortonworks.com:50470\",\n",
      "        \"dfs.blockreport.initialDelay\": \"120\",\n",
      "        \"dfs.replication\": \"1\",\n",
      "        \"dfs.permissions.superusergroup\": \"hdfs\",\n",
      "        \"dfs.datanode.http.address\": \"0.0.0.0:50075\",\n",
      "        \"dfs.domain.socket.path\": \"/var/lib/hadoop-hdfs/dn_socket\",\n",
      "        \"dfs.webhdfs.enabled\": \"true\",\n",
      "        \"dfs.namenode.startup.delay.block.deletion.sec\": \"3600\",\n",
      "        \"dfs.datanode.https.address\": \"0.0.0.0:50475\",\n",
      "        \"dfs.cluster.administrators\": \" hdfs\",\n",
      "        \"dfs.namenode.stale.datanode.interval\": \"30000\",\n",
      "        \"dfs.namenode.avoid.write.stale.datanode\": \"true\",\n",
      "        \"dfs.namenode.handler.count\": \"100\",\n",
      "        \"dfs.datanode.du.reserved\": \"1073741824\",\n",
      "        \"dfs.http.policy\": \"HTTP_ONLY\",\n",
      "        \"dfs.namenode.checkpoint.txns\": \"1000000\",\n",
      "        \"fs.permissions.umask-mode\": \"022\",\n",
      "        \"dfs.block.size\": \"34217472\",\n",
      "        \"dfs.journalnode.edits.dir\": \"/hadoop/hdfs/journalnode\",\n",
      "        \"dfs.namenode.secondary.http-address\": \"sandbox.hortonworks.com:50090\",\n",
      "        \"dfs.datanode.failed.volumes.tolerated\": \"0\",\n",
      "        \"dfs.namenode.name.dir.restore\": \"true\",\n",
      "        \"dfs.client.read.shortcircuit.streams.cache.size\": \"4096\",\n",
      "        \"dfs.namenode.checkpoint.edits.dir\": \"${dfs.namenode.checkpoint.dir}\",\n",
      "        \"dfs.namenode.checkpoint.dir\": \"/hadoop/hdfs/namesecondary\",\n",
      "        \"dfs.nfs.exports.allowed.hosts\": \"* rw\",\n",
      "        \"dfs.support.append\": \"true\",\n",
      "        \"dfs.namenode.write.stale.datanode.ratio\": \"1.0f\",\n",
      "        \"dfs.namenode.name.dir\": \"/hadoop/hdfs/namenode\",\n",
      "        \"dfs.block.access.token.enable\": \"false\",\n",
      "        \"dfs.client.read.shortcircuit\": \"true\",\n",
      "        \"dfs.namenode.avoid.read.stale.datanode\": \"true\",\n",
      "        \"dfs.namenode.accesstime.precision\": \"3600000\"\n",
      "      },\n",
      "      \"properties_attributes\": {\n",
      "        \"final\": {\n",
      "          \"dfs.webhdfs.enabled\": \"true\",\n",
      "          \"dfs.datanode.data.dir\": \"true\",\n",
      "          \"dfs.datanode.failed.volumes.tolerated\": \"true\",\n",
      "          \"dfs.support.append\": \"true\",\n",
      "          \"dfs.namenode.name.dir\": \"true\",\n",
      "          \"dfs.namenode.http-address\": \"true\"\n",
      "        }\n",
      "      },\n",
      "      \"version\": 1,\n",
      "      \"type\": \"hdfs-site\"\n",
      "    }\n",
      "  ],\n",
      "  \"href\": \"http://172.28.128.3:8080/api/v1/clusters/Sandbox/configurations?type=hdfs-site&tag=1\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "### Get current configuration\n",
    "\n",
    "r = s.get(api_url + '/api/v1/clusters/' + cluster + '/configurations?type=hive-site&tag=' + tag)\n",
    "\n",
    "print(r.url)\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Change the configuration\n",
    "\n",
    "config_old = r.json()['items'][0]\n",
    "config_new = r.json()['items'][0]\n",
    "\n",
    "#### The configurations you want to change\n",
    "config_new['properties']['hive.execution.engine'] = 'tez'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Show the differences\n",
    "\n",
    "a = json.dumps(config_old, indent=2).splitlines(1)\n",
    "b = json.dumps(config_new, indent=2).splitlines(1)\n",
    "\n",
    "for line in difflib.unified_diff(a, b):\n",
    "     sys.stdout.write(line)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#### Manipulate the document to match the format Ambari expects\n",
    "\n",
    "#### Adds new configuration tag, deletes fields, and wraps in appropriate json\n",
    "config_new['tag'] = 'version' + str(int(round(time.time() * 1000000000)))\n",
    "del config_new['Config']\n",
    "del config_new['href']\n",
    "del config_new['version']\n",
    "config_new = {\"Clusters\": {\"desired_config\": config_new}}\n",
    "\n",
    "print(json.dumps(config_new, indent=2))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "body = config_new\n",
    "r = s.put(api_url + '/api/v1/clusters/' + cluster, data=json.dumps(body))\n",
    "\n",
    "print(r.url)\n",
    "print(r.status_code)\n",
    "assert r.status_code == 200\n",
    "print(\"Configuration changed successfully!\")\n",
    "print(json.dumps(r.json(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What you'll see from the Ambari UI:\n",
    "\n",
    "![Imgur](http://i.imgur.com/wtSqHyJl.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Examples: Managing Services & Components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#### Create Cluster from Blueprint\n",
    "\n",
    "service = 'GANGLIA'\n",
    "body = {\n",
    "  \"ServiceInfo\": {\n",
    "    \"state\" : \"INSTALLED”\n",
    "  }\n",
    "}\n",
    "\n",
    "r = s.post(api_url + '/api/v1/clusters/ + cluster + 'services/' + service , data=json.dumps(body))\n",
    "print(r.url)\n",
    "print(r.status_code)\n",
    "print(json.dumps(r.json(), indent=2))\n",
    "status_url = r.json()['href']\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PUT /clusters/c1/services/HDFS\n",
    "{\n",
    "  \"ServiceInfo\": {\n",
    "    \"state\" : \"INSTALLED”\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "PUT /api/v1/clusters/c1/services?ServiceInfo/state=INSTALLED \n",
    "{\n",
    "  \"ServiceInfo\": {\n",
    "    \"state\" : \"STARTED”\n",
    "  }\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}