Network Automation – Part 2

As mentioned in Part 1, the Network Interrogation Tool (NIT) is a web service written in Python/Flask that sits in the middle between SolarWinds and Rundeck to provide Rundeck with the various lists of options that the user must select for each job.

I’ll start with a very basic overview of Flask, but any web framework could be used for this layer. The primary goal is to get useful data out of your various tools and format them ready for Rundeck.

Flask is just one of many web frameworks that can be utilised in the Python programming language to build a web-based application. It is a micro framework, as opposed to something like Django which is a full-blown solution that contains all kinds of features already built-in. I would have probably used Django if I hadn’t discovered Rundeck, but as NIT was to be a relatively straightforward project, I decided to keep it simple with Flask.

The Flask quick start guide is a good resource if you want details, but for now it is enough to know that you can use the ‘route’ decorator to tell Flask what URLs should trigger your functions. If you are familiar with MVC, these would be analogous to controllers.


from flask import Flask
app = Flask(__name__)

@app.route('/subnets/list/', defaults={'output': 'html'})
@app.route('/subnets/list/<string:output>')
def subnets_list(output):
# Subnet list code here.

Once the above application is loaded, a user navigating to ‘yourdomain.com/subnets/list’ would trigger the code contained within the subnets_list function. You can also see that we have defined an optional string variable named ‘output’. If nothing is appended to the URL, the string will default to ‘html’. You can capture a number of variables in this way, but in this case we are using it to switch between different outputs such as HTML or JSON.

When /json is appended to the URL, we then use Flask’s ‘jsonify()’ method to format Python lists as JSON and then return them to the browser. If the user doesn’t append anything then it will default to a HTML view. The HTML view is generated using Flask’s ‘render_template()’ method.

def subnets_list(output):
    # Object creation and error handling omitted.

    if output.lower() == 'json':
        # Convert list of Subnet objects into a list of tuples.
        json_subnets = []

        for subnet in subnets.list:
            json_subnets.append((str(subnet.netaddress), subnet.name))

        return jsonify(json_subnets)
    else:
        # Populate the title and description for the view.
        title = 'Subnets'
        description = 'This is a live list of subnets fed from NMS. ' \
                      'Append \'/json\' to the url for the Json endpoint.'

        # Populate the table's headers.
        table_headers = ['Subnet', 'Name']

        # Populate a list of tuples with data.
        table_data = []

        for subnet in subnets.list:
            table_data.append((subnet.netaddress, subnet.name))

        return render_template('table.html',
                               title=title,
                               description=description,
                               table_headers=table_headers,
                               table_data=table_data)

The ‘render_template()’ method takes a template file as a parameter. This template file needs to be written in HTML along with the templating engine Jinja2. The example code snippet below shows a table header being generated from a passed list named ‘table_headers’. By passing our page title, description, table headers and table data into the template, we can dynamically generate pages while using a single template file.


<thead>
    <tr>
        {% for header in table_headers %}
            <th>{{ header }}</th>
        {% endfor %}
    </tr>
</thead>

The HTML/CSS itself utilises the popular CSS library Bootstrap. With Bootstrap you can easily create responsive pages with navigation bars, image carousels etc.

While developing your application locally, Flask will start a basic web server on your PC for testing. This test server shouldn’t be used in production and you will need to integrate your application with a proper web server, such as Apache. To integrate Apache with your Flask application, a Web Server Gateway Interface (WSGI) file is used. The WSGI file (shown below), tells Apache how to start your application.

import sys
sys.path.insert(0, '/opt/scripts/nit')

from nit import app as application

You can then reference the .wsgi file from your httpd.conf file. You will also need to install mod_wsgi and restart Apache.

<VirtualHost *:80>
   ServerName nit.yourdomain.com
   WSGIScriptAlias / /opt/scripts/nit/apache/nit.wsgi
   WSGIDaemonProcess nit.yourdomain.com user=scripts group=scripts processes=2 threads=25
   WSGIProcessGroup nit.yourdomain.com

   <Directory /opt/scripts/nit/apache>
      Require all granted
   </Directory>
</VirtualHost>

Now that we have covered the web front-end of NIT, that front-end needs some data to display. This data is fetched from SolarWinds and manipulated by a set of wrapper classes and their methods. Below is the code that we might use within our previously shown ‘subnets_list’ function to instantiate an object from our ‘Subnets’ class.


# Instantiate a SolarWinds API instance using the
# details from the application config file.
sw_api = solarwinds.Api(app.config['SW_APIURL'], app.config['SW_USERNAME'], app.config['SW_PASSWORD'])

# Pass the above API to the constructor of Subnets.
subnets = nms.Subnets(sw_api)

The ‘Subnets’ class (shown below) takes an instance of the SolarWinds API in its constructor. This API is then used within a list property to fetch all subnets from our IP Address Management (IPAM) system using SQL-like statements that the SolarWinds API expects.

It then iterates through the results, instantiates a new Subnet (singular) object for each record and then adds those new objects to a Python list. Finally it returns the list of ‘Subnet’ objects to the calling function.


class Subnets(object):
    '''Subnets class.
    Attributes:
        sw_api: A SolarWinds API instance.
    '''
    def __init__(self, sw_api=None):
        if sw_api is None:
            raise errors.Error('[%s.%s] - You must provide a SolarWinds API instance.' % (__name__, self.__class__.__name__))
        else:
            self.sw_api = sw_api

    @property
    def list(self):
        '''Get a list of subnets and their details from SolarWinds.
        Returns:
            A list of subnet objects.
        '''
        sw_subnets = self.sw_api.query('SELECT DISTINCT Address, CIDR, FriendlyName FROM IPAM.Subnet ';
                                       'WHERE Address IS NOT NULL AND AddressMask IS NOT NULL')

        subnets_list = []

        for sw_subnet in sw_subnets['results']:
            if '0.0.0.0' not in str(sw_subnet['Address']):
                new_subnet = Subnet(str(sw_subnet['Address']) + '/' + str(sw_subnet['CIDR']), sw_subnet['FriendlyName'])
                subnets_list.append(new_subnet)

        return subnets_list

class Subnet(object):
    '''Subnet class.
    Attributes:
        netaddress: The network address in IP/CIDR format.
        name: The subnet's name, if available.
    '''
    def __init__(self, netaddress=None, name=''):

        if netaddress is None:
            raise errors.Error('[%s.%s] - You must provide a network address for the subnet in CIDR format.';
                               % (__name__, self.__class__.__name__))
        else:
            self.netaddress = IPNetwork(netaddress)
            self.name = name

The ‘list’ property of the class is subsequently used within our decorated Flask functions to output the subnets as JSON that Rundeck can utilise.

NITAvailJson

Another property of the Subnets class is ‘available’, which utilises the netaddr library to return a list of the largest available subnets that can still be utilised. This is the HTML template view without the ‘/json’ appended to the URL.

NITAvailable

Similarly, the class method ‘available_bysize()’ takes a CIDR value (24, 25 etc) and returns a list of available subnets that match the given size.

Your requirements and data repositories may be different, but hopefully this has served as an overview of how you might get data out of your own tools and formatted as JSON endpoints. Once all of your required endpoints are available and are displaying the correct data, you can now use those URLs in Rundeck to provide choices to your users when they are preparing automation jobs.

RundeckRemoteURLJson

The user is now forced to select a valid subnet that we can be confident is available.

RundeckAvailableSubnets

In Part 3 we will discuss what happens when a user executes a job from within Rundeck.

Network Automation – Part 1

As an organisation, we are spread across over 100 sites, ranging from small portacabins to large purpose-built offices. All of these sites are geographically dispersed across an area the size of Belgium.

With budgets tight within the NHS, we are constantly looking to consolidate or get the best value for money from our estate. This results in a high turnover of sites and generates quite a bit of work for our small team.

We have tried to standardise on site configurations for a number of years, but there were always small inconsistencies in configuration, such as switch uplinks on different ports or the odd site where we put in a 48 port switch instead of a 24.

We used configuration templates to a certain degree, but things like subnet and wildcard masks were edited by hand. This usually resulted in multiple hours on the phone between engineers on site and those back at head office trying to diagnose why VPN tunnels were not coming up.

After listening to many hours of the Packet Pushers podcast on the way in to work, the Ansible/Python preaching started to break through. It was time to automate all the things!

allthethings

I wanted a way for our junior engineers to select from a list of available subnets, site codes, bandwidths etc and have the configurations be generated automatically. These could just be emailed to them at first, but ultimately I would like it to push out to the devices directly.

I will start with the user interface part of the solution first, as this will help to explain why some of the other components are required.

As I thought about it, a few requirements for the web front-end started to emerge.

  • Active Directory authentication and job level permissions based upon security groups.
  • Selection lists that can be read from a remote URL as JSON/Text.
  • ‘Default’ and ‘Required’ values.
  • Regex support for validating text inputs.
  • Ability to use entered/selected data as variables within the command line parameters of scripts.
  • Decent looking UI.
  • Log job executions to file so it can be picked up by a syslog collector.
  • Free & open source.

I hunted around for something that fit the above requirements but struggled at first. I started to look into Python web frameworks such as Django and Flask with a view to writing my own. As the scale of the programming task grew, I invested more time in searching for an off-the-shelf package that I could customise. Thankfully I found Rundeck, which is an excellent open source project.

rundeck

Rundeck met all of my requirements and was relatively simple to install. A couple of the optional configuration tasks were a little tricky (namely AD integration and SSL certificates), so I may do a separate post about those.

Shown below is an overview diagram of the various software components and how they integrate.

AutomationIntegration

We use SolarWinds for our network monitoring and IP Address Management (IPAM), so this would be the source of truth for the majority of the required configuration data. Unfortunately it wouldn’t be the correct data, nor would it be in a format that Rundeck could use. For example, the SolarWinds IPAM API could give me a list of used subnets, but not all available /26 or /27 subnets.

This conversion between SolarWinds data and Rundeck is where the Network Interrogation Tool (NIT) comes in, which will be mentioned later in this post, but also gets its own dedicated part in this automation series.

The basic steps for the installation of Rundeck on Fedora are given below, but you should check the latest instructions on the Rundeck website if you install it yourself.

# Install Rundeck and its dependencies.

sudo yum install java-1.8.0
sudo dnf install yum
sudo rpm -Uvh http://repo.rundeck.org/latest.rpm
sudo yum install rundeck
sudo service rundeckd start

# Update the rundeck config file to change the hostname.

cd /etc/rundeck/
sudo nano rundeck-config.properties

Change the config line:
grails.serverURL=http://rundeck.yourdomain.com:4440

# Update the framework.properties file.

sudo nano framework.properties

Change the config lines:
framework.server.name = rundeck.yourdomain.com
framework.server.hostname = rundeck.yourdomain.com
framework.server.port = 4440
framework.server.url = http://rundeck.yourdomain.com:4440

# Add firewall rules.

sudo firewall-cmd --add-port=4440/tcp
sudo firewall-cmd --permanent --add-port=4440/tcp
sudo firewall-cmd --add-port=4443/tcp
sudo firewall-cmd --permanent --add-port=4443/tcp

# Restart the server.

shutdown -r now

Following the successful installation of Rundeck, jobs were created for each of our remote site Network Configuration models, which we have named NC1, NC2, NC3 and so on; because you can never have enough initialisms and acronyms in IT.

You can also connect Rundeck to GitHub so that the job definitions themselves are version controlled. I signed up for a basic GitHub organisation account as I knew that it would be used for other parts of the project (the main difference between the GitHub free and paid plans are that you can have private repositories). Once you have linked Rundeck to GitHub, jobs that have been modified are highlighted until they have been commited.

Shown below is an example job setup screen. Each of the inputs are either fixed drop-down lists or are tested against a regex validation string. This ensures that the generated configurations are always using consistent data. If the site name needs to be in ‘Title Case’ then it won’t let you proceed until it is entered correctly, perfect for anal people like me.

RundeckOptions

When defining a drop-down list of options, you have to point Rundeck at a web address with a JavaScript Object Notation (JSON) feed containing the data. This is where the aforementioned Network Interrogation Tool (NIT) comes in. NIT is a web service written in Python/Flask that sits in the middle between SolarWinds and Rundeck to provide Rundeck with the various lists of options that the user must select for each job.

Once we have all of the required variables, we can call scripts with command line options to perform the various tasks for us.

RundeckWorkflow

As of today we perform the following actions, but they can be expanded easily.

  • Create the DHCP scopes.
  • Create the nodes in NMS for monitoring.
  • Generate the router and switch configurations.
  • Add any tasks that can’t be automated to Asana.

RundeckExecution

I will go in to some of the detail of the Python scripts in a later part of this series.

Rundeck does much more than outlined here, but for our purposes it is enough to define jobs that capture sanitised variables from the user and then call scripts that perform a series of actions.

That’s probably a good place to stop on the front-end side of things.