CLI Scraping

In this lesson, we'll explore how to use the Netmiko library to automate interactions with network devices. We'll cover three scripts that progressively build on each other, demonstrating how to connect to devices, retrieve interface information, and handle multiple devices efficiently.

Prerequisites - venv and Netmiko

Install Netmiko in a new virtual environment

mkdir cli-scraping
cd cli-scraping
uv venv
uv pip install netmiko

What we've done here:

• Created a new directory cli-scraping.
• Changed to the new directory.
• Created a new virtual environment using uv venv.
• Installed the netmiko library using uv pip install netmiko.

What is uv?

uv is a command-line tool that simplifies common tasks in Python development. It is designed to be easy to use and remember. uv installation instructions.

Third-Party Libraries and imports

We have installed netmiko, a third-party library for the first time. Third party libraries are not included in the Python standard library. They are developed by the community and can be installed using pip, uv, pdm and other similar tools.

When we use third party libraries, we need to import them into our scripts. This is done using the import statement. In the following scripts, we import the netmiko library to use its functions.

Importing the netmiko library

import netmiko

The above code import the netmiko module into our script. A module is a library or file containing Python definitions and statements. Modules can contain functions, classes, and variables, and can be imported into other modules or scripts. In the case of netmiko, it is a module that provides wide range of functions and classes to interact with network devices.

In the example today we will use the ConnectHandler class from the netmiko module to establish a connection to a network device.

Running scripts in a virtual environment

When you install a library in a virtual environment, you need to activate the virtual environment to use the library.

Visual Studio Code users can use the integrated terminal to run the scripts - the terminal will automatically activate the virtual environment for you, when you press the play / run button in the top right corner of the editor.

If you like execute the script on the command line in a terminal, you would normally run the script like this:

python script.py

This can still be done, but you need to activate the virtual environment first. To activate the virtual environment, run:

source .venv/bin/activate

To deactivate the virtual environment, run:

deactivate

Even better! Use uv run to execute your scripts

Forget about activating and deactivating the virtual environment. Use uv run to execute your scripts.

uv run myscript.py

Basic Connection and Command Execution with Netmiko

This section demonstrates how to establish a connection to a network device and execute a command to retrieve interface information.

Importing required libraries

import netmiko
from getpass import getpass
from pprint import pprint

As we already established, we installed netmiko to handle SSH connections.
Next, we use the getpass function from the standard library to securely prompt the user for a password. The pprint function is used to print the output in a structured format, to help us build the code step by step.

Getting the connection details

ops = "cisco_ios"
ip_addr = input("Device IP or hostname: ")
user = input("Enter username: ")
pw = getpass("Enter password: ")

the ConnectHandler class requires a few parameters to establish a connection, so we start by collecting the necessary information from the user. We have hardcoded the device type as cisco_ios for now, but this can be changed to a variable if needed.

Establishing a connection

handler = netmiko.ConnectHandler(
    device_type=ops,
    username=user,
    password=pw,
    host=ip_addr,
)

We instantiate Netmiko's ConnectHandler class with the device type, username, password, and IP address. This establishes a connection to the device. The handler object is an instance of the ConnectHandler class used to interact with the device.

Executing a command and getting the output in a structured format

# saves the result to a variable - note the use_textfsm toggle, it gives us structured data rather than a text blob
result = handler.send_command("show ip interface brief", use_textfsm=True)
pprint(result)

After the hard work is done it's pretty simple to send commands to our device, and have the results printed to the screen. In this example we are using the show ip interface brief command, and the use_textfsm=True option to format the output as structured data.

We get a nice list of dicts as output, which is easy to manipulate or use with Python. This is thanks to Netmiko, TextFSM and NTC-templates doing a lot of background work for us, out of the box.

As a final task, we could choose to print only the name and status of the interfaces to practice working with structured data like lists and dictionaries.

Printing only the interface name and status

for interface in result:
    print(f"{interface['interface']} is {interface['status']}")

Example Output

GigabitEthernet0/0 is up
GigabitEthernet0/1 is down
Loopback0 is up

Using a Dictionary for Device Settings

This script improves upon the first by using a dictionary to store device settings, making the code more organized and flexible.

Thanks to the starred expression **device, we can pass the dictionary directly to ConnectHandler instead of specifying each setting in the function call. This requires the dict keys to match the parameter names expected by ConnectHandler. Let's rename the variables to match the expected names and store them as dict keys instead.

Using a Dictionary for Device Settings

import netmiko
from getpass import getpass
from pprint import pprint

# create an empty settings dict
device = {}

# specify the device type
device["device_type"] = "cisco_ios"

# ask the user for a device hostname/ip:
device["host"] = input("Device IP or hostname: ")

# ask the user for credentials
device["username"] = input("Enter username: ")
device["password"] = getpass("Enter password: ")

# the double starred expression allows you to use a dict instead of specifying each setting in the function call,
# this is also the way the netmiko documentation describes it (https://ktbyers.github.io/netmiko/#getting-started-1)
handler = netmiko.ConnectHandler(**device)

# query the device
result = handler.send_command("show ip interface brief", use_textfsm=True)

# print the result in a simple human-readable format
for interface in result:
    print(f"{interface['interface']} is {interface['status']}")

Handling Multiple Devices

This script extends the previous examples to handle multiple devices, demonstrating how to iterate over a list of devices and execute commands on each.

Determine amount of devices to connect to

import netmiko
from getpass import getpass
from pprint import pprint

# create a devices list to hold multiple devices
devices = []

num_devices = int(input("How many devices do you want to connect to? "))

We start by asking the user how many devices they want to connect to, and store the number in a variable. This will be used to determine how many times to run the loop.

We also create an empty list to store the device dictionaries, as we need to create a connection dict for each device.

Building the devices list

# use the range command to create a loop that runs the number of times specified by the user
for _ in range(num_devices):
    # create a settings dict per device
    device = {
        "device_type": "cisco_ios",
        "host": input("Device IP or hostname: "),
        "username": input("Enter username: "),
        "password": getpass("Enter password: "),
    }

    # add the device to the devices list
    devices.append(device)

This is a bit cool, instead of values in the dict, we simply ask the user for the values and store them directly in the dict.

Another thing to notice is the use of the _ variable in the loop. This is a common convention in Python to indicate that the variable is not used in the loop. In this case, we don't need the loop index, so we use _ as a thow-away placeholder.

Storing Results

# create an empty list to store the results
interfaces = []

# iterate over the devices list and connect to each device
for device in devices:
    # for each device, create a handler and execute the command
    handler = netmiko.ConnectHandler(**device)
    result = handler.send_command("show ip interface brief", use_textfsm=True)
    interfaces.append(result)

# print the final result in a structured data format - easy to manipulate or use with Python.
for result in interfaces:
    pprint(result)

Lost in transit?

Unfortunately the script does not correlate device with interface information. This means that you end up with a list of interfaces for each device, but you don't know which device they belong to. Perhaps a dictionary could be used to store the device and its interfaces instead? Or perhaps you should add another "hostname" key to the result dict before appending it, to keep track of the device name? Give it a try!

Summary

In this lesson, we covered how to use Netmiko to automate interactions with network devices. We started with a basic connection script, improved it by using a dictionary for device settings, and finally extended it to handle multiple devices efficiently.

Key Points

• Netmiko simplifies network automation by providing easy-to-use methods for connecting to devices and executing commands.
• Using dictionaries for device settings improves code organization and flexibility.
• Iterating over a list of devices allows for scalable automation.

Try It Yourself

Extend the final script to include additional commands or handle different device types.

Congratulations on completing this lesson on network automation with Netmiko! Keep practicing to enhance your automation skills and streamline your network management tasks.