A lesson in legacy routing using virtual network interfaces

There comes a point in every successful and growing infrastructure where the implementation needs of said network will outgrow its design needs. Fair enough; there's only so much one can account for (especially in this industry), and managing a network to be flexible and change with business needs is what separates the men from the boys (so-to-speak).

Our DC infrastructure hit this point fairly recently. We have a number of db servers split out into shards. Each shard is designed with a master/slave replication topology, keeping one of the slaves as an analysis machine. The analysis machines were originally meant to perform some very limited and specific work, but of course this expanded and evolved as time progressed. Eventually we had the need to upgrade the analysis hardware, but since there were only one in each shard it would be hard to keep them down for the length of time needed to do our work.

The decision was made to move a copy of the analysis database onto a db slave, and migrate the analysis IP to the slave as a virtual IP. In theory, this would be simple. Machines from various VLANs would continue to connect to the "analysis machines" by the same IP addresses they always have, but they would now be temporarily served by the slaves.

Creating the virtual aliases were easy. We could use ifconfig to add an additional interface like so:

# ifconfig bond0:1 10.2.2.59 netmask 255.255.255.0

The trouble now came with the routes. We added several routes to the machine in our usual manner.

route add -net 10.3.3.0/24 gw 10.2.2.1 dev bond0
route add -net 10.4.4.0/24 gw 10.2.2.1 dev bond0
route add -net 10.5.5.0/24 gw 10.2.2.1 dev bond0
route add -net 10.6.6.0/24 gw 10.2.2.1 dev bond0

But when anyone tried to connect from these subnets, their connections would time out.

As a test, I tried connecting to the primary IP over telnet:

$ telnet 10.2.2.52 3306

Immediately I was presented with a MySQL banner. Yet doing the same for the virtual IP would simply time-out. We verified the routes, just to make sure everything seemed apropriate.

$ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref   Use Iface
10.2.2.0        *               255.255.255.0   U     0      0     0   bond0
10.3.3.0        10.2.2.2        255.255.255.0   UG    0      0     0   bond0
10.4.4.0        10.4.4.2        255.255.255.0   UG    0      0     0   bond0
10.5.5.0        10.5.5.2        255.255.255.0   UG    0      0     0   bond0
10.6.6.0        10.6.6.2        255.255.255.0   UG    0      0     0   bond0
default         10.2.2.1        0.0.0.0         UG    0      0     0   bond0

Then we took packet captures on the network to see where our traffic was going. Packets were indeed moving into the db slaves, but none returned. This smelled an awful lot like a routing issue.

As a simple step, we removed our routes, and added them back using the virtual designation as a test.

route add -net 10.3.3.0/24 gw 10.2.2.1 dev bond0:1
route add -net 10.4.4.0/24 gw 10.2.2.1 dev bond0:1
route add -net 10.5.5.0/24 gw 10.2.2.1 dev bond0:1
route add -net 10.6.6.0/24 gw 10.2.2.1 dev bond0:1

The result here was peculiar. Examining the routing table again, we received the same output as before:

$ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref   Use Iface
10.2.2.0        *               255.255.255.0   U     0      0     0   bond0
10.3.3.0        10.2.2.2        255.255.255.0   UG    0      0     0   bond0
10.4.4.0        10.4.4.2        255.255.255.0   UG    0      0     0   bond0
10.5.5.0        10.5.5.2        255.255.255.0   UG    0      0     0   bond0
10.6.6.0        10.6.6.2        255.255.255.0   UG    0      0     0   bond0
default         10.2.2.1        0.0.0.0         UG    0      0     0   bond0

We tried testing conectivity anyway, but unfortunately the result was the same. We next tried removing the routes and adding them in with the iproute2 tools. Again, this was unsuccessful. The primary interface would continue to work, and the virtual interface acted as a virtual black hole. There was a bit of head-scratching next, and quite a bit of reading, but we eventually did come to the bottom of it.

You see, there are a few nasty elements at play here. Back in the golden days of Unix, multiple IPs per interface weren't even a thing, so when the net-tools suite was written (ifconfig, route, netstat, etc) their support didn't exist. Once it became desirable to have them, the idea of creating virtual interfaces was monkey-patched on-top of the existing infrastructure. This was problematic in it's own right, but was compounded by efforts to actually correct this later down the road. The iproute2 suite was eventually written to work more integrally with a completely redesigned network subsystem in the Linux kernel. This subsystem would be far more flexible and advanced that the original model, and as a result much of the administration and support for advanced networking features would become just easier and more reliable to use. In order to facilitate this migration, the old net-tools have been deprectated. In the meantime, they have been mostly patched to map behind the scenes on-top of the new network subsystem. Sadly, this leaves a bunch of legacy components in a sorry state. The new networking subsystem has no real concept of virtual interfaces, so they don't map properly. As a result, trying to use the iproute2 utils with VIPs can have unexpected results (often in the form of virtual interfaces falling back to the primary interface definition: i.e., bond0:1 becomes bond0).

This whole debacle takes one step further and shows broken behaviour with the original net-tools. As it runs out, if you explicitly bind a route to a particular virtual interface, it will always default to the primary interface. However, not defining it allows all interfaces (even virtual interfaces) to use the route. The madening part is, if you don't specify an interface, the routing table still shows you bound to the primary interface. So viewing your routing table will give you identical output whether you have explicity bound the route or not.

The final solution here was to simply remove all of our routes and redefine them without explicit binds:

$ sudo route add -net 10.3.3.0/24 gw 10.2.2.1
$ sudo route add -net 10.4.4.0/24 gw 10.2.2.1
$ sudo route add -net 10.5.5.0/24 gw 10.2.2.1
$ sudo route add -net 10.6.6.0/24 gw 10.2.2.1
$ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref   Use Iface
10.2.2.0        *               255.255.255.0   U     0      0     0   bond0
10.3.3.0        10.2.2.2        255.255.255.0   UG    0      0     0   bond0
10.4.4.0        10.4.4.2        255.255.255.0   UG    0      0     0   bond0
10.5.5.0        10.5.5.2        255.255.255.0   UG    0      0     0   bond0
10.6.6.0        10.6.6.2        255.255.255.0   UG    0      0     0   bond0
default         10.2.2.1        0.0.0.0         UG    0      0     0   bond0

From this point, everything immediately started working. Looking at the routing tables before and after the fix, you wouldn't even notice the difference.

The moral of the story? They heyday of net-tools and VIPs has passed. It's time to just start using iproute2 and multiple address interfaces the way the good developers have intended.

NOTE: Addresses and routes have been altered to protect the original infrastructure, but still accurately reflect the original behaviour for each example.

Suspending and resuming processes

While performing regular administrative duties on any server, you will often find yourself in the position to need to suspend and resume processes. Luckily, nearly all shells on modern systems will have some sort of job control built in to address these needs.

A job is really just another term for a process group. This will commonly be just a single application, but is not exclusive to that idea. Any process group which is not receiving terminal input is considered a background job, while a process group that does receive terminal input is a foreground job. In order for a shell to control these jobs, various IPC calls (termed signals) can be sent to the job which can be interpreted by various signal handlers inside the program.

Conveniently, most programs don't need to define any signal handlers. There is a default set that gets included into each program, and can be generically used.

The most common signals for job control are often as follows: SIGTSTOP SIGCONT SIGINT

There are two main ways we can send these signals. * We can use keyboard shortcuts to instruct the shell to send a particular signal * We can use the kill command to send a signal to a particular process ID

Traditionally, the suspend character has been mapped to Ctrl+Z (SIGSTOP), which will temporarily suspend your process. Similarly, the interrupt character can be signaled by pressing Ctrl+C (SIGINT) on your keyboard.

A lesser known secret is we can send any signal we want to an application using the kill command. It doesn't have to simply interrupt a program. Let's say we have a process with the ID of 1337, and we want to suspend it. We can easily do the following: $ kill -SIGSTOP 1337

Likewise, once we are ready to resume the process, we can use kill to send it the SIGCONT signal: $ kill -SIGCONT 1337

And there you have it! The man pages cover a wide range of additional signals that can be used for various purposes, but mastering even just these simple signal techniques will make a significant difference to your administrative abilities.

Adding timestamps to Bash history

The Bash history feature is an invaluable tool which allows users to recall commands previously entered into their shell with relative ease. This makes it easy to enter repeated commands and keep track of what was done on a system. By default, however, a user is unable to see when these commands were actually entered. When auditing a system, it can sometimes be useful to see this type of information, for example when trying to determine how and when a file may have gone missing on the file system. Since Bash version 3, however, you are able to enable time-stamping of entries for review later.

Applicable versions of Bash provide the environment variable HISTTIMEFORMAT which you can set for this purpose. Although it is empty by default, you can simple assign a time format string to enable time-stamping.

The easiest example is as follows:

export HISTTIMEFORMAT="%F %T "

Any new commands entered after this variable is set will be stamped, and the stamps will be accessible when you use the history command.

$ ls
file1 file2 file3
$ date
Mon Oct 20 11:54:37 EDT 2014
$ cal
    October 2014    
Su Mo Tu We Th Fr Sa
          1 2 3 4
 5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31
$ history
 1001 2014-10-20 11:54:08 export HISTTIMEFORMAT="%F %T "
 1002 2014-10-20 11:54:28 ls
 1003 2014-10-20 11:54:37 date
 1004 2014-10-20 11:54:49 cal
 1005 2014-10-20 11:55:29 history
$ 

To make this persistent, you can easily add the export command to your .bashrc file. You can also look at the man pages to get more information on how to craft custom time format strings: man 3 strftime

It's worth noting that once you enable time-stamping, any existing entries in your .bash_history file will automatically get stamped with the current time. This is important because it will not be possible to recover timestamps of commands that were entered before you enabled time-stamping. You must make sure to enable time-stamping before using your shell if you wish to have this information.

A Foundation for Buffer Overflow Attacks

Buffer overflows are some of the oldest and most important attacks against computer technology. These types of attacks are commonly associated with low level languages (like C and C++), but are not exclusive to them. Despite the importance of understanding this type of attack, there are still a large number of technical people who still don't fully understand it. Hopefully, this article will give you some basic insight into how buffer overflows work and why they are useful/dangerous. This guide will attempt to give you a very basic understanding on the concepts behind these attacks, but please bare in mind: due to a variety of protection mechanisms that are built into modern systems it is actually much more difficult to exploit modern systems using these attacks than it used to be. (If you are reading this to play The Hacker's Sandbox, then everything here is still applicable.)

So what is a buffer overflow?

Buffer overflows are attacks that allow for an unintentional (by design) change in the logical flow of an application. When information needs to be stored in memory, it will either land in the stack, or (for long-term and dynamic data) the heap. (This article will deal specifically with Stack overflows, though similar concepts will apply to the heap as well.) The stack is a region of memory that gets created on every thread that your application is running on. It works using a Last-In, First Out (LIFO) model, where data is said to be either pushed onto or popped off of the stack. When an application wants to store data into a buffer, it will allocate memory on the stack to be filled for that purpose. It can later be manipulated or moved to the heap as needed. The danger comes in when the application tries to write more data to the stack than has been allocated for the buffer. In this instance, an application can overwrite other important locations in memory, causing the program to corrupt or the logical flow of the program to change.

Examining the stack

To understand this a little better, let's take a look at an abstraction of the stack. You can easily visualize the stack using the following parts:

[BUFFER][STACK_FRAME_POINTER][RETURN_ADDRESS]

Image that you have three cards that you want to put down in the buffer. Card A, Card B, and Card C. You can push them down one at a time, first Card A, then Card B, last Card C. You will now have a buffer on the stack that looks like this:

  • Card C
  • Card B
  • Card A

If you wanted to then access Card B, first you would have to pop Card C off of the stack in order to access it. This is the basis for memory management on the stack, and is crucial to understand for understanding buffer overflows. We can take this a step further and see how a real-world function would work with the stack.

I am going to use the mmap() system function exposed by the Linux kernel. Looking at the man pages, you can see the function looks like this:

void *mmap(
 void *addr,
 size_t length,
 int prot,
 int flags,
 int fd,
 off_t offset
 );

If we were to call this function, first we would push the return variable, then we would push each argument onto the stack in reverse order, and finally we would make the call to mmap(). (In this case, the function is void, so no return variable will be pushed.) Abstractly, it would look something like this:

PUSH off_t offset
PUSH int fd,
PUSH int flags,
PUSH int prot,
PUSH size_t length,
PUSH void *addr,
CALL mmap

Once finished, our actual stack will look like this:

void *addr,
size_t length,
int prot,
int flags,
int fd,
off_t offset

Don't worry about the actual data here, the important part is that you understand how the stack works to be able to understand how to exploit it. This is all well and good, but how does it actually help you to control the flow of a program? Well, In addition to holding the buffer, the stack also holds a frame pointer, and the address to return to after a function has finished executing. Let's take another look at that stack:

[BUFFER][STACK_FRAME_POINTER][RETURN_ADDRESS]

Did you notice it? The buffer is placed on the stack before the return address. If we could keep pushing data onto to buffer until it overflows into the RETURN_ADDRESS, we would change where the program thinks it should be jumping back to.

A little bit of math

Ok, so now we know the theory behind overflowing buffers, but how do we know how much data we need to actually exploit this? The truth is, that depends on a few factors, such as your architecture. Computers of different architectures will follow this same stack model, but their memory allocation won't always be the same. You see, every machine architecture has a minimum amount of storage it needs to allocate. Think of it in terms of blocks. A system can only reserve 1 block at a time. If your program were to request only half a block worth of data, the system would need to reserve a full block to satiate that request.

On a 32 bit system, this block is going to be 32 bits. On a 64 bit system, each block will be 64 bits.

So if we were to request 1 byte of data (8 bits), on a 32 bit system, we would end up reserving 4 bytes (32 bits), while on a 64 bit system, we would end up reserving 8 bytes (64 bits).

In order to actually change the RETURN_ADDRESS, first we would need to fill our complete buffer (all of the space reserved for us), then we would need to overflow the stack pointer (this will usually be 1 block of space), and finally we would be able to overwrite the RETURN_ADDRESS (also 1 block in size.)

To complicate this matter more, modern compilers will often use padding on the buffers which will depend on various factors (such as data type and size) which will effect the reserved memory size. Often, the easiest way to determine how large your buffer is would be to open your application in a debugger and actually look at the assembly. If you are playing The Hacker's Sandbox, assume that there is no padding from the compiler.

Some practical examples

The following is a small C application (overflow.c) which will have no measures in place to protect it from buffer overflows.

#include <string.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    char buf[10];
    strcpy(buf,argv[1]);
    return 0;
}

void the_shell()
{
    printf("pwned!
");
    system("/bin/sh");
}

This program is pretty simple. It takes the first argument into the program and places that into a buffer that is 10 bytes long. Notice that there is also a function called the_shell() which never actually gets called. The code exists inside of it, however, to spawn a shell running as the owner of the program. (The truth is, this would be a pretty useless program in real life, but it suits our demonstration very well.)

We could easily try to compile this into a program called overflow. For the purposes of this demonstration, we will be compiling on a 32 bit architecture:

$ gcc -o overflow overflow.c

Note: This compile would be incomplete on a modern machine. Modern compilers, like gcc, would have multiple mechanisms in place, like a stack guard and NX memory regions, which you would need to disable and/or bypass in order to successfully exploit your binaries.

Now, we can try playing around with this for a bit. If we passed an argument of 10 characters (say 0000000000), we will notice the program does nothing, and simply exits. However, if we pass a much longer argument, of say 30 characters, it will crash. The application crashes because key pieces of data become corrupt (like our return address, which likely now points to a nonexistent region of memory), and the logical flow of the program can not continue. So how do we send just enough data to overflow the stack and change the flow of execution without crashing the program? Let's try to fill the stack just enough to trick it into returning to our hidden function the_shell(). Conceptually, we will need to fill the stack so it looks like this on our 32 bit system:

[garbage data] 12 bytes [garbage data] 4 bytes [address of the_shell] 4 bytes

Remember, when the buffer requests 10 bytes, the smallest amount of 32 bit blocks (4 bytes) we can use is 3. Thus, 4 bytes X 3 = 12 bytes.

First, we would fire up our binary in a debugger. We'll use gdb for this example; just launch it against our executable to get an interactive debugging shell:

$ gdb overflow
(gdb)

We know from our source example that we want to try to find the location of the_shell() to fire off our attack. We can simply use gdb's disassembly command to dump out that function, and see what address the beginning of the function has been mapped to.

(gdb) disas the_shell
Dump of assembler code for function the_shell:
   0x080484a4 <+0>: push %ebp
   0x080484a5 <+1>: mov %esp,%ebp
   0x080484a7 <+3>: sub $0x18,%esp
   0x080484aa <+6>: movl $0x8048560,(%esp)
   0x080484b1 <+13>:    call 0x8048350 <puts@plt>
   0x080484b6 <+18>:    movl $0x8048567,(%esp)
   0x080484bd <+25>:    call 0x8048360 <system@plt>
   0x080484c2 <+30>:    leave  
   0x080484c3 <+31>:    ret    
End of assembler dump.
(gdb)

Here, we can see that the entry point for the_shell is 0x080484a4. We're almost there! Before we can execute our attack, we need to push the address onto the stack in the correct order. Remember, our input is being placed on the stack by strcpy() 1 byte at a time. As a result, our bytes are going to look reversed in a dump from the way we would expect to see it. Therefore, we are going to need to input the bytes (one at a time) in reverse order: 0xa4 0x84 0x04 0x08

Putting this all together

Finally, we're ready to launch the attack. We'll need to send garbage data to overflow the buffer (12 bytes for the buffer + 4 bytes for the SFP), and then our payload. We can easily just use 0's for our garbage data. Unfortunately, 0x08 and 0x04 are not printable characters, so we will need to find some way to inject these bytes into our program. A lot of hackers tend to like using a C, Perl, or Python program to do this. Personally, I just tend to use the echo command inside my Bash shell (but feel free to use whatever suits you best). Using the -e flag for echo, I can allow escaped sequences in my strings, and -n will suppress appending newline characters.

$ ./overflow 0000000000000000$(echo -ne "xa4x84x04x08")
pwned!
$

Success! We've been able to alter the flow of execution in our binary to run the hidden shell code.

Again, if if you trying this on a modern system, there are a few more safe guards that need to be taken into consideration before this will work. For example, you would probably need to disable the NX flag from your binaries (you can use execstack on Linux systems to do this.) But if you are playing The Hacker's Sandbox, you won't need to worry about any of that.

For additional information on running buffer overflows against modern systems, I would recommend reading Smashing the Stack in 2011.

Firefox 31+ and sec_error_ca_cert_invalid

Some Background

In early 2014, the Mozilla foundation released a new library for certificate verification called mozilla::pkix. This library was built to be more robust and maintainable than its predecessors, using techniques such as building multiple trust chains to be used in the verification process. With this new library came some changes to the enforcement of some requirements in Mozilla's CA Certificate Policy. Where the most of users may not necessarily notice any differences, some subset of users have definitely been affected. Most specifically, using self-signed certificates can lead to failed verification of the certificate chain. In such instances, the entire connection itself will be refused, and you will be prompted with the following error:

An error occurred during a connection to example.com.

Issuer certificate is invalid.

(Error code: sec_error_ca_cert_invalid)


It is still currently possible to get around this, however, by disabling support for the new mozilla::pkix library. Be aware, however, that doing so may leave you at slightly higher risk to malicious connections being uncaught by verification. Use this method at your own risk.

Disabling mozilla::pix

  • In your Firefox browser, type about:config into the address bar, and hit enter.
  • Search for security.use_mozillapkix_verification and set it to true (you can double click on it to do so)

Now you should be able to reload the page you were trying to connect to and receive your familiar prompt about the unsafe connection. Simply accept the exception to continue on your way.

Dynamic Linker Voodoo: aka How to LD_PRELOAD

The dynamic linker is one of the most important yet widely overlooked components of a modern Operating System. Its job is to load and link-in executable code from shared libraries into executables at run time. There are many intricate details on how the dynamic linker does what it does, but one of the more interesting is the use of the LD_LIBRARY_PATH and LD_PRELOAD environment variables. When defined, these variables will let you override the default behaviour of the dynamic linker to load specific code into your executables. For example, using LD_PRELOAD, you can override the C standard library functions in libc with your own versions (think printf, puts, getc, etc.)

Let's see this in action! We'll start by making a simple program to test the (now deprecated) gets() function. Here, we will create a file called test.c and put the follow contents inside it:

#include <stdio.h>

int main (void)
{
  char str[128];
  printf ("Testing gets()...\n");
  gets(str);
  return 0;
}

Note that this code is not safe and should not be used for production, but it makes a simple test scenario.

Next, we can compile the source with gcc. (Since gets() is deprecated, we're going to throw in the -w flag to suppress warning messages. We don't really care for this example.)

$ gcc -w -o test test.c

Finally, we can run the program and examine it's output:

$ ./test 
Testing gets()...
womp
$

Success! When the executable is run, it links the gets() code from libc into memory and executes that code when we call gets(). Now let's see how we can override libc's implementation with our own. First, we'll write a new version of gets() that we want run. Make a file called mygets.c and enter the following:

char *gets( char *str )
{
  printf("Error: Stop using deprecated functions!\n");
  return "";
}

Once finished, we can compile this into our own shared object library:

gcc -w -fPIC -shared -o mygets.so mygets.c

Finally, let's run the test executable again, but this time we will call it with LD_PRELOAD to load our custom shared library before dynamically linking libc:

$ LD_PRELOAD=./mygets.so ./test 
Testing gets()...
Error: Stop using deprecated functions!
$

As you can see, now our custom code is displaying where we were once being prompted for input. Of course, we could write any code we want to go in here. The only limit is whatever we can think up. This technique could be extremely useful when trying advanced debugging or when trying to replace specific parts of a shared library in your program. You can even take this a step further to create hooks for the original overridden functions.

To illustrate this, let's modify our shared library one more time:

#define _GNU_SOURCE

#include <stdio.h>
#include <dlfcn.h>

char *gets( char *str )
{
  printf("Error: Stop using deprecated functions!\n");
  char *(*original_gets)( char *str );
  original_gets = dlsym(RTLD_NEXT, "gets");
  return (*original_gets)(str);
}

We've done a few things here. We have now referenced the stdio header and we use dlsym to find the original gets() function. Notice that we use the RTLD_NEXT pseudo-handle with dlsym. In order to use this handle, we must include the _GNU_SOURCE test macro (otherwise RTLD_NEXT will not be found). This finds the next occurrence of gets() after the current library and allows us to map it to original_gets(). We can then use it in this function with the mapped name.

We can compile our library again to test out the new code (this time linking the dl lib):

$ gcc -w -fPIC -shared -o mygets.so mygets.c -ldl

Using this method, we can run our test executable again:

$ LD_PRELOAD=./mygets.so ./test 
Testing gets()...
Error: Stop using deprecated functions!
womp
$

At this point, you should notice the custom code we provided for gets(), followed by the prompt by the original libc function. Hopefully this dispels a little bit of the voodoo and gives you another valuable tool to stash in your belt.

Google C++ Style Guide

I am a really huge fan of nicely formatted and presented documentation. Sometimes I can get a little too OCD about this. If you haven't reviewed any of Google's coding style guides you should, they have a lot of nice information in there, and their views are worth considering. Lately I've decided to try and conform to their C++ guide a bit, but perusing through their doc was just ugly for offline viewing. It may not matter to most people, but I made a slightly cleaner version of the 3.274 revision of the guide in PDF format.

Download Google's C++ Style Guide in PDF here

Creating RSA public keys from private keys

Generating a public RSA key is a pretty simple task. More often than not, keys will be generated in pairs. But sometimes you will be given an RSA private key and you will need to create your own public key from it. (For example, when using Amazon EC2). Fortunately, you only need to remember a quick one-liner to do so:

$ ssh-keygen -y -f mykey.pem > mykey.pub

The -y flag tells ssh-keygen to create a public key based off of the private key passed to the -f flag. It's really that simple. by default, this will simple print to stdout, so redirecting output to a file (as in the example here) is your quickest way to saving your new key.

Node.js v0.10 Documentation

Anyone who has worked with Node.js and its documentation will know that the docs are pretty much only hosted on the nodejs.org website these days. Unfortunately, that leaves offline browsing of the docs in a particularly poor state. Since I have had the need to refer to these docs often enough without having an Internet connection, I imagine there are of ton of other people who are having the same issue.

In response, I have created a PDF file of the online docs that you can use for offline use. Download the PDF and happy coding!

Configuring a Cisco router as a terminal server

A Cisco terminal server is a router with multiple, low speed, asynchronous ports that are connected to other serial devices. These are often used to provide out-of-band management to those devices. Using this, you can establish a single point of access to the console ports of multiple devices in your infrastructure, and gain administrative access in scenarios of failed network connectivity.

Cisco usually provides a few types of breakout cables (also referred to as octocables) to connect to your router's async ports. The CAB-OCTAL-ASYNC is a 8-port variant popularly used on the 2500 series routers, and will be explained here. (Make sure to read the documentation for your device and find out which cables are appropriate to use for your setup.)

Let's assume you have a 2500 series router you would like to configure into a terminal server, and you have two 2600 series catalyst switches you will be managing. We should be able to quickly get this setup in a matter of minutes for remote access.

Cabling

Connect your breakout cable to the 2500 router and connect the RJ-45 connectors labeled P0 and P1 to the console ports of your 2600 switches. As long as all of your devices are powered on (and booted up; remember cisco gear can take 5 min or so to fully boot up) you should be ready to continue! (it's that easy.)

Setting your Loopback

You will need to have a loopback device defined in your router to properly administrate your devices on. In IOS, it should be as simple as the following:

Router(config)#interface loop0
Router(config-if)#ip address 172.16.1.1 255.255.255.255
Router(config-if)#^Z

Here, we are defining the device "loop0" and assigning the address of 172.16.1.1/32 to the device. This will be used to connect to your async lines when managing the switches.

Finding your management ports

Cisco gear is a little bit magical here, as they will automatically enumerate ports for your async lines on the loopback device. These ports will be assigned the values of 2000 + P, where P is the label of the breakout cable port number you have plugged in.

For example, we have plugged in P0 and P1 into our test switches, so the enumerated ports will be 2000 and 2001 respectively.

Configure transport access

Telnet is usually defined by default, but it's a good idea (and sometimes necessary) to explicitly define your settings. In order to allow your router to speak out to the switches over the async lines, you must set the transport settings to allow telnet. We can use the following command to do this:

Router(config)#line 0/0/0 0/0/7
Router(config-line)#transport output telnet
Router(config-line)#^Z

This will set the transport output settings on all 8 of your available async lines. Please not that, depending on the hardware you may be using, these lines may be enumerated differently. You can use the show lines command to see your actual available line information and configure accordingly.

Connecting

Now, you can easily connect from your router to either of the switches using telnet.

Router#telnet 172.16.1.1 2000

As usual, Ctrl+Shift+6 then x will bring you back to your terminal server.

If you want to make a quick alias to make connecting easier, you can do so using the ip host commands.

ip host switch1 2000 172.16.1.1
ip host switch2 2001 172.16.1.1

You can now just type switch1 to log into your switch.

From here, you can get fancy and setup aux connections to the router, say if you needed dial-up access to your gear in case of an ISP outage or something similar. But at the very least, this should be enough to get you up and going.