asadbg is a framework of tools to aid in automating live debugging of Cisco ASA devices
asadbg
Preliminary note: we recommend you to use this as part of
asatools but it can also be used
standalone.
asadbg is a framework of tools to aid in automating live debugging of Cisco
ASA devices, as well as automating interaction with the Cisco CLI over
serial/ssh to quickly perform repetitive tasks.
- It wraps gdb
- It supports an
asadbg.cfgconfiguration file to enable debugging different
versions easily - It supports both real hardware and GNS3
- Optionally it uses ret-sync to allow a better debugging experience from IDA Pro
- It supports executing additional gdb scripts at boot
- It provides libraries that can be useful for automating tests across multiple firmware
versions on real hardware
The main tool is asadbg.py and it will execute most of the other helper
scripts. Note that you may need to initially use
asafw to unpack firmware to get the best
flavour of asadbg.
asadbg.py: main tool used to debug ASAsasdbg_hunt.py/asadbg_rename.py: IDA Python script to use with idahunt to import symbols for new targets
The following are automatically imported by asadbg.py but we give you some
information on what they are for:
asa_lib*.py: gdb Python scripts to use
libdlmalloc,
libptmalloc and
libmempool.asa_sync.py: gdb Python script to use
ret-synccomm.py: set of functions to communicate over serial, SSH, telnet.find_lina_pid.py: find the lina PID using SSH commandstemplate_gdbinit: gdbinit template patched byasadbg.pyfor every targets
we debug
Requirements
- Python3 only
- pip3 install pyserial paramiko
- Heavily tested on Linux (but could work on OS X to)
- Preloaded firmware on the flash of real ASA device or ASA emulator configured
in GNS3
You initially need to modify asadbg/env.sh to match your environment. It will
allow you to define paths to the tools used by all the scripts as well as some
variables matching your ASA environment. Note there is a simmilar
asafw/env.sh but only one is required to be used for both projects. We
recommend that you add it to your ~/.bashrc:
source /path/to/asadbg/env.sh
Automated debugging
One of the main benefits of using asadbg is that you can automate your use of
both emulated (GNS3) and real Cisco ASA devices. This can include just
generally automating booting and running various firmwares and configs, or
specifically automating the debugging process of lina using gdb. This is mostly
done through the asadbg.py script. The main idea is to preload some flash on
the ASA device with firmware and configuration files that you want to use on
the device. For each firmware that you want to debug, you can use something
like the asafw tool to mine different symbols and then you can automatically
generate gdbinit files using those symbols. By default asadbg.py will use a
file template_gdbinit to automate building a version-specific gdbinit file at
runtime using whatever symbols are present in the database.
Quickly boot a real device (no debugging)
You can quickly boot a given version by specifying the firmware and configuration
files which must already be on the CF card.
asadbg$ ./asadbg.py --firmware-type normal --serial-port /dev/ttyUSB0 --firmware asa802-k8.bin --config config-802
Debugging a GNS3 device with config file
The following example shows how you can setup the configuration files to debug
a GNS3 device. First we define an asadbg.cfg configuration file similar to
below. It contains the GNS3 IP/port for the configured firewall that we obtain
from GSN3 itself.
[GLOBAL]
gns3_host=192.168.5.1
asadb_file=asadb.json
[asav941200]
version=941200
arch=gns3
rootfs_path= /home/user/_asav941-200.qcow2.extracted/rootfs
gns3_port=12005
attach_gdb=yes
It also points to the ASA database asadb.json file containing the addresses required. What
is important here is that the version=941200 and arch=gns3 specified in
asadbg.cfg allow to match the firmware in asadb.json below. Also we indicate
we want to attach gdb.
{
"ASLR": false,
"addresses": {
"clock_interval": 59514112,
"socks_proxy_server_start": 22139744,
"aaa_admin_authenticate": 418288
},
"fw": "asav941-200.qcow2",
"lina_imagebase": 4194304,
"version": "9.4.1.200",
"arch": 64
}
Now we start debugging and clock_interval will be automatically patched:
asadbg$ ./asadbg.py --name asav941200 --asadbg-config asadbg.cfg
[asadbg] Using config file: asadbg.cfg
[asadbg] Found section: 'asav941200' in config
[asadbg] Using gdb: '/usr/bin/gdb'
[asadbg] Using architecture: gns3
[asadbg] Trying lina: /home/user/_asav941-200.qcow2.extracted/rootfs/asa/bin/lina
[asadbg] Going to debug...
[asadbg] Using GNS3 emulator 192.168.5.1:12005
[asadbg] Starting gdb now...
[gdbinit_941200] Configuring paths...
[gdbinit_941200] Disabling pagination...
[gdbinit_941200] Connecting over TCP/IP...
0x0000003655201190 in ?? () from /home/user/_asav941-200.qcow2.extracted/rootfs/lib64/ld-linux-x86-64.so.2
[gdbinit_941200] Connected.
[gdbinit_941200] Watchdog disabled
[gdbinit_941200] heap debugging plugins loaded
[gdbinit_941200] Additional gdb scripts loaded
[gdbinit_941200] Done.
(gdb) c
Continuing.
Debugging a real device with config file
The following example shows how you can setup the configuration files to debug
a real Cisco ASA device over a serial console.
[GLOBAL]
serial_port=/dev/ttyUSB0
asadb_file=asadb.json
[asa924-gdb]
version=924
arch=32
rootfs_path=/home/user/_asa924-k8.bin.extracted/rootfs
firmware=asa924-k8-debugshell-gdbserver.bin
config=config-924
firmware_type=gdb
attach_gdb=yes
Note also that we need to specify a firmware_type which indicates if the
firmware is unmodified, rooted, has a serial shell enabled or if gdb has
been enabled at boot. This is
because we will do different things at boot depending on the format. Here we
can see that we use a modified firmware asa924-k8-debugshell-gdbserver.bin
which has gdbserver enabled at boot and contains a debug shell.
We see below that when the bootrom starts, asadbg automatically
interrupts the sequence in order to load the firmware and config files already
on the CF card.
asadbg$ ./asadbg.py --name asa924-gdb --asadbg-config asadbg.cfg
[asadbg] Using config file: asadbg.cfg
[asadbg] Found section: 'asa924-gdb' in config
[asadbg] Using gdb: '/usr/bin/gdb'
[asadbg] Using architecture: 32
[asadbg] Trying lina: /home/user/_asa924-k8.bin.extracted/rootfs/asa/bin/lina
[asadbg] Going to debug...
[asadbg] Using serial port: /dev/ttyUSB0
[asadbg] Loading 'asa924-k8-debugshell-gdbserver.bin' with 'config-924'...
[comm] Waiting boot...
[SNIP]
Platform ASA5505
Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.
[SNIP]
rommon #0> boot asa924-k8-debugshell-gdbserver.bin cfg=config-924
Launching BootLoader...
Boot configuration file contains 1 entry.
Loading asa924-k8-debugshell-gdbserver.bin........ Booting...
Platform ASA5505
[SNIP]
SMFW PID: 514, Starting /asa/bin/lina under gdbserver /dev/ttyS0
SMFW PID: 512, started gdbserver on member: 514//asa/bin/lina
SMFW PID: 512, created member ASA BLOB, PID=514
Process /asa/bin/lina created; pid = 517
Remote debugging using /dev/ttyS0
[comm] gdb detected - boot finished.
[comm] Boot should be finished now?
[asadbg] Starting gdb now...
[gdbinit_924] Configuring paths...
[gdbinit_924] Disabling pagination...
[gdbinit_924] Connecting over USB...
0xdc7e2820 in ?? () from /home/user/_asa924-k8.bin.extracted/rootfs/lib/ld-linux.so.2
[gdbinit_924] Connected.
[gdbinit_924] Watchdog disabled
[gdbinit_924] heap debugging plugins loaded
[gdbinit_924] Additional gdb scripts loaded
[gdbinit_924] Done.
(gdb) c
Continuing.
Importing additional symbols
Numerous parts of asadbg rely on a database asadbg/asadb.json that
contains symbols for various firmware versions. We have supplied a small
number of existing symbols, however you can add others manually or use
idahunt
to automatically add new targets. This section details how to do it.
Requirements
The IDA Python scripts in asadbg assume that filelock.py/ida_helper.py are
available. Since they are executed directly from IDA Python, the way we have
done it so far is by creating symlinks in asadbg/. We can’t really create them
on the repo as the symlinks on Windows are different than on Linux. Consequently
it is required to do manually.
On Linux:
asadbg$ ln -s ../idahunt/ida_helper.py ida_helper.py
asadbg$ ln -s ../idahunt/filelock.py filelock.py
On Windows, with Administrator permissions:
C:\asadbg> mklink filelock.py ..\idahunt\filelock.py
C:\asadbg> mklink ida_helper.py ..\idahunt\ida_helper.py
We also need to set the path to the database (automatically done by env.sh):
C:\idahunt>set ASADBG_DB=C:\asadbg\asadb.json
Initial analysis and renaming
We assume you have extracted all the firmware into a directory using e.g.
asafw or that you
extracted only the lina files and dropped them into a hierarchy that indicates
what firmware they come from:
C:\temp>tree /F
Folder PATH listing
Volume serial number is XXXX-XXX
C:.
├───asa924-k8.bin
│ lina
│
├───asa924-smp-k8.bin
│ lina
│
└───asav941-200.qcow2
lina
First you need to do the initial analysis in IDA. We can use --list-only with
any command to check what would be the result of the command.
Note we filter to specific ASA versions (9.2.4) and architecture (32-bit):
C:\idahunt>idahunt.py --verbose --inputdir "C:\temp" --analyse --scripts C:\asadbg\asadbg_rename.py --filter "filters\ciscoasa.py -m 924 -M 924 -v -a 32"
[idahunt] IDA32 = C:\Program Files (x86)\IDA 6.95\idaq.exe
[idahunt] IDA64 = C:\Program Files (x86)\IDA 6.95\idaq64.exe
[idahunt] ANALYSING FILES
[idahunt] Analysing C:\temp\asa924-k8.bin\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq.exe -B -LC:\temp\asa924-k8.bin\lina.log C:\temp\asa924-k8.bin\lina
[ciscoasa] Skipping non 32-bit: C:\temp\asa924-smp-k8.bin\lina
[ciscoasa] Skipping version too high: C:\temp\asav941-200.qcow2\lina
[idahunt] Waiting on remaining 1 IDA instances
[idahunt] EXECUTE SCRIPTS
[idahunt] Executing script C:\asadbg\asadbg_rename.py for C:\temp\asa924-k8.bin\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq.exe -A -SC:\asadbg\asadbg_rename.py -LC:\temp\asa924-k8.bin\lina.log C:\temp\asa924-k8.bin\lina.idb
[ciscoasa] Skipping non 32-bit: C:\temp\asa924-smp-k8.bin\lina
[ciscoasa] Skipping version too high: C:\temp\asav941-200.qcow2\lina
[idahunt] Waiting on remaining 0 IDA instances
Once the analysis is done, you should have a lina.i64 or lina.idb near lina
in the respective folders.
Adding symbols to the database
We use idahunt to add symbols to the database:
C:\idahunt>idahunt.py --verbose --inputdir "C:\temp" --scripts C:\asadbg\asadbg_hunt.py --filter "filters\ciscoasa.py -v"
[idahunt] IDA32 = C:\Program Files (x86)\IDA 6.95\idaq.exe
[idahunt] IDA64 = C:\Program Files (x86)\IDA 6.95\idaq64.exe
[idahunt] EXECUTE SCRIPTS
[idahunt] Executing script C:\asadbg\asadbg_hunt.py for C:\temp\asa924-k8.bin\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq.exe -A -SC:\asadbg\asadbg_hunt.py -LC:\temp\asa924-k8.bin\lina.log C:\temp\asa924-k8.bin\lina.idb
[idahunt] Executing script C:\asadbg\asadbg_hunt.py for C:\temp\asa924-smp-k8.bin\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq64.exe -A -SC:\asadbg\asadbg_hunt.py -LC:\temp\asa924-smp-k8.bin\lina.log C:\temp\asa924-smp-k8.bin\lina.i64
[idahunt] Executing script C:\asadbg\asadbg_hunt.py for C:\temp\asav941-200.qcow2\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq64.exe -A -SC:\asadbg\asadbg_hunt.py -LC:\temp\asav941-200.qcow2\lina.log C:\temp\asav941-200.qcow2\lina.i64
[idahunt] Executing script C:\asadbg\asadbg_hunt.py for C:\temp\asav962-7.qcow2\lina
[idahunt] C:\Program Files (x86)\IDA 6.95\idaq64.exe -A -SC:\asadbg\asadbg_hunt.py -LC:\temp\asav962-7.qcow2\lina.log C:\temp\asav962-7.qcow2\lina.i64
[idahunt] Waiting on remaining 0 IDA instances
Automating executing commands on the CLI
We support executing some useful commands automatically using either SSH, the serial
and theoretically support telnet as well though it has not been heavily tested.
$ comm.py -h
usage: comm.py [-h] [--comm COMM_TYPE] [--port TARGET_PORT] [--ip TARGET_IP]
[--user USER] [--pass PASSWORD] [--reboot] [--version]
[--disable-checkheaps] [--upload] [--force] [--download]
[--delete-ike-sa] [--delete-webvpn-sessions] [--md5]
[--input [INPUT [INPUT ...]]] [--oldssh] [--cmd CMD]
[-C CFG_FILE]
optional arguments:
-h, --help show this help message and exit
--comm COMM_TYPE Communication type {serial(default), telnet, ssh}
--port TARGET_PORT Specify a custom serial (e.g. "/dev/ttyUSB0")/telnet
(e.g. 5000) port)
--ip TARGET_IP Cisco ASA/GNS3 IP address
--user USER User for SSH
--pass PASSWORD Password for SSH
--reboot Reboot router (serial, SSH)
--version Get the version (serial, SSH)
--disable-checkheaps Disable checkheaps default timeout (60 sec)
--upload Upload over SSH
--force Overwrite existing files
--download Download over SSH
--delete-ike-sa Delete IKE SAs (serial, SSH)
--delete-webvpn-sessions
Delete WebVPN sessions (serial, SSH)
--md5 Compute MD5 for files (serial, SSH)
--input [INPUT [INPUT ...]]
List of input files for other commands (eg: file1
file2 file3) (e.g.: --upload, --download, --md5)
--oldssh Specify an old version of SSH (do not know specific
command lines options and do not need them because
unsecure :))
--cmd CMD Command to run on the CLI (debugging)
-C CFG_FILE File containing commands to use (e.g.
config/setup_ssh.cfg)
Getting the ASA version
We use --version and can execute it over serial or SSH:
$ comm.py --comm serial --port /dev/ttyUSB0 --version
[comm] Retrieving version now...
[comm] Serial: /dev/ttyUSB0
[comm] Detected version: 9.2(4)
$ comm.py --comm ssh --ip 192.168.210.77 --port 22 --user user --pass user --version
[comm] Retrieving version now...
[comm] SSH: 192.168.210.77:22
[comm] Detected version: 9.2(4)
Rebooting the ASA
We use --reboot:
$ comm.py --comm ssh --reboot
[comm] Rebooting router now...
[comm] SSH: 192.168.210.77:22
if you have a running CLI, you should see it automatically reboots:
ciscoasa#
***
*** --- START GRACEFUL SHUTDOWN ---
Shutting down isakmp
Shutting down License Controller
Shutting down File system
***
*** --- SHUTDOWN NOW ---
Process shutdown finished
Uploading files to the flash
We can upload several files at a time over SSH:
$ comm.py --upload --comm ssh --oldssh --pass user --input fw/*
[comm] Warning: using forced SSH
[comm] Uploading file...
[comm] SSH: 192.168.210.77:22
[comm] Uploading: asa924-5-k8.bin...
[comm] Executing 'sshpass -p user scp fw/asa924-5-k8.bin [email protected]:asa924-5-k8.bin'...
Connection to 192.168.210.77 closed by remote host.
[comm] Uploading file...
[comm] SSH: 192.168.210.77:22
[comm] Uploading: asa924-k8.bin...
[comm] Executing 'sshpass -p user scp fw/asa924-k8.bin [email protected]:asa924-k8.bin'...
Connection to 192.168.210.77 closed by remote host.
[comm] Finished all tasks.
Note that --oldssh is useful if you are using an old ssh client.
Computing MD5 of files on the flash
It is sometimes useful to check the MD5 of files on the flash to make sure
the files have been correctly uploaded.
$ comm.py --md5 --input asa924-k8.bin asa924-5-k8.bin --comm ssh --oldssh --ip 192.168.210.77
Calculating MD5. Will take 20 seconds... asa924-k8.bin = 4558b27d0dd7ba1751e43b0ba33593a3
Calculating MD5. Will take 20 seconds... asa924-5-k8.bin = 74765d62abb2c4a5e677ed9ca49ebf87
Determining the lina PID/memory mapping in gdb
By default, gdb does not know the lina PID.
(gdb) info proc mappings
Can't determine the current process's PID: you must name one.
We use find_lina_pid.py to find the lina PID.
Then we use info proc mappings <pid> to get the mapping.
(gdb) source find_lina_pid.py
Finding lina PID:
516
You can use the following to see the lina mapping: info proc mappings <pid>
(gdb) info proc mappings 516
process 516
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x8048000 0xa388000 0x2340000 0x0 /asa/bin/lina
0xa388000 0xa38a000 0x2000 0x233f000 /asa/bin/lina
0xa38a000 0xa9aa000 0x620000 0x2341000 /asa/bin/lina
0xa9aa000 0xb74b000 0xda1000 0xa9aa000 [heap]
0xa5ad9000 0xa5bda000 0x101000 0xa5ad9000
0xa5bda000 0xa5bde000 0x4000 0x0 /dev/zero (deleted)
0xa5bde000 0xa5de1000 0x203000 0xa5bde000
0xa5de1000 0xa5de5000 0x4000 0x50004000 /dev/mem
0xa5de5000 0xa5fe7000 0x202000 0xa5de5000
0xa5fe7000 0xa5fe8000 0x1000 0xffb00000 /dev/mem
0xa5fe8000 0xa5ff8000 0x10000 0xfff00000 /dev/mem
0xa5ff8000 0xa6000000 0x8000 0xd8000 /dev/mem
0xa6000000 0xa8724000 0x2724000 0x0 /dev/udma0
0xa8800000 0xab400000 0x2c00000 0x0 /SYSV00000002 (deleted)
0xab400000 0xab800000 0x400000 0x2c00000 /SYSV00000002 (deleted)
0xab800000 0xabc00000 0x400000 0x3000000 /SYSV00000002 (deleted)
0xabc00000 0xac000000 0x400000 0x3400000 /SYSV00000002 (deleted)
0xac000000 0xac400000 0x400000 0x3800000 /SYSV00000002 (deleted)
0xac400000 0xdbc00000 0x2f800000 0x3c00000 /SYSV00000002 (deleted)
0xdbf55000 0xdbf58000 0x3000 0xdbf55000
0xdbf58000 0xdc091000 0x139000 0x0 /lib/libc-2.9.so
0xdc091000 0xdc092000 0x1000 0x139000 /lib/libc-2.9.so
0xdc092000 0xdc094000 0x2000 0x139000 /lib/libc-2.9.so
0xdc094000 0xdc095000 0x1000 0x13b000 /lib/libc-2.9.so
0xdc095000 0xdc098000 0x3000 0xdc095000
0xdc098000 0xdc0a2000 0xa000 0x0 /lib/libgcc_s.so.1
0xdc0a2000 0xdc0a3000 0x1000 0x9000 /lib/libgcc_s.so.1
0xdc0a3000 0xdc0a4000 0x1000 0xa000 /lib/libgcc_s.so.1
0xdc0a4000 0xdc0c8000 0x24000 0x0 /lib/libm-2.9.so
0xdc0c8000 0xdc0c9000 0x1000 0x23000 /lib/libm-2.9.so
0xdc0c9000 0xdc0ca000 0x1000 0x24000 /lib/libm-2.9.so
0xdc0ca000 0xdc0d3000 0x9000 0x0 /lib/libudev.so.0.5.0
0xdc0d3000 0xdc0d4000 0x1000 0x8000 /lib/libudev.so.0.5.0
0xdcfeb000 0xdd000000 0x15000 0xdcfeb000 [stack]
0xffffe000 0xfffff000 0x1000 0x0 [vdso]
0xdc0e0000 0xdc0e1000 0x1000 0xa000 /usr/lib/libcgroup.so.1.0.34
0xdc0e1000 0xdc0e2000 0x1000 0xb000 /usr/lib/libcgroup.so.1.0.34
0xdc0e2000 0xdc6dc000 0x5fa000 0xdc0e2000
0xdc6dc000 0xdc6e3000 0x7000 0x0 /lib/librt-2.9.so
0xdc6e3000 0xdc6e4000 0x1000 0x6000 /lib/librt-2.9.so
0xdc6e4000 0xdc6e5000 0x1000 0x7000 /lib/librt-2.9.so
0xdc6e5000 0xdc6e7000 0x2000 0x0 /lib/libdl-2.9.so
0xdc6e7000 0xdc6e8000 0x1000 0x1000 /lib/libdl-2.9.so
0xdc6e8000 0xdc6e9000 0x1000 0x2000 /lib/libdl-2.9.so
0xdc6e9000 0xdc6fd000 0x14000 0x0 /lib/libpthread-2.9.so
0xdc6fd000 0xdc6fe000 0x1000 0x13000 /lib/libpthread-2.9.so
0xdc6fe000 0xdc6ff000 0x1000 0x14000 /lib/libpthread-2.9.so
0xdc6ff000 0xdc701000 0x2000 0xdc6ff000
0xdc701000 0xdc71c000 0x1b000 0x0 /usr/lib/libexpat.so.1.6.0
0xdc71c000 0xdc71e000 0x2000 0x1a000 /usr/lib/libexpat.so.1.6.0
0xdc71e000 0xdc71f000 0x1000 0x1c000 /usr/lib/libexpat.so.1.6.0
0xdc71f000 0xdc7cd000 0xae000 0x0 /lib/libstdc++.so.6.0.10
0xdc7cd000 0xdc7d1000 0x4000 0xad000 /lib/libstdc++.so.6.0.10
0xdc7d1000 0xdc7d2000 0x1000 0xb1000 /lib/libstdc++.so.6.0.10
0xdc7d2000 0xdc7d8000 0x6000 0xdc7d2000
0xdc7d8000 0xdc7dd000 0x5000 0x0 /usr/lib/libnuma.so.1
0xdc7dd000 0xdc7de000 0x1000 0x4000 /usr/lib/libnuma.so.1
0xdc7de000 0xdc7df000 0x1000 0x5000 /usr/lib/libnuma.so.1
0xdc7df000 0xdc7e0000 0x1000 0xdc7df000
0xdc7e1000 0xdc7e2000 0x1000 0xdc7e1000
0xdc7e2000 0xdc7fe000 0x1c000 0x0 /lib/ld-2.9.so
0xdc7fe000 0xdc7ff000 0x1000 0x1b000 /lib/ld-2.9.so
0xdc7ff000 0xdc800000 0x1000 0x1c000 /lib/ld-2.9.so
0xdcfeb000 0xdd000000 0x15000 0xdcfeb000 [stack]
0xffffe000 0xfffff000 0x1000 0x0 [vdso]