Handling Crashes in InspIRCd

From the makers of InspIRCd.
Jump to: navigation, search
Important Need Update - This page needs to be revised. Information posted here has been viewed as incorrect, incomplete, or obsolete. Anyone is welcome to correct these flaws if this page has not been "protected". Otherwise, contact a Docs Team member and let them know.

Introduction

Before I start, I apologize for some of this manual going off the right of the screen.

If InspIRCd causes a Segmentation Fault this is usually indicated by the message:

[07:25] * Error! Segmentation fault! save meeeeeeeeeeeeee *splat!*

Or maybe in the logfile:

Sat Apr 22 18:16:57 2006: *** fell down a pothole in the road to perfection ***

The best method of bug fixing is to perform certain debugging operations on it. After just coming across a segfault in one of InspIRCd's modules, I thought I would write this tutorial to describe how I tracked down the problem.

Coredumps

NOTE: InspIRCd, at least in later versions, should try to generate coredumps, however using gdb is sometimes more reliable, even if a core dump is made it is generally recommended you use the following method.

But if you're set on loading the core dump yourself, then run:

gdb bin/inspircd /path/to/inspircd.core

then skip to the part of this manual which reads:

Program received signal SIGSEGV, Segmentation fault.

NOTE: Please don't just paste core dumps to our bugtracker, it's very unlikely we'll be able to make use of them on our systems, a backtrace is much more helpful!

Using gdb

If you know what action or command caused InspIRCd to crash, continue reading.

The first thing we need to do is start a debugger. In this case, we will use gdb to assist us. From the inspircd directory, fire up gdb. All versions of InspIRCd past 1.0.3 (or older?) should be able to handle launching gdb for you, simply run:

me@mymachine:~/build/inspircd$ ./inspircd debug

And the launcher script will open gdb and launch InspIRCd for you, the launcher script automatically forces debug logging, so you'll get a great big heap of debug logging. This is good, it helps debug your crash.

DANGER WILL ROBINSON!
Warning:  Log Files
InspIRCd can generate a lot of debug logging, and by a lot, we mean a lot, you'll need to watch out for it filling up your disk.


Your IRCd is now running. You must now connect to it, and force it to crash (always a fun job). When the IRCd crashes, there will be NO OUTPUT on IRC, it will appear to 'hang', but gdb will say otherwise.

Getting Crash Infomation

Now we're in gdb with either a corefile or a just-crashed InspIRCd, we can see where the crash happened:

Program received signal SIGSEGV, Segmentation fault.
0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46
46                      Srv->SendServ(user->fd, buf);

Instantly, this tells us where the problem has occured, the file, and line in question.




IF YOU HAVE SET OPTIMISATION FLAGS ON, THIS DATA WILL NOT BE HERE! PLEASE RE-COMPILE THE IRCD WITH THEM OFF AND TRY AGAIN




If you have received this data, we can get a little more info. The first thing required would be a backtrace, to describe the events leading up to the crash.

 (gdb) bt
 #0  0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46
 #1  0x2835fcd1 in ModuleRandQuote::OnUserConnect (this=0x80f8550, user=0x810b000) at m_randquote.cpp:101
 #2  0x0807fb3a in ConnectUser (user=0x810b000) at inspircd.cpp:2429
 #3  0x0809e5ca in handle_user (parameters=0xbfbd8030, pcnt=4, user=0x810b000) at commands.cpp:1063
 #4  0x08081529 in process_command (user=0x810b000, cmd=0xbfbd8280 "USER") at inspircd.cpp:2872
 #5  0x08081e62 in process_buffer (cmdbuf=0xbfbd84b0 "USER Craig i-br0ked-it 192.168.0.1 :Craig McLure", user=0x810b000) at inspircd.cpp:3009
 #6  0x08084a9d in InspIRCd () at inspircd.cpp:3613
 #7  0x0807dbfd in main (argc=2, argv=0xbfbfeb80) at inspircd.cpp:2067
 (gdb)

Although most of this won't make any sense to you, it is important to the developers. It's also generally helpful to extract variables or data to help with diagnosing the crash, if we take another look at the original crash message:

0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46
46                      Srv->SendServ(user->fd, buf);

The second line indicates (in technical jargon) the method called with two comma separated parameters (user->fd and buf). In a less technical sense, this is the specific instruction which broke. Now, the two aforementioned parameters may be helpful in the debugging process, so it would help us greatly if they could be extracted for review. Fortunately, gdb makes this easy for us by providing the print command. Extracting the parameter contents is as simple as typing in:

  print <Param Name Here>



Please be sure to check your parameter names, as it is highly improbable that they will be identical to the ones you see in this tutorial.



Below is the output of extracting the buf parameter from our crash (truncated for readability):

  (gdb) print buf
  $1 = "NOTICE Craig :Linux is only free if your time is worthless\000ay file is missing.\000EN=307 CHANMODES=...

Try to give all parameters if possible, so in our case here, we would also provide user->fd.


Writing a Bug Report

From all this data, it is now possible to build something very helpful for the bugtracker, here is what i'm about to submit:

I've been experiencing problems with m_randquote crashing with a segfault whenever a user connects. I've used gdb to retreive
some info into the crashing, some output is below:

Program received signal SIGSEGV, Segmentation fault.
0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46
46                      Srv->SendServ(user->fd, buf);
(gdb) bt
#0  0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46
#1  0x2835fcd1 in ModuleRandQuote::OnUserConnect (this=0x80f8550, user=0x810b000) at m_randquote.cpp:101
#2  0x0807fb3a in ConnectUser (user=0x810b000) at inspircd.cpp:2429
#3  0x0809e5ca in handle_user (parameters=0xbfbd8030, pcnt=4, user=0x810b000) at commands.cpp:1063
#4  0x08081529 in process_command (user=0x810b000, cmd=0xbfbd8280 "USER") at inspircd.cpp:2872
#5  0x08081e62 in process_buffer (cmdbuf=0xbfbd84b0 "USER Craig i-br0ked-it 192.168.0.1 :Craig McLure", user=0x810b000) at inspircd.cpp:3009
#6  0x08084a9d in InspIRCd () at inspircd.cpp:3613
#7  0x0807dbfd in main (argc=2, argv=0xbfbfeb80) at inspircd.cpp:2067
(gdb) print buf
$1 = "NOTICE Craig :Linux is only free if your time is worthless\000ay file is missing.\000EN=307 CHANMODES=...
(gdb) print user->fd
$2 = 13
(gdb)

The Text when connecting:
* Your host is frostycoolslug.chatspike.net, running version InspIRCd-1.0(Beta1)
* This server was created 11:56:55 Mar 28 2005
* frostycoolslug.chatspike.net InspIRCd-1.0(Beta1) iowghraAsORVSxNCWqBzvdHtGI lvhopsmntikrRcaqOALQbSeKVfHGCuzN
* MESHED WALLCHOPS MODES=13 CHANTYPES=# PREFIX=(ohv)@%+ MAP SAFELIST MAXCHANNELS=20 MAXBANS=60 NICKLEN=32 TOPICLEN=307 KICKLEN=307 MAXTARGETS=20 :are supported by this server
* AWAYLEN=307 CHANMODES=qaohvb,...
* Message of the day file is missing.
* Craig sets modes [Craig :+x]
* Error! Segmentation fault! save meeeeeeeeeeeeee *splat!*

Including the 005 numeric (the line containing the MESHED token) is always of use, and so sometimes is /modules output as these indicate other modules which may be changing the outcome of the program.

I can almost guarantee that this bug will be corrected in under 24 hours of it being reported, as a lot of information is provided as to the cause making finding a solution easier. just saying "InspIRCd crashes on connect when m_randquote.so is loaded" would mean we would have to spend significant time tracking it down, and doing testing ourselves.


As a final note, this bug was fixed within three hours of adding it to the bugtracker, seems two declerations of Srv were at fault :)


Notes

Using screen

If you're having problems getting InspIRCd to generate corefiles, quite possible on limited shell hosts and the like, then you may want to run Insp inside 'screen', this lets you run it inside gdb without having to have a terminal permanently open, but still allowing you to get backtraces if or when it crashes.

To open a screen, simply run the following command:

screen -S inspircd ./inspircd debug

This creates a screen, and names it (using the -S parameter) 'inspircd' so it can be easily found in the event you choose to use multiple screens. Once it's started, if you wish to drop back to your original terminal whilst it runs, hold Ctrl then sequentially press A followed by D.

Once the IRCd has crashed, you can attach yourself back to your screen by typing:

screen -r inspircd

And you should be able to continue the debugging process as if you'd run ./inspircd debug normally.

PLEASE: Do not leave a screen started, and not return to it. It can be very easy to forget that you've left a screen running, especially once it's been detached, always remember it's there, and check on it frequently!

Logfiles

Just to reiterate here, Insp's debug logfiles can get very, very large, watch out!

Obtaining backtraces using other debuggers

DANGER WILL ROBINSON!
Warning:  Other Debuggers
Please note that the behavior and commands of other debuggers may differ from that of gdb, please read the documentation which comes with your debugger first.


XDB

$ xdb bin/inspircd core
(xdb) t    

DBX

$ dbx bin/inspircd core
(dbx) where

SDB

$ sdb bin/inspircd core
(sdb) t      

DEBUG

$ debug -c core bin/inspircd
debug> stack

DDE

$ dde -ui line core bin/inspircd
dde> bt

ADB

$ adb bin/inspircd core
$c

Note: In this example the literal $c must be typed.