[TriLUG] Apache: Segmentation fault errors

[Robert Dale]

"Before running gdb, be sure that the server is compiled with the -g
option in EXTRA_CFLAGS to include the symbol information in the object
files."

You'll have to recompile and don't let it get 'strip'ped.

There is also the off chance that you have bad ram.

Do you have another machine to test the additional VHs?

-- 
Robert Dale


On 3/8/07, Paul G. Szabady <Paul at thyservice.com> wrote:
> Thanks!
>
> So, here's my core dump trace.  Is the "No symbol table info available."
> output normal or am I missing one or more parameters?
>
>
> # gdb /usr/sbin/httpd /var/tmp/core.59795
> GNU gdb Red Hat Linux (6.3.0.0-1.62rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols
> found)
> Using host libthread_db library "/lib/tls/libthread_db.so.1".
>
> Core was generated by `/usr/sbin/httpd'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /lib/libssl.so.4...(no debugging symbols found)...done.
> Loaded symbols for /lib/libssl.so.4
> Reading symbols from /lib/libcrypto.so.4...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libcrypto.so.4
> Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
> Reading symbols from /usr/kerberos/lib/libkrb5.so.3...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
> Reading symbols from /usr/kerberos/lib/libcom_err.so.3...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
> Reading symbols from /usr/kerberos/lib/libk5crypto.so.3...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
> Reading symbols from /lib/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /lib/libpcre.so.0...
> (no debugging symbols found)...done.
> Loaded symbols for /lib/libpcre.so.0
> Reading symbols from /usr/lib/libpcreposix.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libpcreposix.so.0
> Reading symbols from /usr/lib/libaprutil-0.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libaprutil-0.so.0
> Reading symbols from /lib/libdb-4.1.so...(no debugging symbols found)...done.
> Loaded symbols for /lib/libdb-4.1.so
> Reading symbols from /usr/lib/libexpat.so.0...Reading symbols from
> /usr/lib/debug/usr/lib/libexpat.so.0.4.0.debug...done.
> done.
> Loaded symbols for /usr/lib/libexpat.so.0
> Reading symbols from /usr/lib/libapr-0.so.0...done.
> Loaded symbols for /usr/lib/libapr-0.so.0
> Reading symbols from /lib/tls/librt.so.1...done.
> Loaded symbols for /lib/tls/librt.so.1
> Reading symbols from /lib/tls/libm.so.6...done.
> Loaded symbols for /lib/tls/libm.so.6
> Reading symbols from /lib/libcrypt.so.1...done.
> Loaded symbols for /lib/libcrypt.so.1
> Reading symbols from /lib/libnsl.so.1...done.
> Loaded symbols for /lib/libnsl.so.1
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /lib/tls/libpthread.so.0...done.
> Loaded symbols for /lib/tls/libpthread.so.0
> Reading symbols from /lib/tls/libc.so.6...done.
> Loaded symbols for /lib/tls/libc.so.6
> Reading symbols from /lib/ld-linux.so.2...done.
> Loaded symbols for /lib/ld-linux.so.2
> Reading symbols from /lib/libnss_files.so.2...done.
> Loaded symbols for /lib/libnss_files.so.2
> Reading symbols from /lib/libnss_dns.so.2...done.
> Loaded symbols for /lib/libnss_dns.so.2
> Reading symbols from /usr/lib/httpd/modules/mod_access.so...done.
> Loaded symbols for /etc/httpd/modules/mod_access.so
> Reading symbols from /usr/lib/httpd/modules/mod_auth.so...done.
> Loaded symbols for /etc/httpd/modules/mod_auth.so
> Reading symbols from /usr/lib/httpd/modules/mod_auth_anon.so...done.
> Loaded symbols for /etc/httpd/modules/mod_auth_anon.so
> Reading symbols from /usr/lib/httpd/modules/mod_auth_dbm.so...done.
> Loaded symbols for /etc/httpd/modules/mod_auth_dbm.so
> Reading symbols from /usr/lib/httpd/modules/mod_auth_digest.so...done.
> Loaded symbols for /etc/httpd/modules/mod_auth_digest.so
> Reading symbols from /usr/lib/httpd/modules/mod_include.so...done.
> Loaded symbols for /etc/httpd/modules/mod_include.so
> Reading symbols from /usr/lib/httpd/modules/mod_log_config.so...done.
> Loaded symbols for /etc/httpd/modules/mod_log_config.so
> Reading symbols from /usr/lib/httpd/modules/mod_env.so...done.
> Loaded symbols for /etc/httpd/modules/mod_env.so
> Reading symbols from /usr/lib/httpd/modules/mod_mime_magic.so...done.
> Loaded symbols for /etc/httpd/modules/mod_mime_magic.so
> Reading symbols from /usr/lib/httpd/modules/mod_cern_meta.so...done.
> Loaded symbols for /etc/httpd/modules/mod_cern_meta.so
> Reading symbols from /usr/lib/httpd/modules/mod_expires.so...done.
> Loaded symbols for /etc/httpd/modules/mod_expires.so
> Reading symbols from /usr/lib/httpd/modules/mod_deflate.so...done.
> Loaded symbols for /etc/httpd/modules/mod_deflate.so
> Reading symbols from /usr/lib/httpd/modules/mod_headers.so...done.
> Loaded symbols for /etc/httpd/modules/mod_headers.so
> Reading symbols from /usr/lib/httpd/modules/mod_usertrack.so...done.
> Loaded symbols for /etc/httpd/modules/mod_usertrack.so
> Reading symbols from /usr/lib/httpd/modules/mod_unique_id.so...done.
> Loaded symbols for /etc/httpd/modules/mod_unique_id.so
> Reading symbols from /usr/lib/httpd/modules/mod_setenvif.so...done.
> Loaded symbols for /etc/httpd/modules/mod_setenvif.so
> Reading symbols from /usr/lib/httpd/modules/mod_mime.so...done.
> Loaded symbols for /etc/httpd/modules/mod_mime.so
> Reading symbols from /usr/lib/httpd/modules/mod_dav.so...done.
> Loaded symbols for /etc/httpd/modules/mod_dav.so
> Reading symbols from /usr/lib/httpd/modules/mod_status.so...done.
> Loaded symbols for /etc/httpd/modules/mod_status.so
> Reading symbols from /usr/lib/httpd/modules/mod_autoindex.so...done.
> Loaded symbols for /etc/httpd/modules/mod_autoindex.so
> Reading symbols from /usr/lib/httpd/modules/mod_asis.so...done.
> Loaded symbols for /etc/httpd/modules/mod_asis.so
> Reading symbols from /usr/lib/httpd/modules/mod_info.so...done.
> Loaded symbols for /etc/httpd/modules/mod_info.so
> Reading symbols from /usr/lib/httpd/modules/mod_dav_fs.so...done.
> Loaded symbols for /etc/httpd/modules/mod_dav_fs.so
> Reading symbols from /usr/lib/httpd/modules/mod_vhost_alias.so...done.
> Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so
> Reading symbols from /usr/lib/httpd/modules/mod_negotiation.so...done.
> Loaded symbols for /etc/httpd/modules/mod_negotiation.so
> Reading symbols from /usr/lib/httpd/modules/mod_dir.so...done.
> Loaded symbols for /etc/httpd/modules/mod_dir.so
> Reading symbols from /usr/lib/httpd/modules/mod_imap.so...done.
> Loaded symbols for /etc/httpd/modules/mod_imap.so
> Reading symbols from /usr/lib/httpd/modules/mod_actions.so...done.
> Loaded symbols for /etc/httpd/modules/mod_actions.so
> Reading symbols from /usr/lib/httpd/modules/mod_speling.so...done.
> Loaded symbols for /etc/httpd/modules/mod_speling.so
> Reading symbols from /usr/lib/httpd/modules/mod_alias.so...done.
> Loaded symbols for /etc/httpd/modules/mod_alias.so
> Reading symbols from /usr/lib/httpd/modules/mod_rewrite.so...done.
> Loaded symbols for /etc/httpd/modules/mod_rewrite.so
> Reading symbols from /usr/lib/httpd/modules/mod_proxy.so...done.
> Loaded symbols for /etc/httpd/modules/mod_proxy.so
> Reading symbols from /usr/lib/httpd/modules/mod_proxy_ftp.so...done.
> Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
> Reading symbols from /usr/lib/httpd/modules/mod_proxy_http.so...done.
> Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
> Reading symbols from /usr/lib/httpd/modules/mod_proxy_connect.so...done.
> Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
> Reading symbols from /usr/lib/httpd/modules/mod_cache.so...done.
> Loaded symbols for /etc/httpd/modules/mod_cache.so
> Reading symbols from /usr/lib/httpd/modules/mod_suexec.so...done.
> Loaded symbols for /etc/httpd/modules/mod_suexec.so
> Reading symbols from /usr/lib/httpd/modules/mod_disk_cache.so...done.
> Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
> Reading symbols from /usr/lib/httpd/modules/mod_file_cache.so...done.
> Loaded symbols for /etc/httpd/modules/mod_file_cache.so
> Reading symbols from /usr/lib/httpd/modules/mod_mem_cache.so...done.
> Loaded symbols for /etc/httpd/modules/mod_mem_cache.so
> Reading symbols from /usr/lib/httpd/modules/mod_cgi.so...done.
> Loaded symbols for /etc/httpd/modules/mod_cgi.so
> Reading symbols from /usr/lib/httpd/modules/mod_auth_mysql.so...done.
> Loaded symbols for /etc/httpd/modules/mod_auth_mysql.so
> Reading symbols from /usr/lib/mysql/libmysqlclient.so.10...done.
> Loaded symbols for /usr/lib/mysql/libmysqlclient.so.10
> Reading symbols from /usr/lib/httpd/modules/mod_authz_ldap.so...done.
> Loaded symbols for /etc/httpd/modules/mod_authz_ldap.so
> Reading symbols from /usr/lib/libldap.so.2...done.
> Loaded symbols for /usr/lib/libldap.so.2
> Reading symbols from /usr/lib/liblber.so.2...done.
> Loaded symbols for /usr/lib/liblber.so.2
> Reading symbols from /usr/lib/libsasl.so.7...done.
> Loaded symbols for /usr/lib/libsasl.so.7
> Reading symbols from /usr/lib/libgdbm.so.2...done.
> Loaded symbols for /usr/lib/libgdbm.so.2
> Reading symbols from /lib/libpam.so.0...done.
> Loaded symbols for /lib/libpam.so.0
> Reading symbols from /lib/liblaus.so.1...done.
> Loaded symbols for /lib/liblaus.so.1
> Reading symbols from /usr/lib/httpd/modules/mod_perl.so...done.
> Loaded symbols for /etc/httpd/modules/mod_perl.so
> Reading symbols from
> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE/libperl.so...done.
> Loaded symbols for
> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE/libperl.so
> Reading symbols from /lib/libutil.so.1...done.
> Loaded symbols for /lib/libutil.so.1
> Reading symbols from /usr/lib/httpd/modules/libphp4.so...Reading symbols
> from /usr/lib/debug/usr/lib/httpd/modules/libphp4.so.debug...done.
> done.
> Loaded symbols for /etc/httpd/modules/libphp4.so
> Reading symbols from /usr/local/lib/libsablot.so.0...done.
> Loaded symbols for /usr/local/lib/libsablot.so.0
> Reading symbols from /usr/lib/libjs.so...done.
> Loaded symbols for /usr/lib/libjs.so
> Reading symbols from /usr/local/lib/libiconv.so.2...done.
> Loaded symbols for /usr/local/lib/libiconv.so.2
> Reading symbols from /usr/lib/libpspell.so.4...done.
> Loaded symbols for /usr/lib/libpspell.so.4
> Reading symbols from /usr/lib/libpanel.so.5...done.
> Loaded symbols for /usr/lib/libpanel.so.5
> Reading symbols from /usr/lib/libncurses.so.5...done.
> Loaded symbols for /usr/lib/libncurses.so.5
> Reading symbols from /usr/lib/libgmp.so.3...done.
> Loaded symbols for /usr/lib/libgmp.so.3
> Reading symbols from /usr/lib/libpng12.so.0...done.
> Loaded symbols for /usr/lib/libpng12.so.0
> Reading symbols from /usr/lib/libcurl.so.2...done.
> Loaded symbols for /usr/lib/libcurl.so.2
> Reading symbols from /usr/lib/libbz2.so.1...done.
> Loaded symbols for /usr/lib/libbz2.so.1
> Reading symbols from /usr/lib/libttf.so.2...done.
> Loaded symbols for /usr/lib/libttf.so.2
> Reading symbols from /usr/lib/libfreetype.so.6...done.
> Loaded symbols for /usr/lib/libfreetype.so.6
> Reading symbols from /usr/lib/libjpeg.so.62...done.
> Loaded symbols for /usr/lib/libjpeg.so.62
> Reading symbols from /lib/libcom_err.so.2...done.
> Loaded symbols for /lib/libcom_err.so.2
> Reading symbols from /usr/lib/libxml2.so.2...done.
> Loaded symbols for /usr/lib/libxml2.so.2
> Reading symbols from /usr/lib/libstdc++.so.5...done.
> Loaded symbols for /usr/lib/libstdc++.so.5
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /usr/lib/libltdl.so.3...done.
> Loaded symbols for /usr/lib/libltdl.so.3
> Reading symbols from /usr/lib/libpspell-modules.so.1...done.
> Loaded symbols for /usr/lib/libpspell-modules.so.1
> Reading symbols from /usr/lib/libgpm.so.1...done.
> Loaded symbols for /usr/lib/libgpm.so.1
> Reading symbols from /usr/lib/httpd/modules/mod_python.so...done.
> Loaded symbols for /etc/httpd/modules/mod_python.so
> Reading symbols from /usr/lib/httpd/modules/mod_ssl.so...done.
> Loaded symbols for /etc/httpd/modules/mod_ssl.so
> Reading symbols from /usr/lib/php4/imap.so...Reading symbols from
> /usr/lib/debug/usr/lib/php4/imap.so.debug...done.
> done.
> Loaded symbols for /usr/lib/php4/imap.so
> Reading symbols from /usr/lib/php4/ldap.so...Reading symbols from
> /usr/lib/debug/usr/lib/php4/ldap.so.debug...done.
> done.
> Loaded symbols for /usr/lib/php4/ldap.so
> Reading symbols from /usr/lib/php4/mysql.so...Reading symbols from
> /usr/lib/debug/usr/lib/php4/mysql.so.debug...done.
> done.
> Loaded symbols for /usr/lib/php4/mysql.so
> Reading symbols from /usr/lib/php4/odbc.so...Reading symbols from
> /usr/lib/debug/usr/lib/php4/odbc.so.debug...done.
> done.
> Loaded symbols for /usr/lib/php4/odbc.so
> Reading symbols from /usr/lib/libodbc.so.1...done.
> Loaded symbols for /usr/lib/libodbc.so.1
> Reading symbols from /usr/lib/libodbcpsql.so.2...done.
> Loaded symbols for /usr/lib/libodbcpsql.so.2
> Reading symbols from /usr/lib/php4/pgsql.so...Reading symbols from
> /usr/lib/debug/usr/lib/php4/pgsql.so.debug...done.
> done.
> Loaded symbols for /usr/lib/php4/pgsql.so
> Reading symbols from /usr/lib/libpq.so.3...done.
> Loaded symbols for /usr/lib/libpq.so.3
> Reading symbols from /usr/lib/python2.2/lib-dynload/structmodule.so...done.
> Loaded symbols for /usr/lib/python2.2/lib-dynload/structmodule.so
> Reading symbols from /usr/lib/python2.2/lib-dynload/_codecsmodule.so...done.
> Loaded symbols for /usr/lib/python2.2/lib-dynload/_codecsmodule.so
> #0  0x00000009 in ?? ()
> (gdb) bt full
> #0  0x00000009 in ?? ()
> No symbol table info available.
> #1  0x000008e5 in ?? ()
> No symbol table info available.
> #2  0x0d630e04 in ?? ()
> No symbol table info available.
> #3  0x00000010 in ?? ()
> No symbol table info available.
> #4  0xbffefb88 in ?? ()
> No symbol table info available.
> #5  0xbffefb88 in ?? ()
> No symbol table info available.
> #6  0x00005000 in ?? ()
> No symbol table info available.
> #7  0x00005000 in ?? ()
> No symbol table info available.
> #8  0x00000000 in ?? ()
> No symbol table info available.
> (gdb) thread apply all bt full
>
> Thread 1 (process 59795):
> #0  0x00000009 in ?? ()
> No symbol table info available.
> #1  0x000008e5 in ?? ()
> No symbol table info available.
> #2  0x0d630e04 in ?? ()
> No symbol table info available.
> #3  0x00000010 in ?? ()
> No symbol table info available.
> #4  0xbffefb88 in ?? ()
> No symbol table info available.
> #5  0xbffefb88 in ?? ()
> No symbol table info available.
> #6  0x00005000 in ?? ()
> No symbol table info available.
> #7  0x00005000 in ?? ()
> No symbol table info available.
> #8  0x00000000 in ?? ()
> No symbol table info available.
> (gdb) quit
> #
>
> --
> Paul
> @ Thy Service
>
> > http://httpd.apache.org/dev/debugging.html
> >
> > On 3/7/07, Paul G. Szabady <Paul at thyservice.com> wrote:
> >> James,
> >>
> >> OK, I give up.  How can I make apache dump core?  I'm not finding
> >> anything
> >> valuable through google.  :(
> >>
> >> --
> >> Paul
> >> @ Thy Service
> >>
> >> > On 3/4/07, Paul G. Szabady <Paul at thyservice.com> wrote:
> >> >> Greetings,
> >> >>
> >> >> I was wondering if anyone had any good pointers for trouble-shooting
> >> >> Apache Segmentation fault errors on linux.
> >> > It doesn't matter the application if you are getting a seg fault, look
> >> > at the core file with gdb.  If you are not dropping cores configure
> >> > your box to do so (actually with the 2.6 kernel you can configure
> >> > where cores drop, a really nice feature [solaris had something
> >> > similar]).  Anyway once you have a readable core file type something
> >> > like:
> >> >
> >> >    gcb -c $path_to_core $path_to_executable
> >> >
> >> > Then type:
> >> >
> >> >    bt
> >> >
> >> > This will print a back trace, and that will tell you which subroutine
> >> > the the program was in when it died.   Even if you don't know enough
> >> > to do something with this information, its a great thing to paste into
> >> > a bug report or email to the maintainer (I always start with the
> >> > distro I'm using and start going upstream from there).
> >> > If you do know what to make of a back trace, you can usually (though
> >> > not always) figure out exactly what the problem is, and fix it.
> >> >
> >> > That said not all core dumps are actually bugs in the program.  Many
> >> > times programs will core dump, like the kernel panics when certain
> >> > things are expected to be a certain way and they are not (this is
> >> > called an assert).   The thinking is that to do anything else except
> >> > drop core could cause serious problems.  Some of these asserts catch
> >> > bugs, but others catch problems in the OS, libraries, your sysem setup
> >> > or your hardware (in some cases it could be your just out of memory,
> >> > and it happened that that last malloc was real critical).
> >> >
> >> > All that said most core dumps are due to lazy or tired programmers not
> >> > validating their data properly (well, and no matter how rigorous you
> >> > are in you approach to coding sometimes the neorons simply misfire).
> >> >
> >> > Chers...james
> >> > --
> >> > TriLUG mailing list        :
> >> http://www.trilug.org/mailman/listinfo/trilug
> >> > TriLUG Organizational FAQ  : http://trilug.org/faq/
> >> > TriLUG Member Services FAQ : http://members.trilug.org/services_faq/
> >> >
> >>
> >> --
> >> TriLUG mailing list        :
> >> http://www.trilug.org/mailman/listinfo/trilug
> >> TriLUG Organizational FAQ  : http://trilug.org/faq/
> >> TriLUG Member Services FAQ : http://members.trilug.org/services_faq/
> >>
> >
> >
> > --
> > Robert Dale
> > --
> > TriLUG mailing list        : http://www.trilug.org/mailman/listinfo/trilug
> > TriLUG Organizational FAQ  : http://trilug.org/faq/
> > TriLUG Member Services FAQ : http://members.trilug.org/services_faq/
> >
>
> --
> TriLUG mailing list        : http://www.trilug.org/mailman/listinfo/trilug
> TriLUG Organizational FAQ  : http://trilug.org/faq/
> TriLUG Member Services FAQ : http://members.trilug.org/services_faq/
>