What happens after `ubuntu-bug` has done its thing?
Technically, ubuntu-bug just logs the crash report locally. A separate program, whoopsie, watches for logged reports and uploads them to a central database, where they're automatically grouped to identify overarching problems.
The resulting data is displayed on the Ubuntu error tracker:
Aggregate trends are publicly available, and report details are available to trusted developers. More details are available on the Ubuntu Wiki.
By default, ubuntu-bug doesn't open bugs on Launchpad for crash reports in stable releases, but you can configure it to if you want. After making that change, you can open a bug for an existing crash report by running ubuntu-bug /var/crash/foo.crash.
The collected information or report gets uploaded to a bug tracking system.
If any process in the system dies due to a signal that is commonly referred to as a 'crash' (segmentation violation, bus error, floating point exception, etc.), or e. g. a packaged Python application raises an uncaught exception, the apport backend is automatically invoked.
It produces an initial crash report in a file in /var/crash/ (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, or it belongs to a system process and the user is an administrator, apport informs the user about the crash and offers to report the problem.
If the user leaves the "Send error report" checkbox enabled, Apport uploads the collected information to the bug tracking system. After that it opens the packages' bug filing page with a sensible default bug title and leaves the rest of bug filing process to the web UI.
Ubuntu receives an incredibly large number of bug reports every day through our bug tracking system. Each one of these needs to be read, assessed, and sorted so it can be fixed. This is where we could use your assistance with Helping With Bugs. For a visual representation of the bug triage process, see these nice Flow Charts.
Every bug report is a conversation with the reporter. The first contact any reporter usually has with the Ubuntu community is through a bug triager, who tries to put together a complete bug report. It's very important that we give a good impression, so please be polite and try to use your best English.
Working on simple, untriaged bugs is a good way to get started and become acquainted with the triaging procedure since you'll have to deal with every aspect of the life cycle of a bug. The section Untriaged bugs explains where to find them.
Bug Types
Apport reports
Apport reports are bugs reported via the automated bug reporting program Apport. Reporting bugs using Apport is the preferred way of reporting a bug since it gives the developers a lot of information about the affected system. When Apport is used, less additional information is required, speeding up the whole process.
You can recognize these bugs by the added list of system information in their description. Some programs have hooks for Apport, adding more information when reporting a bug. This information can usually be found in the attachments.
Confirmed bugs
When a bug is marked as 'Confirmed' it is not fully triaged yet. This bug is very close to being marked as 'Triaged', but you need to make sure it is ready for the developers to fix.
Feature Requests
If the bug report is actually a feature request, there are two possibilities. If the requested enhancement is small and well-defined and/or the suggestion concerns an upstream project, the Importance of the bug should be set to 'Wishlist'. When the report is complete the status should be set to 'Triaged'.
Only the members of the Ubuntu Bug Control team can do so. If you're not a member you'll have to ask someone who is to do it for you. Paste the bug number in #ubuntu-bugs and say you think the bug should be set to 'Wishlist'. Someone will notice and set it for you, although not necessarily immediately.
How does it work internally?
Crash interception
Apport uses /proc/sys/kernel/core_pattern to directly pipe the core dump into apport:
$ cat /proc/sys/kernel/core_pattern
|/usr/share/apport/apport %p %s %c
$
Note: that even if ulimit is set to disabled core files (by specyfing a core file size of zero using ulimit -c 0), apport will still capture the crash.
For intercepting Python crashes it installs a /etc/python*/sitecustomize.py to call apport on unhandled exceptions.
Example
Apport is even able to capture core files if PID 1 (Upstart) dies:
- If Upstart detects an internal inconsistency, it raises the SIGABRT signal.
- The Upstart crash handler is called on SIGABRT.
- Upstart crash handler forks a child process.
- The Upstart child process re-raises the signal which results in the child exiting abnormally.
- The kernel detects the child process has exited abnormally and calls apport, piping the core file to apports standard input (due to /proc/sys/kernel/core_pattern).
- apport writes the core file to disk in /var/crash/.
- PID 1 waits for its child to terminate (which only happens once apport has finished writing the core file).
- PID 1 exits.
- kernel panics.
- On next boot, Whoopsie will detect the crash file and process it.
Backend
In order to keep the delay and CPU/IO impact as low as possible, /usr/share/apport/apport only collects data which has to be acquired while the crashed process still exists: information from /proc/pid, the core dump, the executable path, and the signal number. The report is written to /var/crash/executable_path.uid.crash.
Frontend invocation
In Gnome, update-notifier keeps an inotify watch on /var/crash. Whenever there is something new, it calls /usr/share/apport/apport-checkreports. If there are new reports, it calls /usr/share/apport/apport-gtk, which is the frontend shown in the screenshots above.
The frontend then collects additional information like package versions, package file checksums, or OS version, and calls all matching package hooks. To disable this, you can run gsettings set com.ubuntu.update-notifier show-apport-crashes false (as your ordinary desktop user).
Launchpad-based auto-retracer
The Canonical data center runs a service which automatically retrace bugs with apport. By tagging the bugs according to architecture in Launchpad, a retrace will be done and the tag will be removed. Tags that are used are need-i386-retrace or need-amd64-retrace. See the announcement.
Per-package Apport Hooks
It is possible for packages to specify information gathered from the system and included in the bug report. These are done by apport hooks contained in packages. For some useful examples see:
- source_xorg.py - adds additional log files and hardware details to bug reports
- usplash - ignores crashes in specific code paths
- source_totem.py - asks the reporter questions and gathers different information based on responses
in /usr/share/apport/package-hooks. There is also a list of packages providing apport hooks.
If a crash or bug report is submitted through apport, the relevant hooks will be run automatically. If you have an already reported bug that was filed without apport, and you are interested in the information from those hooks, you can ask the bug reporter to use apport-collect bugnumber.
Use the source, Luke!
- You can download the upstream tarball from the Launchpad project page, or the Ubuntu source tarball from the Ubuntu archive.
- apport is developed with the bazaar RCS on Launchpad. If you want to contribute to it or develop your own system based on it, you can get your own branch with bzr get lp:apport for trunk, or debcheckout -a apport for the Ubuntu packaging branch.
Future plans
Various improvements to performance, better tools to work with reports, and integration of more languages (Mono/Python stack traces, assertion messages, etc.) See the relevant specification.
Sources:Apport, How To Triage, and How to enable Apport