OpenSolaris

1. Community: Fault Management
   1. Announcements
   2. Blogs
   3. News
   4. Discussions
   5. Events
   6. Leaders
   7. Observers
   8. Files
   9. SNMP Connector
   10. New Fault Management Features
   11. FMA Aware Device Drivers
   12. Solaris FMA Demo Kit

OpenSolaris Community: Fault Management

View the leaders for this community
Community Observers

• Enable/Enhance Solaris support for Intel Platform
  
• Events Registry
  
• Generic Machine Check Architecture Improvements
  
• Sensor Abstraction Layer
  
• x86 Generic FMA Topology Enumerator
  

About Predictive Self-Healing

Self-healing functionality for users and administrators of a modern operating system provides fine-grained fault isolation and restart where possible of any component—hardware or software—that experiences a problem. To do so, the system must include intelligent, automated, proactive diagnoses of errors that are observed on the system. The diagnosis system is used to trigger targeted automated responses or guided human intervention that mitigate a specific problem or at least prevent it from getting worse. Finally, these new system capabilities are connected to a new model for system administrators oriented around simpler, higher-level abstractions. Sun's first Predictive Self-Healing features are part of Solaris 10 and OpenSolaris and include the Fault Manager and the Service Manager.

About Fault Management

The Solaris Fault Management effort (originally code-named FMA inside of Sun) provides a new architecture for building resilient error handlers, error telemetry, automated diagnosis software, response agents, and a consistent model of system failures for a management stack. Many parts of Solaris are already participating in FMA, including the CPU and Memory error handling for UltraSPARC III and IV, the UltraSPARC PCI HBAs, and more. And a variety of projects are underway, including full support for CPU, Memory, and I/O faults on Opteron, conversion of key device drivers, and integration with various management stacks.

The legacy UNIX failure model was simply to leave error handling up to each subsystem author, and simply provide the ability to emit an error message for a human to the system log in a non-standard format. When a subsystem is converted to participate in Fault Management, error handling is made resilient so that the system can continue to operate despite some underlying failure, and telemetry events are produced that drive automated diagnosis and response. The Fault Management tools and architecture enable development of self-healing content for software and hardware failures, for both microscopic and macroscopic system resources, all with a unified, simple view for administrators and system management software.

Some objectives for the Fault Management Community are to:

• Convert hardware and software subsystems to participate in Fault Management
• Connect Solaris Fault Management to system management protocols and software
• Evolve and enrich the tools and common architecture for Fault Management
• Research underlying failure modes and design and develop automated diagnosis software that is able to build effective self-healing for particular resources

Documentation

• FMA Events and Messages
  • Diagnosis results obtained from the Fault Management software in Solaris contain links to the Knowledge Article Web.
  • The FMA Event Registry is the central repository for all fault management events passed between error handlers, the fault manager and its agents.

• Fault Management for System Administrators. This Fault Management piece is pointed to from Working with the Fault Manager in the Troubleshooting section of the OpenSolaris System Administration Guide,

• Man Pages
  • fmd(1M)
  • fmadm(1M)
  • In OpenSolaris 2008.11, the fmadm repair command has been replaced by the following new commands: fmadm repaired (synonymous with fmadm repair), fmadm replaced, and fmadm acquit.
  • fmdump(1M)
  • In OpenSolaris 2008.11, the preferred method to display fault information and determine the FRUs involved is the fmadm faulty command, not the fmdump command.
  • fmstat(1M)

• Writing Device Drivers for FMA. The Writing Device Drivers guide contains a section "Sun Fault Management Architecture I/O Fault Services" in "Chapter 13, Hardening Solaris Drivers." This section describes the steps and techniques used to write an FMA-aware driver.
  • This content has been pulled out separately. See Hardening Solaris Drivers. We should add documentation to describe the minimum level that all Solaris drivers are required to be hardened to and how to know that you have achieved that. Is there a test suite? Also, we should add information on how to harden specific types of drivers.

• Fault Management Daemon Programmer's Reference Manual. The FMD PRM [PDF] is a description of the internal architecture of the Sun Fault Management Daemon, fmd(1M), and the programming interfaces exported by the daemon.
  • FMDPRM 1.4 April 2008. Added -b option to the fmtopo command. Changed descriptions of TOPO_WALK_CHILD and TOPO_WALK_SIBLING.
  • FMDPRM 1.3 March 2008. Added repaircode to the table of Fault Management Configuration Properties.
  • FMDPRM 1.2 August 2007. Initial post of this document.

• Eversholt Fault Tree Description Language. The Eversholt Fault Tree Description Language [PDF] explains how to use the eversholt language to describe fault trees in the Sun Fault Management Architecture.
  • Eversholt Fault Tree Description Language, 1.8, November 2008. Initial post of this document.

What's New

The OpenSolaris universe of new fault management activities (projects, ARC cases, diagnosis engines, recovery agents and error handlers).

News

Predictive Self-Healing for x64 Feature Story | sun.com | 08/30/2006

Feature story for the front page of sun.com that describes the new Predictive Self-Healing features for x64 systems.

Benefits of Memory Page Retire | Dependable Systems and Networks 2006 | 07/04/2006

Paper for Dependable Systems and Networks Conference describing a quantitative model demonstrating the availability benefits of Solaris Memory Page Retire (MPR), driven by the Solaris Fault Manager.

Predictive Self-Healing and DTrace Receive 2005 Innovation Awards | InfoWorld | 08/01/2005

InfoWorld has announced its 2005 Innovator awards and Sun proudly received awards for Predictive Self-Healing and DTrace, two of the breakthrough technologies in Solaris 10, and the senior engineers who designed and led these projects.

A Diagnosis of Self-Healing Systems | SlashDot | 12/21/2004

SlashDot coverage of the ACM Queue article describing Sun's Predictive Self-Healing architecture.

Self-Healing in Modern Operating Systems | ACM Queue | 12/01/2004

ACM Queue article by Mike Shapiro describing Sun's approach to Predictive Self-Healing, including both Fault Management and Service Management in Solaris 10.

Blogs

Scott Davenport - The x86gentopo Project

Jun 30, 5:14 PM

You've all seen this before...actually, maybe you haven't. In which case....good! You've got a healthy system. But, if your box did have issues, FMA would report something like: # fmadm faulty ...

Scott Davenport - Device Driver Integration with Solaris FMA

Jun 3, 3:23 PM

A lot of work has been put into devising a common rule set for PCI/PCIE devices in Solaris FMA. Known as I/O Fault Services, there's a thorough document detailing how a developer goes about hardening ...

Scott Davenport - OpenSolaris 2009.06, the M3000, and FMA

Jun 1, 10:01 AM

OpenSolaris 2009.06 released today. One of the key features is support for the SPARC product line. A colleague and I gave one of the pre-releases a whirl on a midrange system, an M3000 Now, a silent ...

Scott Davenport - Contract Kills on Memory UEs

May 12, 9:19 AM

On SPARC systems, when there's a memory uncorrectable error (UE), Solaris will determine if the affected page is in user space or kernel space. If in user space, the affected user process is killed - ...

Scott Davenport - Solaris 10 Update 7 Available

May 1, 11:11 AM

Solaris 10 Update 7 is now posted and available for download. And there's been 65+ bug fixes and enhancements for FMA. Here's a few of my favorites (can one have favorite bugs? :) fixed in S10U7: ...