SOLARIS 10 DEEP DIVE: SMF

SOLARIS 10 DEEP DIVE: SMF

Christine Tran US Solaris Adoption Practice Sun Microsystems

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Topics
Part I: SMF Fundamentals
> > > > > > > >

Motivation for SMF Core concepts and terminology Command overview New boot process Quick tour of service administration Managing inetd services Tips for troubleshooting Recovering a repository

Part II: SMF in Action

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Topics
Part III: Advanced SMF
> > > > > > > >

Inside the manifest A little about XML Writing a manifest from scratch Exploring the repository with svccfg and svcprop Security aspects Snaphots Handling signals Portable services

Part IV: More Advanced SMF

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Topics
Part V: SMF Tools and SMF Community
> > > > > > >

Tools: Webmin Tools: xmllint, jedit, Morphon SMF Community on OpenSolaris Resources Review Q&A Closing

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Part I: SMF Fundamentals

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Motivation for SMF

What is a service? Is running a software program the same as delivering a service? What exactly is a service?
> Answer > Answer > Answer

Does there currently exist an OS framework for service support and management?
> Answer > Answer > Answer
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

How are services started today?

init(1M) vi rc scripts in /etc/rc?.d/*
> Long-time running or one-time initializations

inetd(1M) as defined by inetd.conf

> Short-lived to provide network functions

init(1M) as defined by /etc/initab

> Restartable or one-time functions

How are these services grouped? Could they be related? How are the relationships described?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Diagnostic ability
An application fails to start
> > > >

Missing configuration file? Failed to mount a filesystem? Database is late in starting up? Missing another component?

Lack of knowledge of service boundary and service interdependencies limits error handling ability of system. Lacks common framework of error-handling Lacks restart capability

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Management Today

Thousands of different text files, arbitrarily grouped and managed with multiple administration techniques. Undeclared, often unknown dependencies; linear startup is a by-product. Lacks common interface, each service is started a different way. Does not address multiple instances of a service, for example, web servers. Does not address services that may span multiple hosts, for example, a grid.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Motivation for SMF

A Service Oriented Architecture requires a definition of service and a more robust framework to deliver and manage it.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF gives you

A consistent service model in a common framework: command interface, service description, property specification, status view, etc. A meaningful system view. Ability to state dependencies (by-product is parallel boot). Restart capabilities All data stored in persistent, transaction-based repository. Snapshots allow undo and rollback to a working configuration.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service is:
An abstract description of a long-lived software object. May describe object on a collection of machines (grid). May have multiple instances (httpd). Each instance of a service has a well-defined state and a welldefined error boundary [process contract]. > Each service defines methods: start, stop, refresh, etc. > Each service defines dependencies: what it depends on, and may define dependents: what depends on it.
> > > >

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service has a unique identifier called FMRI Fault Management Resource ID

svc://localhost/network/login:rlogin
Scheme svc SMF managed service lrc legacy RC script

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service has a unique identifier called FMRI Fault Management Resource ID

svc://localhost/network/login:rlogin
Location localhost current hostname future release

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service has a unique identifier called FMRI Fault Management Resource ID

svc://localhost/network/login:rlogin
Functional category application system device network milestone platform site
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Functional Categories
Application traditional daemons Device useful for dependencies Milestone similar to run levels Network converted services from inetd.conf Platform platform-specific services System platform-independent system services Site reserved for local site use

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service has a unique identifier called FMRI Fault Management Resource ID

svc://localhost/network/login:rlogin
Service Description related to method or RC script

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

A service has a unique identifier called FMRI Fault Management Resource ID

svc://localhost/network/login:rlogin
Instance default is the default instance

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Core concepts and terminology

FMRI can be addressed by the shortest unique match, for example:
> > > >

svc://localhost/network/login:rlogin svc:/network/login:rlogin network/login:rlogin rlogin

Some common names have changed to a different FMRI, for example: syslog is now system-log

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service states
online the service instance is enabled and has successfully started. offline the service instance is enabled, but the service is not yet running or available to run, usually due to a dependency that has not been satisfied, or an error in the start method. disabled the service instance is not enabled and is not running. maintenance the service instance has encountered an error that must be resolved before it could be started again. There are many reasons why a service could be in this state.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service states
legacy_run the legacy service is not managed by SMF, but the service can be observed.
> Faults not handled by SMF, no automated restart > Administrative error undetected > Software or hardware error results in process death

degraded the service instance is enabled, but is running at a limited capacity. uninitialized this state is the initial state for all services before their configuration has been read.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service dependency
A dependency is a formal description of other services that are required to start a service. A service can be dependent on another service or files. When a dependency is not met, the service stays offline. Whether a dependency is satisfied depends on its type:
> require_all all services are running or all files are present. > require_any at least one is running or at least one file is

present. > optional_all all are running, disabled, in maintenance, or not present. For files, this type is the same as require_all. > exclude_all all are disabled, in maintenance, or when files are not present.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Dependency actions
Once running (online or degraded states are considered running) if a service dependency is stopped or refreshed, SMF determines whether to restart the service depending on the restart_on attribute specified.
restart_on attribute Error Restart Refresh yes yes yes no yes yes no no yes

Reason for dependency stop Error Non-error stop Refresh

None no no no

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service restarter
/lib/svc/bin/svc.startd is the master process starter and restarter. It
> reads the Service Configuration Repository and start services > > > >

on boot; restart services that have failed; shutdown services whose dependencies are no longer satisfied; run legacy RC scripts at run-level transitions; provides system view of service status.

There is a delegated restarter, inetd.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Delegated restarter
Assumes responsibility for executing a set of services. Manages service faults; may gracefully handle conditions such as signals or core dumps. May define/require additional service configuration. Often provides functionality common to a specific set of services, for example: inetd

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF Manifest
Description of the initial configuration of a service in XML. Loaded into the SMF repository at boot time. Sun-delivered services live in /var/svc/manifest. For ISVs, manifests should be placed in the appropriate subdirectory /var/svc/manifest. Manifests for anything that is specific to the customer's site only can go in /var/svc/manifest/site. This is reserved for local use.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF Manifest
A manifest identifies, at the very least:
> the service name > what the service is dependent on > ways to start and stop the service

Can contain other attributes, such as the delegated restarter, config files, log files and others.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Configuration Repository

Located in /etc/svc. Distributed between local memory (volatile) and local file (repository.db). repository.db holds the persistent service description read from the manifests. volatile is a tmpfs allocated from swap containing transient data (lock files, init state, log files). The underlying DB engine is sqlite 2.8 Zones have their own repository. Changes to services should be made against repository.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Profile and Archive

Profile a set of service instances and their enabled or disabled state. Useful for copying service states between systems. Generated by svccfg extract Archive a complete set of persistent data for all service instances. Useful for copying service definitions between systems. Generated by svccfg archive Output is an XML file similar to manifests.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF Snapshots
snapshot a complete collection of properties for a service instance. It provides a historical view of a service and simplifies rollback of service configuration changes. No snapshot-at-will capability yet. Snapshot types:
> > > > >

initial taken on the first import of the manifest last_import taken during last import running properties of the currently running service instance previous properties of the service instance just before running start taken at the last successful service start
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Key files and directories

/var/svc manifests, logs and profiles /lib/svc binaries, methods, seed, support files /etc/svc/repository.db global configuration database /lib/svc/seed/global.db minimal configuration database /lib/libscf.so.1 repository APIs

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

New filesystems
/etc/svc/volatile
> Directory where svc.startd stores log files in early stages of

boot, before /var is mounted read-write. Contains transient data.

/system/contracts
> Primary interface into contract subsystem, to allow service

instances to be restarted. Indirectly related to SMF.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Command overview
General commands
> > > >

svcs(1) svcadm(1M) svccfg(1M) svcprop(1)

service status listings administrative actions general property manipulation property reporting administrative actions/property modification conversion of legacy inetd.conf entries

inetd management commands

> inetadm(1M) > inetconv(1M)

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Command overview
Daemons
> svc.startd(1M) dependency engine, master restarter > svc.configd(1M) repository, authorization > inetd(1M) delegated restarter

Contracts subsystem
> > > >

ctrun(1M) ctstat(1M) ctwatch(1M) libcontract(3LIB)

execute with process contract display active contracts monitor contract events contract APIs

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

New boot process

Instead of booting to run level, SMF introduces the concept of milestone. milestone a service which specifies a collection of services which declare a specific state of systemreadiness. You can boot to specific milestone, services not part of that milestone are temporarily disabled. The default milestone is the milestone you will transition to at every boot. It is usually milestone all. init S, boot -s and friends still work.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Milestones and run levels

SVR4 Run Level s, S 2 3 -

SMF Milestone none single-user multi-user multi-user-server all

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Legacy services
Milestones are services, as such, they have manifests which have defined dependencies. After all its dependencies have been satisfied, the start method of the SMF major milestone executes each 'S' script within /etc/rc?.d/ with the argument 'start'. /var/svc/manifest/milestone/single-user.xml specifies a series of dependencies, and the start method is /sbin/rcS start. SMF services first, legacy services last, run once only.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Part II: SMF in Action

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service administration
svcs reports service status svcs -a show all services svcs -x show services not running, impact svcs -d show dependencies (what this depends on) svcs -D show dependents (what depends on this) svcs -p show member processes svcs -l/-v show additional details

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -a STATE legacy_run online STIME Oct_12 Oct_12 FMRI lrc:/etc/rc2_d/S20sysetup svc:/network/ssh:default

disabled Oct_12 svc:/application/print/server:default offline Oct_12svc:/application/print/rfc1179:default ...

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -x svc:/application/print/server:default (LP print server) State: disabled since Wed Oct 12 16:18:38 2005 Reason: Disabled by an administrator. See: http://sun.com/msg/SMF-8000-05 See: lpsched(1M) Impact: 2 dependent services are not running. -v for list.) (Use

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -d http STATE online online STATE STIME Oct_12 Oct_12 STIME FMRI svc:/network/loopback:default svc:/network/physical:default FMRI

# svcs -D loopback disabled Oct_12 svc:/network/dns/client:default disabled Oct_12 svc:/network/dns/server:default disabled Oct_12 svc:/network/rarp:default

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -p sendmail STATE online STIME 16:10:32 16:10:32 FMRI 23724 sendmail 23725 sendmail 16:09:32 svc:/network/smtp:sendmail

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcs
# svcs -l rarp fmri svc:/network/rarp:default name Reverse Address Resolution Protocol (RARP) server enabled state next_state state_time restarter false disabled none Wed Oct 12 16:18:41 2005 svc:/system/svc/restarter:default

dependency require_all/error file://localhost/etc/nsswitch.conf (online) dependency require_any/error

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Administration
svcadm administer services
-t -r -s temporary recursive synchronously waits for completion

svcadm enable/disable svcadm enable/disable -t svcadm restart svcadm refresh svcadm clear/mark svcadm milestone svcadm -d milestone

enable/disable services temporarily, until rebooted restart specified service re-read service configuration clear/mark maintenance state transition to a milestone transition and set as default

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm
# svcs sendmail STATE online STIME FMRI 16:49:59 svc:/network/smtp:sendmail

# svcadm disable -t sendmail # svcs sendmail STATE disabled STIME FMRI 16:51:27 svc:/network/smtp:sendmail

Does not show service in temporary state. The presence of this property tells us that this setting is temporary.
# svcprop sendmail | grep general_ovr/enabled general_ovr/enabled boolean false

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs keyserv STATE STIME FMRI disabled Oct_12 svc:/network/rpc/keyserv:default # svcadm enable keyserv # svcs keyserv STATE STIME FMRI maintenance 16:58:57 svc:/network/rpc/keyserv:default

Keyserv is in maintenance mode, why?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs -x svc:/network/rpc/keyserv:default (RPC encryption key storage) State: maintenance since Wed Oct 19 16:58:57 2005 Reason: Start method exited with $SMF_EXIT_ERR_CONFIG. See: http://sun.com/msg/SMF-8000-KS See: keyserv(1M) See: /var/svc/log/network-rpc-keyserv:default.log Impact: This service is not running.

Actually, it is because the domain name is not set on this host.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# domainname test.com # domainname test.com

There is a service called domain, which takes care of system identifying info; it has to be restarted.
# svcs -o fmri,desc domain FMRI # svcs domain STATE online STIME Oct_12 FMRI svc:/system/identity:domain DESC
svc:/system/identity:domain system identity(domainname)

# svcadm restart domain

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm keyserv
# svcs keyserv STATE STIME FMRI maintenance 16:58:57 svc:/network/rpc/keyserv:default

Keyserv is still in maintenance, why?

# svcadm clear keyserv # svcs keyserv STATE STIME FMRI online 17:02:48 svc:/network/rpc/keyserv:default

Will this survive a reboot?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
What milestone am I in?
# svcprop -p options_ovr/milestone system/svc/restarter:default svcprop: Couldn't find property `options_ovr/milestone' for instance `svc:/system/svc/restarter:default'.

If you see this, you are in milestone all.

# svcadm milestone single-user # Requesting System Maintenance Mode Console login service(s) cannot run Oct 19 17:19:00 zone1 syslogd:going down on signal 15 svc.startd: The system is coming down for administration. Please wait. svc.startd: Killing user processes: done.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
svc.startd: Killing user processes: done. Root password for system maintenance (control-d to bypass): single-user privilege assigned to /dev/console. Entering System Maintenance Mode # svcprop -p options_ovr/milestone system/svc/restarter:default svc:/milestone/single-user:default # reboot # svcprop -p options_ovr/milestone system/svc/restarter:default svcprop: Couldn't find property `options_ovr/milestone' for instance `svc:/system/svc/restarter:default'.

I'm back in milestone all.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svcadm milestone
# svcadm milestone -d single-user # Requesting System Maintenance Mode # reboot # svcprop -p options_ovr/milestone system/svc/restarter:default svc:/milestone/single-user:default

Every subsequent reboot will get me back here. To get back to full service, set default milestone to be all.
# svcadm milestone -d all

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Managing inetd services

Definition for inetd-started services were in inetd.conf but inetd.conf is now deprecated. Look how short it is! inetd.conf entries are converted to SMF manifests and imported into the repository on initial boot. The old way of adding a service by modifying inetd.conf and HUPing inetd no longer works. If you have an application that depends on what's in inetd.conf, there is a tool called inetconv that will convert the inetd.conf entry to an SMF manifest.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetd
inetd is now a delegated restarter for inet services. It manages inet-specific properties such as
> tcp_wrappers > max connection rate

It manages state for inet services. It listens for connections, etc.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetconv
# grep tftp inetd.conf # TFTPD - tftp server (primarily used for booting) # tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot

It's commented out. Uncomment the entry. You can HUP inetd but take my word that that won't do anything.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetconv
# inetconv inetconv: Notice: Service manifest for 100235/1 already generated as / var/svc/manifest/network/rpc/100235_1rpc_ticotsord.xml, skipped tftp -> /var/svc/manifest/network/tftp-udp6.xml Importing tftp-udp6.xml ...Done inetconv: Notice: Service manifest for 100083/1 already generated as / var/svc/manifest/network/rpc/100083_1-rpc_tcp.xml, skipped inetconv: Notice: Service manifest for 100068/2-5 already generated as / var/svc/manifest/network/rpc/100068_2-5-rpc_udp.xml, skipped

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetconv
# svcs tftp svcs: Pattern 'tftp' doesn't match any instances # svcs -l udp6 fmri name enabled state next_state state_time restarter svc:/network/tftp/udp6:default tftp true online none Wed Oct 19 17:40:16 2005 svc:/network/inetd:default

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Administration
inetadm administer inetd-started services inetadm list all services managed by inetd inetadm -p show default inetd service property values inetadm -l show all properties for a service inetadm -e/-d enable/disable a service inetadm -m modify property values inetadm -M modify inetd default property values

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetadm
# inetadm -l udp6 SCOPE NAME=VALUE name="tftp" endpoint_type="dgram" proto="udp6" isrpc=FALSE wait=TRUE exec="/usr/sbin/in.tftpd -s /tftpboot" user="root" ...

Let's say you want to run with some other option, -X instead of -s.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetadm
# inetadm -m udp6 exec="/usr/sbin/in.tftpd -X / tftpboot" # inetadm -l udp6 SCOPE NAME=VALUE name="tftp" endpoint_type="dgram" proto="udp6" isrpc=FALSE wait=TRUE exec="/usr/sbin/in.tftpd -X /tftpboot" user="root"

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetadm
Can also use inetadm to manage inetd-services as well as svcadm.
# svcs udp6 STATE STIME FMRI online 17:49:44 svc:/network/tftp/udp6:default # inetadm -d udp6 # svcs udp6 STATE STIME FMRI disabled 18:01:53 svc:/network/tftp/udp6:default

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

inetadm
Usually if you don't use a service, it's enough to disable it using svcadm. But you want it really gone then delete it from your repository using svccfg delete FMRI.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Administration
svcprop get service configuration properties -p property FMRI fetch a property -s snapshot FMRI fetch property from the snapshot
# svcprop -p inetd_start/exec tftp/udp6 /usr/sbin/in.tftpd\ -s\ /tftpboot # svcprop -p restarter/state tftp/udp6 online

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Service Administration
svccfg modify service configurations select FMRI select a service listprop show properties for a service setprop change a property value for a service delprop delete a property value for a service delete delete a service export|import export|import a service description in XML extract|apply extract|apply a profile archive dump a full XML service description for all services
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svccfg
# svccfg -s print/server listprop 'lpsched/fd_limit' lpsched/fd_limit # svccfg svc:> select print/server svc:/application/print/server> listprop lpsched/fd_limit lpsched/fd_limit count 0 svc:/application/print/server> setprop lpsched/fd_limit = 8192 svc:/application/print/server> exit # svcadm refresh print/server # svccfg -s print/server listprop 'lpsched/fd_limit' lpsched/fd_limit count 8192 count 0

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svccfg
Can be used in batch or interactive mode. Understands globbing to form regex We'll work through a more pertinent example later.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Tips for troubleshooting

If a single service is broken, -x is your friend. svcs -x will display the service logfile, if it exists. See service start message:
> boot -m verbose.

For a system that hangs during boot:

> boot -m verbose,milestone=none > log in at prompt > svcadm milestone all

Watch system progress with svcs.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Tips for troubleshooting

truss the service starting up by trussing the restarter Or modify the start method, wrapping it in truss
# svcprop -p start/exec apache2 /lib/svc/method/http-apache2 start # svccfg -s apache2 svc:/network/http:apache2> setprop start/exec = /usr/bin/truss -ealfo /tmp/apache2.truss /lib/svc/method/http-apache2 start

Don't forget svcadm clear once fault is rectified.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Last resorts
Have changes been made to repository?
> How many services are affected? > What snapshots exist? > Can you revert back to a previous snapshot?

Read /lib/svc/share/README. Restore repository from backup.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Revert a snapshot
# svccfg -s svc:/network/dns/server:default svc:/network/dns/server:default> listsnap initial last-import previous running start svc:/network/dns/server:default> revert start svc:/network/dns/server:default> exit # svcadm refresh dns/server # svcadm restart dns/server

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Recovering a repository
The SMF repository can be recovered in case of corruption. You may want to save the current repository in /etc/svc/repository.db for post mortem. Restore script is /lib/svc/bin/restore_repository. If you restore from seed, next boot would be the initial boot, you will see SMF reading and importing manifests.
# zlogin -C zone1 [Connected to zone 'zone1' console] 40/104

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

restore_repository
# /lib/svc/bin/restore_repository The following backups of /etc/svc/repository.db exist, from oldest to newest: boot-20051012_125317 manifest_import-20051012_130044 boot-20051012_161835 boot-20051019_172301 boot-20051019_180815 # Please enter one of: 1) boot, for the most recent post-boot backup 2) manifest_import, for the most recent manifest_import backup. 3) a specific backup repository from the above list
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

restore_repository
4) -seed-, the initial starting repository. customizations will be lost.) 5) -quit-, to cancel. Enter response [boot]: -seedAfter confirmation, the following steps will be taken: svc.startd(1M) and svc.configd(1M) will be quiesced, if running. /etc/svc/repository.db -- renamed --> /etc/svc/repository.db_old_20051019_182048 /lib/svc/seed/nonglobal.db -- copied --> /etc/svc/repository.db and the system will be rebooted with reboot(1M). Proceed [yes/no]? yes
Sun Proprietary/Use by Permission Only

(All

SOLARIS 10 DEEP DIVE: SMF

Part III: Advanced SMF

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Manifest revisited
Manifest is the description of the initial configuration of a service, delivered in XML. Automatically imported into repository on boot, install, upgrade, and pkgadd by svc:/system/manifest-import. Ignored once imported unless modified. Do not modify Sun-delivered manifests! If you need to change service properties, do it in the repository. We'll talk more about this.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Order is important
Service bundle Service name Create default instance Single instance Dependencies Exec methods Property groups Stability level Template
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

A little about XML

XML markups contain data about data, metadata, on the information they bracket.
> <time>1200</time>

Tells you at 1200 is a time unit, instead of weight or distance. Whatever processes XML data can recognize that data for what it is. Can contain any kind of data. How do we differentiate? An XML file of train schedule will have different formatting than an XML file of TV schedule, not just formatting, but different processing and different usage. Yet, <time>1200</time> can appear in both XML files.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Introducing DOCTYPE
Look at the top of each manifest, they all have this line:
> <!DOCTYPE service_bundle SYSTEM

'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>

DTD Document Type Definition, contrast with DOCTYPE for HTML. Tells you whether you (especially if you are an XMLprocessing program) are reading a train schedule or a TV Guide listing. DTD is one example of a type of markup called declaration. Specifically, the DTD is used to constrain the grammar of an XML file.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Grammar?
Grammar governs the correctness of an XML document. A train schedule must have: source, terminus, track number, departure and arrival time. It won't have channel, show name, show time, duration, starring, rating. Looking at just data, they look very similar. The DTD defines and enforces how they must be marked up in XML, one way and one way only. DTD is a collection of rules, or declarations, describing markup objects. An element declaration defines what it can contain and in which order, also known as a content model.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Define your own language

Strictly speaking, XML is not a markup language. It has no tag with absolute meaning. <time> can mean absolute time or elapsed time. Contrast this with <b></b>, this always means bold. XML lays down framework for you to construct your own markup language. Adding elements in the DTD adds objects to your language. An element not declared in the DTD is illegal. An element containing anything not declared in the DTD is illegal. We'll see why this is important later on.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

What's in the DTD?

A bunch of things! None resembling what you might find in a manifest.
> <!-- comments --> are comments > <!ELEMENT> or <!ATTLIST> are declarations > <blah /> is same as <blah></blah>

Ordering in DTD is not same as the required ordering in manifest. For example, declaration for property group is before declaration of service FMRI, and service bundle is declared last.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Here's an snippet
<!ELEMENT dependency ( service_fmri*, stability?, ( propval | property )* ) > <!ATTLIST dependency name grouping exclude_all | restart_on none ) #REQUIRED type delete CDATA #REQUIRED ( require_all | require_any | optional_all ) #REQUIRED ( error | restart | refresh | CDATA #REQUIRED ( true | false ) "false" >
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Why XML?
Think about what's in /etc/rc?.d today. Anyone can write anything in any format and drop it in /etc/rc?.d. In fact, what's in /etc/rc?.d is closer to methods than manifests. A unified service management tool depends on having uniform data to process. The repository is a database which relies on having correct data in the correct cell. With XML you get a parser and a validator for free. Anything that XML can't validate won't get imported. The parser is in a library called libxml2. It comes with xmllint, which is a generic XML validator.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

A quick tour inside the manifest

We said a manifest must have at least
> > > >

the service name what the service is dependent on ways to start and stop the service in theory you need a restarter too but there is the default svc.startd. Lights Depends on Power On/Off switch on the wall, flip up or down Robot is the default restarter
Sun Proprietary/Use by Permission Only

House-keeping robot analogy

> > > >

SOLARIS 10 DEEP DIVE: SMF

tftp-udp6.xml
service name='network/tftp/udp6' type='service' <exec_method type='method' name='inetd_start' exec='/usr/sbin/in.tftpd -s /tftpboot' <exec_method type='method' name='inetd_offline' exec=':kill_process'

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

tftp-udp6.xml
<restarter> <service_fmri value='svc:/network/inetd:default' /> </restarter>

Note that the method isn't start, but inetd_start. Wait! No dependencies? Well, having inetd as your delegated restarter implicitly marks you as dependent on inetd. Who starts inetd, then? It's started in /var/svc/profile/generic_open.xml If you look at /var/svc/manifest/network/nfs/server.xml, you will see that there is no explicitly defined restarter.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Writing a manifest from scratch

Not all Sun products are delivered in a manifest, the reason seems to be time and effort. The README file for MySQL still says to link a startup script in /etc/rc3.d to the mysql.server script. If no manifest is delivered, the service can still run as a legacy service. We're going to convert MySQL into an SMF service. You may want to do this in a zone. Remember that each zone has their own manifests and SMF repository.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Setting up MySQL database

First, we need to set up the database. Instructions are in /etc/sfw/mysql/README.solaris.mysql. The startup script is /etc/sfw/mysql/mysql.server.
root@mysql~> /usr/sfw/bin/mysql_install_db Preparing db table Preparing host table Preparing user table Preparing func table Preparing tables_priv table Preparing columns_priv table Installing all prepared tables 13:38:46 /usr/sfw/sbin/mysqld: Shutdown Complete
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

MySQL
Add users and groups
root@mysql~> groupadd mysql root@mysql~> useradd -g mysql mysql root@mysql~> chgrp -R mysql /var/mysql root@mysql~> chmod -R 770 /var/mysql

Add configuration files

root@mysql~> installf SUNWmysqlr /var/mysql d 770 root mysql root@mysql~> cp /usr/sfw/share/mysql/my-medium.cnf /var/mysql/my.cnf

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

MySQL
Start MySQL manually to see that everything works
root@mysql~> /usr/sfw/sbin/mysqld_safe --user=mysql & Starting mysqld daemon with databases from /var/mysql

Set the passwords

root@mysql~> /usr/sfw/bin/mysqladmin -u root password 'mysql' root@mysql~> /usr/sfw/bin/mysqladmin -u root -h mysql password 'mysql'

Check the status of MySQL

root@mysql~> /usr/sfw/bin/mysqladmin -u root -pmysql status Uptime: 37 Threads: 1 Questions: 1 Slow queries: 0 Opens: 6 Flush tables: 1 Open tables: 0 Queries per second avg: 0.02
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Copy and modify

Great! Everything works. Now you are ready to write your manifest! Find a manifest similar to your application and tweak it. Since MySQL is an application, I'll look in /var/svc/manifest/application. The print/server.xml seems closest to what I want, so I'll start from that. Remember that site-specific manifests go into site. Most everything is already filled out, I just change a few relevant properties.
root@mysql~> cp /var/svc/manifest/application/print/server.xml /var/svc/manifest/site/mysql.xml

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Name
Change the name
<service_bundle type='manifest' name='SUNWsqlr:mysql'> <service name='application/mysql' type='service' version='1'>

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Dependency
I leave the existing dependencies alone. I may want to add a dependency on the config file, so MySQL doesn't start if it's missing.
<dependency name='database' grouping='require_all' restart_on='none' type='path'> <service_fmri value='file://localhost/var/mysql/my.cnf'/> </dependency>

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

This just in
File dependencies going may be going away. CR 6240573
> Synopsis: File dependencies are currently overused by SMF

consumers. The problem is that a file dependency means "wait for this file to become available before starting". That's not the usual case. Most users of file dependencies are trying to express the concept that "if this file is missing, there is a problem."

No file event to signal SMF if a file has changed. Actions in a file dependency doesn't do anything, yet.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Methods
Change the start/stop methods
<exec_method type='method' name='start' exec='/etc/sfw/mysql/mysql.server start' timeout_seconds='60' /> <exec_method type='method' name='stop' exec='/etc/sfw/mysql/mysql.server stop' timeout_seconds='60' />

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Description and documentation

For now, delete the property groups as they don't apply to mysql Change the description and documentation location
<loctext xml:lang='C'> MySQL server </loctext> <documentation> <manpage title='mysql' section='1M' manpath='/usr/sfw/man' /> </documentation>

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Importing your manifest

You're done! Now import it.
root@mysql~> svccfg import mysql.xml root@mysql~> svcs -l mysql fmri name enabled state next_state state_time restarter svc:/application/mysql:default MySQL server false disabled none Tue Jun 07 19:20:51 2005 svc:/system/svc/restarter:default

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Importing your manifest

dependency require_all/none svc:/system/filesystem/local (online) dependency require_all/none svc:/system/filesystem/usr (online) dependency require_all/refresh svc:/system/identity:domain (online) dependency (online) optional_all/none svc:/system/system-log

dependency require_all/none file://localhost/var/mysql/my.cnf (online root@mysql~> svcadm enable mysql root@mysql~> svcs mysql STATE STIME FMRI online 19:21:19 svc:/application/mysql:default
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Done!
Congratulations! You have successfully written a service manifest. Now MySQL is an SMF-managed service.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF on OpenSolaris
Collection of manifests and methods are at http://www.opensolaris.org/os/community/smf/manifests/ There is a different manifest for MySQL submitted there.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Exploring the repository

We said the repository is a database containing all descriptions of all services SMF knows about, and their states. Make changes to service against its content in the repository, not in the manifest. The days of looking in a text file to find what's running is over. In fact, what's in a manifest can differ vastly with the running configuration. We'll fix Bug ID 6270245 as an example. Bug ID: 6270245
> Synopsis: apache2 needs svc:/system/filesystem/local

dependency when Sun Volume Manager metadevices are in use

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Two ways to fix

Two ways to fix: Delete the service from the repository, fix the manifest and re-import. This is not the recommended way. Fix the currently running service configuration. This is the recommended way, and we'll talk about why after we've finished the exercise. We're going to do this both ways, in order to better understand the repository and the manifest. In this case the service can be addressed as http or apache2, http is the service, apache2 is the instance. Disable the service, then delete it from the repository.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Fix 1: deleting http from repository

# svccfg svc:> list network/http network/http svc:> delete network/http svc:> exit Or # svccfg delete apache2 # svcs apache2 svcs: Pattern 'apache2' doesn't match any instances STATE STIME FMRI

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Modifying manifest
Once again, look around for something to copy from, I find my filesystem/local dependency in inetd-upgrade.xml and ssh.xml, respectively:
inetd-upgrade.xml: <dependency name='filesystem' ssh.xml: <dependency name='fs-local'

Note the name, filesystem and fs-local. These are just common names, they are not FMRI. In fact, these can be anything descriptive. Add this to my /var/svc/manifest/network/http-apache2.xml Import your fixed manifest and check.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

filesystem/local dependency
dependency name='myfs-local' grouping='require_all' restart_on='none' type='service'> <service_fmri value='svc:/system/filesystem/local'/> </dependency>

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Compare with DTD

<!ELEMENT dependency ( service_fmri*, stability?, ( propval | property )* ) > <!ATTLIST dependency name | CDATA #REQUIRED grouping ( require_all | require_any | exclude_all optional_all ) #REQUIRED restart_on ( error | restart | refresh | none ) #REQUIRED type delete CDATA #REQUIRED ( true | false ) "false" >
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

In depth
name is myfs-local, a descriptive name. grouping is require_all, self-explanatory, but when is grouping not require_all? When the dependency is a milestone, for example. restart_on is none. If you depend on a service, and its state changes, you may want to re-examine your own service. Refer back to slide reason for dependency stop type is service; can be service or file.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

restart_on
restart_on none: do not restart, you require this dependency to be satisfied once, on startup, that's all. restart_on error: restart when this service faults. Why? Opposite of previous case, you depend on this service all the time. If it faults (restart is implicit) and does not come back up, you need to restart and check your dependency. What state would you be in? restart_on restart: restart when this service has an error stop, or a non-error stop. restart_on refresh: restart on all the above conditions, and when this service re-read its configuration file.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

svccfg import
# svccfg import http-apache2.xml # svcs -l apache2
fmri name enabled state next_state state_time restarter svc:/network/http:apache2 Apache 2 HTTP server false disabled none Mon Jun 06 18:26:41 2005 svc:/system/svc/restarter:default

dependency require_all/error svc:/network/loopback:default (online) dependency optional_all/error svc:/network/physical:default (online) dependency (online) require_all/none svc:/system/filesystem/local
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Fix 2: modify properties in repository

Fix by modifying current service properties in the repository. Now that I've written the manifest, I know better what to do. Basically, take that block of dependency and some how get it into the running service configuration of apache2. Disable and delete apache2 from the repository. Re-import the original, unrepaired manifest; it should only have two dependencies.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Put back original manifest

# svccfg -s apache2 delete # svccfg import http-apache2.xml # svccfg -s apache2 listpg loopback physical start stop refresh httpd startd general restarter restarter_actions dependency dependency method method method application framework framework framework NONPERSISTENT framework NONPERSISTENT
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

What I'm interested in

# svccfg svc:> help General commands: help set repository end Manifest commands: inventory validate import export archive Profile commands: Entity commands: apply extract list select unselect add delete

Snapshot commands: listsnap selectsnap revert Property group commands: listpg addpg delpg Property commands: listprop setprop delprop editprop Property value commands: addpropvalue delpropvalue setenv unsetenv svc:>
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Adding properties
Add a new dependency to my property group
svc:/network/http:apache2> addpg myfs-local dependency

In your manifest, this conforms to

<dependency name='myfs-local'

Add other properties

svc:/network/http:apache2> setprop myfslocal/grouping = astring: require_all svc:/network/http:apache2> setprop myfslocal/restart_on = astring: none svc:/network/http:apache2> setprop myfs-local/type = astring: servicr

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Oops! A typo.
svc:/network/http:apache2> listprop myfs-local myfs-local/group myfs-local/restart_on myfs-local/type dependency astring astring astring require_all none servicr

svc:/network/http:apache2> editprop (invokes vi) # Property group "myfs-local" # delprop myfs-local # addpg myfs-local dependency # setprop myfs-local/group = astring: (require_all) # setprop myfs-local/restart_on = astring: (none) # setprop myfs-local/type = astring: (servicr)
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Add the FMRI

Change this line to
setprop myfs-local/type = astring: service

Upon quitting vi, the temp file is executed. This corresponds to

grouping='require_all' restart_on='none' type='service'>

Now add the FMRI of the dependency

svc:/network/http:apache2> setprop myfs-local/ entities = fmri: svc:/system/filesystem/local

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Check your configuration

svc:/network/http:apache2> listprop ... filesystem-local filesystem-local/group filesystem-local/restart_on filesystem-local/type dependency astring astring astring require_all none service

filesystem-local/entities fmri svc:/system/filesystem/local ... svc:/network/http:apache2> exit

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Refresh the service

# svcadm refresh apache2 # svcs -l apache2 fmri name ... restarter svc:/system/svc/restarter:default dependency require_all/error svc:/network/loopback:default (online) dependency optional_all/error svc:/network/physical:default (online) dependency require_all/none svc:/system/filesystem/local (online) svc:/network/http:apache2 Apache 2 HTTP server

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

New property added

# svccfg -s apache2 listpg loopback physical start stop refresh httpd startd general restarter restarter_actions myfs-local dependency dependency method method method application framework framework framework NONPERSISTENT framework NONPERSISTENT dependency
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

The reason why ...

You've modified the service properties for apache2 in the repository. Don't be confused the next time you look at http-apache2.xml and wonder what happened. Manifests are read and imported only once (unless they are modified.) Don't modify manifests in place. Upon boot svc:/system/manifest-import searches /var/svc/manifest for new XML files and ones which have changed and import them. This allows us to upgrade service properties (via a patch, for example) by dropping new manifest in, instead of manipulating the repository using svccfg.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

The reason why ...

When svccfg import is run on a new manifest for services which already exist, it first compares the current values to those in the last-import snapshot. If values don't differ, it assumes that there's been no customization, and upgrades the properties to what's in the manifest. Otherwise it issues warnings to the user, but leaves the customizations in place. So any patch which alters a manifest will cause manifestimport to try to upgrade the corresponding service, no post-install action necessary.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

The reason why ...

If you had modified that manifest in place and re-imported it, or even if you copied it somewhere else, modified it, and re-imported it, svccfg will think that the modified version is the original, official version (via the last-import snapshot comparison), will will override all your customization after the patch. The moral of the story: leave those manifests alone. Make changes against the repository with svccfg. man smf_bootstrap(5) for the gory details.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Part IV: More Advanced SMF

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF security
SMF uses RBAC to delegate service management responsibility. We'll talk about managing specific services first. SMF provides these propval to be put into individual manifest to authorize specific services:
> action_authorization can execute method > value_authorization can change value of existing properties > modify_authorization change, add, delete properties

These are fixed keywords. Let's take a look smtp-sendmail.xml and ssh.xml as examples.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

action_authorization
smtp-sendmail.xml: <property_group name='general' type='framework'> <propval name='action_authorization' type='astring' value='solaris.smf.manage.sendmail' /> </property_group> ssh.xml: <property_group name='general' type='framework'> <!-- to start stop sshd --> <propval name='action_authorization' type='astring' value='solaris.smf.manage.ssh' /> </property_group>

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

prof_auth
Where will you find this? /etc/security/prof_auth Mail Management:::Manage sendmail & queues:auths=solaris.smf.manage.sendmail;help=RtMail Mngmnt.html Network Security:::Manage network and host security:auths=solaris.smf.manage.ssh,solaris.network.*; help=RtNetSecure.html These RBAC roles have this privilege, which privilege is this? The one defined by the keyword action_authorization, which allows you to execute methods.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Delegating to a user
But you haven't set up roles properly, you want to delegate service administration to a user.
# echo christine::::auths=solaris.smf.manage.ssh >> /etc/user_attr

You have to re-login for authorization to take effect.

# su christine $ id uid=85228(christine) gid=10(staff)

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Delegating to a user
$ svcs ssh STATE online $ date Fri Oct 21 15:47:40 EDT 2005 $ svcadm restart ssh $ tail /var/svc/log/network-ssh:default.log [ Oct 21 15:47:42 Stopping because service restarting. ] [ Oct 21 15:47:42 Executing stop method (:kill) ] [ Oct 21 15:47:42 Executing start method ("/lib/svc/method/sshd start") ] [ Oct 21 15:47:42 Method "start" exited with status 0 ]
Sun Proprietary/Use by Permission Only

STIME Oct_19

FMRI svc:/network/ssh:default

SOLARIS 10 DEEP DIVE: SMF

A little problem
$ svcadm disable ssh svcadm: svc:/network/ssh:default: Permission denied. $ svcadm disable -t ssh $ svcs ssh STATE disabled STIME FMRI 15:55:43 svc:/network/ssh:default

$ svcprop ssh |grep general_ovr general_ovr/enabled boolean false

Service can only be temporarily disabled, why? See BugID 6303496.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Needs value_authorization
Bug ID 6303496
> Synopsis cron cannot be smf managed with

solaris.smf.manage.cron

It turns out that you also need value_authorization, the keyword to change existing property values. Why? Because you're actually changing the value of some property. Which property? And exactly where is that general_ovr property listed? general and general_ovr appear to be property groups of type framework. Check the DTD. We'll talk more about instances later.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Looking for general_ovr

# svccfg -s ssh svc:/network/ssh> listprop general* general general/action_authorization solaris.smf.manage.ssh general/entity_stability general/single_instance # svccfg -s ssh:default svc:/network/ssh:default> listprop general* general general/enabled general_ovr NONPERSISTENT general_ovr/enabled
Sun Proprietary/Use by Permission Only

framework astring astring boolean Unstable true

framework boolean true framework boolean false

SOLARIS 10 DEEP DIVE: SMF

value_authorization
# svccfg -s ssh svc:/network/ssh> listprop general/action_authorization solaris.smf.manage.ssh astring

Do you need to addpg?

svc:/network/ssh> setprop general/value_authorization = astring: "fooblat" svc:/network/ssh> exit

You still need to make one change for this to work.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

value_authorization
# tail /etc/user_attr christine::::auths=solaris.smf.manage.ssh,fooblat # svcadm refresh ssh # su - christine $ svcadm disable ssh $ svcs ssh STATE disabled $ STIME FMRI 16:42:47 svc:/network/ssh:default

$ svcprop ssh|grep general_ovr

Yes, it's a permanent change.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Value can be anything

The value of the auth used has no specific meaning except that it must exist in /etc/user_attr. In the manifest
<propval name='value_authorization' type='astring' value='fooblat' />

... is equal to
<propval name='value_authorization' type='astring' value='solaris.smf.manage.ssh' />

... but the later is more descriptive. In fact:

# auths christine solaris.smf.manage.ssh,fooblat,solaris.device.cdrw,s
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF Security
If you look at /etc/security/prof_attr, you will see Service Management:::Manage services:auths=solaris.smf.manage, solaris.smf.modify These are two main authorizations in SMF:
> solaris.smf.modify add, delete, modify services > solaris.smf.manage restart, refresh, or other state

modification

They don't appear in any manifest because they apply all services. There are others: method, dependency, application, and framework. man smf_security(5) for the gory details.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Blueprint
Restricting Service Administration in the Solaris 10 Operating System by Glenn Brunette (June 2005)
This Sun BluePrints Cookbook describes how to use the Solaris 10 Service Management Facility (SMF) to require specific authorizations for certain types of operations. Using this capability, it is possible to delegate access to core service management functions based on the concept of least privilege--if a user or service does not strictly need to have some degree of privilege, then that privilege should not be granted. SMF allows organizations to have much finer grained access control policies than was possible before the Solaris 10 Operating System.

http://www.sun.com/blueprints/0605819-2887.pdf

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Snapshots
# svccfg -s ssh svc:/network/ssh> listsnap Instance not selected. svc:/network/ssh> select default svc:/network/ssh:default> listsnap last-import initial running

Where are snapshot start and previous?

# svcs ssh STATE disabled STIME Oct_21 FMRI svc:/network/ssh:default
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

What's the difference?

# svccfg -s ssh listprop '*general*' general general/entity_stability general/single_instance general/action_authorization solaris.smf.manage.ssh general/value_authorization general general/enabled general_ovr framework boolean false framework NONPERSISTENT framework astring boolean astring astring yadayada Unstable true

# svccfg -s ssh:default listprop '*general*'

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

instance
Instances are units of administrations. Services are mechanism for sharing properties. Instance can have properties, and its parent service can have properties. Instances have snapshots, services do not. If unnamed, that instance is called default. Snapshots are composite view of the instance and service properties at one point in time.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

revert
svc:/network/ssh:default> listsnap last-import running initial start

svc:/network/ssh:default> selectsnap initial [initial]svc:/network/ssh:default> revert svc:/network/ssh:default> unselect svc:/network/ssh> listprop general* general general/action_authorization solaris.smf.manage.ssh general/entity_stability general/single_instance framework astring astring boolean Unstable true

value_authorization has disappeared.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Snapshots
Snapshots are not good for rolling back a single change because
> No control over when snapshots are taken > Alters the service properties as well

To undo a property change, it's simpler to use svccfg setprop commands Can use svcprop or svccfg export to take textual snapshots. However, when you use svccfg export, you use the service name, even though the output file contains the description of the instance(s).
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

ignore_error property
From http-apache2.xml:
<property_group name='startd' type='framework'> <!-- sub-process core dumps shouldn't restart session --> <propval name='ignore_error' type='astring' value='core,signal' /> </property_group>

SMF is instructed to ignore signals sent to a service or core dumps by a service; otherwise SMF will try to restart the service. man svc.startd(1M) for the gory details; description at the startd/ignore_error paragraph.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Signaling httpd
# ps -ef|grep http webservd 29855 29853 0 11:46:46 ? /usr/apache2/bin/httpd -k start root 29853 27175 0 11:46:45 ? /usr/apache2/bin/httpd -k start webservd 29857 29853 0 11:46:46 ? /usr/apache2/bin/httpd -k start webservd 29856 29853 0 11:46:46 ? /usr/apache2/bin/httpd -k start webservd 29858 29853 0 11:46:46 ? /usr/apache2/bin/httpd -k start webservd 29854 29853 0 11:46:46 ? /usr/apache2/bin/httpd -k start # kill -HUP 29853 <-- started by root
Sun Proprietary/Use by Permission Only

0:00 0:00 0:00 0:00 0:00 0:00

SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/apache2/logs/error_log (nothing ...) # tail -f /var/svc/log/network-http:apache2.log [ Oct 26 11:46:44 Executing start method ("/lib/svc/method/http-apache2 start") ] [ Oct 26 11:46:45 Method "start" exited with status 0 ] (nothing ... )

What would happen, by the way, if you kill just httpd processes started by webservd?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Keep on killing httpd

# ps -ef|grep http webservd 29854 27175 0 11:46:46 ? /usr/apache2/bin/httpd -k start # kill -9 29854 <-- last one # tail -f /var/apache2/logs/error_log [Wed Oct 26 12:46:53 2005] [notice] Digest: generating secret for digest authentication ... [Wed Oct 26 12:46:53 2005] [notice] Digest: done [Wed Oct 26 12:46:54 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations 0:00

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

httpd restarted by SMF

# tail -f /var/svc/log/network-http:apache2.log Stopping because all processes in service exited. ] [ Oct 26 11:55:05 Executing stop method ("/lib/svc/method/http-apache2 stop") ] httpd (pid 29853?) not running [ Oct 26 11:55:05 Method "stop" exited with status 0 ] [ Oct 26 11:55:05 Executing start method ("/lib/svc/method/http-apache2 start") ] [ Oct 26 11:55:06 Method "start" exited with status 0 ] # svcs apache2 STATE online STIME FMRI 12:46:53 svc:/network/http:apache2
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Removing ignore_error property

# svccfg -s apache2 svc:/network/http:apache2> listprop startd* startd startd/ignore_error framework astring core,signal

svc:/network/http:apache2> delprop startd/ignore_error svc:/network/http:apache2> listprop startd* startd framework svc:/network/http:apache2> exit # svcadm refresh apache2

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Log of httpd refreshing

# tail -f /var/apache2/logs/error_log
[Wed Oct 26 12:55:30 2005] [notice] Graceful restart requested, doing restart [Wed Oct 26 12:55:30 2005] [warn] (128)Network is unreachable: connect to listener [Wed Oct 26 12:55:30 2005] [notice] Digest: generating secret for digest authentication ... [Wed Oct 26 12:55:30 2005] [notice] Digest: done [Wed Oct 26 12:55:31 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations

# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 12:55:30 Rereading configuration. ] [ Oct 26 12:55:30 Executing refresh method ("/lib/svc/method/http-apache2 refresh") ] [ Oct 26 12:55:30 Method "refresh" exited with status 0 ]
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

httpd without ignore_error

# kill -9 29999 <-- started by root # tail -f /var/svc/log/network-http:apache2.log [ Oct 26 13:13:02 Stopping because process received fatal signal from outside the service. ] [ Oct 26 13:13:02 Executing stop method ("/lib/svc/method/http-apache2 stop") ] httpd (pid 29999?) not running [ Oct 26 13:13:02 Method "stop" exited with status 0 ] [ Oct 26 13:14:02 Method or service exit timed out. Killing contract 720 ] # svcs apache2 STATE maintenance STIME FMRI 13:14:02 svc:/network/http:apache2
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Clearing and restarting

# svcadm clear apache2 # tail -f /var/svc/log/network-http:apache2.log [ Oct 26 13:16:04 Leaving maintenance because clear requested. ] [ Oct 26 13:16:04 Enabled. ] [ Oct 26 13:16:04 Executing start method ("/lib/svc/method/http-apache2 start") ] [ Oct 26 13:16:05 Method "start" exited with status 0 ]

What would happen if you kill any httpd processes started by webservd, now that you no longer have the ignore_error property?
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/apache2/logs/error_log [Wed Oct 26 13:24:23 2005] [notice] caught SIGTERM, shutting down [Wed Oct 26 13:24:25 2005] [notice] Digest: generating secret for digest authentication ... [Wed Oct 26 13:24:25 2005] [notice] Digest: done [Wed Oct 26 13:24:26 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations

SMF stops and restarts httpd:apache because svc.startd will interpret any externally-generated signal as a failure of the service, and will execute the stop and start method to try and recover.
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Logs
# tail -f /var/svc/log/network-http:apache2.log [ Oct 26 13:24:23 Stopping because process received fatal signal from outside the service. ] [ Oct 26 13:24:23 Executing stop method ("/lib/svc/method/http-apache2 stop") ] [ Oct 26 13:24:23 Method "stop" exited with status 0 ] [ Oct 26 13:24:24 Executing start method ("/lib/svc/method/http-apache2 start") ] [ Oct 26 13:24:25 Method "start" exited with status 0 ]

What's different though? Why isn't it in maintenance mode?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Portable SMF
Perhaps you have a large number of servers, or even zones, on which you'd like to imprint an identical set of services. Well, this is not easy. This is provisioning, and SMF is a service management tool. You can't just pack up the database, or the XML dump generated by svccfg archive, or all the manifests ... Why not? What about the methods, the config files, the binaries, everything else that makes up the apps?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Duplicating service properties

If, however, you have an application already installed, and you would like to transfer service properties (such as dependencies, the exec method, anything that might be defined by the manifest), you can do that by exporting and then importing a manifest.
# svccfg export svc:/network/http > apache2.export

Move apache2.export to another system.

# svccfg import apache2.export

Note: the argument to svccfg export is a service, not an instance, although the XML file will contain definition of the instance(s).
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Duplicating service states

You can get a profile, or an XML file that describes the state of all services by using svccfg extract. This XML file can be used to duplicate the on|off state of services from system to system. Use svccfg apply.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Profiles
A profile is an XML description of all service instance states on a system. Profiles are in /var/svc/profile. Profiles are applied in order: generic, platform, site Never modify existing profiles, use site.xml for local customization. An example of something that may create a use a profile is the Solaris Security Toolkit 4.2 (aka JASS).

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Part V: SMF Tools and Community

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Webmin

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

About Webmin
A web-based interface for UNIX system administration It comes with Solaris 10, or get it at http://www.webmin.com Version 1.190 an later supports SMF, even has a feature to create a new manifest.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SMF in Webmin

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Create new service

This button will walk you through creating a new service manifest. You don't need to know XML, but you do need to understand what goes into a manifest. Will ask you a series of questions such as:
> > > > >

Name of service What are the dependencies, Which is the delegated restarter Service properties Man pages

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

GUI Tools
With Webmin you can monitor and manage SMF services through a browser. A GUI interface to SMF is being developed. Watch for a SunMC SMF module due near the end of this year.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

More tools
Use xmllint to check for errors in the manifest file without importing the manifest. svccfg validate does something similar, without changing the repository. Jedit is a free text editor with XML with syntax checker and validator. Comes with Solaris 10 (/usr/demo/jds/bin/jedit) or get it from http://www.jedit.org. Morphon Technologies has a good and free XML editor, but sadly no longer in development, get it from http://www.morphon.com.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SFM community on OpenSolaris

http://www.opensolaris.org/os/community/smf/ Submit manifests or methods you've written here: http://www.opensolaris.org/os/community/smf/manifests/ greenline-interest@sun.com migrating to smf-discuss@opensolaris.org Subscribe here: http://www.opensolaris.org/os/community/smf/discussions Read it here: http://www.opensolaris.org/jive/forum.jspa?forumID=24 File bugs here: http://bugs.opensolaris.org/
Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Resources
Internal > Greenline project page: http://greenline.eng > greenline-interest alias. External > BigAdmin: http://www.sun.com/bigadmin > Especially the Service Developer Intro at http://www.sun.com/bigadmin/content/selfheal/sdev_intro.html > SMF community: http://opensolaris.org/os/community/smf/ > SMF mailing list: smf-discuss@opensolaris.org Blogs > Liane Praza's blog: http://blogs.sun.com/roller/page/lianep > Stephen Hahn's blog: http://blogs.sun.com/sch

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Review

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Questions?

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

And lastly ...

Thank you!

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

References
Presentations L. Praza. smf(5): Service Management Facility. R. Manus, G. Hiregoudar. Solaris Management Framework Workshop.

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

Changes
Last modified: 03/10/2005

Sun Proprietary/Use by Permission Only

SOLARIS 10 DEEP DIVE: SMF

SOLARIS 10 DEEP DIVE: SMF

christine.tran@sun.com US Solaris Adoption Practice Sun Microsystems

Sun Proprietary/Use by Permission Only