--- /dev/null
+SAS Layer
+---------
+
+The SAS Layer is a management infrastructure which manages
+SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The
+layout is as follows: while SCSI Core is concerned with
+SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
+phy/OOB/link management, the SAS layer is concerned with:
+
+ * SAS Phy/Port/HA event management (LLDD generates,
+ SAS Layer processes),
+ * SAS Port management (creation/destruction),
+ * SAS Domain discovery and revalidation,
+ * SAS Domain device management,
+ * SCSI Host registration/unregistration,
+ * Device registration with SCSI Core (SAS) or libata
+ (SATA), and
+ * Expander management and exporting expander control
+ to user space.
+
+A SAS LLDD is a PCI device driver. It is concerned with
+phy/OOB management, and vendor specific tasks and generates
+events to the SAS layer.
+
+The SAS Layer does most SAS tasks as outlined in the SAS 1.1
+spec.
+
+The sas_ha_struct describes the SAS LLDD to the SAS layer.
+Most of it is used by the SAS Layer but a few fields need to
+be initialized by the LLDDs.
+
+After initializing your hardware, from the probe() function
+you call sas_register_ha(). It will register your LLDD with
+the SCSI subsystem, creating a SCSI host and it will
+register your SAS driver with the sysfs SAS tree it creates.
+It will then return. Then you enable your phys to actually
+start OOB (at which point your driver will start calling the
+notify_* event callbacks).
+
+Structure descriptions:
+
+struct sas_phy --------------------
+Normally this is statically embedded to your driver's
+phy structure:
+ struct my_phy {
+ blah;
+ struct sas_phy sas_phy;
+ bleh;
+ };
+And then all the phys are an array of my_phy in your HA
+struct (shown below).
+
+Then as you go along and initialize your phys you also
+initialize the sas_phy struct, along with your own
+phy structure.
+
+In general, the phys are managed by the LLDD and the ports
+are managed by the SAS layer. So the phys are initialized
+and updated by the LLDD and the ports are initialized and
+updated by the SAS layer.
+
+There is a scheme where the LLDD can RW certain fields,
+and the SAS layer can only read such ones, and vice versa.
+The idea is to avoid unnecessary locking.
+
+enabled -- must be set (0/1)
+id -- must be set [0,MAX_PHYS)
+class, proto, type, role, oob_mode, linkrate -- must be set
+oob_mode -- you set this when OOB has finished and then notify
+the SAS Layer.
+
+sas_addr -- this normally points to an array holding the sas
+address of the phy, possibly somewhere in your my_phy
+struct.
+
+attached_sas_addr -- set this when you (LLDD) receive an
+IDENTIFY frame or a FIS frame, _before_ notifying the SAS
+layer. The idea is that sometimes the LLDD may want to fake
+or provide a different SAS address on that phy/port and this
+allows it to do this. At best you should copy the sas
+address from the IDENTIFY frame or maybe generate a SAS
+address for SATA directly attached devices. The Discover
+process may later change this.
+
+frame_rcvd -- this is where you copy the IDENTIFY/FIS frame
+when you get it; you lock, copy, set frame_rcvd_size and
+unlock the lock, and then call the event. It is a pointer
+since there's no way to know your hw frame size _exactly_,
+so you define the actual array in your phy struct and let
+this pointer point to it. You copy the frame from your
+DMAable memory to that area holding the lock.
+
+sas_prim -- this is where primitives go when they're
+received. See sas.h. Grab the lock, set the primitive,
+release the lock, notify.
+
+port -- this points to the sas_port if the phy belongs
+to a port -- the LLDD only reads this. It points to the
+sas_port this phy is part of. Set by the SAS Layer.
+
+ha -- may be set; the SAS layer sets it anyway.
+
+lldd_phy -- you should set this to point to your phy so you
+can find your way around faster when the SAS layer calls one
+of your callbacks and passes you a phy. If the sas_phy is
+embedded you can also use container_of -- whatever you
+prefer.
+
+
+struct sas_port --------------------
+The LLDD doesn't set any fields of this struct -- it only
+reads them. They should be self explanatory.
+
+phy_mask is 32 bit, this should be enough for now, as I
+haven't heard of a HA having more than 8 phys.
+
+lldd_port -- I haven't found use for that -- maybe other
+LLDD who wish to have internal port representation can make
+use of this.
+
+
+struct sas_ha_struct --------------------
+It normally is statically declared in your own LLDD
+structure describing your adapter:
+struct my_sas_ha {
+ blah;
+ struct sas_ha_struct sas_ha;
+ struct my_phy phys[MAX_PHYS];
+ struct sas_port sas_ports[MAX_PHYS]; /* (1) */
+ bleh;
+};
+
+(1) If your LLDD doesn't have its own port representation.
+
+What needs to be initialized (sample function given below).
+
+pcidev
+sas_addr -- since the SAS layer doesn't want to mess with
+ memory allocation, etc, this points to statically
+ allocated array somewhere (say in your host adapter
+ structure) and holds the SAS address of the host
+ adapter as given by you or the manufacturer, etc.
+sas_port
+sas_phy -- an array of pointers to structures. (see
+ note above on sas_addr).
+ These must be set. See more notes below.
+num_phys -- the number of phys present in the sas_phy array,
+ and the number of ports present in the sas_port
+ array. There can be a maximum num_phys ports (one per
+ port) so we drop the num_ports, and only use
+ num_phys.
+
+The event interface:
+
+ /* LLDD calls these to notify the class of an event. */
+ void (*notify_ha_event)(struct sas_ha_struct *, enum ha_event);
+ void (*notify_port_event)(struct sas_phy *, enum port_event);
+ void (*notify_phy_event)(struct sas_phy *, enum phy_event);
+
+When sas_register_ha() returns, those are set and can be
+called by the LLDD to notify the SAS layer of such events
+the SAS layer.
+
+The port notification:
+
+ /* The class calls these to notify the LLDD of an event. */
+ void (*lldd_port_formed)(struct sas_phy *);
+ void (*lldd_port_deformed)(struct sas_phy *);
+
+If the LLDD wants notification when a port has been formed
+or deformed it sets those to a function satisfying the type.
+
+A SAS LLDD should also implement at least one of the Task
+Management Functions (TMFs) described in SAM:
+
+ /* Task Management Functions. Must be called from process context. */
+ int (*lldd_abort_task)(struct sas_task *);
+ int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
+ int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
+ int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
+ int (*lldd_I_T_nexus_reset)(struct domain_device *);
+ int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
+ int (*lldd_query_task)(struct sas_task *);
+
+For more information please read SAM from T10.org.
+
+Port and Adapter management:
+
+ /* Port and Adapter management */
+ int (*lldd_clear_nexus_port)(struct sas_port *);
+ int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
+
+A SAS LLDD should implement at least one of those.
+
+Phy management:
+
+ /* Phy management */
+ int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
+
+lldd_ha -- set this to point to your HA struct. You can also
+use container_of if you embedded it as shown above.
+
+A sample initialization and registration function
+can look like this (called last thing from probe())
+*but* before you enable the phys to do OOB:
+
+static int register_sas_ha(struct my_sas_ha *my_ha)
+{
+ int i;
+ static struct sas_phy *sas_phys[MAX_PHYS];
+ static struct sas_port *sas_ports[MAX_PHYS];
+
+ my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
+
+ for (i = 0; i < MAX_PHYS; i++) {
+ sas_phys[i] = &my_ha->phys[i].sas_phy;
+ sas_ports[i] = &my_ha->sas_ports[i];
+ }
+
+ my_ha->sas_ha.sas_phy = sas_phys;
+ my_ha->sas_ha.sas_port = sas_ports;
+ my_ha->sas_ha.num_phys = MAX_PHYS;
+
+ my_ha->sas_ha.lldd_port_formed = my_port_formed;
+
+ my_ha->sas_ha.lldd_dev_found = my_dev_found;
+ my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
+
+ my_ha->sas_ha.lldd_max_execute_num = lldd_max_execute_num; (1)
+
+ my_ha->sas_ha.lldd_queue_size = ha_can_queue;
+ my_ha->sas_ha.lldd_execute_task = my_execute_task;
+
+ my_ha->sas_ha.lldd_abort_task = my_abort_task;
+ my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
+ my_ha->sas_ha.lldd_clear_aca = my_clear_aca;
+ my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
+ my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
+ my_ha->sas_ha.lldd_lu_reset = my_lu_reset;
+ my_ha->sas_ha.lldd_query_task = my_query_task;
+
+ my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
+ my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
+
+ my_ha->sas_ha.lldd_control_phy = my_control_phy;
+
+ return sas_register_ha(&my_ha->sas_ha);
+}
+
+(1) This is normally a LLDD parameter, something of the
+lines of a task collector. What it tells the SAS Layer is
+whether the SAS layer should run in Direct Mode (default:
+value 0 or 1) or Task Collector Mode (value greater than 1).
+
+In Direct Mode, the SAS Layer calls Execute Task as soon as
+it has a command to send to the SDS, _and_ this is a single
+command, i.e. not linked.
+
+Some hardware (e.g. aic94xx) has the capability to DMA more
+than one task at a time (interrupt) from host memory. Task
+Collector Mode is an optional feature for HAs which support
+this in their hardware. (Again, it is completely optional
+even if your hardware supports it.)
+
+In Task Collector Mode, the SAS Layer would do _natural_
+coalescing of tasks and at the appropriate moment it would
+call your driver to DMA more than one task in a single HA
+interrupt. DMBS may want to use this by insmod/modprobe
+setting the lldd_max_execute_num to something greater than
+1.
+
+(2) SAS 1.1 does not define I_T Nexus Reset TMF.
+
+Events
+------
+
+Events are _the only way_ a SAS LLDD notifies the SAS layer
+of anything. There is no other method or way a LLDD to tell
+the SAS layer of anything happening internally or in the SAS
+domain.
+
+Phy events:
+ PHYE_LOSS_OF_SIGNAL, (C)
+ PHYE_OOB_DONE,
+ PHYE_OOB_ERROR, (C)
+ PHYE_SPINUP_HOLD.
+
+Port events, passed on a _phy_:
+ PORTE_BYTES_DMAED, (M)
+ PORTE_BROADCAST_RCVD, (E)
+ PORTE_LINK_RESET_ERR, (C)
+ PORTE_TIMER_EVENT, (C)
+ PORTE_HARD_RESET.
+
+Host Adapter event:
+ HAE_RESET
+
+A SAS LLDD should be able to generate
+ - at least one event from group C (choice),
+ - events marked M (mandatory) are mandatory (only one),
+ - events marked E (expander) if it wants the SAS layer
+ to handle domain revalidation (only one such).
+ - Unmarked events are optional.
+
+Meaning:
+
+HAE_RESET -- when your HA got internal error and was reset.
+
+PORTE_BYTES_DMAED -- on receiving an IDENTIFY/FIS frame
+PORTE_BROADCAST_RCVD -- on receiving a primitive
+PORTE_LINK_RESET_ERR -- timer expired, loss of signal, loss
+of DWS, etc. (*)
+PORTE_TIMER_EVENT -- DWS reset timeout timer expired (*)
+PORTE_HARD_RESET -- Hard Reset primitive received.
+
+PHYE_LOSS_OF_SIGNAL -- the device is gone (*)
+PHYE_OOB_DONE -- OOB went fine and oob_mode is valid
+PHYE_OOB_ERROR -- Error while doing OOB, the device probably
+got disconnected. (*)
+PHYE_SPINUP_HOLD -- SATA is present, COMWAKE not sent.
+
+(*) should set/clear the appropriate fields in the phy,
+ or alternatively call the inlined sas_phy_disconnected()
+ which is just a helper, from their tasklet.
+
+The Execute Command SCSI RPC:
+
+ int (*lldd_execute_task)(struct sas_task *, int num,
+ unsigned long gfp_flags);
+
+Used to queue a task to the SAS LLDD. @task is the tasks to
+be executed. @num should be the number of tasks being
+queued at this function call (they are linked listed via
+task::list), @gfp_mask should be the gfp_mask defining the
+context of the caller.
+
+This function should implement the Execute Command SCSI RPC,
+or if you're sending a SCSI Task as linked commands, you
+should also use this function.
+
+That is, when lldd_execute_task() is called, the command(s)
+go out on the transport *immediately*. There is *no*
+queuing of any sort and at any level in a SAS LLDD.
+
+The use of task::list is two-fold, one for linked commands,
+the other discussed below.
+
+It is possible to queue up more than one task at a time, by
+initializing the list element of struct sas_task, and
+passing the number of tasks enlisted in this manner in num.
+
+Returns: -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
+ 0, the task(s) were queued.
+
+If you want to pass num > 1, then either
+A) you're the only caller of this function and keep track
+ of what you've queued to the LLDD, or
+B) you know what you're doing and have a strategy of
+ retrying.
+
+As opposed to queuing one task at a time (function call),
+batch queuing of tasks, by having num > 1, greatly
+simplifies LLDD code, sequencer code, and _hardware design_,
+and has some performance advantages in certain situations
+(DBMS).
+
+The LLDD advertises if it can take more than one command at
+a time at lldd_execute_task(), by setting the
+lldd_max_execute_num parameter (controlled by "collector"
+module parameter in aic94xx SAS LLDD).
+
+You should leave this to the default 1, unless you know what
+you're doing.
+
+This is a function of the LLDD, to which the SAS layer can
+cater to.
+
+int lldd_queue_size
+ The host adapter's queue size. This is the maximum
+number of commands the lldd can have pending to domain
+devices on behalf of all upper layers submitting through
+lldd_execute_task().
+
+You really want to set this to something (much) larger than
+1.
+
+This _really_ has absolutely nothing to do with queuing.
+There is no queuing in SAS LLDDs.
+
+struct sas_task {
+ dev -- the device this task is destined to
+ list -- must be initialized (INIT_LIST_HEAD)
+ task_proto -- _one_ of enum sas_proto
+ scatter -- pointer to scatter gather list array
+ num_scatter -- number of elements in scatter
+ total_xfer_len -- total number of bytes expected to be transferred
+ data_dir -- PCI_DMA_...
+ task_done -- callback when the task has finished execution
+};
+
+When an external entity, entity other than the LLDD or the
+SAS Layer, wants to work with a struct domain_device, it
+_must_ call kobject_get() when getting a handle on the
+device and kobject_put() when it is done with the device.
+
+This does two things:
+ A) implements proper kfree() for the device;
+ B) increments/decrements the kref for all players:
+ domain_device
+ all domain_device's ... (if past an expander)
+ port
+ host adapter
+ pci device
+ and up the ladder, etc.
+
+DISCOVERY
+---------
+
+The sysfs tree has the following purposes:
+ a) It shows you the physical layout of the SAS domain at
+ the current time, i.e. how the domain looks in the
+ physical world right now.
+ b) Shows some device parameters _at_discovery_time_.
+
+This is a link to the tree(1) program, very useful in
+viewing the SAS domain:
+ftp://mama.indstate.edu/linux/tree/
+I expect user space applications to actually create a
+graphical interface of this.
+
+That is, the sysfs domain tree doesn't show or keep state if
+you e.g., change the meaning of the READY LED MEANING
+setting, but it does show you the current connection status
+of the domain device.
+
+Keeping internal device state changes is responsibility of
+upper layers (Command set drivers) and user space.
+
+When a device or devices are unplugged from the domain, this
+is reflected in the sysfs tree immediately, and the device(s)
+removed from the system.
+
+The structure domain_device describes any device in the SAS
+domain. It is completely managed by the SAS layer. A task
+points to a domain device, this is how the SAS LLDD knows
+where to send the task(s) to. A SAS LLDD only reads the
+contents of the domain_device structure, but it never creates
+or destroys one.
+
+Expander management from User Space
+-----------------------------------
+
+In each expander directory in sysfs, there is a file called
+"smp_portal". It is a binary sysfs attribute file, which
+implements an SMP portal (Note: this is *NOT* an SMP port),
+to which user space applications can send SMP requests and
+receive SMP responses.
+
+Functionality is deceptively simple:
+
+1. Build the SMP frame you want to send. The format and layout
+ is described in the SAS spec. Leave the CRC field equal 0.
+open(2)
+2. Open the expander's SMP portal sysfs file in RW mode.
+write(2)
+3. Write the frame you built in 1.
+read(2)
+4. Read the amount of data you expect to receive for the frame you built.
+ If you receive different amount of data you expected to receive,
+ then there was some kind of error.
+close(2)
+All this process is shown in detail in the function do_smp_func()
+and its callers, in the file "expander_conf.c".
+
+The kernel functionality is implemented in the file
+"sas_expander.c".
+
+The program "expander_conf.c" implements this. It takes one
+argument, the sysfs file name of the SMP portal to the
+expander, and gives expander information, including routing
+tables.
+
+The SMP portal gives you complete control of the expander,
+so please be careful.
git://git.onelab.eu / linux-2.6.git / blobdiff