.\" $Id: mm_data.5,v 6.8 2004/06/05 22:35:46 btompkin Exp $ Copyright (c) 2004, LEGATO Software, a division of EMC"
.\" Copyright (c) 2000 Legato Systems, Incorporated.
.\" All rights reserved.
.TH MM_DATA 5 "Aug 23, 06" "StorEdge EBS 7.3.2"
.SH NAME
mm_data \- Sun StorEdge EBS media multiplexor data (tape and disk) format
.SH DESCRIPTION
This documents the data format that the
Sun StorEdge EBS
media multiplexor daemon,
.BR nsrmmd (8),
writes to long term storage media such as tapes and optical disks.
See
.BR nsr_device (5)
and
.BR nsrmm (8)
for a discussion of supported device families and types.
The format described here applies to any fixed record device, such
as raw disks, or fixed record tape devices with file marks.
Sun StorEdge EBS uses the \fIeXternal Data Representation\fP (XDR) standard to write
media which can be interchanged among a wide variety of machines.
Only the mechanism used to multiplex save set streams onto the storage
media is described here; the formats of save set streams depend on
the type of Sun StorEdge EBS client, and are described in
.BR nsr_data (5).
.LP
A
.I volume
is one physical piece of media such as a tape reel or disk cartridge.
A tape volume is made up of multiple media
.IR file s,
and each media file may contain several media
.IR record s.
These media files and records should not be confused with a client's
(for example UNIX or DOS) user files or records;
the two do not necessarily correspond.
For example, a given media file or even a single media record may contain
many small client user files. On the other hand, a single large client file
may be split across several media files, and even across several volumes.
Media files do not span volume boundaries.
Save sets may span media files and even volumes.
.LP
On most tapes, media files
can be skipped very quickly by the device's hardware or
associated device driver software,
and the hardware can detect when an end of a
file has been reached.
On some tapes, records can also be quickly skipped forward.
Otherwise, access to the media is sequential.
.LP
Media records are described by the
.B mrecord
structure.
Label records are fixed in size,
.BR MINMRECSIZE
bytes. Other records can potentially be a larger size that must be
some constant for the rest of the volume.
Sun StorEdge EBS always writes, reads and skips data in
units of full-sized media records.
Each
.B mrecord
contains zero or more
.BR mchunks
These 
.B mrecords
are used for storing one or more
client save sessions or 
used by Sun StorEdge EBS for synchronization and labelling.
The
.SM XDR
format of a media file's
.BR mrecord "s and"
.BR mchunk s
are as follows:
.\".RS
.LP
.nf
\fR\s10
.ta 4n 32n
const MINMRECSIZE = 32768;		/* minimum media record size */
const MMAXCHK = 2048;		/* maximum number of chunks in record */
const MHNDLEN = 120;		/* private area length for handlers */

enum mrec_version {		/* mrecord version */
	MREC_VER5 = 0,		/* older format mrecord */
	MREC_VER6 = 6		/* current format mrecord */
};

/*
 * For media record format version 5, the data types lgui_t, lg_off64_t,
 * and lg_time64_t are defined as:
 */
typedef struct lgui_t unsigned long;
typedef struct lg_off64_t unsigned long;
typedef struct lg_time64_t unsigned long;

/*
 * For media record format version 6, the data types lgui_t, lg_off64_t,
 * and lg_time64_t are defined as:
 */
typedef struct lgui_t {		/* XDR encoded Unique Id. */
	char _bytes[20];
} lgui_t;
typedef struct lg_off64_t unsigned long long;
typedef struct lg_time64_t unsigned long long;


typedef lgui_t ssid_t;			/* save set id */
typedef lgui_t volid_t;			/* key for the volume database */

struct  mchunk {
	ssid_t mc_ssid;	/* owning save set id */
	lg_off64_t mc_low;	/* 1st byte, relative to save stream */
	opaque mc_data<MINMRECSIZE>;	/* chunk's data */
};

struct  mrecord {
	opaque mr_handler[MHNDLEN];	/* private to media handler */
	mrec_version mr_version;	/* Media record version number */
	u_long mr_orec;		/* record size */
	volid_t mr_volid;	/* encompassing volume's id */
	u_long mr_fn;	/* encompassing file number */
	u_long mr_rn;	/* record number within the file */
	u_long mr_len;	/* record byte length */
	mchunk mr_chunk<MMAXCHK>;	/* chunks of save streams */
};
\fP\s0
.fi
.LP
.RE
The first field of an
.BR mrecord ,
.BR mr_handler ,
is reserved for media-specific data (currently it is not used
by any implementation).
The
.B mr_version
field is the version number of the media record format.
The size of the rest of the fields in the media record depends on
the version number.
The
.B mr_orec
field is the size of the current record.
A media record's header fields,
.BR mr_volid ", " mr_fn ", "
and
.BR mr_rn ,
are used to check the tape position and the data read from the record.
The file numbers and record numbers start at zero and increment sequentially.
The record number is reset each time the file number is incremented.
On disks, file numbers are always zero.
The
.B mr_len
field is the actual number of valid bytes in this
record, as opposed to the size of the device's read or write request.
.LP
If file or record skipping is unreliable, Sun StorEdge EBS can still recover
from isolated errors, at worst by rewinding and reading the tape from the start.
If a volume can be physically unmounted or
mounted without notice to the media management daemon,
then the volume identifier in each record provides a quick
way of verifying when this happens, without the need for a full rewind and
reading of the label in most cases.
.LP
The
.BR mchunk s
within an
.B mrecord
contain client data from one or more save sessions.
The
.BR mc_ssid " and "
.B mc_low
values are used to reconstruct the save streams from the
chunks within the records.
The
.B mc_data
field holds the actual data of each chunk.
For a given save set,
.B mc_low
plus the length of
.B mc_data
should equal the following chunk's value for
.B mc_low.
Save sets may by intermingled arbitrarily within media records.
.LP
The first chunk of the first
record of the first media file
on the volume encapsulates the
.B volume
.B label
information;
for some media, the second chunk contains
additional volume information, for example,
the media pool the volume belongs to:
Subsequent data in the first file is reserved for future
expansion.
The label may be duplicated in a second file for redundancy, in case the
first copy of the label gets accidentally overwritten.
The formats of the
volume label and additional label information are described by the following
.SM XDR
data structures:
.\".RS
.LP
.nf
\fR\s10
const MVOLMAGIC = 0x070460;	/* volume magic number */
const NSR_LENGTH = 64;	/* length of several strings */
const RAP_MAXNAMELEN = 64;	/* maximum length of attribute name */

struct mvollabel {
	u_long  mvl_magic;	/* medium volume verification number */
	lg_time64_t mvl_createtime; /* time at which volume labeled */
	lg_time64_t mvl_expiretime; /* time for volume to expire */
	u_long  mvl_recsize;	/* expected size of mrecords */
	volid_t mvl_volid;	/* medium volume id */
	string  mvl_volname<NSR_LENGTH>;	/* medium volume name */
};

struct vallist {
	vallist *next;
	string value<>;	/* attribute value */
};

struct attrlist {
	attrlist *next;
	string name<RAP_MAXNAMELEN>;	/* attribute name */
	vallist *values;	/* attribute values */
};

/*
 * Additional information may includes the following attributes
 * (listed by the name they are stored with):
 * "volume pool" : the media pool
 */
struct mvolinfo {
	struct attrlist *mvi_attributes;	/* any other information */
};
\fP\s0
.fi
.LP
.RE
The
.B mvl_magic
field must be equal to
.B MVOLMAGIC
in order for the chunk to represent a valid volume label.
If the volume label changes in the future, the new format will have another
``magic'' number, but the format described here must still be allowed.
The
.B mvl_volid
is an internal identifier assigned and managed by the media manager.
The
.B mvl_volname
is the volume name that is assigned when the media is first labeled.
The time fields are in UST format \- the number of seconds elapsed since
00:00 GMT, January 1, 1970.
The
.B mvl_recsize
is the size of all subsequent media records found on the tape.
.LP
The
.B mvp_pool
is the pool name that is assigned when the media is first labeled.
Different media pools allow administrators to segregate their data onto
sets of volumes.
Media cannot be reassigned from one media pool to another.
Pool names are a maximum of
.B NSR_LENGTH
characters long.
.LP
Synchronization marks, called
.BR schunks ,
are also written periodically to the media for each save set.
Synchronization chunks are used by
.BR scanner (8)
when verifying or extracting directly from a volume.
They are also used by
.B nsrmmd
when trying to recover from media errors during file recovery.
The following XDR data structure describes a synchronization chunk:
.\".RS
.LP
.nf
\fR\s10
.ta 4n 35n
typedef lgui_t clientid_t;

struct	ssclone_t {
	lg_time64_t ssc_cloneid;	/* unique UST stamp wrt ss_ssid */
	u_long ssc_flag;	/* lots of status buried here	*/
	u_long ssc_frag;	/* not used, always 0 */
};

/*
 * Synchronization chunk of the newer MREC_VER6 media format.
 */
struct schunk {
	u_long ssi_gen;	/* Not used. */
	ssid_t ssi_ssid;	/* save set identifier */
	ssid_t ssi_prev;	/* non-zero iff continuation */
	u_long ssi_level;	/* backup level	*/
	lg_time64_t ssi_time;	/* save time on client */
	lg_time64_t ssi_create;	/* creation time on server */
	lg_time64_t ssi_insert;	/* insertion time on server */
	lg_time64_t ssi_complete;	/* completion time on server */
	clientid_t ssi_clientid;	/* client name identifier */
	u_long ssi_flags;	/* more details about this ss */
	string ssi_host<>;	/* client name - save set owner */
	string ssi_name<>;	/* symbolic name, for example "/usr" */
	uint64_t ssi_size;	/* actual number of bytes saved */
	uint64_t ssi_nfiles;	/* number of client files saved */
	u_long ssi_browse;	/* browse time offset */
	u_long ssi_recycle;	/* recycle time offset */
	struct attrlist *ssi_al;	/* generic RAP attribute list */
	ssclone_t ssi_clones<>;	/* information about this clone */
};

/*
 * Synchronization chunk of the older MREC_VER5 media format.
 */
struct old_schunk {
	opaque ssi_host[NSR_LENGTH];	/* save set host */
	opaque ssi_name[NSR_LENGTH];	/* symbolic name */
	u_long ssi_time;	/* save time */
	u_long ssi_expiry;	/* expiration date */
	u_long ssi_size;	/* actual size saved */
	u_long ssi_nfiles;	/* number of files */
	ssid_t ssi_ssid;	/* ssid for this save set */
	u_long ssi_flag;	/* various flags, see below */
	u_long ssi_info;	/* volid or ssid, see below */
};
.ta 24n 35n

#define SSI_START	1	/* start of a save set */
#define SSI_SYNC	2	/* synchronization point */
#define SSI_CONT	3	/* continued from another volume */
#define SSI_END	4	/* end of this save set */
#define SSI_SSMASK	0x0000000f	/* save set sync chunk type */
#define SSI_LBIAS	0x10000000	/* the level is included in the flags */
#define SSI_LMASK	0xff000000	/* mask to cover bits for level */
#define SSI_LSHIFT	24	/* shift amount for the level */
#define SSI_INCOMPLETE	0x00010000	/* not finished (aborted) */
#define SSI_CONTINUED	0x00800000	/* continued save set series */
.ft R
.fi
.LP
.RE
The
.B ssi_ssid
is the save set identifier of this save set.
The
.B ssi_time
field contains the create time of the save set in UST based
on the client's clock.
The
.B ssi_create
field contains the create time of the save set in UST based
on the server's clock.
The
.B ssi_insert
field contains the time the save set was inserted into the media database
in UST based on the server's clock.
The
.B ssi_complete
field contains the completion time of the save set in UST based
on the server's clock.
The
.BR ssi_clientid " and " ssi_host
are the client identifier and name of the index which contains this save set.
Traditionally this is the client identifier and name of the client where the
save set originated.
The
.B ssi_name
is the save set name to be presented to the user.
These are both null-terminated strings, even though the fields are fixed length
in the older version media records.
The
.BR ssi_size " and " ssi_nfiles
are the number of bytes and number of files saved so far for this save set.
The
.B ssi_browse
is the time offset in seconds from the save set insertion time to the time
this save set is no longer browsable.
The
.B ssi_recycle
is the time offset in seconds from the save set insertion time to the time
this save set becomes recyclable.
The
.BR ssi_al
is the generic save set attribute.
.LP
The
.B ssi_flag
indicates the type of this synchronization chunk
and other information about the save set.
In the older version synchronization chunk,
this field also contains the level of this save set.
There are four basic types of synchronization marks
that can be found from examining
.BR "ssi_flag & SSI_SSMASK" .
.B SSI_START
is used to mark the beginning of a save set.
.B SSI_SYNC
marks a periodic synchronization point and is only
written at an exact file boundary in the save set.
.B SSI_CONT
indicates that this is the continuation of a save
set that started on a different volume.
When 
.B "ssi_flag & SSI_SSMASK"
is
.BR SSI_CONT ,
.BR ssi_prev " or " ssi_info
contains the volume identifier for the save set's preceding volume.
These synchronization chunks are used when a save set spans a volume boundary.
.B SSI_END
marks the end of a save set.
.LP
On the new version of synchronization chunk, the
.B ssi_level
field contains the save set backup level.
On the older version of synchronization chunk. Should the
.B SSI_LBIAS
bit be set then
.BR ssi_flag " &"
.B SSI_LMASK
shifted to the right by
.B the value of SSI_LSHIFT
specifies the level of the save set.
The
.B SSI_INCOMPLETE
bit indicates that this save set did not finish properly.
This could be caused by a user interrupting an in progress save.
.LP
The
.B SSI_CONTINUED
bit indicates that this save set is logically
continued to or from another save set.
These continued save sets are used to handle very large save sets.
If the
.B SSI_CONTINUED
bit is set and
.B "ssi_flag & SSI_SSMASK"
is
.BR SSI_START ,
then
.BR ssi_prev " or " ssi_info
gives the previous save set id that this save set was continued from.
If the
.B SSI_CONTINUED
bit is set and
.B "ssi_flag & SSI_SSMASK"
is
.BR SSI_END ,
then
.BR ssi_prev " or " ssi_info
gives the next save set id that this save set is continued to.
.LP
The
.B ssi_expiry
field is the expiration date, in UST, for this save set.
This field is zero if an explicit save set expiration
time was not specified when the save set was created.
This field no longer exists in the new synchronization chunk.
.SH "SEE ALSO"
.na
.BR nsr_device (5),
.BR nsr_data (5),
.BR nsrmm (8),
.BR nsrmmd (8),
.BR nsrmmdbd (8),
.BR nsr (8),
.BR scanner (8).
.TP 20
RFC 1014 XDR: External Data Representation Specification