|
Access
to Mass Storage at FNAL
(dCache and enstore)
At Fermilab
there are two cooperating systems: enstore and dCache.
Enstore is a network attached tape system allowing sequential access.
dCache is a disk farm allowing for random access. All access to the
mass storage should be done through dCache.
As shown in the
diagram above, dCache serves as a front end to Enstore. All files
written into dCache will be automatically written to tape (enstore) and
if a file doesn't reside on disk in dCache the file will automatically
be staged from enstore (tape) to disk (dCache). Some features of the
enstore dCache combination are:
-
dCache makes a multi-terabyte server farm look like one coherent and
homogeneous storage system.
- Rate adaptation between the application and the tertiary storage
resources.
- Optimized usage of expensive tape robot systems and drives by
coordinated read and write requests.
- No explicit staging is necessary to access the data (but pre staging
possible and in some cases desirable).
- The data access method is unique independent of where the data
resides.
- High performance and fault tolerant transport protocol between
applications and data servers.
- Can be accessed directly from your application (e.g. ROOT: TDCacheFile
class).df
In addition the user can use resilient
dCache which makes the data disk space distributed among worker
nodes of the CMS computing farm available. It uses pnfs to provide a
single name space structure. Resilient
dCache assumes that individual worker nodes are unreliable and
can go down at
any time, therefore it makes replicas of data files on multiple farm
nodes. Files that are put into resilient dCache are not migrated to
tape, so no staging from tape is ever necessary.
Enstore
Enstore system
status and information
dCache
dCache project
information
Fermilab
USCMS dCache status page
dCache mailing
list: dcache@listserv.fnal.gov
Both
systems use
the same name space and can be accessed via the pnfs pseudo file
system. For example:
ls
/pnfs/cms
lists
the files
in the cms top directory. Note: The /pnfs/cms
directory has been declared read only. To create new directories and to
copy new files into pnfs use /pnfs/cms/WAX/2/ as the
root of your path. The following command create a new directory:
mkdir
/pnfs/cms/WAX/2/<username>
To
put files
into resilient dCache users do
the following:
mkdir
/pnfs/cms/WAX/resilient/<username>
Note!
pnfs might look like a
regular file system but it's not!
Restrict commands to:
mkdir, rmdir,
rm, dccp and srcmp
(don't
cd into /pnfs, don’t run find
commands, don't run "ls -r").
If you would like a list of files in pnfs see this page.
Handling
of many small files might
present problems to the
system therefore it is strongly recommended to tar collections of small
files before moving them into dCache!
For files over 2GB the ls
command will show 1k for the files size. To get the true file size use
the special dcache commands dcsize
dcls
and dcdu
To copy
files from dCache with dccp:
To copy the
file over to your local disk, do
dccp
/pnfs/cms/WAX/2/username/myfile.data $PWD
dCache can also be
accessed (read only) via a URL like syntax in case you don't have
access to the pnfs file system
dccp
dcap://cmsgridftp.fnal.gov:24125/pnfs/fnal.gov/usr/cms/WAX/2/username/myfile.data
$PWD
Note: /pnfs/cms maps to /pnfs.fnal.gov/usr/cms
on cmsgridftp.fnal.gov.
To copy files into dCache with srmcp:
NOTE: srmcp requires authentication. Please see dCache/SRM file transfer through
srmcp for instructions on getting your grid-certificate and
registering with the CMS VOMS so that you are authorized to copy files
into dCache with srmcp.
First make sure
you have Java 1.5.0 in your path at the beginning:
export
PATH=/usr/java/jdk1.5.0_10/bin/:$PATH
or
setenv PATH /usr/java/jdk1.5.0_10/bin/:$PATH
Then use srcmp to copy yhe file into dcache
srmcp
"file:///$PWD/myfile.dat" "srm://cmssrm.fnal.gov:8443/resilient/username/myfile.data"
or, to copy the
file from dDache to your local disk, do
srcmp "srm://cmssrm.fnal.gov:8443/resilient/username/myfile.data"
"file:///$PWD/myfile.dat"
Note: You must first create the
path on /pnfs/cms/WAX and make it group writable with chmod "775". The
SRM automatically add /pnfs/cms/WAX to the url path. For example, if
the path of a
file on /pnfs is /pnfs/cms/WAX/resilient/username/myfile.dat
it maps to the url srm://cmssrm.fnal.gov:8443/resilient/username/myfile.dat
Removing
/pnfs/cms/WAX from the path gives the path for the url.
Note:
LFNs beginning with /store
map to PFNs /pnfs/cms/WAX/11/store.
Accessing
files in dCache using a ROOT application:
One nice feature of
the dCache system is the fact that files can be written and read
directly with a ROOT application. Just replace TFile with
TDCacheFile. The code fragment below demonstrates the use of the
TDCacheFile class, it also shows both ways of accessing the files (pnfs
and URL).
#include
<TDCacheFile.h>
int testdcache()
{ TFile *hfile = new TDCacheFile
("dcap://cmsgridftp.fnal.gov:24125/pnfs/fnal.gov/usr/cms/wenzel/hsimple_dcache.root","READ","Demo
ROOT file with histograms",0);
hfile->ls();
hfile->Print();
hpx->Draw();
return 0;}
ROOT is used by
CMSSW therefore it is setup with the CMSSW runtime environment.
To set
this up: run scramv1
list CMSSW to list CMSSW versions which are available and their
location, change directories to the location of a version of CMSSW,
then
run
eval `scramv1
runtime -csh`
An example is shown
below:
scramv1 list CMSSW
cd /uscmst1/prod/sw/cmssw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_4_3
eval `scramv1 runtime -csh`
cd -
root
The commands can be
executed directly
root [0] .L
${ROOTSYS}/lib/libDCache.so
root [1] TFile *hfile = new TDCacheFile
("dcap://cmsgridftp.fnal.gov:24125/pnfs/fnal.gov/usr/cms/wenzel/hsimple_dcache.root","READ","Demo
ROOT file with histograms",0);
root [2] hfile->ls();
root [3] hfile->Print();
root [4] hpx->Draw();
For questions
concerning this page and dCache send email to: cms-dcache@fnal.gov
|
