Software and Computing Subproject

Access to Mass Storage at FNAL
(dCache and enstore)

At Fermilab there are two cooperating systems: enstore and dCache. Enstore is a network attached tape system allowing sequential access. dCache is a disk farm allowing for random access. All access to the mass storage should be done through dCache.

As shown in the diagram above, dCache serves as a front end to Enstore. All files written into dCache will be automatically written to tape (enstore) and if a file doesn't reside on disk in dCache the file will automatically be staged from enstore (tape) to disk (dCache). Some features of the enstore dCache combination are:

-dCache makes a multi-terabyte server farm look like one coherent and homogeneous storage system.
- Rate adaptation between the application and the tertiary storage resources.
- Optimized usage of expensive tape robot systems and drives by coordinated read and write requests.
- No explicit staging is necessary to access the data (but pre staging possible and in some cases desirable).
- The data access method is unique independent of where the data resides.
- High performance and fault tolerant transport protocol between applications and data servers.
- Can be accessed directly from your application (e.g. ROOT: TDCacheFile class).df

In addition the user can use resilient dCache which makes the data disk space distributed among worker nodes of the CMS computing farm available. It uses pnfs to provide a single name space structure. Resilient dCache assumes that individual worker nodes are unreliable and can go down at any time, therefore it makes replicas of data files on multiple farm nodes. Files that are put into resilient dCache are not migrated to tape, so no staging from tape is ever necessary.

Enstore

Enstore system status and information

dCache

dCache project information
Fermilab USCMS dCache status page

dCache mailing list: dcache@listserv.fnal.gov

Both systems use the same name space and can be accessed via the pnfs pseudo file system. For example:

ls /pnfs/cms

lists the files in the cms top directory. Note: The /pnfs/cms directory has been declared read only. To create new directories and to copy new files into pnfs use /pnfs/cms/WAX/2/ as the root of your path. The following command create a new directory:

mkdir /pnfs/cms/WAX/2/<username>

To put files into resilient dCache users do the following:

mkdir /pnfs/cms/WAX/resilient/<username>

Note! pnfs might look like a regular file system but it's not!
Restrict commands to:
mkdir, rmdir, rm, dccp and srcmp
(don't cd into /pnfs, don’t run find commands, don't run"ls -r").
If you need to see a list of files in pnfs see this page.

Handling of many small files might present problems to the system therefore it is strongly recommended to tar collections of small files before moving them into dCache!

For files over 2GB the ls command will show 1k for the file size. To get the true file size use the special dcache commands dcsize dcls and dcdu

To copy files from dCache with dccp:

To copy the file over to your local disk, do

dccp /pnfs/cms/WAX/2/username/myfile.data $PWD

dCache can also be accessed (read only) via a URL like syntax in case you don't have access to the pnfs file system

Note: /pnfs/cms maps to /pnfs.fnal.gov/usr/cms on cmsgridftp.fnal.gov.

To copy files into dCache with srmcp:

NOTE: srmcp requires authentication. Please see dCache/SRM file transfer through srmcp for instructions on getting your grid-certificate and registering with the CMS VOMS so that you are authorized to copy files into dCache with srmcp.

First make sure you have Java 1.5.0 in your path at the beginning:

export PATH=/usr/java/jdk1.5.0_10/bin/:$PATH
or
setenv PATH /usr/java/jdk1.5.0_10/bin/ $PATH

Then use srcmp to copy yhe file into dcache

srmcp "file:///$PWD/myfile.dat" "srm://cmssrm.fnal.gov:8443/resilient/username/myfile.data"

or, to copy the file from dDache to your local disk, do

srcmp "srm://cmssrm.fnal.gov:8443/resilient/username/myfile.data" "file:///$PWD/myfile.dat"

Note: You must first create the path on /pnfs/cms/WAX and make it group writable with chmod "775". The SRM automatically add /pnfs/cms/WAX to the url path. For example, if the path of a file on /pnfs is /pnfs/cms/WAX/resilient/username/myfile.dat it maps to the url srm://cmssrm.fnal.gov:8443/resilient/username/myfile.dat
Removing /pnfs/cms/WAX from the path gives the path for the url.

Note: LFNs beginning with /store map to PFNs /pnfs/cms/WAX/11/store.

Accessing files in dCache using a ROOT application:

One nice feature of the dCache system is the fact that files can be written and read directly with a ROOT application.  Just replace TFile with TDCacheFile. The code fragment below demonstrates the use of the TDCacheFile class, it also shows both ways of accessing the files (pnfs and URL).

#include <TDCacheFile.h>
  int testdcache() 
{ TFile *hfile = new  TDCacheFile ("dcap://cmsgridftp.fnal.gov:24125/pnfs/fnal.gov/usr/cms/
wenzel/hsimple_dcache.root","READ","Demo ROOT file with histograms",0);
hfile->ls(); 
hfile->Print();
hpx->Draw();
return 0;}

ROOT is used by CMSSW therefore it is setup with the CMSSW runtime environment. To set this up: run scramv1 list CMSSW to list CMSSW versions which are available and their location, change directories to the location of a version of CMSSW, then run eval `scramv1 runtime -csh`

An example is shown below:

scramv1 list CMSSW
cd /uscmst1/prod/sw/cmssw/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_4_3
eval `scramv1 runtime -csh`
cd -
root

The commands can be executed directly

root [0] .L ${ROOTSYS}/lib/libDCache.so
root [1] TFile *hfile = new TDCacheFile ("dcap://cmsgridftp.fnal.gov:24125/pnfs/fnal.gov/usr/cms/
wenzel/hsimple_dcache.root","READ","Demo ROOT file with histograms",0);
root [2] hfile->ls();
root [3] hfile->Print();
root [4] hpx->Draw();

For questions concerning this page and dCache send email to: cms-dcache@fnal.gov