Email from Jon Bakken (15 Nov 2006)

Hi Alan,

1. The
http://www-stken.fnal.gov/enstore/tape_inventory/COMPLETE_FILE_LISTING_cms
is run every 4 hours, starting at midnight.  The script takes a variable
amount of time to complete.  This list is run on pnfs database backups, so
the information in the file can be up to 5 hours old even when the file is
newly available on the web site.  This makes it up to 9 hours old just
before the next one is available.  The 1st line of the file has the time
stamp.

2. I didn't think that this would satisfy your needs, so I negotiated
another file,
http://www-stken.fnal.gov/enstore/tape_inventory/RECENT_FILE_CMS
This file is made every hour, and only lists the files made during the
last 12 hours.  Therefore, the combination of the 2 listings should give
you an accurate representation of the files in pnfs on tape once an hour.
This job runs at 40 past the hour, and it is requested that you get it at
45 past the hour to ensure it has finished.

3. Generating the COMPLETE_FILE_LISTING_cms has many steps, and some can
fail without impacting the integrity of the tape handling system.
Therefore, although it is expected that this file will be new every 4
hours, there may be some cases where a new version won't be available
until another cronjob has run.  No one from the tape system will come in
to fix problems with this file.

4. The RECENT_FILES_CMS listing is from the live database, and it is
unlikely that it will fail, at least not in a way that would not also
cause the tape system integrity to be in trouble.  Therefore, although
there isn't a guarantee on this file either, it should be unlikely for
failures.

5. After having said there isn't any guarantee on the files, I can tell
you that the database person for the tape system is a very responsible
person and he frequently monitors the system during off-hours, so problems
may be fixed quickly even on nights and weekends.

This is the best that we can do for you right now.  It would, as you also
pointed out, be best that all these things be put into cms standard
databases and all these listing files be eliminated.

Let me know if you think you need anything else.

Jon Bakken

**************************************************************************

Note from Alan Stone (17 Nov 2006)

Using a Python script in a cronjob, I am parsing the Complete File Listing
to get the "PNFS MTCC File List".  The "Recent File List" shows all traffic
including HCAL, Monte Carlo, etc. 

**************************************************************************
Email from Yujun Wu [28 Jun 2007]

Hi All,

The new list is ready and it is at:

http://cmsdcam2.fnal.gov/dcache/LASTHALFDAY_FILES_CMS

It will be updated every 10 minutes. Please note Jon has asked me to put all
the newly arrived files in the last half day to this list (not only Global run files).
So, you may have to do some parse yourself.  

I will write some new script for updating this list. However, the list itself should
not change. If you have some suggestions on the format of the list, please just 
let me know.

Regards,
Yujun