Functions RED have to be potentially modified.
Uploading files to the CVMFS server includes taking a hash of the contents of the file.
1. User sends data to the spooler.
2. From the spooler, there are two modes in which CVMFS files can be uploaded:
1. Direct mode :
Example .cvmpublish : The file is not broken down into chunks. The file is directly sent to the uploader which uploads the files to the server.
2. Indirect mode :
The file is broken down into chunks, depending on whether it is a text file or a binary file, appropriate file breaking function is called. This is done by the File Processor.
Current Scope : We propose to handle garbage collection of files only in Indirect mode.
Extended Scope : We may go for removal of duplicate Direct Mode files by performing a diff operation.
** For upload , abstractSpooler calls Upload (upload.h)
** For dividing the file into chunks, AbstractSpooler calls process() (upload.h) that would divide the file into chunks and send the chunks one by one to the server. (The chunking is configurable by setting allow_chunking parameter.)
The callback function is called only when all of the file has been uploaded to the server.
** Remove - delete a file
** WaitForUpload - Do not take any other jobs to be processed.
** WaitForTermination - No more jobs are scheduled after this function is called.
** TearDown - performs global cleanup work after spooler object has been deleted.
** JobDone - Called after upload job gets completed.
** Processing Callback - callback function from FileProcessor().
1. ProcessingCallback -
a) keep a local cache containing hashes of recently uploaded files.
b) before calling the Upload function, check whether hash is present in the cache. If yes, do not call upload.
c) Ensure all or none upload - make sure that either all the chunks of the file are rejected or all file .
PROS : Saves Network bandwidth.
CONS : Slower Upload of the files on the server. Quering for duplicate chunks should be faster.
1. Upload - Uploads the local_path file to a remote_path location.
2. Upload - Uploads the local_path file to remote_path location and appends localhost path to content hash under which chunk is stored.
3. Remove : Remove a file from backend Storage.
4. Respond : calls the callback function associated with finishing a job.
Chunk Object < upload_file_chunker.h>
Chunk object : Add a field marked - set to zero. This would be marked delete by Preprocessor function if the chunk is a duplicate. (This field can be used for garbage collector.)
Uploading files to the CVMFS server includes taking a hash of the contents of the file.
1. User sends data to the spooler.
2. From the spooler, there are two modes in which CVMFS files can be uploaded:
1. Direct mode :
Example .cvmpublish : The file is not broken down into chunks. The file is directly sent to the uploader which uploads the files to the server.
2. Indirect mode :
The file is broken down into chunks, depending on whether it is a text file or a binary file, appropriate file breaking function is called. This is done by the File Processor.
Current Scope : We propose to handle garbage collection of files only in Indirect mode.
Extended Scope : We may go for removal of duplicate Direct Mode files by performing a diff operation.
Spooler :: <upload.h>
** For upload , abstractSpooler calls Upload (upload.h)
** For dividing the file into chunks, AbstractSpooler calls process() (upload.h) that would divide the file into chunks and send the chunks one by one to the server. (The chunking is configurable by setting allow_chunking parameter.)
The callback function is called only when all of the file has been uploaded to the server.
** Remove - delete a file
** WaitForUpload - Do not take any other jobs to be processed.
** WaitForTermination - No more jobs are scheduled after this function is called.
** TearDown - performs global cleanup work after spooler object has been deleted.
** JobDone - Called after upload job gets completed.
** Processing Callback - callback function from FileProcessor().
LIMITING GARBAGE GENERATION :
1. ProcessingCallback -
a) keep a local cache containing hashes of recently uploaded files.
b) before calling the Upload function, check whether hash is present in the cache. If yes, do not call upload.
c) Ensure all or none upload - make sure that either all the chunks of the file are rejected or all file .
PROS : Saves Network bandwidth.
CONS : Slower Upload of the files on the server. Quering for duplicate chunks should be faster.
Upload < upload_facility.h>
1. Upload - Uploads the local_path file to a remote_path location.
2. Upload - Uploads the local_path file to remote_path location and appends localhost path to content hash under which chunk is stored.
3. Remove : Remove a file from backend Storage.
4. Respond : calls the callback function associated with finishing a job.
Chunk Object < upload_file_chunker.h>
Chunk object : Add a field marked - set to zero. This would be marked delete by Preprocessor function if the chunk is a duplicate. (This field can be used for garbage collector.)
upload_file_chunker_xor32 <AMBIGUOUS- need to read again>
upload_file_processor.h
1. Temporary File Chunk - has the File path appended to the chunk structure.
2. Pending File - Intermediate between uploading file chunks one by one by calling AbstractUploader(), finishing all chunks and calling FileProcessor with final results of upload.
3. Upload CallBack() called after every file chunk upload.
