Class BlockUploader


  • public class BlockUploader
    extends java.lang.Object
    Works with an UploadMatcher dividing the input table into chunks and uploading them separately to produce an arbitrarily large result while each upload/match operation is of a limited size.
    Since:
    15 May 2014
    Author:
    Mark Taylor
    • Constructor Summary

      Constructors 
      Constructor Description
      BlockUploader​(UploadMatcher umatcher, int blocksize, long maxrec, java.lang.String outName, uk.ac.starlink.table.JoinFixAction uploadFixAct, uk.ac.starlink.table.JoinFixAction remoteFixAct, ServiceFindMode serviceMode, boolean oneToOne, boolean uploadEmpty)
      Constructor.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      uk.ac.starlink.table.StarTable runMatch​(uk.ac.starlink.table.StarTable inTable, QuerySequenceFactory qsFact, uk.ac.starlink.table.StoragePolicy storage)
      Performs an upload join in blocks.
      void setTruncationAdvice​(java.lang.String truncationAdvice)
      Sets a string that can be issued to the user as additional advice if there are truncations in block results.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • BlockUploader

        public BlockUploader​(UploadMatcher umatcher,
                             int blocksize,
                             long maxrec,
                             java.lang.String outName,
                             uk.ac.starlink.table.JoinFixAction uploadFixAct,
                             uk.ac.starlink.table.JoinFixAction remoteFixAct,
                             ServiceFindMode serviceMode,
                             boolean oneToOne,
                             boolean uploadEmpty)
        Constructor.
        Parameters:
        umatcher - upload matcher
        blocksize - maximum number of rows per uploaded block
        maxrec - maximum number of output rows accepted in total
        outName - name of output table
        uploadFixAct - name deduplication policy for upload table
        remoteFixAct - name deduplication policy for remote table
        serviceMode - upload match mode
        oneToOne - true iff output rows match 1:1 with input rows
        uploadEmpty - determines behaviour if there are no input rows: true means attempt the match anyway, false means throw an IOException
    • Method Detail

      • runMatch

        public uk.ac.starlink.table.StarTable runMatch​(uk.ac.starlink.table.StarTable inTable,
                                                       QuerySequenceFactory qsFact,
                                                       uk.ac.starlink.table.StoragePolicy storage)
                                                throws java.io.IOException
        Performs an upload join in blocks.

        As currently implemented, the input table needs to have random access, and the output table has random access. It would be possible to stream on input and output to some extent, though each input chunk will still have to be random access.

        The row indices given by ConeQueryRowSequence.getIndex calls associated with the supplied qsFact object must correspond to the row indices in the supplied inTable.

        Parameters:
        inTable - input table, must have random access
        qsFact - object to generate positional queries when applied to a table
        storage - storage policy for storing raw result table
        Throws:
        java.io.IOException
      • setTruncationAdvice

        public void setTruncationAdvice​(java.lang.String truncationAdvice)
        Sets a string that can be issued to the user as additional advice if there are truncations in block results.
        Parameters:
        truncationAdvice - user-readable advice string