Last week, I was doing some testing using the Python API interface for sentinel-1 toolbox with Snappy and the SNAP desktop to see the time taken by each of them. The workflow for processing the data is described here. The results were astonishing. Each and every steps were significantly slower, especially the co-registration process that took 8-10 mins in the SNAP desktop took around 2.75 hours, which was very disappointing. There are few discussions on the SNAP STEP Forum, but most of them, at the end, started using the Graph Processor Tool (GPT) to process the data. GPT is based in JAVA, and has parallel processing that makes it utilize the available resources of the machine to speed up the process. The SNAP Desktop on the background uses the GPT as well.

Turns out, switching from using Snappy to GPT is fairly simple. Usually, a graph looks like below. This is an example graph representing Ellipsoid Correction through Geo-location Grid.

<graph id="someGraphId">
    <version>1.0</version>
    <node id="someNodeId">
        <operator>Ellipsoid-Correction-GG</operator>
        <sources>
            <source>${source}</source>
        </sources>
        <parameters>
            <sourceBands>string,string,string,...</sourceBands>
            <imgResamplingMethod>string</imgResamplingMethod>
            <mapProjection>string</mapProjection>
         </parameters>
    </node>
</graph>

The pre-coregistration gpt xml (lets call this precoregister.xml) looks as below for my use-case.

<graph id="preprocessingSentinel1">
    <version>1.0</version>
    <node id="ApplyOrbitFileNode">
        <operator>Apply-Orbit-File</operator>
        <sources>
            <source>${sourceProducts}</source>
        </sources>
        <parameters>
            <continueOnFail>false</continueOnFail>
        </parameters>
    </node>
    <node id="CalibrationNode">
        <operator>Calibration</operator>
        <sources>
            <source>ApplyOrbitFileNode</source>
        </sources>
        <parameters>
            <outputSigmaBand>false</outputSigmaBand>
            <outputGammaBand>false</outputGammaBand>
            <outputBetaBand>true</outputBetaBand>
        </parameters>
    </node>
    <node id="MultilookNode">
        <operator>Multilook</operator>
        <sources>
            <source>CalibrationNode</source>
        </sources>
        <parameters>
            <nRgLooks>3</nRgLooks>
            <nAzLooks>3</nAzLooks>
            <outputIntensity>true</outputIntensity>
            <grSquarePixel>true</grSquarePixel>
        </parameters>
    </node>
    <node id="TerrainFlatteningNode">
        <operator>Terrain-Flattening</operator>
        <sources>
            <source>MultilookNode</source>
        </sources>
        <parameters>
            <demName>SRTM 1Sec HGT</demName>
            <demResamplingMethod>BICUBIC_INTERPOLATION</demResamplingMethod>
            <oversamplingMultiple>1.5</oversamplingMultiple>
            <additionalOverlap>0.1</additionalOverlap>
        </parameters>
    </node>
</graph>

The post-coregistration gpt xml (lets call this postcoregister.xml) looks as below for my use-case.

<graph id="preprocessingSentinel1">

    <version>1.0</version>
    <node id="MasterSlaveReaderNode">
        <operator>ProductSet-Reader</operator>
        <sources/>
        <parameters>
            <fileList></fileList>
        </parameters>
    </node>
    <node id="CoregistrationNode">
        <operator>DEM-Assisted-Coregistration</operator>
        <sources>
            <source>MasterSlaveReaderNode</source>
        </sources>
        <parameters>
            <demName>SRTM 1Sec HGT</demName>
            <demResamplingMethod>BICUBIC_INTERPOLATION</demResamplingMethod>
            <resamplingType>BICUBIC_INTERPOLATION</resamplingType>
            <tileExtensionPercent>100</tileExtensionPercent>
            <maskOutAreaWithoutElevation>true</maskOutAreaWithoutElevation>
        </parameters>
    </node>
    <node id="SpeckleFilterNode">
        <operator>Speckle-Filter</operator>
        <sources>
            <source>CoregistrationNode</source>
        </sources>
        <parameters>
            <filter>Lee Sigma</filter>
            <enl>4.0</enl>
            <numLooksStr>4</numLooksStr>
            <windowSize>9x9</windowSize>
            <sigmaStr>0.9</sigmaStr>
            <targetWindowSizeStr>5x5</targetWindowSizeStr>
        </parameters>
    </node>
    <node id="EllipsoidCorrectionNode">
        <operator>Ellipsoid-Correction-GG</operator>
        <sources>
            <source>SpeckleFilterNode</source>
        </sources>
        <parameters>
            <imgResamplingMethod>BILINEAR_INTERPOLATION</imgResamplingMethod>
        </parameters>
    </node>
</graph>

The master-slave product are dynamic for the use-case. So I am using lxml library to manipulate the xml files. The python script to call the gpt along with the xml manipulation looks as:

import os
import subprocess
from lxml import etree

output_path = 'D:/Documents/gpt-test-output/'

graph_path = 'D:/Documents/precoregister.xml' # shown above
coregister_path = 'D:/Documents/postcoregister-master-slave.xml'
coregister_template = 'D:/Documents/postcoregister.xml' # shown above
gpt_path = 'C:/snap/bin/gpt.exe'

exec_cmd = '%s %s -t %s{}_Orb_Cal_ML_TF.dim {}.zip' % (gpt_path, graph_path, output_path)
coreg_cmd = '%s %s -t %s{0}_Orb_Cal_ML_TF_Stack_Spk_EC.dim' % (gpt_path, coregister_path, output_path)

def parse(source):
    try:
        parser = etree.XMLParser(
            huge_tree = True,
            no_network = False,
            remove_blank_text = True,
        )
        result = etree.parse(source, parser)
        return result
    except Exception as e:
        print("XML Parse error: {}".format(e))
        return None

def tostring(tree, xml_declaration=False, pretty_print=True):
    string = etree.tostring(
        tree,
        xml_declaration = xml_declaration,
        encoding = "utf-8",
        pretty_print = pretty_print,
    )
    return string

def get_nodes():
    registration_tree = parse(coregister_template)
    master_slave_node = registration_tree.findall("./node[@id='MasterSlaveReaderNode']")[0]

    return registration_tree, master_slave_node

def list_to_string(list):
    return ','.join(list)

def main():

    data = ['D:/Documents/S1A_IW_GRDH_1SDV_20180823T001920_20180823T001945_023366_028ABB_4C9B','D:/Documents/S1A_IW_GRDH_1SDV_20180827T122210_20180827T122235_023432_028CD5_5821','D:/Documents/S1A_IW_GRDH_1SDV_20180830T001122_20180830T001147_023468_028E02_AF51']

    intersecting_slaves = ['D:/Documents/S1A_IW_GRDH_1SDV_20180830T001122_20180830T001147_023468_028E02_AF51_Orb_Cal_ML_TF.dim', 'D:/Documents/S1A_IW_GRDH_1SDV_20180827T122210_20180827T122235_023432_028CD5_5821_Orb_Cal_ML_TF.dim', 'D:/Documents/S1A_IW_GRDH_1SDV_20180823T001920_20180823T001945_023366_028ABB_4C9B_Orb_Cal_ML_TF.dim']

    for _data in data:
        print('started file: {}'.format(_data))

        file_name = _data.split('D:/Documents/')[1]
        before_coregistration = exec_cmd.format(file_name, _data)
        result = subprocess.check_output(before_coregistration, shell=True)
        print(result)

        registration_tree, master_slave_node = get_nodes()
        master_slave_list = master_slave_node.xpath('//fileList')[0]
        master_slave_list.text = list_to_string(intersecting_slaves)

        with open('D:/Documents/postcoregister-master-slave.xml','w') as f:
            f.write(tostring(registration_tree, pretty_print=True))

        after_coregistration = coreg_cmd.format(file_name)
        result = subprocess.check_output(after_coregistration, shell=True)
        print(result)

        os.remove('D:/Documents/postcoregister-master-slave.xml')

        print('**********************************************')
        print('end file')


if __name__ == '__main__':
    main()