Reduce the amount of database update by the pipeline
During the data transfer, the pipeline updates the transfer progress each time the Read
/Write
function is called (so roughly once for each SFTP packet, or once per R66 block). This is way too much, and in the case of large file transfers with rapid connections, it may lead to a severe performance penalty.
Thus the pipeline should be changed to only update the database at fixed time interval, so that the database does not bottleneck the data transfer.