I found that libusb calls are mainly responsible for long idle times.
For some improvement/workaround look at E-Mail with topic: ftdi_write_chunksize, block_erasers and spi_read_status_register proposals
But still looking for further improvement.
Regards,
Simon