I've written a file sync and verify program in Python. I use it to backup important files, some of which are hosted on a network drive. When my network utilization is high enough, copying a file from the network drive can sometimes fail. For this reason, I implemented a retry mechanism.

Since retries are so basic to software development, I looked around for a reusable library. I had good results with tenacity.

To leverage tenacity, just install it and reference it in your app. Then add a "@retry" decorator above any functions or methods that need retries. Make sure that they throw exceptions when they require retries.

For example, in the source code below, you'll find this @retry decorator:

@retry(wait=wait_fixed(Constants.RETRY_WAIT), stop=stop_after_attempt(Constants.MAX_RETRIES),
           before=copy_file_before_callback)

This specifies that the method immediately after the decorator will be retried if it throws any exceptions. There will be a fixed wait time before each retry, and retries will stop after a specified number of attempts. The copy_file_before_callback function function is called before each retry.

I strongly recommend that you use the before= callback mechanism in your code to log retries. That way, during testing, you'll know if retries are actually happening.

import os
import win32file
import win32api
import win32con
import pywintypes
import shutil
import platform
from tenacity import *
from StringLiterals import StringLiterals
from AppException import AppException
from Constants import Constants
from Globals import app_globals


def copy_file_before_callback(retry_state):
    if retry_state.attempt_number > 1:
        app_globals.log.print(f'***** FileSystemUtils.copy_file: attempt_number: {retry_state.attempt_number} file path: {retry_state.args[1]}')


class FileSystemUtils:
    
    ...

    @staticmethod
    @retry(wait=wait_fixed(Constants.RETRY_WAIT), stop=stop_after_attempt(Constants.MAX_RETRIES),
           before=copy_file_before_callback)
    def copy_file(source_file_path, destination_file_path):
        if os.path.exists(destination_file_path):
            os.remove(destination_file_path)

        shutil.copy2(source_file_path, destination_file_path)

I'm ready to retry this walk!