Need to create a "Hot Folder" to automatically copy new files

lego

lego

lego

Soldato
Joined
4 Apr 2004
Posts
5,626
Location
Pontypridd
This will be run on a local windows 10 machine, essentially all I need is when a new file is created in a selected folder, that file is automatically copied to another folder.

The use for this is for a new RIP we are using, with our old RIP we could drop a list of files into it and it would add them to the RIP in file order. However the new RIP seems to take them at random (And there's no way of stopping this). This creates a big slow down when trimming and sorting the media for shipping.

As the files are created every 15-90 seconds, it gives enough time for the RIP to load them into the list in the correct order.

Any help welcome
 
robocopy {source} {dest} /MOV /MON:1
Moves the file from source directory to dest directory, monitors the source and runs if there is one change (new file created)

Would that do what you need?
 
Sort of, though that's moving the files, I guess I could just take the /MOV command out and it would just copy new and updated files?
 
The copy seems to work, my only other problem is I want it to only copy new files from the time of monitoring.

For example : I want run robocopy on c:\test1 and copy any new files to c:\test2, with the current command above it copies all current files in the folder and then starts monitoring, is there a way for it to scan the files first, not copy them, then only start copying new / edited files?
 
lego said:
The copy seems to work, my only other problem is I want it to only copy new files from the time of monitoring.

For example : I want run robocopy on c:\test1 and copy any new files to c:\test2, with the current command above it copies all current files in the folder and then starts monitoring, is there a way for it to scan the files first, not copy them, then only start copying new / edited files?
Click to expand...
You could add /MAXAGE:1 which would only copy those from today, or replace the 1 with today's date
or /M those with an Archive bit set (any modified file) and then re-set it once copied
 
Andre said:
You could add /MAXAGE:1 which would only copy those from today, or replace the 1 with today's date
or /M those with an Archive bit set (any modified file) and then re-set it once copied
Click to expand...

This works but I've come across another problem that may result in robocopy not working, due to the workflow that I use, the destination folder for the copied files is actually the RIPs own hot folder, unfortunately the RIP processes then deletes each file, so with the above command it's getting a copy of each new file created added to the rip each minute (meaning many duplicates)

Relentless81 said:
Microsoft Synctoy?
Click to expand...

I did have a look and the main issue is the the sync file it leaves in the destination folder gets spat out by the RIP
 
Pho said:
What about Powershell? You can use a FileSystemWatcher to monitor a folder and be notified instantly of any changes. In the notification you can move / rename / do whatever you want with the file.
https://community.idera.com/databas...osts/using-filesystemwatcher-correctly-part-2
Click to expand...

Thanks, I'll have a quick look this morning, not really done anything like this in the past but I'm willing to give it a go, hopefully I can get it going and it can save me a lot of time in the long run
 
Actually, plan b. I'd probably use Python or write a little utility in C# (which would also use the same underlying FileSystemWatcher) over Powershell as Powershell can be flaky for stuff like this.

Try this that I knocked together.

  1. Install python
  2. Open a command prompt and:
  3. pip install watchdog
  4. pip install argparse

Save the below code to say watcher.py and run with:
  • python watcher.py --source "c:\temp\source" --dest "c:\temp\dest"

Any file written to c:\temp\source will be moved to c:\temp\dest. You can test it out by copying and pasting a file into the source folder.


Code: 
import argparse
import sys
import time
import logging
import shutil
import time
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

class EventHandler(LoggingEventHandler):
    def __init__(self, dest, max_retries, retry_sleep):
        self.dest = dest
        self.max_retries = max_retries
        self.retry_sleep = retry_sleep
       
    #def on_moved(self, event):
    #    super(LoggingEventHandler, self).on_moved(event)
    #    logging.info("Moved %s: from %s to %s", what, event.src_path, event.dest_path)

    def on_created(self, event):
        super(LoggingEventHandler, self).on_created(event)
        logging.info("Created %s: %s", self.what(event), event.src_path)       
        self.move_file(event.src_path)

    #def on_deleted(self, event):
    #    super(LoggingEventHandler, self).on_deleted(event)
    #    logging.info("Deleted %s: %s", self.what(event), event.src_path)

    #def on_modified(self, event):
    #    super(LoggingEventHandler, self).on_modified(event)
    #    logging.info("Modified %s: %s", self.what(event), event.src_path)

    def what(self, event):
        return 'directory' if event.is_directory else 'file'
       
    def move_file(self, path, attempt=1):
        try:
            logging.debug("Moving %s to %s" % (path, self.dest))
            shutil.move(path, self.dest)
        except PermissionError:
            if (attempt > self.max_retries):
                raise
            logging.warn("Attempt %d/%d: file locked - might still be getting written" % (attempt, self.max_retries))
            time.sleep(self.retry_sleep)
            self.move_file(path, attempt+1)
        except:
            logging.error("Unexpected error:", sys.exc_info()[0])

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='File System Watcher')
    parser.add_argument('-s', '--source', type=str, help='Source path', required=True)
    parser.add_argument('-d', '--dest', type=str, help='Destination path', required=True)
    parser.add_argument('-r', '--retries', type=int, help='Maximum retry attempts', default=12)
    parser.add_argument('-z', '--retry-sleep', dest='retry_sleep', type=int, help='Retry sleep time', default=5)

    args = parser.parse_args()

    logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
    event_handler = EventHandler(args.dest, args.retries, args.retry_sleep)
    observer = Observer()
    observer.schedule(event_handler, args.source, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(5)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()
 
Last edited:
Wow Pho, thanks for that, it's all up and running, the only thing is it seems to be moving the new files and not copying them. I did try to change line #23 to a copy command, but it threw up an error.

Is this something easily fixed?
 
Oh sorry I thought I'd read you wanted to move them :o

Try replacing it with the following which adds a copy_file function:

Code: 
import argparse
import sys
import time
import logging
import os
import shutil
import time
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

class EventHandler(LoggingEventHandler):
    def __init__(self, dest, max_retries, retry_sleep):
        self.dest = dest
        self.max_retries = max_retries
        self.retry_sleep = retry_sleep
       
    #def on_moved(self, event):
    #    super(LoggingEventHandler, self).on_moved(event)
    #    logging.info("Moved %s: from %s to %s", what, event.src_path, event.dest_path)

    def on_created(self, event):
        super(LoggingEventHandler, self).on_created(event)
        logging.info("Created %s: %s", self.what(event), event.src_path)       
        self.copy_file(event.src_path)

    #def on_deleted(self, event):
    #    super(LoggingEventHandler, self).on_deleted(event)
    #    logging.info("Deleted %s: %s", self.what(event), event.src_path)

    #def on_modified(self, event):
    #    super(LoggingEventHandler, self).on_modified(event)
    #    logging.info("Modified %s: %s", self.what(event), event.src_path)

    def what(self, event):
        return 'directory' if event.is_directory else 'file'

    def copy_file(self, path, attempt=1):
        try:
            destination_filename=os.path.join(self.dest, os.path.basename(path))
            logging.debug("Copying %s to %s" % (path, destination_filename))
            shutil.copyfile(path, destination_filename)
        except PermissionError:
            if (attempt > self.max_retries):
                raise
            logging.warn("Attempt %d/%d: file locked - might still be getting written" % (attempt, self.max_retries))
            time.sleep(self.retry_sleep)
            self.copy_file(path, attempt+1)
        except:
            logging.error("Unexpected error:", sys.exc_info()[0])

    def move_file(self, path, attempt=1):
        try:
            logging.debug("Moving %s to %s" % (path, self.dest))
            shutil.move(path, self.dest)
        except PermissionError:
            if (attempt > self.max_retries):
                raise
            logging.warn("Attempt %d/%d: file locked - might still be getting written" % (attempt, self.max_retries))
            time.sleep(self.retry_sleep)
            self.move_file(path, attempt+1)
        except:
            logging.error("Unexpected error:", sys.exc_info()[0])

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='File System Watcher')
    parser.add_argument('-s', '--source', type=str, help='Source path', required=True)
    parser.add_argument('-d', '--dest', type=str, help='Destination path', required=True)
    parser.add_argument('-r', '--retries', type=int, help='Maximum retry attempts', default=12)
    parser.add_argument('-z', '--retry-sleep', dest='retry_sleep', type=int, help='Retry sleep time', default=5)

    args = parser.parse_args()

    logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
    event_handler = EventHandler(args.dest, args.retries, args.retry_sleep)
    observer = Observer()
    observer.schedule(event_handler, args.source, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(5)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()
 
Works like a charm, thanks so much. Not sure I can offer any services in return, but I do work in Printing so if there's anything you might need just send me a trust.

Thanks again
 
Glad it works :). You should be able to run it via a scheduled task or something if you want to run in the background always.

Can you print bank notes that pass as legit? ;)
 
I've just saved the execution line as a bat file and will run it when I'm designing.

On the bank note side of things, it is something we can do with our Textronics wax printer, but it's all out of red wax for some reason which has nothing to do with the big stack of fifties in the corner! :P
 
Cool. Hopefully it won't cause you any problems.

Wonder if anyone actually bothers to fraud £50 notes given everyone's already highly suspicious of them as it is.
 
You must log in or register to reply here.
Back
Top Bottom