Problems Synchronising Large Folders with OneDrive

• Posted by:
• Olly Lennox
• No Comments

The OneDrive client generally does a good job of keeping your files and folders synchronised between your devices and the cloud. However it is not without it’s problems, especially when trying to sync large files and folders containing millions of files and folders or 100s of gigabytes.

We often have to work with large datasets for our clients as part of migrations from on-premise file servers into the cloud. As a result we’ve picked up a few tips and tricks when working with the OneDrive client application which I’ll try and cover in this article. I should caveat that these are simply things which have helped us in the past and seem to work with the current iteration of OneDrive. These tips are not “Microsoft approved” and I’d always recommend performing your own research and testing first before bringing this to a production environment.

Part 1:  Other tools are available

Before even looking at OneDrive it’s worth being aware of some other tools out there which can sometimes be more useful with certain tasks:

SharePoint Migration Tool (SPMT)

If you don’t already know about SPMT, then I’d suggest taking a look before reading any further (https://docs.microsoft.com/en-us/sharepointmigration/introducing-the-sharepoint-migration-tool). This tool is designed to push up large volumes of data into SharePoint and does the job significantly better than the OneDrive client does. If you’re only task is to push data up then I’d strongly recommend abandoning the the OneDrive client and moving to this tool instead.

It is also worth nothing that OneDrive (personal) still uses SharePoint as the backend so this tool can be used to push data into a User’s OneDrive in addition to SharePoint sites/libraries. I’m not sure if that still applies to home users as we only ever work in a business/O365 context.

 

The SharePoint Web Interface

Another tool to bear in mind is the SharePoint web interface which is accessible by browsing to the document library / OneDrive library through a web browser. This interface is quite limited when it comes to file/folder operations but is significantly quicker when it comes to large move or copy operations from one library to another or when moving folders within the same library.

The main limitation of the web interface is in the total number of files/folders it can process (as opposed to the size of those files/folders). For example if you have a handful of very large files/folders then it can be much faster to go through the web GUI than attempting to sync these through the OneDrive client. Even with very large folders, the web GUI can often still work fine if you split your folder tree up and work in batches. As a rule of thumb I’d say up to ~100gb is possible but if you start seeing errors, try breaking the biggest folders in half and moving them in two stages. The SharePoint storage metrics page (available through the site settings) can be very useful to determine where the majority of the data is stored to help you split the job down.

I wouldn’t recommend using the upload function to push up large volumes of files through the Web GUI. You will frequently hit the timeout limit or encounter random crashes which will put you in limbo where it’s hard to know what has been uploaded and what hasn’t

 

Mover.IO / Third-Party Cloud Tools

Microsoft recently bought out the site Mover.IO (https://mover.io) which is a third-party tool for copying cloud files and uploading to SharePoint. The great news is they’ve made the tool free to Microsoft users meaning that you can use it to transfer any amount of data without spending a penny. If your main task is to sync another cloud service (like DropBox) or merge from another O365 tenancy then this tool is very good and works very well.  There are other tools available (BitTitan, CloudHQ) so go with whatever works for you and your budget.

 

Part 2: Know your enemy

The first and most important thing to know about OneDrive is that it can really screw up your data. In theory this should never happen, the app is well established (and I’m sure that the internal sync logic in there is fine) but it does happen, especially with very large data sets. If this is an important project and you are able to use any of the tools listed above instead then I’d strongly recommend that approach over the OneDrive client.

If you are forced down the OneDrive route then there are a few things worth checking before you start

File/Folder Paths

The internal limit for a file path on a Windows device is 260 characters. Any file that goes above this (for example, when one folder tree is moved below another) becomes inaccessible to the operating system and you will find that common operations like delete, copy, etc. start failing. You should be mindful of this when naming your SharePoint sites/libraries and when choosing the default location to synchronise on your PC. For example if your site is titled “My Company Document Library” and your document library is “All Executive Documents”. then by default your OneDrive path could be something like:

C:\Users\Oliver Lennox\Lennox IT\My Company Document Library\All Executive Documents\

Already that’s added nearly 1/3 of your total limit. You can reduce this by choosing the root of the drive as your default path and keeping your site/library names short. e.g.

C:\Lennox IT\Docs\Exec\

once you’ve done your work with OneDrive you can then rename things in SharePoint again if you want to be more readable.

Note that illegal file paths don’t completely crash OneDrive and it will plough on through but it does seem to slow everything down quite dramatically and obviously if your PC is unable to read the file/directory then it won’t be uploaded to your SharePoint.

 

Illegal Characters

To be honest, the latest version of the client seems to do a pretty good job of handling illegal characters in file / folder names so we’ve not encountered too many problems here. Still, it is worth familiarising yourself with these characters in case your data set is particularly exposed:

https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa

 

Computer Resources

Any operation which has to trawl very large data sets is always going to put stress on a PC. The bigger the dataset, the more problems you’re going to have. I’d recommend the following:

• Disable the Windows Search service and any real-time Virus or Malware protection. You don’t want these services kicking in trying to index/scan every file that’s going through the OneDrive app
• Minimum of 8gb memory. 4gb will often max out and start paging excessively which will slow everything down. 8gb seems to be OK but with very large sets you might still run out
• Ideally use a spare PC for this if you can so there is nothing to conflict with OneDrive and you can leave it running overnight
• An SSD will speed things up quite a bit, especially for the system drive. We’ve actually found that the drive which stores your actual OneDrive files is rarely a bottleneck so having a small SSD for the system paired with a large HDD for the files actually works fine
• A fast internet connection will obviously help but a stable connection is actually more important. The jobs can take several days and are often throttled at Microsoft’s end so stability is usually more important that raw speed

 

Throttling

Microsoft do (and will) throttle your connection to SharePoint online / 365 so with a large job, expect it to start running extremely slowly after a while. You can usually tell because the client on your PC will keep a high memory footprint but CPU, Disk and Network will drop right off.

According to the Microsoft website (https://docs.microsoft.com/en-us/sharepointmigration/sharepoint-online-and-onedrive-migration-speed). Throttling should mostly occur during the main business hours for your region so letting OneDrive sync outside of these hours should boost performance for the job.

 

32bit / 64bit OneDrive Client

At the time of writing this article (August 2021), Microsoft is in the process of rolling out the 64bit OneDrive client application which should improve performance on 64bit devices (i.e. pretty much all devices these days). Right now this version is only available as a preview build but you can download it and in our testing, there did seem to be slight performance gains (though only slight).

Regardless, it is strongly recommended that your OneDrive client is patched up as far as it will go before you start. The later versions are a lot more stable and work a lot better.

 

Waiting Time

If you are expecting OneDrive to sync a very large number of files and folders then expect to wait a long time. A very long time. We have had jobs which can take months to complete so be prepared to manage your client expectations about how long their project might take. Where the bottleneck is and why it take so long, I have no idea but patience is very important.
 

 

Part 3:   Tips and Tricks

Once you’ve got your OneDrive environment setup as best as you can then these are some of the other little things you can use to monitor your migration and help it along

Pause / Resume toggle

I’ll start with the most basic.  For reasons known only to Microsoft, pausing the client (through the settings menu), waiting for it to pause, and then resuming again can often jump start it back into life. I’m not sure what goes on in the background here or why this works but often it does so give it a go, you might find that a sync which has been running for 12 hours magically finishes.

Illegal Folder Locations

One of the most common reasons for the OneDrive sync client to completely give up is where you have files/folders saved locally to a location that is either nonsense in the context or SharePoint or where your account doesn’t have permission to access it.

For example if you have saved files directly into your company folder (the blue building icon) then this is illegal because the folders directly under here must correspond to a SharePoint site. I have seen the OneDrive client give up the sync entirely when this happens.

The red cross icon is often an indicator when files are saved to a place you don’t have permission to access (like a read-only folder). If this has happened you should either move the folder/file to a valid location or delete them from the OneDrive. This problem can often happen when an admin moves things on the server or when you make group/permission changes to an account in O365.

Don’t trust the progress window / recent file list

When you click the little OneDrive icon near the clock you will bring up status information like the number of files to process or recent activity. For long jobs, this information is often wrong or ambiguous and I wouldn’t recommend relying on it as a basis for your decision making. A far better indicator is Task Manager and seeing what the OneDrive.exe process is doing:

• OneDrive.exe – High CPU + Disk Activity – This is good, it means that OneDrive is working away doing it’s thing
• OneDrive.exe – Low CPU + Disk. Memory > 200mb – This usually means it’s working but is on the go slow. Be prepared to wait.
• OneDrive.exe – Low CPU + Disk + Memory – This could point to an issue. Try the Pause/Resume toggle trick or look for problems in the OneDrive folder. Try a ChkDsk

 

OneDriveTemp folder

OneDrive uses a hidden folder called OneDriveTemp as a bit of a scratch pad when it syncs. This folder is located in the root of the drive that holds your OneDrive folder (e.g. C:\). Often times files will get stuck in here for no real reason and OneDrive won’t seem to process any further. An indicator of this is when OneDrive reports “50 files updating (63.7 mb)”

The Good Old pause/resume trick will often get things moving out of this folder and it’s worth a go. Be wary of abandoning your job when things are in this folder because there is a risk of losing data if do.

 

PS Explorer

The free PS Explorer tool from Microsoft can be very helpful to find out which files OneDrive is currently operating on (which can give you a clue where problems are). Use the “Find Handle / DLL” function to search the OneDrive folder which will show locked files. https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer

 

Extreme Patience

As per the previous section, OneDrive can take a very long time and the interface will give you no clue what it’s doing or why. Be patient and leave the client ticking along, eventually it will get there (or it will give you a hint as to what is going wrong) so don’t be tempted to start moving files about or making rash changes which might corrupt the job.

 

---

 

I hope this article helps you with your OneDrive syncs. If you need help with a large file migration to Office 365 then don’t hesitate to contact us.