文件同步的主要问题 [英] A major problem with file sync
问题描述
我想将文件系统存储从本地办公室同步到服务器目的地。 ·目标文件存储空间很大。在2400万个文件和文件夹中大约1 TB。 ·源只有几MB。 ·我面临的问题是
目标机器上的knowlegde文件很大（大约200MB）。我的意思是syncfile.metadata正如我从sych算法中理解的那样，当源要将文件发送到目的地时，此文件被发送到源我看到每次需要时在网络上发送整个
200MB告诉消息来源它的知识。这个解决方案对我不利，因为有大约50个源尝试同步服务器。如果我运行基于同步
框架的Synctoy 2.1，它不会发送这个知识文件，速度要快得多。我应该改变什么才能让它为一个巨大的目的地正常而快速地工作？如果Synctoy 2.1能够做到这一点并且它被包裹在sych框架中，我必须改变什么才能在我的大环境中使用这个解决方案？ （BTW–我没有使用哈希进行文件比较）以下是我正在使用的重要代码片段：
public void SyncFileSystemReplicasOneWay（字符串sourceReplicaRootPath，字符串destinationReplicaRootPath，FileSyncScopeFilter过滤器，FileSyncOptions选项）
{
logger.Debug（" SyncFileSystemReplicasOneWay - Started"） ;
FileSyncProvider sourceProvider = null;
FileSyncProvider destinationProvider = null;
try {
sourceProvider = new FileSyncProvider（sourceReplicaRootPath，filter，options）;
destinationProvider = new FileSyncProvider（destinationReplicaRootPath，filter，options）;
destinationProvider.AppliedChange + = new EventHandler（OnAppliedChange）;
destinationProvider.SkippedChange + = new EventHandler（OnSkippedChange）;
sourceProvider.DetectingChanges + = new EventHandler（sourceProvider_DetectingChan ges）;
sourceProvider.DetectedChanges + = new EventHandler（sourceProvider_DetectedChanges）;
destinationProvider.DetectingChanges + = new EventHandler（destinationProvider_DetectingChanges）;
destinationProvider.DetectedChanges + = new EventHandler（destinationProvider_DetectedChanges）; //使用SyncCallbacks
冲突的项目。 SyncCallbacks destinationCallbacks = destinationProvider.DestinationCallbacks;
destinationCallbacks.ItemConflicting + = new EventHandler（OnItemConflicting）;
destinationCallbacks.ItemConstraint + = new EventHandler（OnItemConstraint）;
SyncOrchestrator agent = new SyncOrchestrator（）;
agent.LocalProvider = sourceProvider;
agent.RemoteProvider = destinationProvider;
//将源同步到目的地
agent.Direction = SyncDirectionOrder.Upload; logger.Info（"同步对副本的更改："+ destinationProvider.RootDirectoryPath）;
Console.WriteLine（"同步对副本的更改："+ destinationProvider.RootDirectoryPath）;
agent.SessionProgress + = new EventHandler（agent_SessionProgress）;
agent.Synchronize（）;
Console.WriteLine（" Synchronizing ended"）;
logger.Info（" Synchronizing ended"）; }
最后
{
//释放资源if（sourceProvider！= null）
sourceProvider.Dispose（）;
if（destinationProvider！= null）
destinationProvider.Dispose（） ;
logger.Debug（" SyncFileSystemReplicasOneWay - Ended"）;
}
}
选项包括：fileSystemSynchronizer.Options = FileSyncOptions.RecycleDeletedFiles | FileSyncOptions.RecyclePreviousFileOnUpdates | FileSyncOptions.RecycleConflictLoserFiles;
谢谢，Pini
< blockquote>
您好，
我认为您的同步应用程序是从本地办公室计算机运行的。如果要避免在每次同步时通过网络发送.metadata文件，请使用不同的FileSyncProvider构造函数将目标元数据文件夹显式设置为ocal office machine的
文件夹。当FileSyncProvider检测到目标的本地更改时，它需要访问.metadata文件。此文件不仅包含知识，还包含所有本地元数据更改跟踪信息，例如每个文件/目录的
文件路径，属性，文件大小和先前更改检测中的文件内容哈希值。 这些更改跟踪metatdata应该占用大部分200MB大小而不是知识。
由于您有50个客户端需要与服务器同步，因此在每个客户端中将单独的服务器.metadata保持为不同的同步对需要支付额外费用。如果您的客户从未直接相互同步，则可以通过这种方式设置它们。
它与SyncToy文件夹&pair对概念相同。
谢谢，
Dong
I want to synchronize a file system storage from a local office to a server destination. · The destination file storage is huge. About 1 TB in 24 million files and folders. · The source has only a few MB. · The problem I am facing is that the knowlegde file on the destination machine is huge (about 200MB). I mean the syncfile.metadata As I understood from the sych algorithm this file is sent to the source when the source want to send files to the destination I am seeing that the entire 200MB is sent on the network each time it needs to tell the source it’s knowledge. This solution is not good for me since there are about 50 sources trying to sync the server. And if I am running the Synctoy 2.1 which is based on the synchronization framework it doesn’t send this knowledge file and is much much faster. What should I change in order to make it work normal and fast for a huge destination? If the Synctoy 2.1 is capable of doing it and it is wrapped aroung the sych framework what do i have to change to use this solution in my large enviorment? (BTW – I am not using hash for file comparison) Here are the improtant code snippets that I am using:
public void SyncFileSystemReplicasOneWay(string sourceReplicaRootPath, string destinationReplicaRootPath, FileSyncScopeFilter filter, FileSyncOptions options)
{
logger.Debug( "SyncFileSystemReplicasOneWay - Started");
FileSyncProvider sourceProvider = null;
FileSyncProvider destinationProvider = null;
try {
sourceProvider = new FileSyncProvider(sourceReplicaRootPath, filter, options);
destinationProvider = new FileSyncProvider(destinationReplicaRootPath, filter, options);
destinationProvider.AppliedChange += new EventHandler(OnAppliedChange);
destinationProvider.SkippedChange += new EventHandler(OnSkippedChange);
sourceProvider.DetectingChanges += new EventHandler(sourceProvider_DetectingChanges);
sourceProvider.DetectedChanges += new EventHandler(sourceProvider_DetectedChanges);
destinationProvider.DetectingChanges += new EventHandler(destinationProvider_DetectingChanges);
destinationProvider.DetectedChanges += new EventHandler(destinationProvider_DetectedChanges); // Use SyncCallbacks for
conflicting items. SyncCallbacks destinationCallbacks = destinationProvider.DestinationCallbacks;
destinationCallbacks.ItemConflicting += new EventHandler(OnItemConflicting);
destinationCallbacks.ItemConstraint += new EventHandler(OnItemConstraint);
SyncOrchestrator agent = new SyncOrchestrator();
agent.LocalProvider = sourceProvider;
agent.RemoteProvider = destinationProvider;
// Sync source to destination
agent.Direction = SyncDirectionOrder.Upload; logger.Info( "Synchronizing changes to replica: " + destinationProvider.RootDirectoryPath);
Console.WriteLine("Synchronizing changes to replica: " + destinationProvider.RootDirectoryPath);
agent.SessionProgress += new EventHandler(agent_SessionProgress);
agent.Synchronize();
Console.WriteLine("Synchronizing ended");
logger.Info( "Synchronizing ended"); }
finally
{
// Release resources if (sourceProvider != null)
sourceProvider.Dispose();
if (destinationProvider != null)
destinationProvider.Dispose();
logger.Debug( "SyncFileSystemReplicasOneWay - Ended");
}
}
The options are: fileSystemSynchronizer.Options = FileSyncOptions.RecycleDeletedFiles | FileSyncOptions.RecyclePreviousFileOnUpdates | FileSyncOptions.RecycleConflictLoserFiles;
Thanks, Pini
Hi,
I assume that your sync app is running from local office machine. If you want to avoid the .metadata file to be sent across the network in every sync, please use different FileSyncProvider constructor to explicit set the destination metadata folder to a folder of the ocal office machine. When FileSyncProvider detects the local changes for destination, it need to access the .metadata file. This file doesn't only contain knowledge, it also contains all local metadata change tracking info such as each file/directory's file path, attributes, file size and file content hash value in previous change detection. These change tracking metatdata should take most of the 200MB size instead of Knowledge.
Since you have 50 clients that need to sync with server, keeping a seperate server .metadata in each client as different sync pair is additional cost. If your clients never sync to each other directly, it is ok to set them up in this way. It is same as the SyncToy folder pair concept.
Thanks,
Dong
这篇关于文件同步的主要问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
