SFTP can sync 1021 files only

Welcome to BestSync Support Forum!
If you have any questions, comments, concerns, suggestion, please summit here, we'll try to reply you in short time.
Thank you!
Post Reply
dDIMA
Posts: 6
Joined: Tue Feb 10, 2015 8:45 am

SFTP can sync 1021 files only

Post by dDIMA »

Hi.

I use BestSync to archive files from different SFTP servers. Best Sync can't download more than 1021 file. I need to restart task again to make sync complete. For example, if remote server has 2500 new files I need to restart task 3 times.
File #1022 and all next files have an error message in the log:

1==>2,\20150322065046-000598-001E.G.037.game.log.xz, 22.03.2015 10:05:01,3572,SFTP Error:
OpenFile:
sftpOpenFile:
remotePath: /...[truncate]... /20150322065046-000598-001E.G.037.game.log.xz
access: readOnly
createDisposition: openExisting,accessBackup
v3Flags: 0x1
Sen

All tasks have the same options:
- one way sync from 1 to 2.
- no backup, no include/exclude, no filters
- copy new empty folders, use intrermediate, verify file by the available methods
- retry 3 times
- no encryption, no renaming
- do nothing on delete
- run daily
- no speed control, no extra applications
- create _bestsync_meta.dat

Remote servers have vsftpd installed. Other SFTP clients download all files in the single session without any errors.
What can be wrong with either BestSync settings or with vsftpd settings?
Attachments
sync1021.jpg
sync1021.jpg (127.83 KiB) Viewed 10528 times
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: SFTP can sync 1021 files only

Post by RiseFly »

BestSync has not file number limit.
Please confirm if you are using newest version (10.0.0.18), if not, please download and try the new version to see if the problem can be resolved.
dDIMA
Posts: 6
Joined: Tue Feb 10, 2015 8:45 am

Re: SFTP can sync 1021 files only

Post by dDIMA »

RiseFly wrote:
> BestSync has not file number limit.
> Please confirm if you are using newest version (10.0.0.18), if not, please
> download and try the new version to see if the problem can be resolved.

I have version:
BestSync 2015
English Version Ver10.0.1.9(64Bit)

the problem is still exists.
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: SFTP can sync 1021 files only

Post by RiseFly »

We have used a new version of SFTP library, and the released version is BestSync 2015 (ver 10.0.2.0). We have confirmed by our SFTP server that the new version can upload more than 1021 files in one time synchronization.
dDIMA
Posts: 6
Joined: Tue Feb 10, 2015 8:45 am

Re: SFTP can sync 1021 files only

Post by dDIMA »

RiseFly wrote:
> We have used a new version of SFTP library, and the released version is
> BestSync 2015 (ver 10.0.2.0). We have confirmed by our SFTP server that the
> new version can upload more than 1021 files in one time synchronization.

I have upgraded BestSynt to "English Version Ver10.0.2.0(64Bit)". The issue is still actual. :roll:
Attachments
1021.png
1021.png (3.66 KiB) Viewed 10499 times
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: SFTP can sync 1021 files only

Post by RiseFly »

BestSync has not this kind of limit, and we have confirmed by our SFTP server that BestSync can upload/download more than 1021 files in one synchronization. Please check if your server has the limit or problem.
dDIMA
Posts: 6
Joined: Tue Feb 10, 2015 8:45 am

Re: SFTP can sync 1021 files only

Post by dDIMA »

Hello

The issue is still actual.
It seems that BestSync keeps files opened until end of task.

This is a log from source FTP server while sync task is active
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
1440 0 2469363
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
1440 0 2469363
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
1488 0 2469363
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
1728 0 2469363
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
1837 0 2469363
1837 is a max value. Best Sync can't syncronize more files.

This is a log from source FTP server when BestSync task was aborted.
ddima@STATUS:~$ cat /proc/sys/fs/file-nr
816 0 2469363

Note 1837-816 is equal to 1021.
I can increase limits but I think it could be better to close already processed files immediately after sync.
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: SFTP can sync 1021 files only

Post by RiseFly »

We have added a closefile command after upload a file to SFTP server.
Please try the new version 10.0.3.2 to see if the problem can be resolved.
dDIMA
Posts: 6
Joined: Tue Feb 10, 2015 8:45 am

Re: SFTP can sync 1021 files only

Post by dDIMA »

Yes. Now it works good.
Thank you very much. The issue is resolved.
Post Reply