Sync problem to S3

Welcome to BestSync Support Forum!
If you have any questions, comments, concerns, suggestion, please summit here, we'll try to reply you in short time.
Thank you!
Post Reply
john.tyter
Posts: 4
Joined: Mon Nov 23, 2015 1:44 pm

Sync problem to S3

Post by john.tyter »

I synchronize the local data to Amazon S3 by one-way mirroring.
By 1st syncronization process, 500,000 files in local drive were uploaded to S3 successfully.
In next syncronization process, I expected that diffrentioal synchronization will be excuted.
But as checking preview, These files already uploaded to S3 became not existed, So started to be uploaded to S3.
Checking files by S3 management console, I can recognize all files however,
So I doubt that Bestsync2015 cannot recognize files normally.

And,I make another backet, Syncronizing every 100 files to S3.
Then,after exceeding number of file over 1000, These files already uploaded to S3 became not be recognized again.

What is the cause resulting in this problem?
Please give me some advise.
Thanks.
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: Sync problem to S3

Post by RiseFly »

Please start the preview of the task to see if there is any error when list the files in S3.
You can find the detailed log at the log window of the preview.
john.tyter
Posts: 4
Joined: Mon Nov 23, 2015 1:44 pm

Re: Sync problem to S3

Post by john.tyter »

While verifying behavior of Syncronization, some logs were output like below.

1. never using Double-byte character to folder,files, and syncronizing those 1200 files to S3, checked the preview.

Result:1200 files were recognized successfully, objects to be syncronize is 0 file.

Log:
>List Bucket.
>GET /
<200 OK
>GET ?prefix=test_folder/&max-keys=1000
<200 OK
<Get 0 folders, 999 files.
>GET ?prefix=test_folder/&marker=test_folder%2Ftest_file0999.txt&max-keys=1000
<200 OK
<Get 0 folders, 201 files.

2. containing Double-byte character to folder,files, and syncronizing those 1200 files to S3, checked the preview.

Result:999 files were recognized successfully, yet objects to be syncronize is 201 files.

Log:
>List Bucket.
>GET /
<200 OK
>GET ?prefix=test_folder2/&max-keys=1000
<200 OK
<Get 0 folders, 999 files.
>GET ?prefix=test_folder2/&marker=test_folder2%2F%E7%B9%9D%E3%83%BB%E3%81%9B%E7%B9%9D%E5%8C%BB%E3%83%B5%E7%B9%A7%EF%BD%A1%E7%B9%A7%EF%BD%A4%E7%B9%9D%EF%BD%AB0999.txt&max-keys=1000
<200 OK
<Get 0 folders, 0 files.

The problem now I facing is 2.
When I used Double-byte character to folder and files, consequence of GET became 0.
So it looks like only 999 files were recognized as objects to be syncronize in 1st GET.

Please let me know some workarouond.
For example:change the Character code in Advanced setting of Bestsync.

Thanks.
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: Sync problem to S3

Post by RiseFly »

Please try the new version 10.0.4.0 to see if the problem can be resolved.
john.tyter
Posts: 4
Joined: Mon Nov 23, 2015 1:44 pm

Re: Sync problem to S3

Post by john.tyter »

Thank you for your reply.
I tried the new version of 10.0.4.0 from 10.0.3.9.
Logs output were changed like below, but it seems that the part of Double-byte character got garbled.
Consequently, it is not changed that files after 999th in S3 are not recognized normally.
For your infomation, I am using Japanese language.

Log(got garbled):
>List Bucket.
>GET /
<200 OK
>GET ?prefix=test_folder2/&max-keys=1000
<200 OK
<Get 0 folders, 999 files.
>GET ?prefix=test_folder2/&marker=test_folder2/繝・せ繝医ヵ繧。繧、繝ォ0998.txt&max-keys=1000
<200 OK
<Get 0 folders, 0 files.

When I deleted all files in S3 and uploaded them as new ones,
those files were recognized as Double-byte character successfully.

Log(recognized successfully):
>List Bucket.
>GET /
<200 OK
>GET ?prefix=test_folder2/&max-keys=1000
<200 OK
<Get 0 folders, 0 files.
>[1]Upload /[Bukect Name]/test_folder2/テストファイル0998.txt
>[1]PUT /test_folder2/%E3%83%86%E3%82%B9%E3%83%88%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB0001.txt
john.tyter
Posts: 4
Joined: Mon Nov 23, 2015 1:44 pm

Re: Sync problem to S3

Post by john.tyter »

When I excuted GET and filename of 1000th contains Double-byte character,
the command could not find the file and fails.
But When I excuted GET and file of 1000th contains only 1-byte character,
the command suuceed.
Do you have some plan to fix this problem?

Thanks.
RiseFly
Site Admin
Posts: 1077
Joined: Tue Nov 03, 2009 2:51 pm

Re: Sync problem to S3

Post by RiseFly »

Sorry for replying you late.
On Bestsync Ver10.0.4.1, We have tested a folder with more than 1500 files with unicode double-byte file names, all worked fine.
Please tell on the latest version. If there are old files with strange name in S3, please delete them first before test.
Post Reply