Threaded Mode | Linear Mode

08-15-2015, 10:35 AM

@xwcwt report:

20150815/xwcwt 10:12AM: 发现/mnt/dstore无故没有挂载，疑是出现故障。当时执行的操作有将Fedora镜像源软件包rsync到dstore中（该操作已经持续2天，之前未出现该状况）。

08-15-2015, 12:03 PM

I usually restart the nfs3 daemon of HDFS for this situation. The method here is better and more graceful than restarting nfs3 daemon.

Please note this down in the D-store instructions.

08-15-2015, 01:14 PM

(08-15-2015 12:03 PM)zma Wrote: I usually restart the nfs3 daemon of HDFS for this situation. The method here is better and more graceful than restarting nfs3 daemon.

Please note this down in the D-store instructions.

Have added this info on http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=2736&page=2&highli

08-15-2015, 05:16 PM

Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt  fedora/ dstore@10.88.14.2:/mnt/dstore/

08-15-2015, 05:33 PM

(08-15-2015 05:16 PM)xwcwt Wrote: Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt fedora/ dstore@10.88.14.2:/mnt/dstore/

Could you try to use our specified command in http://tab.d-thinker.org/showthread.php?tid=2479 ? And report problem if you encountered problems.

08-15-2015, 05:52 PM

(08-15-2015 05:16 PM)xwcwt Wrote: Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt fedora/ dstore@10.88.14.2:/mnt/dstore/

HDFS does not support rewriting existing files, while rsync tries to do so with many common options. And HDFS' file attribute is not update every time the files is modified within a small time span after last modification, which causes rsync think the existing files in D-store is older some times.

Please try the one as in the post whose link @amoners just posted.

08-15-2015, 06:05 PM

I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/

and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

08-20-2015, 09:29 AM

(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

08-20-2015, 09:44 AM

(08-20-2015 09:29 AM)amoners Wrote:
(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

The 2MB/s is downloading through multiple ways. Not directly to D-store.

My guess is the 'sync' option causes the dstore mounting slow.

Please try 'async'. But be aware the dstore mounting may be stuck at a higher probability.

08-20-2015, 10:56 AM

(08-20-2015 09:44 AM)zma Wrote:
(08-20-2015 09:29 AM)amoners Wrote:
(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

The 2MB/s is downloading through multiple ways. Not directly to D-store.

My guess is the 'sync' option causes the dstore mounting slow.

Please try 'async'. But be aware the dstore mounting may be stuck at a higher probability.

@xwcwt: I have changed mount option to 'async', please try again.