d-store down on TBG9 - Printable Version

d-store down on TBG9 - Printable Version

+- Technology and Beyond (TaB) (http://tab.d-thinker.org)
+-- Forum: Data Thinker (/forumdisplay.php?fid=896)
+--- Forum: Resources (/forumdisplay.php?fid=211)
+---- Forum: Completed work (/forumdisplay.php?fid=275)
+---- Thread: d-store down on TBG9 (/showthread.php?tid=4023)

d-store down on TBG9 - amoners - 08-15-2015 10:35 AM

@xwcwt report:

20150815/xwcwt 10:12AM: 发现/mnt/dstore无故没有挂载，疑是出现故障。当时执行的操作有将Fedora镜像源软件包rsync到dstore中（该操作已经持续2天，之前未出现该状况）。

RE: d-store down on TBG9 - zma - 08-15-2015 12:03 PM

I usually restart the nfs3 daemon of HDFS for this situation. The method here is better and more graceful than restarting nfs3 daemon.

Please note this down in the D-store instructions.

RE: d-store down on TBG9 - amoners - 08-15-2015 01:14 PM

(08-15-2015 12:03 PM)zma Wrote: I usually restart the nfs3 daemon of HDFS for this situation. The method here is better and more graceful than restarting nfs3 daemon.

Please note this down in the D-store instructions.

Have added this info on http://tab.d-thinker.org/showthread.php?tid=2736&page=2&highlight=dstore

RE: d-store down on TBG9 - xwcwt - 08-15-2015 05:16 PM

Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5)

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt  fedora/ dstore@10.88.14.2:/mnt/dstore/

RE: d-store down on TBG9 - amoners - 08-15-2015 05:33 PM

(08-15-2015 05:16 PM)xwcwt Wrote: Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt fedora/ dstore@10.88.14.2:/mnt/dstore/

Could you try to use our specified command in http://tab.d-thinker.org/showthread.php?tid=2479 ? And report problem if you encountered problems.

RE: d-store down on TBG9 - zma - 08-15-2015 05:52 PM

(08-15-2015 05:16 PM)xwcwt Wrote: Some errors when I re-exec rsync cmd:

Code:

rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragon-4.11.3-1.fc20.x86_64.rpm.tcURLN" -> "linux/releases/20/Everything/os/Packages/d/dragon-4.11.3-1.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.dragonegg-3.3-11.fc20.x86_64.rpm.Nr9UDn" -> "linux/releases/20/Everything/os/Packages/d/dragonegg-3.3-11.fc20.x86_64.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-1.0-11.fc20.noarch.rpm.9dhLm1" -> "linux/releases/20/Everything/os/Packages/d/drascula-1.0-11.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-de-1.0-8.fc20.noarch.rpm.q8yKey" -> "linux/releases/20/Everything/os/Packages/d/drascula-de-1.0-8.fc20.noarch.rpm": Input/output error (5) rsync: rename "/mnt/dstore/linux/releases/20/Everything/os/Packages/d/.drascula-es-1.0-8.fc20.noarch.rpm.YWD2Ec" -> "linux/releases/20/Everything/os/Packages/d/drascula-es-1.0-8.fc20.noarch.rpm": Input/output error (5)

The cmd used is :

Code:

rsync -avrt fedora/ dstore@10.88.14.2:/mnt/dstore/

HDFS does not support rewriting existing files, while rsync tries to do so with many common options. And HDFS' file attribute is not update every time the files is modified within a small time span after last modification, which causes rsync think the existing files in D-store is older some times.

Please try the one as in the post whose link @amoners just posted.

RE: d-store down on TBG9 - xwcwt - 08-15-2015 06:05 PM

I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/

and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

RE: d-store down on TBG9 - amoners - 08-20-2015 09:29 AM

(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

RE: d-store down on TBG9 - zma - 08-20-2015 09:44 AM

(08-20-2015 09:29 AM)amoners Wrote:
(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

The 2MB/s is downloading through multiple ways. Not directly to D-store.

My guess is the 'sync' option causes the dstore mounting slow.

Please try 'async'. But be aware the dstore mounting may be stuck at a higher probability.

RE: d-store down on TBG9 - amoners - 08-20-2015 10:56 AM

(08-20-2015 09:44 AM)zma Wrote:
(08-20-2015 09:29 AM)amoners Wrote:
(08-15-2015 06:05 PM)xwcwt Wrote: I change to use

Code:

rsync --inplace -r -z -v -h --size-only --progress --timeout=600 -e "ssh" fedora/ dstore@tgb9:/mnt/dstore/
and the speed is very slow, just about 200-300KB/s, the transport is in the intranet.

@zma: Is there any mechanism to test performance of d-store? How many speed in d-store cluster on HK? How many speed sync gene files from HK to NB, in my memory, you said around 2MB/s, is it? (May I confused.)

The 2MB/s is downloading through multiple ways. Not directly to D-store.

My guess is the 'sync' option causes the dstore mounting slow.

Please try 'async'. But be aware the dstore mounting may be stuck at a higher probability.

@xwcwt: I have changed mount option to 'async', please try again.

RE: d-store down on TBG9 - lingu - 08-30-2015 01:25 AM

This'd better be stored with tbg9 so that we group everything about it in one place, assuming there is not access constraints. We can create a board for tbg9.