Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
D-store user instructions
05-21-2015, 09:07 PM (This post was last modified: 10-12-2017 11:01 AM by zma.)
Post: #1
D-store user instructions
D-Store is a reliable data storage system. It stores data with 1, 2, 3 or multiple replications and can be expanded to large capacity.

D-store is usually mounted to `/thinker/dstore/`. Subdirectories under it are created according to the usage.

Common operations and suggested tools

1. Uploading files to a remote D-store through the network:

Use the `upload` command of the `ds` tool which is a command line tool that works the best with D-store. Check the manual for how to use it http://tab.d-thinker.org/showthread.php?tid=4878

2. Downloading files from a remote D-store through the network:

Use the `download` command of the `ds` tool http://tab.d-thinker.org/showthread.php?tid=4878 .

3. Storing files to D-store mounted locally:

Use the `cp` command of the `ds` tool.

A short tutorial on usage of `ds cp` and other tools http://tab.d-thinker.org/showthread.php?tid=9594 .

4. Reading files from D-store locally.

You can use common tools like cp/less to read files on D-store.



Other tools supported

We suggest use the above methods. But some other tools are supported. Examples are as follows for your references.

If you are using Linux

1. Mount you remote directory with sshfs on a node on D-store like dstore-171:

Code:
sshfs -o ro -o follow_symlinks username@your.server:/path/to/be/mounted/ /path/to/mount/point/on/dstore/node/

2. Then use ds tools to directly upload the data

Code:
ds cp /path/to/data/directory/  /thinker/dstore/your/chosen/dir/

Run programs on D-store:

Best practices:

1. Store files to be written in some local directories instead of the D-store directory. After the processing is done, you can copy the result files back to D-store for storing them reliably.

How to build software in D-store: http://tab.d-thinker.org/showthread.php?tid=4696

Related documents
Changes in this article may imply necessary changes in the following documents:
TBM
GM manual

-----
20160306/gl: Add a section for related docs.
Visit this user's website Find all posts by this user
Quote this message in a reply
02-10-2016, 11:54 AM
Post: #2
RE: D-store user instructions
How to start/stop D-store?

Can we make the part in TBM or D-book available as the official user manual?
Find all posts by this user
Quote this message in a reply
02-24-2016, 02:31 PM
Post: #3
RE: D-store user instructions
(02-10-2016 11:54 AM)lingu Wrote:  Can we make the part in TBM or D-book available as the official user manual?

Most of the content is consistent with the TBM. There are more info/methods here. We may merge some info here to TBM.

Generally, this thread can be considered the upon source of the section in TBM.
Visit this user's website Find all posts by this user
Quote this message in a reply
03-06-2016, 03:09 PM
Post: #4
RE: D-store user instructions
(02-10-2016 11:54 AM)lingu Wrote:  How to start/stop D-store?

Can we make the part in TBM or D-book available as the official user manual?

Wangjun - please work on this. First the info about starting/stopping D-store is written here, then we organize it into the manuals. Note that the update of the manuals should be very careful - please "design" updates then implement the endorsed designs.
Find all posts by this user
Quote this message in a reply
03-07-2016, 09:03 AM
Post: #5
RE: D-store user instructions
(03-06-2016 03:09 PM)lingu Wrote:  
(02-10-2016 11:54 AM)lingu Wrote:  How to start/stop D-store?

Can we make the part in TBM or D-book available as the official user manual?

Wangjun - please work on this. First the info about starting/stopping D-store is written here, then we organize it into the manuals. Note that the update of the manuals should be very careful - please "design" updates then implement the endorsed designs.

Actually, It had been added into manual, refer to http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=5084&pid=1525

In my understanding, you point is that the contents should be here, then d-store manual include it, am I right? If so, I will move some info here.
Quote this message in a reply
06-20-2016, 04:16 PM
Post: #6
RE: D-store user instructions
Save a copy of old info.

(05-21-2015 09:07 PM)zma Wrote:  D-Store is a reliable data storage system. It stores data with 3-copy replications and can be expanded to large capacity.

To use it, users can connect its Secure FTP interface at the connection point defined by Host, Part and Username.

For dstore-171, the following example shows the commands that may be used.

(one example here; for other D-store instances, please ask the administrators)

Host: wiles.baijia.info
Port: 23171
Username and password: please contact the administrators.

D-store is usually mounted to `/mnt/dstore/`. Subdirectories under it are created for different users.

Examples of usage with various tools:

D-store provides a `ds` command line tool which works the best.

ds

Check the manual for how to use it http://tab.d-thinker.org/showthread.php?tid=4878

For rsync:

Code:
rsync -avxP -b --suffix=.backup-$(date "+%Y%m%d-%H%M%S") --timeout=600 -e "ssh -p 23171" \
/path/to/local/dir/or/file \
username@wiles.baijia.info:/mnt/dstore/assigned-dir/your/preferred/path/

The 'username' and the 'assigned-dir' are those assigned to you by admins.

For scp:

Code:
scp -P 23171 -r \
/path/to/local/dir/or/file \
username@wiles.baijia.info:/mnt/dstore/assigned-dir/your/preferred/path/

Other methods if you are using Linux

1. Mount you remote directory with sshfs on a node on D-store like dstore-171:

sshfs -o ro -o follow_symlinks username@on.your.server:/path/to/be/mounted/ /path/to/mount/point/on/dstore/node/

2. Then use hdfs tools to directly upload the data

Code:
hdfs -put /path/to/data/directory/  /YourStorageDirOnDStore/your/chosen/dir/

Note there is no '/mnt/dstore/' prefix in the destination directory.

If the destination directory is a directory with specific replication factor settings such as $glad_root/store/world/ (2 copy on disk) in http://tab.d-thinker.org/showthread.php?tid=5114 , you may also use command like:

Code:
hdfs dfs -Ddfs.replication=1 -put ...

The data replication will be later set to the one as in the destination directory.

Run programs on D-store:

Best practices:

1. Store files to be written in some local directories instead of the D-store directory. After the processing is done, you can copy the result files back to D-store for storing them reliably.

How to build software in D-store: http://tab.d-thinker.org/showthread.php?tid=4696

Related documents
Changes in this article may imply necessary changes in the following documents:
TBM
GM manual

-----
20160306/gl: Add a section for related docs.
Visit this user's website Find all posts by this user
Quote this message in a reply
10-12-2017, 10:56 AM
Post: #7
RE: D-store user instructions
(05-21-2015 09:07 PM)zma Wrote:  A short tutorial on usage of `ds cp` and other tools http://tab.d-thinker.org/showthread.php?tid=9594 .

added a tutorial for very new users.
Visit this user's website Find all posts by this user
Quote this message in a reply
10-12-2017, 10:58 AM
Post: #8
RE: D-store user instructions
save a copy.

(05-21-2015 09:07 PM)zma Wrote:  D-Store is a reliable data storage system. It stores data with 1, 2, 3 or multiple replications and can be expanded to large capacity.

D-store is usually mounted to `/thinker/dstore/`. Subdirectories under it are created according to the usage.

Common operations and suggested tools

1. Uploading files to a remote D-store through the network:

Use the `upload` command of the `ds` tool which is a command line tool that works the best with D-store. Check the manual for how to use it http://tab.d-thinker.org/showthread.php?tid=4878

2. Downloading files from a remote D-store through the network:

Use the `download` command of the `ds` tool http://tab.d-thinker.org/showthread.php?tid=4878 .

3. Storing files to D-store mounted locally:

Use the `cp` command of the `ds` tool.

A short tutorial on usage of `ds cp` and other tools http://tab.d-thinker.org/showthread.php?tid=9594 .

4. Reading files from D-store locally.

You can use common tools like cp/less to read files on D-store.



Other tools supported

We suggest use the above methods. But some other tools are supported. Examples are as follows for your references.

Following examples use a D-store instance as the example:

Host: wiles.baijia.info
Port: 23171
D-store mount point: /mnt/dstore

For rsync:

Code:
rsync -avxP -b --suffix=.backup-$(date "+%Y%m%d-%H%M%S") --timeout=600 -e "ssh -p 23171" \
/path/to/local/dir/or/file \
username@wiles.baijia.info:/mnt/dstore/assigned-dir/your/preferred/path/

The 'username' and the 'assigned-dir' are those assigned to you by admins.

For scp:

Code:
scp -P 23171 -r \
/path/to/local/dir/or/file \
username@wiles.baijia.info:/mnt/dstore/assigned-dir/your/preferred/path/

Other methods if you are using Linux

1. Mount you remote directory with sshfs on a node on D-store like dstore-171:

sshfs -o ro -o follow_symlinks username@on.your.server:/path/to/be/mounted/ /path/to/mount/point/on/dstore/node/

2. Then use hdfs tools to directly upload the data

Code:
hdfs -put /path/to/data/directory/  /YourStorageDirOnDStore/your/chosen/dir/

Note there is no '/mnt/dstore/' prefix in the destination directory.

If the destination directory is a directory with specific replication factor settings such as $glad_root/store/world/ (2 copy on disk) in http://tab.d-thinker.org/showthread.php?tid=5114 , you may also use command like:

Code:
hdfs dfs -Ddfs.replication=1 -put ...

The data replication will be later set to the one as in the destination directory.

Run programs on D-store:

Best practices:

1. Store files to be written in some local directories instead of the D-store directory. After the processing is done, you can copy the result files back to D-store for storing them reliably.

How to build software in D-store: http://tab.d-thinker.org/showthread.php?tid=4696

Related documents
Changes in this article may imply necessary changes in the following documents:
TBM
GM manual

-----
20160306/gl: Add a section for related docs.
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: