Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Use 'make install' to install Data Thinker
02-08-2016, 12:12 PM (This post was last modified: 08-21-2017 07:42 PM by lingu.)
Post: #1
Use 'make install' to install Data Thinker
To install a Data Thinker system (a thinker), store the dt installation package on think_portal and run 'make install' in the 'dt' dir in the decompressed code tree as think_user.

To prepare the installation package, run 'make package' in the code tree's top directory (dt) to generate a package, then copy this package to think_portal.

Prerequisite
Let's use $thinker_user to denote the user that runs the thinker. The user $thinker_user should be able to passwordlessly log to all nodes.
You can use AddClusterUser ( http://tab.d-thinker.org/showthread.php?tid=7548 ) to do it like this:
Code:
$ AddClusterUser --username $thinker_user --password 123456

Usage
The synopsis of "make install":
Code:
make install [mic=<MIC file path>]

'make install' accepts "mic" option in the form of shevars (shell variables) added to the make command. The "mic" option would specifies the path of the Master Installation Config file (mic.pcf), which contains your defined installation parameters. The make logic also learns configurations from /thinker/etc/soft/think/mic.pcf if such a file exists.

The following parameters are available in the mic.pcf. These options are optional. They would use the default value if not specified.
ips: <ip_list_file> the file containing the IPs of the nodes
think_portal: <portal> the specification of the portal
think_user: <username> the username of the user who runs the thinker
force_append_bashrc: <True|False> Whether forcing append installation vars into ~/.bashrc. Usually for automatically installation by script, force_append_bashrc needs to be "Ture". Otherwise, you have to input a "Y" into the interactive shell.

A typical mic.pcf would like this:
Quote:ips: ~/my.ips
think_portal: 10.5.5.5
think_user: gene
force_append_bashrc: True

Design
The installation process includes the mic.pcf, and invokes cod://dt/release/bin/deploy to copy files and configure a thinker system:
release/bin/deploy [--force-append-bashrc] --newme --ips=[$ips] [$think_user@][think_portal]

The workflow in the deploy script is described below.
Code:
dependency_checking
files_bin=a list of files in dir release/bin to be copied
files=some thinker system files in dir release
files_relaese=$files_bin $files
files_extra=$libfiles $testfiles $lr_demo $kmeans_demo
set version info
prepare and send a tar ball including $release_files, files_extra to the dest as $think_ship_pack
on dest:
  unpack think_ship_pack
if $basic_setup {
  on dest:
    run setup_basic
}
precompile alphabet
if $ip_list_files set {
  on dest:
    run nconfig
}

nconfig sets up the thinker configuration for a set of nodes.

In the DECENT paradigm, we may attempt to install think on multiple nodes. But the nodes should behave differently -- the portal node should do more work and other nodes do less or none. The installation process can know which IP the think_portal is. Then, we can get the node's own official IP as used in its DECENT install process by bmc2node. If those two IPs don't match, the node is not the portal, and it exits the installation with status 0. If the two IPs match, the installation process as before.

Guardian: lingu

-----
20170821/gl: Add the decent installation function.
20170127/zli: Add pseudo code for pre-compilation during "make install"
20170126/gl: Specify the use of the site pcf.
20170102/gl: Specify that the installation should use the package and on the portal.
20161215/gl: Link to multi-node setup info.
20160917/gl: Use think_user and think_portal
20160507/gl: Add a design info.
20160621/yxj: Add make package info.
20160710/gl: Add the user creation method.
Find all posts by this user
Quote this message in a reply
03-28-2016, 01:03 PM
Post: #2
RE: Use 'make install' to install Data Thinker
An example: install DT on hosts whose IPs are listed in /thinker/conf/ips.cfg for user gene6 and set the portal to be 10.28.1.2.

Code:
make install ips=/thinker/conf/ips.cfg portal=10.28.1.2 user=gene6
Find all posts by this user
Quote this message in a reply
03-29-2016, 11:32 AM
Post: #3
RE: Use 'make install' to install Data Thinker
`make help` does not have info of the `mkae install`:

$ make help
Usage: make [options]
Options:
BIGMSG_COMMON_H,
BIN_TRANSLATOR_IMPORTS,
CLEAN_ALL,
CONTAINER_FILES,
CONTAINER_SMALLSYNC,
help Print this message.
MEM_HOME_O,
quiz,
RUNTIME_FILES,
RUNTIME_CONST,
RUNTIME_SMALLSYNC,
RUNTIME_IMPORTS,
SCHEDULER_IMPORTS,
smallsync,
test,
testall,
VPC_O,
major variable:
CLEAN, CLEAN=no enables to clean all upstream CTs before smallsync
SKIP, SKIP=yes enables to skip upstream CTs when they do not exist

For more info and bug reporting, please visit
http://tab.d-thinker.org/showthread.php?tid=5046


At least it should have a link to this thread so that users know how to use the `make install` command.
Quote this message in a reply
03-29-2016, 10:10 PM
Post: #4
RE: Use 'make install' to install Data Thinker
(03-29-2016 11:32 AM)zma Wrote:  `make help` does not have info of the `mkae install`:

$ make help
Usage: make [options]
Options:
BIGMSG_COMMON_H,
BIN_TRANSLATOR_IMPORTS,
CLEAN_ALL,
CONTAINER_FILES,
CONTAINER_SMALLSYNC,
help Print this message.
MEM_HOME_O,
quiz,
RUNTIME_FILES,
RUNTIME_CONST,
RUNTIME_SMALLSYNC,
RUNTIME_IMPORTS,
SCHEDULER_IMPORTS,
smallsync,
test,
testall,
VPC_O,
major variable:
CLEAN, CLEAN=no enables to clean all upstream CTs before smallsync
SKIP, SKIP=yes enables to skip upstream CTs when they do not exist

For more info and bug reporting, please visit
http://tab.d-thinker.org/showthread.php?tid=5046


At least it should have a link to this thread so that users know how to use the `make install` command.

Indeed the help info is not very easy to read and is perhaps not complete. Directly pointing to the TaB discussion is an easy and perhaps effective solution.

I don't know who implemented 'make help'. Usually, we can read the history but in this case the history is truncated. So I am cc'ing Wentao and Wangjun and hope one of them knows about it and will take care of it.

My impression is that 'make help' was implemented before 'make install'. This is perhaps why 'install' info is missing.
Find all posts by this user
Quote this message in a reply
03-29-2016, 11:46 PM
Post: #5
RE: Use 'make install' to install Data Thinker
(03-29-2016 10:10 PM)lingu Wrote:  
(03-29-2016 11:32 AM)zma Wrote:  `make help` does not have info of the `mkae install`:

$ make help
Usage: make [options]
Options:
BIGMSG_COMMON_H,
BIN_TRANSLATOR_IMPORTS,
CLEAN_ALL,
CONTAINER_FILES,
CONTAINER_SMALLSYNC,
help Print this message.
MEM_HOME_O,
quiz,
RUNTIME_FILES,
RUNTIME_CONST,
RUNTIME_SMALLSYNC,
RUNTIME_IMPORTS,
SCHEDULER_IMPORTS,
smallsync,
test,
testall,
VPC_O,
major variable:
CLEAN, CLEAN=no enables to clean all upstream CTs before smallsync
SKIP, SKIP=yes enables to skip upstream CTs when they do not exist

For more info and bug reporting, please visit
http://tab.d-thinker.org/showthread.php?tid=5046


At least it should have a link to this thread so that users know how to use the `make install` command.

Indeed the help info is not very easy to read and is perhaps not complete. Directly pointing to the TaB discussion is an easy and perhaps effective solution.

I don't know who implemented 'make help'. Usually, we can read the history but in this case the history is truncated. So I am cc'ing Wentao and Wangjun and hope one of them knows about it and will take care of it.

My impression is that 'make help' was implemented before 'make install'. This is perhaps why 'install' info is missing.

info can be found on: http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=4203&pid=1172

I can take care of it.
Quote this message in a reply
03-30-2016, 12:17 AM
Post: #6
RE: Use 'make install' to install Data Thinker
(03-29-2016 11:46 PM)xwcwt Wrote:  info can be found on: http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=4203&pid=1172

I can take care of it.

Thanks for locating the info and taking initiative to fix it. +8
Find all posts by this user
Quote this message in a reply
05-04-2016, 05:24 PM
Post: #7
RE: Use 'make install' to install Data Thinker
`make install` write the "think_vpc_num" to ~$think_user/.bashrc, it's better to append to set-env.sh .

set-env.sh contains default values. The .bashrc is usually used by the $think_user only.
Quote this message in a reply
05-07-2016, 04:38 PM
Post: #8
RE: Use 'make install' to install Data Thinker
The installation process invokes cod://dt/release/bin/deploy to copy files and configure a thinker system. The workflow is described below.

Code:
dependency_checking
files_bin=a list of files in dir release/bin to be copied
files=some thinker system files in dir release
files_relaese=$files_bin $files
files_extra=$libfiles $testfiles $lr_demo $kmeans_demo
set version info
prepare and send a tar ball including $release_files, files_extra to the dest as $think_ship_pack
on dest:
  unpack think_ship_pack
if $basic_setup {
  on dest:
    run setup_basic
}
if $ip_list_files set {
  on dest:
    run nconfig
}
Find all posts by this user
Quote this message in a reply
05-08-2016, 02:59 PM (This post was last modified: 05-08-2016 03:01 PM by lingu.)
Post: #9
RE: Use 'make install' to install Data Thinker
(05-04-2016 05:24 PM)zma Wrote:  `make install` write the "think_vpc_num" to ~$think_user/.bashrc, it's better to append to set-env.sh .

set-env.sh contains default values. The .bashrc is usually used by the $think_user only.

Good idea. But the think_vpc_num should respect the value from the identical envar.

We may replace think_vpc_num to be think_size, and both DT and GLAD users can manipulate it.

I suggest we keep the currnet think_vpc_num as before, but start to use think_size with the new logic. When there is not a think_size value, we use the value from think_vpc_num, if present. If both think_size and think_vpc_num are set, the former takes precedence.

Please get this work done ASAP. Currently, I have a lot of headache on tbg6 which uses the thinker in gene6 because of the global setting.
Find all posts by this user
Quote this message in a reply
05-08-2016, 07:11 PM
Post: #10
RE: Use 'make install' to install Data Thinker
(05-08-2016 02:59 PM)lingu Wrote:  
(05-04-2016 05:24 PM)zma Wrote:  `make install` write the "think_vpc_num" to ~$think_user/.bashrc, it's better to append to set-env.sh .

set-env.sh contains default values. The .bashrc is usually used by the $think_user only.

Good idea. But the think_vpc_num should respect the value from the identical envar.

It did so and should be doing so (if there is not bug in recent changes).

Quote:We may replace think_vpc_num to be think_size, and both DT and GLAD users can manipulate it.

I suggest we keep the currnet think_vpc_num as before, but start to use think_size with the new logic. When there is not a think_size value, we use the value from think_vpc_num, if present. If both think_size and think_vpc_num are set, the former takes precedence.

I don't think there is urgent need to add a new variable yet. think_vpc_num can do what we need so far. Adding a new variables doing something an old one does plus a little bit new things will increase maintenance cost and likely introduce several bugs not covered in test cases.

Without a new interface-like variable, to achieve the needs:

The detailed change will be

in set-env-template-local.sh

# default number of containers
if [[ x$think_vpc_num == x ]]; then
export think_vpc_num="1"
fi

--->

# default number of containers
if [[ x$think_vpc_num == x ]]; then
think_vpc_num_use_conf="true"
export think_vpc_num="1"
fi

in bin/nconfig.sh

cat $(dirname $0)/checkAppend.py | ssh $thinker_user@$portal "python - ~$thinker_user/.bashrc '
export think_vpc_num=$ipscnt
'"

-->

cat $(dirname $0)/checkAppend.py | ssh $thinker_user@$portal "python - ~$thinker_user/.bashrc '
if [[ "\$think_vpc_num_use_conf\" == "true" ]]; then
export think_vpc_num=$ipscnt
fi
'"

Quote:Currently, I have a lot of headache on tbg6 which uses the thinker in gene6 because of the global setting.

What's your specific problem? Did you update DT and GLAD to the latest version on it? That can be a good use case for the changes here.

So far with my experience, even with DT of old version, controlling think_vpc_num using envar works quite well.
Quote this message in a reply
Post Reply 


Forum Jump: