Map Vs List Comprehension

Case 1. Map Vs List comprehension for user defined function

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)' 'def ch(x): return x+10' 'map(ch ,xs)'

1000000 loops, best of 3: 1.3 usec per loop

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)' 'def ch(x): return x+10' '[ch(x) for x in xs]'

1000000 loops, best of 3: 1.39 usec per loop

Case 2. Map Vs List comprehension for inbuilt function

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)'  'map(hex,xs)'

1000000 loops, best of 3: 0.926 usec per loop

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)'  '[hex(x) for x in xs]'

1000000 loops, best of 3: 1.14 usec per loop

Case 3. Map Vs List comprehension for function built by lambda

python -mtimeit -s'xs = xrange(10)'  'map(lambda x: x+10,xs)'

1000000 loops, best of 3: 1.29 usec per loop

python -mtimeit -s'xs = xrange(10)'  '[lambda x: x+10 for x in xs]'

1000000 loops, best of 3: 0.958 usec per loop

Time taken by map function is lesser in Case1&2, whereas higher in Case3

                                                        Map                     ListComprehension

1. User defined function                  1.3 usec                      1.39 usec

2. Inbuilt function                           0.926 usec                  1.14 usec   

3. function built by lambda            1.29 usec                    0.958 usec

MySQL for beginners

Install mysql-server and mysql-client  using the command for example in ubuntu sudo apt-get install mysql-server mysql-client, this will prompt for admin password, provide the password and remember the password.

logging into the mysql shell for id “root” and password “password"

mysql -u root -ppassword

To list the available databases

> show databases; 

To create database

> create database if not exists django_tut; 

Selecting database

> use django_tut

To delete database

> drop database if exists django_tut;

To list down all the tables in the database

> Show Tables;

Creating a table

> create table if not exists ex_table( p_id INT, P_code varchar 20)

verifying the table

>describe ex_table;

deleting the table “ex_table1"

> drop table if exists ex_table1;

Inserting a records in the table ex_table

> insert into ex_table value (100, “hundred”)

> insert into ex_table value (200, “two hundred”)

Inserting a records in the table ex_table1

> insert into ex_table value (100, “hundred”)

> insert into ex_table value (300, “three hundred”)

Updating a record
>update ex_table set p_code="HUNDRED" where p_id=100

Querying the table

> select * from ex_table;

Querying for a particular column in the table

> select p_id from ex_table;

Querying based on condition

> select p_id from ex_table where p_id =100;

Alter table adding a column in the table

> alter table ex_table add column p_description varchar(50);

>describe table;

Joining a table

> select * from ex_table join ex_table1 on ex_table.p_id = ex_table1.p_id;

taking backup (below command prompt for password)

$mysqldump -u root -p —databases django_tut > ~/django_tut_bkup.sql

restoring backup(below command prompt for password)

$mysql -u root -p django_tut < ~/django_tut_bkup.sql

GIT tag basics

• Clone the repository => git clone https://github.com/Jadatravu/Tutorials.git Tut
• Goto the checkout repository  =>cd Tut/
• To list all the available tags=> git tag -l 
• To list all the available branches git branch
• Creating a tag “first_tag” with the commit revision => git tag -a first_tag 673b992
• Pushing all the created tags => git push origin --tags
• Checking out the tag “first_tag” by creating a local branch “first_tag_branch” => git checkout -b first_tag_branch first_tag

[Hadoop] Importing/Exporting Database MySQL <==> HDFS using Sqoop

• Import all MySQL tables from "training" database  into the HDFS directory "hdfsdir"

$]sqoop import-all-tables --connect jdbc:mysql://localhost/training --username training --password training --warehouse-dir hdfsdir

• Loginto the hive shell
• create the external table(without location) with the schema as of the MySQL database table "movies"

hive> external table movies(eid int, ename String, code String) row format delimited fields terminated by ',';

• Load data from the hdfs location into the table movies

hive>load data inpath 'hdfs://localhost/user/training/hdfsdir/Movies/part-*' into table movies;

• Run Hive Queries

hive>select  eid, name, codefrom movies where eid >1000;

• Overwrite table "movies" with a filter, this will delete "movies" directory in "/user/hive/warehouse" and recreates the directory "/user/hive/warehouse/movies" with a file 0000000_0. Contents of this file would the records of the filter

hive> insert overwrite table movies select * from movies where eid>1670; 

• Create a table in MySQL with the same scheme

mysql>  CREATE TABLE movies1 ( eid int(11) NOT NULL, ename char(255), code char(255));

• export data from the HIVE/HDFS into the MySQL

$] sqoop export --connect jdbc:mysql://localhost/training --username training --password training --table movies1 -m 4 --export-dir /user/hive/warehouse/movies

BASH Script to Get the Child Process IDs recursively.

#!/bin/bash
function return_child_pid
{
   pd=$1
   count=0
   for p in `ps -ef | awk '$3=="'"$pd"'" {print $2}'`;
   do
      count=$(($count+1))
      return_child_pid $p
   done
   if [ $count == 0 ]; then
     echo $1
     return $1
   fi
}

return_child_pid $1

Web Interface to Subversion log

1. Install pysvn, BeautifulSoup, httplib2 python modules, also install php, php support packages for apache2/httpd

2. Create directory with read,write permissions for apache2/httpd user "/usr/local/share/WebSVNLog/"

3. Place the files in apache2/httpd default folder for example "/var/www/html"

4. Place the SVN path and access credentials(user_name,path) in svn_log.py and svn_branches.py files 

5. Run the svn_branches.py, will create/update svn_branches.json file in   "/usr/local/share/WebSVNLog/" directory

6. Access the WebSVNLog from the browser http://127.0.0.1/Web_Log.php

Note: This is tested in linux OS, needs configuration changes in to enable in other OS. Also this scripts can be placed in SVN server or any other machine which has access to the Subversion server.

Source code accessible from WebSVNLog

Docker Machine [VirtualBox] Steps

1. Install Virtualbox in the Hostmachine

2. Download the "linux"  binary from the URL

• Linux: https://github.com/docker/machine/releases/download/0.0.1/linux

   Rename the linux named binary as machine & chmod u+x machine

3. Create a virtualbox VM from the command

./machine create -d virtualbox dev

4.start  the Virtualbox  and observe the newly created vm in the virtualbox list.

5. Power on the created VM

6. Once the created VM is up, play with the docker commands

 docker run busybox echo helloworld