Neo4j Spark connector Tutorial

Neo4j having python drivers could be availed from this  location
Neo4j spark connector available at location
It is having drivers for java and scala but not for python

Below are the steps to setup Neo4j- spark scala connector application

Setup:
1. neo4j spark connector
Connector jar could be downloaded from the location
2. scala
download rpm from location
and install using rpm -i
3. sbt
Download sbt to /opt from location
update /etc/profile with export PATH=$PATH:/opt/sbt/bin

Code:
Write scala code or checkout example from github.com location
Update neo4j credentials

Build & Run:
Follow steps in the Readme to build & run the application 

MIcroservice Case study

Microservices could be implemented in several way. In this case below is the stack used.

• Django
• Flask
• Node.js
• mysql
• mongo
• docker
• kubernetes
• ubuntu 

In this usecase API searches keyword "search_term" in the available public repositories and displays top 5 active repositories

There are 3 app services here i.e., Django (DAPP) service, FLASK(FAPP) service, NODE.JS(Nodeapp) service

There are 2 database services Mysql(ms-mysql)service and Mongo(ms-mongo)service.

All five docker images are available at location Docker Hub

Preparing the environment

Install docker and kubernetes as mentioned in the location 

Checkout and start kubernetes files from github location

Starting and Stopping services

Start the kubernetes services/pods/deployments with below steps with the files located in dhub

sh ./start0.sh

sh ./start.sh

For stopping kubernetes services/pods/deployments use below step

sh ./stop.sh

Consuming APIs

DAPP consumes FAPP, MySql services and exposes API

http://<HOST-IP>:8000/search/?search_term=flask

FAPP consumes Mongo service and exposed API

http://<HOST-IP>:9000/api/v1/githubsearch/?search_term=kubernetes&employee=nexi_user3

Nodeapp consumes Mongo service

http://<HOST-IP>:8081/

Deployin django in nginx & uwsgi environment

1. Install uwsgi , nginx 
apt-get install uwsgi, nginx

2. Clone django helloworld program
git clone https://github.com/Jadatravu/Tutorials/

3. Copy the helloworld program into /opt folder
cp -r Tutorials/dj_nx_wsgi  /opt

sudo chown -R www-data:www-data /opt/dj_nx_wsgi

4. Create a python virtual environment and install django
cd /opt
virtualenv env
source /opt/env/bin/activate
pip install django

5. Test uwsgi
(env) osboxes@ubuntu:/opt$ uwsgi --http  :8001 --wsgi-file /opt/dj_ng_wsgi/dj_ng_wsgi/wsgi.py  --virtualenv /opt/env --chdir /opt/dj_ng_wsgi/

6. Access from browser
http://192.168.91.129:8001/hello/

7. Configure nginx
copy nginx.conf from dj_nj_wsgi folder to /etc/nginx/sites-available
make soft link
ln -s /etc/nginx/sites-available/nginx.conf /etc/nginx/sites-enabled/nginx.conf

8.restart nginx
service nginx restart

9. start wsgi
(env) osboxes@ubuntu:/opt$ uwsgi --socket :8001 --wsgi-file /opt/dj_ng_wsgi/dj_ng_wsgi/wsgi.py  --virtualenv /opt/env --chdir /opt/dj_ng_wsgi/

10.from browser access the url.
http://192.168.91.129:8000/hello/

pylint, python logger

pylint --disable=R,C,W,I,E,F test.py --msg-template='{msg_id}:{line:3d},{column}: {obj}: {msg}' --disable=import-error --rcfile ./to-path/.pylintrc

import logging

import sys

root = logging.getLogger()

root.setLevel(logging.DEBUG)

fo= open("/Users/sadatravu/Surya/py_scripts/text.log", "w")

ch= logging.StreamHandler(fo)

#ch = logging.StreamHandler(sys.stdout) # this is for logging into stdout

ch.setLevel(logging.DEBUG)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

ch.setFormatter(formatter)

root.addHandler(ch)

def main():

    for i in range(10):

       logging.debug("message [%d]"%i)

main()

SPA using Angular

Single Page Application using Angular

Video

knockout-vs-backbone-vs-angular

• Angular and knockout comes with data binding
• Backbone is also quite easy to grasp.
• Angular is very powerful and deals with complex models

Original blog

DJANGO FAQ

1. Django project files significance

a. manage.py => enables the django project settings path to the environment
b. settings.py => this file contains the settings of django project like debug mode, database settings, templates path, static_url path etc.,
c. urls.py this file is the controller file which maps the web url to the respective view function/class
d. wsgi.py file => this file will have the environment settings like virtual environment site-packages path, django project path. path of this wsgi.py should be included in the apache configuration file apache2.conf
e. models.py => this file will have the information models and members of the models
f. views.py => this file will have the views functions/classes

2. Django commands

django-admin start-project my_pro => Creates django project
django-admin startapp myapp => creates django application inside the django project

python ./manage.py runserver 127.0.0.1:8000 => runs the development server

python ./manage.py makemigrations => creates the database migration scripts

python ./manage.py migrate => applies the migrations

python ./manage.py inspectdb > model.py  => Creates the models wrt to the already existing database

python ./manage.py collectstatic => collects all the static files in the django project

python ./manage.py dbshell  => Opens the database shell

python ./manage.py shell => opens the django/python shell

Map Vs List Comprehension

Case 1. Map Vs List comprehension for user defined function

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)' 'def ch(x): return x+10' 'map(ch ,xs)'

1000000 loops, best of 3: 1.3 usec per loop

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)' 'def ch(x): return x+10' '[ch(x) for x in xs]'

1000000 loops, best of 3: 1.39 usec per loop

Case 2. Map Vs List comprehension for inbuilt function

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)'  'map(hex,xs)'

1000000 loops, best of 3: 0.926 usec per loop

:~ sadatravu$ python -mtimeit -s'xs = xrange(10)'  '[hex(x) for x in xs]'

1000000 loops, best of 3: 1.14 usec per loop

Case 3. Map Vs List comprehension for function built by lambda

python -mtimeit -s'xs = xrange(10)'  'map(lambda x: x+10,xs)'

1000000 loops, best of 3: 1.29 usec per loop

python -mtimeit -s'xs = xrange(10)'  '[lambda x: x+10 for x in xs]'

1000000 loops, best of 3: 0.958 usec per loop

Time taken by map function is lesser in Case1&2, whereas higher in Case3

                                                        Map                     ListComprehension

1. User defined function                  1.3 usec                      1.39 usec

2. Inbuilt function                           0.926 usec                  1.14 usec   

3. function built by lambda            1.29 usec                    0.958 usec

MySQL for beginners

Install mysql-server and mysql-client  using the command for example in ubuntu sudo apt-get install mysql-server mysql-client, this will prompt for admin password, provide the password and remember the password.

logging into the mysql shell for id “root” and password “password"

mysql -u root -ppassword

To list the available databases

> show databases; 

To create database

> create database if not exists django_tut; 

Selecting database

> use django_tut

To delete database

> drop database if exists django_tut;

To list down all the tables in the database

> Show Tables;

Creating a table

> create table if not exists ex_table( p_id INT, P_code varchar 20)

verifying the table

>describe ex_table;

deleting the table “ex_table1"

> drop table if exists ex_table1;

Inserting a records in the table ex_table

> insert into ex_table value (100, “hundred”)

> insert into ex_table value (200, “two hundred”)

Inserting a records in the table ex_table1

> insert into ex_table value (100, “hundred”)

> insert into ex_table value (300, “three hundred”)

Updating a record
>update ex_table set p_code="HUNDRED" where p_id=100

Querying the table

> select * from ex_table;

Querying for a particular column in the table

> select p_id from ex_table;

Querying based on condition

> select p_id from ex_table where p_id =100;

Alter table adding a column in the table

> alter table ex_table add column p_description varchar(50);

>describe table;

Joining a table

> select * from ex_table join ex_table1 on ex_table.p_id = ex_table1.p_id;

taking backup (below command prompt for password)

$mysqldump -u root -p —databases django_tut > ~/django_tut_bkup.sql

restoring backup(below command prompt for password)

$mysql -u root -p django_tut < ~/django_tut_bkup.sql

GIT tag basics

• Clone the repository => git clone https://github.com/Jadatravu/Tutorials.git Tut
• Goto the checkout repository  =>cd Tut/
• To list all the available tags=> git tag -l 
• To list all the available branches git branch
• Creating a tag “first_tag” with the commit revision => git tag -a first_tag 673b992
• Pushing all the created tags => git push origin --tags
• Checking out the tag “first_tag” by creating a local branch “first_tag_branch” => git checkout -b first_tag_branch first_tag

[Hadoop] Importing/Exporting Database MySQL <==> HDFS using Sqoop

• Import all MySQL tables from "training" database  into the HDFS directory "hdfsdir"

$]sqoop import-all-tables --connect jdbc:mysql://localhost/training --username training --password training --warehouse-dir hdfsdir

• Loginto the hive shell
• create the external table(without location) with the schema as of the MySQL database table "movies"

hive> external table movies(eid int, ename String, code String) row format delimited fields terminated by ',';

• Load data from the hdfs location into the table movies

hive>load data inpath 'hdfs://localhost/user/training/hdfsdir/Movies/part-*' into table movies;

• Run Hive Queries

hive>select  eid, name, codefrom movies where eid >1000;

• Overwrite table "movies" with a filter, this will delete "movies" directory in "/user/hive/warehouse" and recreates the directory "/user/hive/warehouse/movies" with a file 0000000_0. Contents of this file would the records of the filter

hive> insert overwrite table movies select * from movies where eid>1670; 

• Create a table in MySQL with the same scheme

mysql>  CREATE TABLE movies1 ( eid int(11) NOT NULL, ename char(255), code char(255));

• export data from the HIVE/HDFS into the MySQL

$] sqoop export --connect jdbc:mysql://localhost/training --username training --password training --table movies1 -m 4 --export-dir /user/hive/warehouse/movies

BASH Script to Get the Child Process IDs recursively.

#!/bin/bash
function return_child_pid
{
   pd=$1
   count=0
   for p in `ps -ef | awk '$3=="'"$pd"'" {print $2}'`;
   do
      count=$(($count+1))
      return_child_pid $p
   done
   if [ $count == 0 ]; then
     echo $1
     return $1
   fi
}

return_child_pid $1

Web Interface to Subversion log

1. Install pysvn, BeautifulSoup, httplib2 python modules, also install php, php support packages for apache2/httpd

2. Create directory with read,write permissions for apache2/httpd user "/usr/local/share/WebSVNLog/"

3. Place the files in apache2/httpd default folder for example "/var/www/html"

4. Place the SVN path and access credentials(user_name,path) in svn_log.py and svn_branches.py files 

5. Run the svn_branches.py, will create/update svn_branches.json file in   "/usr/local/share/WebSVNLog/" directory

6. Access the WebSVNLog from the browser http://127.0.0.1/Web_Log.php

Note: This is tested in linux OS, needs configuration changes in to enable in other OS. Also this scripts can be placed in SVN server or any other machine which has access to the Subversion server.

Source code accessible from WebSVNLog

Docker Machine [VirtualBox] Steps

1. Install Virtualbox in the Hostmachine

2. Download the "linux"  binary from the URL

• Linux: https://github.com/docker/machine/releases/download/0.0.1/linux

   Rename the linux named binary as machine & chmod u+x machine

3. Create a virtualbox VM from the command

./machine create -d virtualbox dev

4.start  the Virtualbox  and observe the newly created vm in the virtualbox list.

5. Power on the created VM

6. Once the created VM is up, play with the docker commands

 docker run busybox echo helloworld

Dockerfile for the Django application, MySQL as database

Below steps to prepare docker image with the Django with helloworld application installed/setup, MySQL as a database

1. Save the Below dockerfile contents to the directory.

2. Build the image as sudo docker build -t django_hw_img .
(there is a . at the end).

On successful image creation, docker image will created. it will be listed in the list from the command sudo docker images

3. Run the image as 

sudo docker run -i -t -p 80:80 django_hw_img:latest /bin/bash

4. On container shell start the apache2 service service apache2 start

5.Access the application from the http://<your_host_ip_address>/helloworld/

+===================Dockerfile======================+

#################################################

# Dockerfile to build Python-Django WSGI Application Containers

# Based on Ubuntu:latest

#################################################

# Set the base image to Ubuntu

FROM ubuntu

# File Author / Maintainer

MAINTAINER surya.janardhan@gmail.com

# Update the sources list

RUN apt-get update

# Install basic applications

RUN apt-get install -y git build-essential libapache2-mod-wsgi apache2

# Install Python and Basic Python Tools

RUN apt-get install -y python python-pip

# Get pip to download and install requirements:

RUN pip install django

#install MySQL in noninteractive way

RUN export DEBIAN_FRONTEND=noninteractive

RUN apt-get -q -y install mysql-server python-mysqldb

#create the working dir and set the working directory

CMD mkdir /usr/local/share/apps

WORKDIR /usr/local/share/apps

#clone the tutorials from the github.com

RUN git clone https://github.com/Jadatravu/Tutorials /usr/local/share/apps/tutorials

# set the permissions to the app directory

CMD cd /usr/local/share/apps && chown -R www-data:www-data tutorials

WORKDIR  /usr/local/share/apps/tutorials/django_tutorials/helloworld/

# set the database for the helloworld database django_tut

RUN echo "CREATE DATABASE django_tut;" > /usr/local/share/apps/create_database.txt && service mysql start  && mysql -u root < /usr/local/share/apps/create_database.txt &&  python manage.py migrate

# Configure apache configuration for the helloworld application

RUN  echo "WSGIScriptAlias / /usr/local/share/apps/tutorials/django_tutorials/helloworld/helloworld/wsgi.py" >> /etc/apache2/apache2.conf &&  echo "WSGIPythonPath /usr/local/share/apps/tutorials/django_tutorials/helloworld/" >> /etc/apache2/apache2.conf &&  echo "<Directory /usr/local/share/apps/tutorials/django_tutorials/helloworld/>" >> /etc/apache2/apache2.conf &&  echo "<Files wsgi.py>" >> /etc/apache2/apache2.conf &&  echo "Order deny,allow" >> /etc/apache2/apache2.conf &&  echo "Require all granted" >> /etc/apache2/apache2.conf &&  echo "Satisfy Any" >> /etc/apache2/apache2.conf &&  echo "</Files>" >> /etc/apache2/apache2.conf &&  echo "</Directory>" >> /etc/apache2/apache2.conf

#expose the port

EXPOSE 80

+===================Dockerfile======================+

Django Static files tutorial

1. Create a Django project and app, Edit settings.py for database settings
2. Place the static files "files1.txt", "files2.txt" in the directory for example "/home/notroot/django_tut/staticfiles/sfiles"
3. create the directory files in the path for example "/home/notroot/django_tut/staticfiles/"
4. edit section  STATICFILES_DIRS in settings.py file with the value
"/home/notroot/django_tut/staticfiles/sfiles"
5. edit section in STATIC_ROOT in settings.py file with the value "/home/notroot/django_tut/staticfiles/files"
6. set the value for the variable in settings.py STATIC_URL = '/static/'
7. write a template in the template dir for example index.html
{% load staticfiles %}
  <a href="{% static 'files1.txt' %}">link1</a>
  <a href="{% static 'files2.txt' %}">link2</a>
8. update the TEMPLATE_DIRS option in the settings.py.
9. Add a view in the views.py and update urls.py file
10. In the project root folder run the command "python manage.py collectstatic"
out put would be

You have requested to collect static files at the destination
location as specified in your settings.

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes

0 static files copied, 73 unmodified.
11. run the server, access the url from the browser

http://127.0.0.1:8000/staticfiles/
  
12 Download the tutorial from the github

Note: This is compatible with django 1.5.4

Django Logging Tutorial

1.Update LOGGING section in settings.py with logger,handler,formatter fields
  Logger which specifies the handler, in this case it is file
 'loggers': {
        'logapp':{
            'handlers': ['file'],
            'level': 'DEBUG',
        },
    }
  File handler specifies the filename "logapp.log", also specifies theformatter in this case it is simple.
  'handlers': {
        'file':{
            'level':'DEBUG',
            'class':'logging.FileHandler',
            'filename':'logapp.log',
            'formatter':'simple'
        },
    }
  Formatter specifies the formatter in this case it is 'simple'.
 'formatters':{
        'simple':{
           'format':'%(levelname)s %(message)s'
        }
    },
2. import logging modules and add the logger message lines whereever it is required to have log message.

    logger.debug("This is debug message")
    logger.error("This is error message")

3. log messages will be printed in the log file in this case "logapp.log"
   http://127.0.0.1:8000/logproject/

4.Source Code can be downloaded from the github
https://github.com/Jadatravu/Tutorials/tree/master/django_tutorials/logproject


Note: This tutorial is compatible with django 1.5.4, python 2.7.4

Django Helloworld tutorial        
1. Install django from pip or apt-get/yum
2. Install mysql, python-mysqldb pacakge from apt-get/yum
3. Create database in mysql using steps mentioned in the url creating-database   
4. Creating mysql user and granting database permissions to the created user using the steps mentioned in the url adding-users
5. Edit helloworld/settings.py for database settings [ Database(mysql), Database name, Database user, Database user password]
6. Run server in the helloworld directory python manage.py runserver

7. Open the url http:/127.0.0.1:8000/helloworld

8.Source Code can be downloaded from the github
https://github.com/Jadatravu/Tutorials/tree/master/django_tutorials/helloworld

Note: This tutorial is compatible with django 1.5.4, python 2.7.4

Deploying Django application. http://tinyurl.com/odz9hfx

1. Install libapache2-mod-wsgi using apt-get utility.

2. Add below lines to /etc/conf/apache2.conf

WSGIScriptAlias / /home/notroot/django_tut/helloworld/helloworld/wsgi.py
WSGIPythonPath /home/notroot/django_tut/helloworld/
<Directory /home/notroot/django_tut/helloworld/>
<Files wsgi.py>
Order deny,allow
Require all granted
Satisfy Any
</Files>
</Directory>

3. reload apache2 "service apache2 reload"

4. access url/view from the browser example : http://127.0.0.1/helloworld

Database relationships are very well explained. http://tinyurl.com/yktcdlm

• One to One Relationships
• One to Many and Many to One Relationships
• Many to Many Relationships
• Self Referencing Relationships