Celery with django

Celery can use with django for the pupose of running background tasks. Using celery, we can avoid python threads inside view function in django.

First,
Creating django project "celerydjango"

django-admin startproject celerydjango

project structure
-------------------
|-- celerydjango
|   |-- __init__.py
|   |-- settings.py
|   |-- urls.py
|   `-- wsgi.py
`-- manage.py

cd celerydjango

create django app "celeryapp"

------------------------------------

django-admin startapp celeryapp

project structure

-------------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

in settings.py

----------------

add 'celeryapp' in INSTALLED_APPS tuple.

check whether app is ready:

python manage.py runserver

Performing system checks...

System check identified no issues (0 silenced).

January 16, 2015 - 17:01:59

Django version 1.7.3, using settings 'celerydjango.settings'

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C.

yeah it is working...

Now we have to setup celery

sudo pip install celery

then create a celery.py file in project/project directory

----------------------------------------------------------------

touch celerydjango/celery.py

proj structure

----------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

then,

cd celerydjango

in celery.py

--------------

from __future__ import absolute_import

import os

from celery import Celery

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celerydjango.settings')

app = Celery('celerydjango')

app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

@app.task(bind=True)

def debug_task(self):

    print('Request: {0!r}'.format(self.request))

Now we have to install rabbitmq which is used as queue for celery

Now a little point---what celery actually do is, it act as a consumer for the queue.When a task is inserted to the queue, it begin to run that task.That is celery is always looking for queue whether new task is coming in the queue. In django it uses a default queue(name is celery and user is guest) if queue is not explicitily defined.

So install rabbitmq

-----------------------

sudo apt-get install rabbitmq-server

then run celery

-----------------------

celery -A celerydjango worker -l info

you can see

[tasks]

  . celerydjango.celery.debug_task

to test a task

--------------

in celery.py, add

--------------------

@app.task

def test():

    print('testing celery task')

then run

-----------

celery -A celerydjango worker -l info

you can see

--------------

[tasks]

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah the celery setup is ready now..

Now its time to add task from django views

-----------------------------------------------------

Here i am not going to tell html or static part.Just how can use celery with django.

Another matter about celery is....

celery will check all tasks defined in tasks.py file inside all apps defined in INSTALLED_APPS.

so let us create a tasks.py

touch celeryapp/tasks.py

proj stucture

--------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tasks.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   |-- wsgi.py

`-- manage.py

in tasks.py

-------------

add,

from celery import task

@task

def test1():

    print "testing..."

check whether task is ok.

------------------------------

celery -A celerydjango worker -l info

output:

---------

[tasks]

  . celeryapp.tasks.test1

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah our task is taken by celery....

one thing for checking whether queue is created

-------------------------------------------------------

sudo rabbitmqctl list_queues

Listing queues ...

celery 0

yeah default celery queue is created...

Then i am setting an url for celery testing

in celerydjango/urls.py

------------

from django.conf.urls import patterns, include, url

from django.contrib import admin

from celeryapp import views

urlpatterns = patterns('',

    url(r'^admin/', include(admin.site.urls)),

    url(r'^testcelery/$', views.testcelery, name='testcelery'),

)

in celeryapp/views.py

--------------------------

from django.shortcuts import render

from django.http import HttpResponse

from celeryapp.tasks import test1

def testcelery(request):

    test1.delay()

    return HttpResponse("celery is running now, you can check in celery tab")

Now all setups are ready for testing....

run celery in  new tab.

---------------------------

celery -A celerydjango worker -l info

run django in new tab

--------------------------

python manage.py runserver

go to  http://localhost:8000/testcelery/

then check tab where celery run.

we can find..

[2015-01-16 18:46:16,707: WARNING/MainProcess] celery@latheef-Ideapad-S100 ready.

[2015-01-16 18:46:21,508: INFO/MainProcess] Received task: celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0]

[2015-01-16 18:46:21,514: WARNING/Worker-2] testing...

[2015-01-16 18:46:21,517: INFO/MainProcess] Task celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0] succeeded in 0.00535557200055s: None

Yeah it is working....

Now some points..

passing arguments

@task

def test1(a, b):

    #stuff here with a, b

then call as test1.delay(3, 4)

checking task status

result = test1.delay()

check status by

from celery.result import AsyncResult

AsyncResult(result.id).state gives status of task

Naming custom task id

result = test1.apply_async(args=[], kwargs={}, task_id='customid')

AsyncResult('customid').state gives status of task

Now you are ready to use celery. Enjoy with its powerful functionalities....

Mongo Authentication is really Simple :)

Lets talk about Mongo Auth. We know that an "authentication" should be based on username,password,roles, etc.

First start mongod without auth

sudo mongod

OR

service mongodb start

Then take mongo shell

$mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin

>db.addUser('user','pass')
{
"user" : "user",
"readOnly" : false,
"pwd" : "e0c4a7b97d4db31f5014e9694e567d6b",
"_id" : ObjectId("53fc7c5f4e77b510f6f4046f")
}

Then, Start mongod instance in --auth mode.

sudo mongod --auth

OR

add auth=true in mongodb.conf in /etc/ and restart mongodb service.


Then take mongo shell

$mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin

>show collections
Tue Aug 26 17:53:04.682 error: {
"$err" : "not authorized for query on admin.system.namespaces",
"code" : 16550
} at src/mongo/shell/query.js:128

>db.auth('user','pass')

1

yeah...now you are authenticated.You can access all databases.

> db.logout()

{ "ok" : 1 }

Since admin database is a default in every mongodb and act as admin.Users in admin can access all databases.Another thing is admin user can set readOnly by passing 'true' as argument.

db.addUser('user','pass','true')

Important thing is,

In mongodb, Users are created for databases.

for example,if you want to create user for database 'testdatabase':

>use testdatabase

testdatabase

db.addUser('user','pass')

Here you are ready.You can add roles for users

Mongodb (NoSQL) in Python-PyMongo

          Mongodb is an opensource document database in which data are stored as documents.That is,a database includes many collections and collections includes many documents.A document is  field and value pairs.Value may be other document,arrays,or arrays of documents.For a python programmer,document may be considered as as a dictionary and collection can be considered as a list of dicts.

example of a document:

{"author": "Mike","text": "My first blog post!"}

Here i am defining how to connect mongo db using Python program.This might help in Flask framework.

First you need to install pymongo

sudo pip install pymongo

In flask route program,

import pymongo
from pymongo import MongoClient

Then create mongo instance,

client=MongoClient()  # here in flask,it comes below of  app=Flask(__name__)

db=client.testdatabase  # testdatabase is the database in mongodb

to connect with a collection,

collection=db.testcollection  # testcollection is a collection in testdatabase.

Insertion of data:

data={"author": "Mike","text": "My first blog post!"}
collection.insert(data)

Remove of data:

collection.remove()

retrieve of data:

collection.find()

for a specific document(filtering),

collection.find({'author':'Mike'})

Operations can be done as what doing with list of dict in python.for eg:

for i in collection.find():
print i

This gives each documents(dict in python) in the collection(list in python)

I designed a Paint App in Flask with Mongodb as database using a save button.
To see the code Click here

Deploy Django app in heroku

Here is how a Django app deploy to heroku.Django app should be success in your local host.Then read carefully,it is something confusing.But leave it,i will help you..


Your application should be like this..

project
-------project
--------------__init.py
--------------setings.py
--------------urls.py
--------------wsgi.py
-------manage.py


This setup will get using commands

django-admin.py startproject project

then,

cd project
django-admin.py startapp paint  #here i am adding a paint app

Care that we are in the main project folder,

Then Whole project should be like this..


project
-------project
--------------__init.py
--------------setings.py
--------------urls.py
--------------wsgi.py
-------paint
--------------__init__.py
--------------models.py
--------------tests.py
--------------views.py
-------manage.py
-------requirements.txt
-------Procfile
-------venv


Don't be upset,we should cover how these gets...

In settings.py we have to add some codes..

import urlparse
import dj_database_url
DATABASES['default'] =  dj_database_url.config()
# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# Allow all host headers
ALLOWED_HOSTS = ['*']
# Static asset configuration
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = 'staticfiles'
STATIC_URL = '/static/'
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)


Add this line also

here = lambda * x: os.path.join(os.path.abspath(os.path.dirname(__file__)), *x)
PROJECT_ROOT = here("..")
root = lambda * x: os.path.join(os.path.abspath(PROJECT_ROOT), *x)
TEMPLATE_DIRS = (
    root('templates'),
)

Delete already existing template directory in your settings.py.But dont edit already existing database

your wsgi.py shoulbe

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())

Then run command

Heroku login     #to enter username and password
heroku create websitename   #should like www.websitename.herokuapp.com
heroku addons:add heroku-postgresql:dev
heroku config:set HEROKU=1
heroku pg:promote HEROKU_POSTGRESQL_PINK   #PINK is here only your's can be find using
virtualenv venv --distribute   #virtual environment for flask
source venv/bin/activate
pip freeze > requirements.txt

requirements.txt should be like this

Django==1.6
argparse==1.2.1
distribute==0.6.34
dj-database-url==0.2.2
dj-static==0.0.5
gunicorn==18.0
psycopg2==2.5.1
static==0.4
wsgiref==0.1.2

then,

pip install -r requirements.txt
git init

Create a Procfile using vi Procfile ('P' is capital) and add following line in it

web: gunicorn project.wsgi

Also add a .gitignore file using vi .gitignore and add following lines in it

venv
*.pyc
*.py~

Then,

git add .
git commit -m "hosting"
git push heroku master 

After launching it,

heroku pg:promote HEROKU_POSTGRESQL_PINK   #PINK is here only your's can be find using

heroku config | grep HEROKU_POSTGRESQL

heroku run python manage.py syncdb

Then,'

heroku ps:scale web=1
heroku restart --app websitename

heroku open

Then you can see it is working......!!!!!!

heroku logs #for further info

If u have any doubts just clone my simple paint app using,

heroku git:clone -a latheefdjangopaint

and verify the codes and structure of Django app

Deploy Flask app in Heroku

In a flask app,database let us consider postgresql

Now we have to deploy Flask app in heroku which is success in local host.First,

In our app,database connection is  like this

con = psycopg2.connect(database='firstdb') 
cur = con.cursor()

Instead of this,use....

urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["DATABASE_URL"])
con=psycopg2.connect(database=url.path[1:],user=url.username,password=url.password,host=url.hostname,port=url.port)

also import some modules

import os
import psycopg2
import urlparse

wherever database to connect,we should connect like above in each function..

Next in routes.py or the routing python program.our app includes

app.run(debug=True)

Instead of this...

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app.run(host='0.0.0.0', port=port)


Then also add

app.config.from_object(__name__)

in the below of app = Flask(__name__)..so it should be like this..

app = Flask(__name__)
app.config.from_object(__name__)


Then,

Create account on heroku.com
open terminal and enter

Heroku login     #to enter username and password
cd appfloder      #appfolder includes route.py,templates etc...
heroku create websitename   #should like www.websitename.herokuapp.com
heroku addons:add heroku-postgresql:dev   #creating database
heroku config:set HEROKU=1
virtualenv venv --distribute   #virtual environment for flask
source venv/bin/activate
pip freeze > requirements.txt

requirements.txt should be like this

Flask==0.10.1
Jinja2==2.7.1
MarkupSafe==0.18
Werkzeug==0.9.4
argparse==1.2.1
distribute==0.6.34
gunicorn==18.0
itsdangerous==0.23
psycopg2==2.5.1
wsgiref==0.1.2

then,

pip install -r requirements.txt
git init

Create a Procfile using vi Procfile ('P' is capital) and add following line in it

web: python routeprogam.py #here route program is yours

Also add a .gitignore file using vi .gitignore and add following 3 lines in it.This is for avoiding the virtual env files to deploy.

venv
*.pyc
*.py~

Then,


git add .
git commit -m "hosting"
git push heroku master 

After launching it,

heroku pg:promote HEROKU_POSTGRESQL_PINK   #PINK is here only your's can be find using

heroku config | grep HEROKU_POSTGRESQL

Then,'

heroku ps:scale web=1
heroku restart --app websitename

heroku open

Then you can see it is working......!!!!!!

heroku logs #for further info

If u have any doubts just clone my simple microblog app using,

heroku git:clone -a latheefmicroblog

and verify the codes and structure of Flask app

Unit testing in Python

Unit testing may be define as testing each units of a program.Unit testing in python has a separate testing python program for original program.

Let us explain with an example:
i want to create a program which accepts a number and check whether odd or even.

In python unit testing testing,we must assume the original program to predict requirements,inputs,outputs,errors etc..

Let us assume original program is oddeven.py and test program is testoddeven.py.

First we must know the requirements,inputs,outputs,errors..
Input may be int,string,float etc..
So errors may arise when input is string,float,zero.

So lets start our testoddeven.py

import oddeven #our original program
import unittest #python inbuilt module

class KnownValues(unittest.TestCase):
knownValues=((1,'odd'),(2,'even'),(11,'odd'))

Here some input and expected output is defined in a class

import oddeven #our original program
import unittest #python inbuilt module

class KnownValues(unittest.TestCase):
knownValues=((1,'odd'),(2,'even'),(11,'odd'))
def testoddeven(self):
for i,j in self.knownValues:
result=oddeven.testoe(i)
self.assertEqual(j,result)

Here checking whether the output equals to output of our oddeven.py Now we have to design errors or assume errors.Here possible errors are
zero,float value,string.Now i am defining expected errors.

import oddeven
import unittest

class KnownValues(unittest.TestCase):
knownValues=((1,'odd'),(2,'even'),(11,'odd'))
 
           def testoddeven(self):
for i,j in self.knownValues:
result=oddeven.testoe(i)
self.assertEqual(j,result)
class OddEvenBadInput(unittest.TestCase):
def testZero(self):
self.assertRaises(oddeven.ZeroError, oddeven.testoe,0)
def testNonInteger(self):
self.assertRaises(oddeven.NonIntegerError, oddeven.testoe,6.5)
def testString(self):
self.assertRaises(oddeven.StringError, oddeven.testoe,"abc")

if __name__ == "__main__":
unittest.main() 


Then start coding oddeven.py Then define the errors in original program sa below.

class ZeroError(Exception):pass
class NonIntegerError(Exception):pass
class StringError(Exception):pass


Then define each line and test each time using python testoddeven.py Rewrite the program until 'ok' gets...




class ZeroError(Exception):pass
class NonIntegerError(Exception):pass
class StringError(Exception):pass

def testoe(n):
if type(n)==int:
if n==0:raise ZeroError,"zero..."
elif n%2==0:
return "even"
else:
return "odd"
elif type(n)==str:
raise StringError,"string...!!!!"
else: raise NonIntegerError,"Non int...."

unit testing is a method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use. Unit tests are created by programmers or occasionally by white box testers during the development process.One can view a unit as the smallest testable part of an application.Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended.

To view the codes.Click here

Python flask microblog application using postgresql

   
As in the previous example,there used sqlite3.Here database is postgresql

    Blog app act as a small scale blogspot account which provides options such as posting commenting etc.Blog app uses postgresql database to store the post and comments. psycopg2 module provided by python is used for it.Posting can be done only by the admin. Admin has to log in to post to avoid fake users.

documentation on flask


These kind of things takes a lot of work to code in normal networking in python.  Flask makes everything easier.Also flask helps the debugging much easier. The blog app is my first work in flask and it didn't give me much headache.
Let me show you some basic items in flask.

@app.route('url')
def func_name():
.....

Here first line tells us 'Where to go'.For example @app.route('/') means home page, @app.route('/post') means post link(ie home/post/).

Under that we define the function to tell the app 'how to behave' when these links are provided. We can render an html page, read posts from forum, save data etc.

In short if have a 'not bad' knowledge in flask then web based apps can be easily made. Flask is a great web framework provided by python where rookies can have a shot too.

Postgresql setup:

    sudo apt-get install libpq-dev python-dev

    sudo apt-get install postgresql

    sudo -u postgres createuser --superuser $USER

    sudo -u postgres psql 

    sudo -u postgres createdb $USER

Now you can create database in terminal using command psql

    create database databasename;

    \c databasename;   #to connect database

    create table posts(id integer,title TEXT,text INT);

    \d   #show tables

    \d tablename #to show corresponding table

    then use select query and more...

Connection:

To connect postgresql,we must install psycopg2

    sudo apt-get install python-psycopg2

 
In a program connection done through this lines.for example:

import psycopg2
con = psycopg2.connect(database='firstdb')
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS paintstore1")
cur.execute("CREATE TABLE paintstore1(id serial,title text,imagedata text)")
con.commit()
con.close()

To view the codes: Click here
To view the site i deployed to heroku,click here