Celery with django

Celery can use with django for the pupose of running background tasks. Using celery, we can avoid python threads inside view function in django.

First,
Creating django project "celerydjango"

django-admin startproject celerydjango

project structure
-------------------
|-- celerydjango
|   |-- __init__.py
|   |-- settings.py
|   |-- urls.py
|   `-- wsgi.py
`-- manage.py

cd celerydjango

create django app "celeryapp"

------------------------------------

django-admin startapp celeryapp

project structure

-------------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

in settings.py

----------------

add 'celeryapp' in INSTALLED_APPS tuple.

check whether app is ready:

python manage.py runserver

Performing system checks...

System check identified no issues (0 silenced).

January 16, 2015 - 17:01:59

Django version 1.7.3, using settings 'celerydjango.settings'

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C.

yeah it is working...

Now we have to setup celery

sudo pip install celery

then create a celery.py file in project/project directory

----------------------------------------------------------------

touch celerydjango/celery.py

proj structure

----------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

then,

cd celerydjango

in celery.py

--------------

from __future__ import absolute_import

import os

from celery import Celery

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celerydjango.settings')

app = Celery('celerydjango')

app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

@app.task(bind=True)

def debug_task(self):

    print('Request: {0!r}'.format(self.request))

Now we have to install rabbitmq which is used as queue for celery

Now a little point---what celery actually do is, it act as a consumer for the queue.When a task is inserted to the queue, it begin to run that task.That is celery is always looking for queue whether new task is coming in the queue. In django it uses a default queue(name is celery and user is guest) if queue is not explicitily defined.

So install rabbitmq

-----------------------

sudo apt-get install rabbitmq-server

then run celery

-----------------------

celery -A celerydjango worker -l info

you can see

[tasks]

  . celerydjango.celery.debug_task

to test a task

--------------

in celery.py, add

--------------------

@app.task

def test():

    print('testing celery task')

then run

-----------

celery -A celerydjango worker -l info

you can see

--------------

[tasks]

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah the celery setup is ready now..

Now its time to add task from django views

-----------------------------------------------------

Here i am not going to tell html or static part.Just how can use celery with django.

Another matter about celery is....

celery will check all tasks defined in tasks.py file inside all apps defined in INSTALLED_APPS.

so let us create a tasks.py

touch celeryapp/tasks.py

proj stucture

--------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tasks.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   |-- wsgi.py

`-- manage.py

in tasks.py

-------------

add,

from celery import task

@task

def test1():

    print "testing..."

check whether task is ok.

------------------------------

celery -A celerydjango worker -l info

output:

---------

[tasks]

  . celeryapp.tasks.test1

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah our task is taken by celery....

one thing for checking whether queue is created

-------------------------------------------------------

sudo rabbitmqctl list_queues

Listing queues ...

celery 0

yeah default celery queue is created...

Then i am setting an url for celery testing

in celerydjango/urls.py

------------

from django.conf.urls import patterns, include, url

from django.contrib import admin

from celeryapp import views

urlpatterns = patterns('',

    url(r'^admin/', include(admin.site.urls)),

    url(r'^testcelery/$', views.testcelery, name='testcelery'),

)

in celeryapp/views.py

--------------------------

from django.shortcuts import render

from django.http import HttpResponse

from celeryapp.tasks import test1

def testcelery(request):

    test1.delay()

    return HttpResponse("celery is running now, you can check in celery tab")

Now all setups are ready for testing....

run celery in  new tab.

---------------------------

celery -A celerydjango worker -l info

run django in new tab

--------------------------

python manage.py runserver

go to  http://localhost:8000/testcelery/

then check tab where celery run.

we can find..

[2015-01-16 18:46:16,707: WARNING/MainProcess] celery@latheef-Ideapad-S100 ready.

[2015-01-16 18:46:21,508: INFO/MainProcess] Received task: celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0]

[2015-01-16 18:46:21,514: WARNING/Worker-2] testing...

[2015-01-16 18:46:21,517: INFO/MainProcess] Task celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0] succeeded in 0.00535557200055s: None

Yeah it is working....

Now some points..

passing arguments

@task

def test1(a, b):

    #stuff here with a, b

then call as test1.delay(3, 4)

checking task status

result = test1.delay()

check status by

from celery.result import AsyncResult

AsyncResult(result.id).state gives status of task

Naming custom task id

result = test1.apply_async(args=[], kwargs={}, task_id='customid')

AsyncResult('customid').state gives status of task

Now you are ready to use celery. Enjoy with its powerful functionalities....

Mongo Authentication is really Simple :)

Lets talk about Mongo Auth. We know that an "authentication" should be based on username,password,roles, etc.

First start mongod without auth

sudo mongod

OR

service mongodb start

Then take mongo shell

$mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin

>db.addUser('user','pass')
{
"user" : "user",
"readOnly" : false,
"pwd" : "e0c4a7b97d4db31f5014e9694e567d6b",
"_id" : ObjectId("53fc7c5f4e77b510f6f4046f")
}

Then, Start mongod instance in --auth mode.

sudo mongod --auth

OR

add auth=true in mongodb.conf in /etc/ and restart mongodb service.


Then take mongo shell

$mongo
MongoDB shell version: 2.4.9
connecting to: test
> use admin
switched to db admin

>show collections
Tue Aug 26 17:53:04.682 error: {
"$err" : "not authorized for query on admin.system.namespaces",
"code" : 16550
} at src/mongo/shell/query.js:128

>db.auth('user','pass')

1

yeah...now you are authenticated.You can access all databases.

> db.logout()

{ "ok" : 1 }

Since admin database is a default in every mongodb and act as admin.Users in admin can access all databases.Another thing is admin user can set readOnly by passing 'true' as argument.

db.addUser('user','pass','true')

Important thing is,

In mongodb, Users are created for databases.

for example,if you want to create user for database 'testdatabase':

>use testdatabase

testdatabase

db.addUser('user','pass')

Here you are ready.You can add roles for users

Mongodb (NoSQL) in Python-PyMongo

          Mongodb is an opensource document database in which data are stored as documents.That is,a database includes many collections and collections includes many documents.A document is  field and value pairs.Value may be other document,arrays,or arrays of documents.For a python programmer,document may be considered as as a dictionary and collection can be considered as a list of dicts.

example of a document:

{"author": "Mike","text": "My first blog post!"}

Here i am defining how to connect mongo db using Python program.This might help in Flask framework.

First you need to install pymongo

sudo pip install pymongo

In flask route program,

import pymongo
from pymongo import MongoClient

Then create mongo instance,

client=MongoClient()  # here in flask,it comes below of  app=Flask(__name__)

db=client.testdatabase  # testdatabase is the database in mongodb

to connect with a collection,

collection=db.testcollection  # testcollection is a collection in testdatabase.

Insertion of data:

data={"author": "Mike","text": "My first blog post!"}
collection.insert(data)

Remove of data:

collection.remove()

retrieve of data:

collection.find()

for a specific document(filtering),

collection.find({'author':'Mike'})

Operations can be done as what doing with list of dict in python.for eg:

for i in collection.find():
print i

This gives each documents(dict in python) in the collection(list in python)

I designed a Paint App in Flask with Mongodb as database using a save button.
To see the code Click here