Celery with django

Celery can use with django for the pupose of running background tasks. Using celery, we can avoid python threads inside view function in django.

First,
Creating django project "celerydjango"

django-admin startproject celerydjango

project structure
-------------------
|-- celerydjango
|   |-- __init__.py
|   |-- settings.py
|   |-- urls.py
|   `-- wsgi.py
`-- manage.py

cd celerydjango

create django app "celeryapp"

------------------------------------

django-admin startapp celeryapp

project structure

-------------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

in settings.py

----------------

add 'celeryapp' in INSTALLED_APPS tuple.

check whether app is ready:

python manage.py runserver

Performing system checks...

System check identified no issues (0 silenced).

January 16, 2015 - 17:01:59

Django version 1.7.3, using settings 'celerydjango.settings'

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C.

yeah it is working...

Now we have to setup celery

sudo pip install celery

then create a celery.py file in project/project directory

----------------------------------------------------------------

touch celerydjango/celery.py

proj structure

----------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   `-- wsgi.py

`-- manage.py

then,

cd celerydjango

in celery.py

--------------

from __future__ import absolute_import

import os

from celery import Celery

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celerydjango.settings')

app = Celery('celerydjango')

app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

@app.task(bind=True)

def debug_task(self):

    print('Request: {0!r}'.format(self.request))

Now we have to install rabbitmq which is used as queue for celery

Now a little point---what celery actually do is, it act as a consumer for the queue.When a task is inserted to the queue, it begin to run that task.That is celery is always looking for queue whether new task is coming in the queue. In django it uses a default queue(name is celery and user is guest) if queue is not explicitily defined.

So install rabbitmq

-----------------------

sudo apt-get install rabbitmq-server

then run celery

-----------------------

celery -A celerydjango worker -l info

you can see

[tasks]

  . celerydjango.celery.debug_task

to test a task

--------------

in celery.py, add

--------------------

@app.task

def test():

    print('testing celery task')

then run

-----------

celery -A celerydjango worker -l info

you can see

--------------

[tasks]

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah the celery setup is ready now..

Now its time to add task from django views

-----------------------------------------------------

Here i am not going to tell html or static part.Just how can use celery with django.

Another matter about celery is....

celery will check all tasks defined in tasks.py file inside all apps defined in INSTALLED_APPS.

so let us create a tasks.py

touch celeryapp/tasks.py

proj stucture

--------------

|-- celeryapp

|   |-- admin.py

|   |-- __init__.py

|   |-- migrations

|   |   `-- __init__.py

|   |-- models.py

|   |-- tasks.py

|   |-- tests.py

|   `-- views.py

|-- celerydjango

|   |-- celery.py

|   |-- __init__.py

|   |-- settings.py

|   |-- urls.py

|   |-- wsgi.py

`-- manage.py

in tasks.py

-------------

add,

from celery import task

@task

def test1():

    print "testing..."

check whether task is ok.

------------------------------

celery -A celerydjango worker -l info

output:

---------

[tasks]

  . celeryapp.tasks.test1

  . celerydjango.celery.debug_task

  . celerydjango.celery.test

yeah our task is taken by celery....

one thing for checking whether queue is created

-------------------------------------------------------

sudo rabbitmqctl list_queues

Listing queues ...

celery 0

yeah default celery queue is created...

Then i am setting an url for celery testing

in celerydjango/urls.py

------------

from django.conf.urls import patterns, include, url

from django.contrib import admin

from celeryapp import views

urlpatterns = patterns('',

    url(r'^admin/', include(admin.site.urls)),

    url(r'^testcelery/$', views.testcelery, name='testcelery'),

)

in celeryapp/views.py

--------------------------

from django.shortcuts import render

from django.http import HttpResponse

from celeryapp.tasks import test1

def testcelery(request):

    test1.delay()

    return HttpResponse("celery is running now, you can check in celery tab")

Now all setups are ready for testing....

run celery in  new tab.

---------------------------

celery -A celerydjango worker -l info

run django in new tab

--------------------------

python manage.py runserver

go to  http://localhost:8000/testcelery/

then check tab where celery run.

we can find..

[2015-01-16 18:46:16,707: WARNING/MainProcess] celery@latheef-Ideapad-S100 ready.

[2015-01-16 18:46:21,508: INFO/MainProcess] Received task: celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0]

[2015-01-16 18:46:21,514: WARNING/Worker-2] testing...

[2015-01-16 18:46:21,517: INFO/MainProcess] Task celeryapp.tasks.test1[d9540751-eaa2-471e-a13c-66c9667783f0] succeeded in 0.00535557200055s: None

Yeah it is working....

Now some points..

passing arguments

@task

def test1(a, b):

    #stuff here with a, b

then call as test1.delay(3, 4)

checking task status

result = test1.delay()

check status by

from celery.result import AsyncResult

AsyncResult(result.id).state gives status of task

Naming custom task id

result = test1.apply_async(args=[], kwargs={}, task_id='customid')

AsyncResult('customid').state gives status of task

Now you are ready to use celery. Enjoy with its powerful functionalities....