A generic method for python nested dictionary

In recent project i was working to integrate the third party library. In response to call i have a json object which holds the data of all the trips. It was a list of dictionaries, each dictionary holding the data about a trip and it was a nested dictionary.

Now situation was at different part of the code has to look out the trip dictionary for given key(s). So i wrote a generic method which will take the key and results the value of that key.  

Following is a simplification of scenario as i described above. 

Here is a python nested dictionary,

dic = {"a": 1, {"b": {"c": 2}}.

Now have generic method which operates on this dictionary, and this method can be called from different places of the program passing key of dictionary.

def get_value(dic, key):
      # return value of key

In gerenal one can get the value from dictionary like:

dic['a']   ---> 1

If i had to fetch the value of 2 from dic , it is quite simple.

dic['b']['c']  ---> 2

Since it is a nested dictionary, how get_value function is to return 2 ?
Here the reduce function comes to help:

reduce(dic.get, ['b', 'c'], dic) 

What it does is,

dic.get('b')  ---> {'c': 2},     #now dic is updated to {'c': 2}
dic.get('c')  ----> 2 

Using reduce helped me to wrote the generic method which can be operated on any python dictionary.

Thanks falsetru helped me to understand this.

My introduction of REST

Colleague: "hey Naveen,  we have to write the API back-end for mobile developers".
Me: "OK,  i can write the API calls".
Colleague: "Make it RESTful".
Me: "RESTful .. ha ....  mmm".

Although i have worked with other REST APIs but never wrote myself.  So it was a good time to understand RESTful concepts before writing code.

REST means Representational state transfer.

History of REST
REST was developed during 1994-1995 as a part of web concepts. HTTP protocol 1.0 and HTTP1.1 lays the conceptual foundation of REST although it has significantly improved a lot over time.
It was initially referred as a "HTTP object model".

Understanding REST
REST system in which a developer has a whole system at his parcel and on applies the constraints. Implementation details of the system does not matters while developing REST interface of the system.
I would like elaborate on the specific meaning of the "Representational state transfer".
RESTful system works as a state machine. A user progress through the application by  choosing the link or passing some parameters from one state to another. User is presented with the state of the application for each action.

REST architectural constraints
Here is the very brief overview of these constraints
Client-Server : client-server model is being used.
Stateless: each request from client to server contains enough information to process itself and does not uses any stored context of server.
Cache: if response is catchable , then client can use that response for equivalent request.
Uniform Interface: identification of resources, manipulation of resource
through representations, self-descriptive messages, and hypermedia as the engine of application state.
Layered system: architecture of the system can be composed in a hierarchical style. There is complete isolation between the hierarchies.
Code on demand: server can transfer the piece of code to client. e.g. javascript code to be excuted by client.


Lot to be shared on REST, keep reading.

Intresting bits from source code

It has been really exciting to dig into the source code of a language  or framework. I have been going through te souce code of python2.7 and Django1.6 from quite a while. Digging the source code certainly enrich his understanding of the system and learn new things right under the hood.

In this post i'm going to share some of those things.

Python2.7: module 'keyord'
Finding an element in a list.

daylist = ["sun", "mon", "tue", "wed", "thrus", "fri", "sat"] 
isday = frozenset(daylist).__contains__

isday("fri")  >> True
isday("abc") >> False

isday is a function object which can be for a string to check if it is present in list or not.


Using logical operators to do if.. else..

>>truth_value = 6 > 33
>>truth_value and "yeah, its true"  or "its a big fat lie"
"its a big fat lie"

This is how its working:
exp1 and exp2
'and' returns a exp1 if exp1 is evaluated to False otherwise evaluates exp2 and return it.

>>truth_value and "yeah, its true"  
False

now the expression becomes like

>>False or "its a big fat lie"
>>"its a big fat lie"

 'or' returns a exp1 if exp1 is evaluated to True otherwise evaluates exp2 and return it.



In Django shell auto completion for python keywords, modules and other imported objects works using rlcompleter module.

>>import rlcompleter
>>import readline
>>readline.parse_and_bind("tab: complete")
>>readline. <TAB PRESSED>
readline.__doc__          
readline.get_line_buffer(  
readline.read_init_file(
readline.__file__         
readline.insert_text(      readline.set_completer(
readline.__name__         
readline.parse_and_bind(

code module provides facilities to interact with the interpreter.
Running python interpreter using python code:

>>import code
>>code.interact()

Similarly if you have ipython or bpython installed

>>import ipython
>ipython.embed()

If you want the interpreter to have some objects at start , try this

>>code.interact(local={'hack': 'hack is fun'})
>>hack
'hack is fun'

So that's it for this post, will post more interesting stuff.

A look at python slicing

In this post i'm going to take a look at basic of python slicing which can be quite useful.
Let say we have ptr = "POINTER" a string field, i'll use that string to show the different types of slicing.

Get first n elemenst using [:n]

ptr[:1]    --> "P"
ptr[:2]   --> "PO"
ptr[:3]   --> "POI"

Remove first n elements using [n:]

ptr[1:]  --> "OINTER"
ptr[2:]  --> "INTER"
ptr[3:]  --> "NTER"

Get last n elements using [-n:]

ptr[-1:]  --> "R"      
ptr[-2:]  --> "ER"    
ptr[-3:]  --> "TER"   

Remove last n elements using [:-n]

ptr[:-1]  --> "POINTE"
ptr[:-2]  --> "POINT" 
ptr[:-3]  --> "POIN"

  
Get sub elemens using [n:m]
It given sub elements from n (inclusive) m(exclusive).

ptr[0:3] --> "POI"

Get sub elemens using [n:-m]
a). first apply [:-m] .i.e. removes last m elements,
b). then show all elements from n (inclusive).
ptr[0:-1]  --> "POINTE"
ptr[2:-3]  --> "IN

 Get sub elemens using [-n:m]
a). first apply [:m] .i.e. get first m elements,
b). get n last elements.
c) get common elements from a) and b)

example :  ptr[-1:7]  -->  'R'

a) ptr[:7] ----> X = 'POINTER' 
b) ptr[-1]  --->  Y = 'R'
c) apply  intersection(X, Y) --> 'R' 

example :  ptr[-1:6]   --> ""

a) ptr[:6]  ----> X = 'POINTE'
b) ptr[-1]  --->  Y = 'R'
c) apply  intersection(X, Y) --> nothing.

One intesting observation in here is :

[-7:1] => 'P'
[-6:2] => 'O'
[-5:3] => 'I'
........
........
 [-1:7] => 'R'

Following code implement above behaviour
ptr_len=len(ptr)+1
for i in range(1, ptr_len):
  ptr_len=ptr_len-1
  print ptr[-ptr_len:i]


Thats it. I hope,  i have demonstrated in very simple to get you understand python slicing.

Creating a excel file in python

To Create a Excel file using python , the xlsxwriter module found pretty cool
                    pip install xlsxwriter

after installing it , lets write a code to create a file.

Create a file and associate it with worksheet

             workbook = xlsxwriter.Workbook('person.xlsx')
             worksheet = workbook.add_worksheet()

In a worksheet, column and row in a worksheet are referenced by their index number, which starts with 0.

Set a width  of all columns : 
      worksheet.set_column(0, index_last_column, number)
above line can also be used to set width of particular column.

Creating a headers in bold :
     bold = workbook.add_format({'bold': 1})
      worksheet.write('A', 'Title', bold)
      worksheet.write('B', 'Name', bold)
      worksheet.write('C', 'Age', bold)
first row whose index is 0 is filled with above headers. 
     
Filling Records
person_data = list(Person.objects.all())
let say we have a list of Person objects in person_data
                row =1 // its not 0, since first row is a header
                col =0
           for person in person_data:
                worksheet.write_string(row, col,        person.title)
                worksheet.write_string(row, col+1,   person.first_name)
                worksheet.write_string(row, col+2,   person.last_name)
                r = r+1


Close the file object :
           workbook.close()

checkout docs for more fun.

CSRF verification failed while making POST request from REST client

Problem :  CSRF verification failed
Recently i was using the chrome REST extension for testing the API calls.

When a POST request is made for a django url using extension, i got 403 forbidden error with message

CSRF verification failed. Request aborted

Why this problem?
I dig to the django docs and found that with every POST request a CSRF token is required.
So it was needed to pass that CSRF token as a value of X-CSRFToken Http header as a part of POST request.

Solution
find the the value of CSRF token and pass it as a value X-CSRFToken header in POST request.
Pretty easy to find CSRF token value by using request object.
 >> request.COOKIES['csrftoken']
'ofDXgUB8kChf7pRkUypjDdtGrQsK8xYb'
or
 >> request.META['CSRF_COOKIE']
'ofDXgUB8kChf7pRkUypjDdtGrQsK8xYb'

Source code : django.http.request

Learning from source code is a best way to grasp the concepts behind  a framework, library or a language.
So I got the copy Django source code and started with module django.http.request module. In this post i'm going to share what i have learned from that.

1.  DEBUG and ALLOWED_HOSTS
When DEBUG is False make sure you have initialized the ALLOWED_HOSTS, otherwise get the SuspiciousOperation error.

A host is either HTTP_HOST or HTTP_X_FORWARDED_HOST. If  DEBUG is True that host is used and if DEBUG is False then host is validated from ALLOWED_HOSTS list.
        # There is no hostname validation when DEBUG=True
        if settings.DEBUG:
            return host
        domain, port = split_domain_port(host)
        if domain and validate_host(domain, settings.ALLOWED_HOSTS):
            return host
        else:
            msg = "Invalid HTTP_HOST header: %r." % host
            if domain:
                msg += "You may need to add %r to ALLOWED_HOSTS." % domain
            else:
                msg += "The domain name provided is not valid according to RFC 1034/1035"
            raise DisallowedHost(msg)

When that host name is not found in ALLOWED_HOSTS it throws error :
SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): localhost:8000
Solution is to add a hostname string in ALLOWED_HOSTS list.


2. Finding request scheme (i.e. http or https)
Inelegant attempt is to check the request.META for 'wsgi.url_scheme'.
 >> request.META[''wsgi.url_scheme']
'http'

Ideally 'scheme' attribute should be related to the request object, something like request.scheme
Up-to the Django version 1.6 there is no 'scheme' attribute for request object.
At this point i was a bit excited to suggest a django ticket for that.
But i found that in development version they are going to associate scheme to request object.


So these two stuff i found , hope to share more in coming posts.

Awesome..... motherfucker

Adaption of one of the most iconic scene from Pulp Fiction for all who loves programming, programming-motherfucker.

From Zedshaw. ..... the man behind the learnpythonthehardway .
 

Using Django Social Auth

I have been trying to integrate a facebook login for my side-project site. There are various modules available but that works for me is django-social-auth.
This module provides the integration of various social platforms like facebook, twitter, google etc.
Please install that module before preceding . It also had a good documentation, which ca be found here.

 In this post i'm just posting about stuff to how to integrate facebook login in your Django app.

Step 1. Create a facebook app.
facebook app which will gives you App ID/API Key and App Secret which will be used in Django settings.

Please make sure that  following app settings: 

  >> Sandbox Mode is ON

  >>  Site URL is a http://127.0.0.1:8000/ . 
        This should be the url from which you going to make a login request.

Step 2. Django Setting for facebook intergation.

AUTHENTICATION_BACKENDS = (
    'social_auth.backends.facebook.FacebookBackend',
    'django.contrib.auth.backends.ModelBackend',
)

FACEBOOK_APP_ID =  "app_id"
FACEBOOK_API_SECRET = "app_secret"

LOGIN_URL          = '/login-form/'
LOGIN_REDIRECT_URL = '/logged-in/'
LOGIN_ERROR_URL    = '/login-error/'
LOGOUT_REDIRECT_URL = '/'



SOCIAL_AUTH_DEFAULT_USERNAME = 'auth_user'
SOCIAL_AUTH_UID_LENGTH = 16
SOCIAL_AUTH_ASSOCIATION_HANDLE_LENGTH = 16
SOCIAL_AUTH_NONCE_SERVER_URL_LENGTH = 16
SOCIAL_AUTH_ASSOCIATION_SERVER_URL_LENGTH = 16
SOCIAL_AUTH_ASSOCIATION_HANDLE_LENGTH = 16
SOCIAL_AUTH_ENABLED_BACKENDS = ('facebbok',)
FACEBOOK_EXTENDED_PERMISSIONS= ['email']

Step 3. Login link in your html page.
 <a href="{% url socialauth_begin 'facebook' %}">Login with Facebook</a> 
you can make that link a bit fancy using facebook login button. but for now that will serve our purpose.

Step 4. Add url for social-auth.

Add follwoing url for social-auth login in urls.py .
url(r'', include('social_auth.urls')),

and 
following url will take to a view , once user is logged successfully.

url(r'^logged-in/$', 'pollsite.views.home', name='home'),

in above line you can use , LOGIN_REDIRECT_URL which is same as 'logged-in'.



When you click on that link, a familiar facebook login page will appear.

Thats it.  If you encounter any problem , please post it as a comment and i'll be happy to help you out.
 And keep reading.

Make existing project as a github repository

Recently i was working on side project on Django and then decided to put it on the github.
First of all you need to create a repository on github with same name as folder you want to put on git.
Then follow following steps :

Step 1: >> git init

Step 2: >> git add .

Step 3: >> git commit -m "setup commit"
               above command will print list of all files.

Step 4:  >>git remote add origin https://github.com/username/reponame.git

Step 5 : >> git push origin master

       if error :
                  To https://github.com/navyad/pollsite.git  ! [rejected]        master -> master (non-fast-forward) error: failed to push some refs to 'https://github.com/navyad/pollsite.git' To prevent you from losing history, non-fast-forward updates were rejected Merge the remote changes (e.g. 'git pull') before pushing again.  See the 'Note about fast-forwards' section of 'git push --help' for details.
       then :
              >> git pull origin master
       
Step 6: >>git push origin master
       

Its done , you can check the git repo.

Custom HTTP 404 error page in Django

Django default HTTP 404 error page looks like this , provided in DEBUG=True in settings.py file.

Following error message is fine for the development , but when you are going for production then you need lot better than that.

In following steps describes how can you serve your custom 404.html page for HTTP 404 error.

Step 1: Create a 404.html file having your error message.

Step 2 : Place 404.html in directory pointed by TEMPLATE_DIRS .

Step 3: In urls.py set handler404 = 'app.views.custom_404'. this will be a you custom view which will return the 404.html page when Http 404 error occurs.

Step 4: Create you custom view in views.py , 

          def custom_404(request):

                    return render_to_response('404.html')

Step: 5 in settings.py , ALLOWED_HOSTS = ['hostname'],  this will be a IP address from which you are running the django project.

Step 6: very important , in settings.py , make DEBUG=False, Only then you will able to see the custom error page.

Hope that now you will easily able to see you custom error page.

Python Code : Part 1

Python is amazing in terms of how once can code in just few lines. I'm starting a series which includes the code snippets from various resources on internet . The objective is to learn something new or something which is hidden in python.

Swap two variables
>> a=7
>> b=9
>> b, a=a, b       // tuple packing and unpacking is happening here.
on right side of = tuple packing is done (a, b) --> (7, 9)
on left side of = tuple unpacking is being done and values are swapped (b,a ) <--- (7 ,9)
>> a
9
>> b
7

Re-calculating a variable based on other variable.
>> a = 3
>> b = a + 3
now if i keep changing value of a then value of b should also be changed for example if a is 5 then b should be 8.
>> b = lambda : a + 3
 >> a =4
>> b() ............. it will pass current value of a to lambda expression.
7
>> a =2
>> b()
5

Apply operator on 2 and 2*3 to make it 6.
>> val = 2 and 2*3
>> val
6
for and operator , if both operand are true then it will return value of last operand , otherwise value of operand which is evaluating to be False.
>> val = 2 and 0   or( 0 and 2)
>> val
0
'and' and 'or' operator does not return True or False but value of expression !!

Tweaking filter() of python

Following is basic list comprehension example which filter out the even numbers from a list.
>> [ x for x in [1,2,3,4] if  if n%2==0]
[2, 4]

Same can be achieved from filter(func, sequence). And for that we have to write the filter logic in a function, say poi().

>> def pp(n):
...     if n%2==0:
...             return True
...     else:
...             return False
...

>> filter(pp, [1,2,3,4])
[2, 4]

Upto here it seems that the filter(func, seq) takes a function as a first argument and sequence type (list, tuple , string, unicode or xrange() ) as a second argument. And for each element of sequence type it runs the function and returns elements of type seq.
The elements of returned list are initialized from the function, for which function returns True.

To see how filter() behaves ,return type of poi() has been played with.
Tweak 1: what if return type is False or None ?
def poi(x):
    return X    //X =False or None
>> filter(poi, [1,2,3,4])
[]

Tweak 2: what if i return type is string ?

Lets say for 2 in list , i want to return "two". 

def poi(x):
    if x==2:
        return "two"
>> filter(poi, [1,2,3,4])
[1, 2, 3]

Tweak 3: what if i return type is int ?
Lets say for 2 in list , i want to return 4.
def poi(x):
    if x==2:
        return x*x
>> filter(poi, [1,2,3,4])
[1, 2, 3]

What i learned :

1. filter(func, seq) will create a list of elements for which the function , poi() , returns True.
2. if poi() returns False or None for an element then resultant list will be an empty list.
3. if poi() returns other than boolean,  then it will return given list of filter.

What i understood:

say i have a sequence S having n elements and i want to filter the elements on some boolean condition, then filter() can rescue me.  Note that resultant sequence type will have all or some elements of given sequence for which boolean condition is True.