positional-only-parameter in Python

In recent release of python 3.8, new feature  positional-only-parameter is introduced.
I am going to cover motivations behind positional-only-parameter.
There are already some functions which takes positional arguments only like range(), id(), len().

def add(x, y):
    return x+y

Motivations:
1. Programmer does not want to expose the arguments to caller.
  in add(x=3, y=4)  x and y directly exposed to caller.

2. Tomorrow if programmer want to refactor the code and changes parameters name then existing call to the function will break.
if add(x, y) is changed to add(a, b) then add(x=3, y=4) will break. As a result of this programmer has to maintain the parameter's name.

3. add(*args, **kwargs) is not clear to caller as *args can take number of arguments without specifying any meaning of those.

4 When subclass overrides method of base class and changes the name of parameters.
class Sup(object):
  def add_one(self, x)
    return x+1

class Sub(Sup):
  def add_one(self, y)
    return y+1

Sub().add_one(x=1)
which results in error:
TypeError: add_one() got an unexpected keyword argument 'x'

5. Maintaining the logical order of arguments.
def get_args(a, b, c):
   return (a, b, c)

get_args(a=1, c=3, b=2)
get_args(c=9, a=7, b=5)

We can clearly see order of arguments is not followed in calls. positional-only-parameter will allow to maintain the order of calls.


Hope you like this article, Thanks for reading.

Feature request for python's requests library

Recently as apart of day job I had to count the number of API requests were made to keep check of throttling limits. As a standard for python codebase, requests library has been used.
I wrote some code which will maintain the API URLs and their respective counts.
That got thing done.

After a while I realized, there should be a way to intercept the request. Just like in there is middleware in Django for request. Then checked the documentation of requests and found event-hooks.

Going through the docs found out that hook is available only for response.  And they have intentions to add other hooks as well mentioned in TODO.
Found this opportunity to contribute one of the best libraries of python and without any delay I created github issue

Hopefully library maintainers will have a look and will add as a feature request officially.

python: new-style and old-style class

A new-style class inherits from object (or type) where as old-style class does not. In python 3.0 old-style class is removed.

In this post I am going to explore differences between the two.

1: types

Here we can seen that type of obj is not of class but of built-in type instance.  Attribute __class__ of obj gives information about class to which it belongs.
So type and __class__ attributes differ for old-style class.

Lets see what happens with New-style class (python 3.6): 

type and __class__ attributes does not differ for new-style class.


2: MRO (Method Resolution Order)

In addition to above differences, old-style class does not support mro.

whereas new-style class does have that:

Also one can see that new-style class inherits from a object.


3: super
super() does not works with old-style classes.

Works with new-style

the reason being is that super() expect type object for class In case of old-style type(ABC) is classobj not type But in case of new-style type(NewABC) is type.

4: descriptor protocol
New-style classes have implemented descriptor protocol whereas old-style does not.


5: raising an exception
In new style objects cannot be raised as exception unless derived from Exception.

Thanks for reading.

repr and str in python

The purpose of repr() and str() is to display information about object. But there is slight distinction between the two. In this post we will explore that.

By definition repr() method is official representation of object. That means if i want to create an object repr should be descriptive enough to give that information.
Lets take an example of datetime object.

>> from datetime import datetime
>> now_datetime = datetime.now()
>> repr(now_datetime)

prints

>> 'datetime.datetime(2017, 10, 2, 11, 0, 7, 382330)'

so if a want to create a new datetime object repr should be referred for that matter.

new_datetime = datetime.datetime(2017, 11, 12, 15, 56, 9, 343831)

Thats exactly the purpose of repr.

As per doc str() is a 'informal' representation of object. It has be string and you can put whatever you feel like.

>> str(now_datetime)
>> '2017-10-02 11:00:07.382330'


Now the difference
str() it should only return valid string object whereas repr() should either return string or a python expression. 
In case of python expression which can be used to create object.

>> make_this_repr = "x+1"
>> repr(make_this_repr)
>> "'x+1'"
>> x =10
>> eval(make_this_repr)
>> 11


 
Hope you find it useful, thanks for reading.

A generic method for python nested dictionary

In recent project i was working to integrate the third party library. In response to call i have a json object which holds the data of all the trips. It was a list of dictionaries, each dictionary holding the data about a trip and it was a nested dictionary.

Now situation was at different part of the code has to look out the trip dictionary for given key(s). So i wrote a generic method which will take the key and results the value of that key.  

Following is a simplification of scenario as i described above. 

Here is a python nested dictionary,

dic = {"a": 1, {"b": {"c": 2}}.

Now have generic method which operates on this dictionary, and this method can be called from different places of the program passing key of dictionary.

def get_value(dic, key):
      # return value of key

In gerenal one can get the value from dictionary like:

dic['a']   ---> 1

If i had to fetch the value of 2 from dic , it is quite simple.

dic['b']['c']  ---> 2

Since it is a nested dictionary, how get_value function is to return 2 ?
Here the reduce function comes to help:

reduce(dic.get, ['b', 'c'], dic) 

What it does is,

dic.get('b')  ---> {'c': 2},     #now dic is updated to {'c': 2}
dic.get('c')  ----> 2 

Using reduce helped me to wrote the generic method which can be operated on any python dictionary.

Thanks falsetru helped me to understand this.

A look at python slicing

In this post i'm going to take a look at basic of python slicing which can be quite useful.
Let say we have ptr = "POINTER" a string field, i'll use that string to show the different types of slicing.

Get first n elemenst using [:n]

ptr[:1]    --> "P"
ptr[:2]   --> "PO"
ptr[:3]   --> "POI"

Remove first n elements using [n:]

ptr[1:]  --> "OINTER"
ptr[2:]  --> "INTER"
ptr[3:]  --> "NTER"

Get last n elements using [-n:]

ptr[-1:]  --> "R"      
ptr[-2:]  --> "ER"    
ptr[-3:]  --> "TER"   

Remove last n elements using [:-n]

ptr[:-1]  --> "POINTE"
ptr[:-2]  --> "POINT" 
ptr[:-3]  --> "POIN"

  
Get sub elemens using [n:m]
It given sub elements from n (inclusive) m(exclusive).

ptr[0:3] --> "POI"

Get sub elemens using [n:-m]
a). first apply [:-m] .i.e. removes last m elements,
b). then show all elements from n (inclusive).
ptr[0:-1]  --> "POINTE"
ptr[2:-3]  --> "IN

 Get sub elemens using [-n:m]
a). first apply [:m] .i.e. get first m elements,
b). get n last elements.
c) get common elements from a) and b)

example :  ptr[-1:7]  -->  'R'

a) ptr[:7] ----> X = 'POINTER' 
b) ptr[-1]  --->  Y = 'R'
c) apply  intersection(X, Y) --> 'R' 

example :  ptr[-1:6]   --> ""

a) ptr[:6]  ----> X = 'POINTE'
b) ptr[-1]  --->  Y = 'R'
c) apply  intersection(X, Y) --> nothing.

One intesting observation in here is :

[-7:1] => 'P'
[-6:2] => 'O'
[-5:3] => 'I'
........
........
 [-1:7] => 'R'

Following code implement above behaviour
ptr_len=len(ptr)+1
for i in range(1, ptr_len):
  ptr_len=ptr_len-1
  print ptr[-ptr_len:i]


Thats it. I hope,  i have demonstrated in very simple to get you understand python slicing.

Creating a excel file in python

To Create a Excel file using python , the xlsxwriter module found pretty cool
                    pip install xlsxwriter

after installing it , lets write a code to create a file.

Create a file and associate it with worksheet

             workbook = xlsxwriter.Workbook('person.xlsx')
             worksheet = workbook.add_worksheet()

In a worksheet, column and row in a worksheet are referenced by their index number, which starts with 0.

Set a width  of all columns : 
      worksheet.set_column(0, index_last_column, number)
above line can also be used to set width of particular column.

Creating a headers in bold :
     bold = workbook.add_format({'bold': 1})
      worksheet.write('A', 'Title', bold)
      worksheet.write('B', 'Name', bold)
      worksheet.write('C', 'Age', bold)
first row whose index is 0 is filled with above headers. 
     
Filling Records
person_data = list(Person.objects.all())
let say we have a list of Person objects in person_data
                row =1 // its not 0, since first row is a header
                col =0
           for person in person_data:
                worksheet.write_string(row, col,        person.title)
                worksheet.write_string(row, col+1,   person.first_name)
                worksheet.write_string(row, col+2,   person.last_name)
                r = r+1


Close the file object :
           workbook.close()

checkout docs for more fun.

Python Code : Part 1

Python is amazing in terms of how once can code in just few lines. I'm starting a series which includes the code snippets from various resources on internet . The objective is to learn something new or something which is hidden in python.

Swap two variables
>> a=7
>> b=9
>> b, a=a, b       // tuple packing and unpacking is happening here.
on right side of = tuple packing is done (a, b) --> (7, 9)
on left side of = tuple unpacking is being done and values are swapped (b,a ) <--- (7 ,9)
>> a
9
>> b
7

Re-calculating a variable based on other variable.
>> a = 3
>> b = a + 3
now if i keep changing value of a then value of b should also be changed for example if a is 5 then b should be 8.
>> b = lambda : a + 3
 >> a =4
>> b() ............. it will pass current value of a to lambda expression.
7
>> a =2
>> b()
5

Apply operator on 2 and 2*3 to make it 6.
>> val = 2 and 2*3
>> val
6
for and operator , if both operand are true then it will return value of last operand , otherwise value of operand which is evaluating to be False.
>> val = 2 and 0   or( 0 and 2)
>> val
0
'and' and 'or' operator does not return True or False but value of expression !!

Tweaking filter() of python

Following is basic list comprehension example which filter out the even numbers from a list.
>> [ x for x in [1,2,3,4] if  if n%2==0]
[2, 4]

Same can be achieved from filter(func, sequence). And for that we have to write the filter logic in a function, say poi().

>> def pp(n):
...     if n%2==0:
...             return True
...     else:
...             return False
...

>> filter(pp, [1,2,3,4])
[2, 4]

Upto here it seems that the filter(func, seq) takes a function as a first argument and sequence type (list, tuple , string, unicode or xrange() ) as a second argument. And for each element of sequence type it runs the function and returns elements of type seq.
The elements of returned list are initialized from the function, for which function returns True.

To see how filter() behaves ,return type of poi() has been played with.
Tweak 1: what if return type is False or None ?
def poi(x):
    return X    //X =False or None
>> filter(poi, [1,2,3,4])
[]

Tweak 2: what if i return type is string ?

Lets say for 2 in list , i want to return "two". 

def poi(x):
    if x==2:
        return "two"
>> filter(poi, [1,2,3,4])
[1, 2, 3]

Tweak 3: what if i return type is int ?
Lets say for 2 in list , i want to return 4.
def poi(x):
    if x==2:
        return x*x
>> filter(poi, [1,2,3,4])
[1, 2, 3]

What i learned :

1. filter(func, seq) will create a list of elements for which the function , poi() , returns True.
2. if poi() returns False or None for an element then resultant list will be an empty list.
3. if poi() returns other than boolean,  then it will return given list of filter.

What i understood:

say i have a sequence S having n elements and i want to filter the elements on some boolean condition, then filter() can rescue me.  Note that resultant sequence type will have all or some elements of given sequence for which boolean condition is True.

Catching an Exception in Python

Catching a exception without name: 
try:
      x = 1/0
except:
     print "it will be catched"

To get information about a exception,  use sys.exc_info() in except clause.:
excep_info = sys.exc_info()   // it returns a tuple
excep_info[0]   == type of exception
excep_info[1]  == exception message or arguments
excep_info[2]  == exception object value.


Catching a exception by name:
 try:
      x = 1/0
except ValueError:
     print "it will be catched"

or  (to get exception information)

 try:
      x = 1/0
except ValueError as exp:
     print "it will be catched"
     print type(exp)    // type of exception
     print exp.args     //exception message or argumenst


Catching exceptions by names , incorrectly :
 try:
      x = 1/0
except ZeroDivisionError, TypeError:
     print "it will be catched"

There are two issues with above code:
1) It will not catch the  ZeroDivisionError exception in a wrong way.
    Reason: except statement is interpreted  in a wrong way.
    except ZeroDivisionError as  TypeError:
    It is because in old python except ZeroDivisionError, e, anything after comma is interpreted as        alias to the exception before comma.

 TypeError is used as a reference to the ZeroDivisionError as following code shows.
      print "it will be catched", TypeError
output
     it will be catched integer division or modulo by zero

2) It will not catch the TypeError.
    Reason: since TypeError will be used as a reference.


Catching exceptions by names , correctly :
In this case either of the exception will be catched.
  try:
      x = 1/0
except ( ZeroDivisionError, TypeError ):
     print "it will be catched"

Boolean Value of a Data Types in Python

Boolean values in python are True, False and None. While going through the pages of DiveIntoPython , i found a sentence which says that :

You can use virtually any expression in a boolean context.

then i thought ANY expression ?  so i tried following code :

if "poi":
     print "yes"    
else:
    print "no"

My first guess was that it should give error, since "poi" is not evaluates to be a boolean value. It didn't give any error and prints the "yes". but .....Why ?

Every datatype in python has a boolean associated value with it. So string, list , tuple and dictionary are all can be used in the boolean context.

Then what evaluates true or False for these data types ?

Well, whenever these data types are empty, their boolean value will be False.
>>var=""
>>bool(var)
False
Same is true for [], (), {} i.e. non-empty list, tuple and dictionary.

And when these data types are not empty having atleast one element , their boolean value will be True.
>>var="python"
>>bool(var)
True
Same is true for [3], (4,5), {"key": 3} i.e. non-empty list, tuple and dictionary.

For numbers ,
None evaluates to be False.
non-zero number evaluates to be True.
0 (or 0.0) evaluates to be False.

Zen of Python

The guideline principles behind the python language design by Tim Peters.

>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!


while coding the python above points should be considered. Then on can say that it is a pythonic code.

Python Collection: Counter Class

Conceptual idea of Counter is class is to represents the multiset of mathematics. Mutiset is set which allows the a member(element) to appear more than once like [a,a,a,b,b,b,b,c,c].
Number of times a element appear is said to be multiplicity of element.
Where this multiset can be useful ?
Prime factor of 144 is [2,2,2,2,3,3],  which is a multiset !!

In python terms
Counter is a subclass of dict which stores the element as a key and count as a value. It is unordered collection type.
This class supports the mathematical operations like addition, subtraction, union and intersection between two objects. To use this class one needs to import Counter class from collections package.
In this post i'm going to work around this class for basic operations.

Use case 1:
Let say we have a list  ['red', 'blue', 'red', 'green', 'blue', 'blue']. from which we want to find out the number of occurrence of a element.
>>> color_list = ['red', 'blue', 'red', 'green', 'blue', 'blue']
>>> cnt = Counter()

>>> for word in color_list:

...     cnt[word] += 1
>>> cnt
Counter({'blue': 3, 'red': 2, 'green': 1})

creating the counter object :
>>> from collections import Counter             //importing Counter
>>> c = collections.Counter(a=3, b=2, c=5)

printing counter object :
>>> c
Counter({'c': 5, 'a': 3, 'b': 2})
>>> c.items()
[('a', 3), ('c', 5), ('b', 2)]

list all elements in a counter object :
>>> list(c.elements())
['a', 'a', 'a', 'c', 'c', 'c', 'c', 'c', 'b', 'b']

finding the count for an element :
>> c['a']
3
one interesting thing here is if we tries to find the element which is not in c , say c['k']
>> c['k']
0         // returns zero not KeyError,  which is a case when using the dictionary.

get the all keys and values in counter object :

>>> c.keys()

['a', 'c', 'b']

>>> c.values()

[3, 5, 2]

finding the most common elements and their count using most_common(n) method, where n is number of elements :

>>> c.most_common(1)   // n is 1

[('c', 5)]

and 

>>> c.most_common(2)  // n is 2

[('c', 5), ('a', 3)]

updating the counter object : 

>> c2 = {'a': 5}

>> c.update(c2)

>> c

Counter({'a': 8, 'b': 5, 'b': 2})    // value of a element is updated to (3+5=8).

Use case 2 : Find out the most common words in a file 

>>> import re
>>> words = re.findall(r'\w+', open('hamlet.txt').read().lower())
>>> Counter(words).most_common(10)
[('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631),
 ('you', 554),  ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)]

No JSON object could be decoded: line 1 column 0 (char 0)

I have written a python code which makes the HTTP requests to fetch the json data.
The json data seems like :
{    
  "items": [

          {
            "question_id": 18384375,
            "answer_id": 18388044,
            "creation_date": 1377195687,
            "last_activity_date": 1377195687,
            "score": 0,
            "is_accepted": false,
            "owner": {
                "user_id": 1745001,
                "display_name": "Ed Morton",
                "reputation": 10453,
                "user_type": "registered",
                "link": "http://stackoverflow.com/users/1745001/ed-morton"
                 }
          },

      ]
}

Code : 

>>url="https://api.stackexchange.com/2.1/answers?order=desc&sort=activity&site=stackoverflow"

>>obj = urllib.urlopen(url)
>>response_str = obj.read()
>>response_json = simplejson.loads(response_str)

Error : 

The last line above shoots following  error.
JSONDecodeError at /
No JSON object could be decoded: line 1 column 0 (char 0)

when i tried to change the simplejson to json object for loads() then following  error occurred
ValueError at /
No JSON object could be decoded

I tried to check the json format using the Jsonlint which gave me error :

The error suggested that problem is of encoding type of the response . Then i inspects the response information like
    >>obj.info()

         Server: nginx
         Date: Fri, 23 Aug 2013 04:51:01 GMT
         Content-Type: application/json; charset=utf-8
         Connection: close
         Cache-Control: private
         Content-Encoding: gzip
         Access-Control-Allow-Origin: *
         Access-Control-Allow-Methods: GET, POST
         Access-Control-Allow-Credentials: false
         Content-Length: 2334

Solution:

The problems seems to be of content Content-Encoding of the response which is gzip format and simplejson or json  is failed to correctly parse that. After many searches on stackoverflow i found the following the solution(s) of problem.
Solution: better to use requests object.

1) using only requests

   import requests

   response_json = requests.get(url).json()

2)  using only requests with simplejson or json.

     import requests

     response = requests.get(url)

     response_json = simplejson.loads(response.text)

     print response_json

Above both the codes  returning the correct Content-Encoding for json.
>>reposnse_json.encoding
     utf-8
>>response_json.headers['content-type']
    "application/json; charset=utf8'

Reading http JSON response in Python

Make http request that returns JSON response.               
                   >>fileObj = urllib.urlopen("http://www.omdbapi.com/?t=Fight Club")


Get JSON response from fileObj
                    >>jsonData =  fileObj.read()

 if you print this jsonData , it will look like this.
{
    "Title":"Fight Club",
    "Year":"1999",
    "Rated":"R",
    "Released":"15 Oct 1999",
    "Runtime":"2 h 19 min",
    "Genre":"Drama",
    "Director":"David Fincher",
    "Writer":"Chuck Palahniuk, Jim Uhls",
    "Actors":"Brad Pitt, Edward Norton, Helena Bonham Carter, Meat Loaf",
    "Plot":"An insomniac office worker looking for a way to change his life crosses paths with a devil-may-care soap maker and they form an         underground fight club that evolves into something much, much more...",
    "Poster":"http://ia.media-imdb.com/images/M/MV5BMjIwNTYzMzE1M15BMl5BanBnXkFtZTcwOTE5Mzg3OA@@._V1_SX300.jpg",
    "imdbRating":"8.9",
    "imdbVotes":"727,988",
    "imdbID":"tt0137523",
    "Type":"movie",
    "Response":"True"
}


Now you have whole JSON response , what if you are only interested in some particular response field(s) ,say 'Director'  ?
Lets see how we can get particular field(s) of JSON response.

Convert json object into python dictionary
                  >>import simplejson
                  >>response_dict = simplejson.loads(jsonData)


Use json data from response_dict to get field
                >>response_dict['Director']
                     David Fincher


So thats how you can read JSON response ,  by the way that was damn good movie....!!.

Reverse in python

Reverse using reversed(seq) function
To reverse a sequence one can use the reversed(seq) built-in-function. what is sequence ? str, list and tuple are sequence. This function can take a sequence as a argument and returns an iterator. Lets dig deep into it.
A  reversed(seq) method will call the __reversed__() method of sequence which will reverse the elements of sequence and  and return a iterator for reverse sequence. So the reversed(seq) not just return a iterator but a reverse iterator.

 Feel from example:

>>list=[1,2,3,4,5]

>>reverse_iterator =  reversed(list)

>>for ele in reverse_iterator:

>>....    print ele                         // 5,4,3,2,1

We can also use the next() method of reverse_iterator object to call 'successive' elements.
>>reverse_iterator.next()         //5
>>reverse_iterator.next()         //4
>>reverse_iterator.next()         //3
>>reverse_iterator.next()         //2
>>reverse_iterator.next()         //1
>>reverse_iterator.next()         //StopIteration exception, since no more elements are left


Reverse using extended slice, [start:end:step]
Extended slice is another way to reverse str, list and tuple. First, lets get familiar with what extended slice is all about.  
Sometimes it may be desirable to iterate over only certain elements of a type. for example,
L = [11 22, 33, 44, 55, 66, 77, 88, 99, 100] , it may be required that one wants to iterate over only even or odd numbers thereby skipping certain elements from the list.
>>L = [11 22, 33, 44, 55, 66, 77, 88, 99, 100]
>>L[1:10:2]       -----> 2,4,6,8,10     // all even numbers of L
>>L[0:10:2]       ------> 1,3,5,7,9      // all odd numbers of L

Now let answer,  how ?
Syntax for extended slice is [start:end:step]. start specifies the index (of first element) from which it will start picking the elements, end specifies upto which index (of last element) it will pick elements and step specifies the number of elements to skip.

The default value of start is 0, of end is length of item in used ( length of L) and of step is 1.
So if you try like L[::] it will print the L as it is, assuming it as L[0:10:1]
Note that the step cannot be 0  because you have to take step to iterate (i made that up ). Therefore following will give error.
>>> L[0:10:0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: slice step cannot be zero



Here is important thing, the step part also defines the order in which an item is iterated. if you provide a negative value then it start iterating in reverse order.
>>> L[::-1]
[100, 99, 88, 77, 66, 55, 44, 33, 22, 11]
what L[::-1] means ?
It means that Iterate over L from 0 to length of L in reverse order.

So that all about reverse in python, i hope you find it useful.