Feature request for python's requests library

Recently as apart of day job I had to count the number of API requests were made to keep check of throttling limits. As a standard for python codebase, requests library has been used.
I wrote some code which will maintain the API URLs and their respective counts.
That got thing done.

After a while I realized, there should be a way to intercept the request. Just like in there is middleware in Django for request. Then checked the documentation of requests and found event-hooks.

Going through the docs found out that hook is available only for response.  And they have intentions to add other hooks as well mentioned in TODO.
Found this opportunity to contribute one of the best libraries of python and without any delay I created github issue

Hopefully library maintainers will have a look and will add as a feature request officially.

Learning Elixir with side project

Recently I have been attracted to functional programming , the idea of creating an application which is entirely composed of functions with no side-effects and working with immutable data structures.
Just bumped into Elixir, tried couple of days and yeah really liked it.

How to learn
Idea is to identify problem(s) which I care about and solve those, it could very tiny thing.
This way I need not to struggle with motivation to keep carry on.

Identifying problem
I maintain IMDB watchlist and look for torrent sites if that movie from my watchlist is available or not. It's a mundane thing and i do not want spend time just to check that.
So I choose this to automate that and start my learning path for Elixir.

Building project
I created issues on project and started work on those.
One part of my brain telling me  is it correct way to do ? Other part was saying just make it work.  
More so coming from python background and writing code in it for more than six years, first thing comes to mind while writing code is to make pythonic since I know how to make it work.
But with new language had to suppress the urge to write idiomatic code from Day 1,  I can improve it while doing refactor.
It took a couple of weeks doing small improvements and iterating over. Whenever I was stuck on something took help from StackOverflow.

Learnings
1. Pattern matching is super cool. It really helps to write clean code.
2. Pipe operator. Another programming construct which helps to write clean code.
3. Mix, a build tool. Need to explore it more.
4. Tried out third-party libraries like HTTPoison, Floki and others.
5. Learning ideas from Joe Armstrong thesis

Feedback
Once the stable version of project is build, I decided to take feedback from elixir community.I approached  elixir-lang,  elixir-forum, gitter channels  lisbon-elixir, awesome-elixir,
local-elixir-meetup-group
Really appreciate their time and feedback on project.

Thanks for reading.

JWT: how token is created

In this post i am going to give a thoughts about how JWT token is got created. If do not aware of JWT token and its use-cases then please read this and come back.


Ok, now you have basic introduction about JWT token so i can go ahead of steps that involves creating token.

Let say we are creating token with followings:

algorithm is HS384
payload is {'some': 'payload'}
secret key is "secret"
and
header is {"custom_header": "custom_val"}


So lets follow through the creation of JWT token.

 1. algorithm is checked if it is a supported or not.

2. payload is encoded in utf-8 and converted into a byte string.
     b_payload = b'{"some":"payload"}'


3. header is encoded in utf-8 and converted into a byte string.
    b_header = b'{"typ":"JWT", "alg":"HS384", "custom_header": "custom_val"}'

4. b_payload is encoded into a base64 encoding:
    en_payload = base64_encode(b_payload)
    looks like
    b'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzM4NCIsImhrIjoiaGVhZC12YWwifQ'

5. b_header is encoded into a base64 encoding
    en_header = base64_encode(b_header)
    looks like
    b'eyJzb21lIjoicGF5bG9hZCJ9'

6.  A signing string is created by concatenating en_payload and en_header with dot (.)
    signing_input = b'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzM4NCIsImhrIjoiaGVhZC12YWwifQ' .b'eyJzb21lIjoicGF5bG9hZCJ9'

7. signature is created by signing an algorithm with signing_input and key
    signature = sign(signing_input, key)     # alg_obj is an algorithm object
    looks like
    b'\xc1\x7f\x7f\xfb\x96\xb3\x0fc\x1e\x84.\x02\xe5\xf5\xfd\xbb\xb2\x9bf0\x9ea\xec\x06U\x15-]\xca;\x1f\xfb\xa6J\xc7pv\xdf\x0cu;j`o\xa6ia\x9d'

8. Now, signature is encoded into a base64 encoding
   en_signature = base64_encode(signature)
   looks like
   en_sign = b'wX9_-5azD2MehC4C5fX9u7KbZjCeYewGVRUtXco7H_umSsdwdt8MdTtqYG-maWGd'

9. And finally all three components, en_payload , en_header and en_signature,  are concatenated by with dot (.) which results in token
   looks like
 b'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzM4NCIsImhrIjoiaGVhZC12YWwifQ.eyJzb21lIjoicGF5bG9hZCJ9.wX9_-5azD2MehC4C5fX9u7KbZjCeYewGVRUtXco7H_umSsdwdt8MdTtqYG-maWGd'


This is very basic of how token is created,  hope you find it useful.

On the fly: namedtuple

Python has a namedtuple from collections module.  It is interesting to look how it got created because it is created for user defined name.


namedtuple(typename,  field_names, verbose=False, rename=False)
for example:
point_obj = namedtupe('Point', ['x', 'y'])


I would like to share few observations about namedtuple after going through the source code of same.

1. a class definition is stored as a string literal which is to be executed.

2. A subclass of tuple is created with name as a typename. i.e.: point
     class Point(tuple):
        pass

3. property and operator.itemgetter applied on field_names. the Point will have x and y as a property
x and y will be attributes of point.

4. A class definition is created which involves 2. and 3. which looks like this.

5. exec() executes the class definition which returns object of Point.


You can have look at this code,  thanks for reading.

Django Internal: create()

A model instance is corresponds to a record in the database once saved. In this post i am going to
explore what happens behind the scene when you call  create(). It create a instance of model and save that instance which creates a record in the database.

let say we have a model defined as:

class Author(models.Model):
   name = models.CharField(max_length=100)
   age = models.IntegerField()


Lets go step by step to deconstruct Author.objects.create()

Creating instance of model

 instance = Author(name="roy", age=47)

1. signals invoked
Once starts it send signal pre_init and on successful completion of this method post_init is called.

2. check for given model fields not values.
check can fail if you more arguments than number of fields or attribute which does not present on model.
for example:
instance = Author(name="roy", age=47, code='python')
TypeError: 'code' is an invalid keyword argument for this function.

Each model has attribute called ._meta through which gives related details for models

As for information the models.Model  has an abstract base class called ModelBase which add attributes on a model itself like ._meta, errors (DoesNotExist, MultipleObjectsReturned) and validating model itself for its concrete definition.

Saving model instance
model method ._save_base() is called by save() to save model instance.  this where pre_save() and post_save() signals are invoked.

There are couple of important actors involved in here, I would like brief them.
 1. InsertQuery: instance of class is initiated for actual SQL' s raw query by passing model fields objects and values.
2. SQLInsertCompiler: This is SQL compiler for InsertQuery. It is a place where django models fields and values are parsed into SQL raw statement.
3. DatabaseWrapper: It is an wrapper for mysql-python client. It creates connection to SQL server and uses cursor to execute the raw SQL statement.

Queryset private method _insert() is place where things kick off at low level.  creating a record means INSERT SQL query. This happens inside a atomic transaction.
Basically django creates insert query gives that to compiler and compiler invoke database's wrapper method execute() by passing raw SQL statement as

     cursor.execute('INSERT INTO `author` (`name`, `age`) VALUES (%s, %s)', ['roy', 46])

Once above code is executed successfully record is being is created in database. And thereafter post_save signal is invoked. And that s how an model instance creates an record in database. 

Hope you found it interesting, keep reading.

Django Manager's methods

In this post i am going to explore how django.db.models.manager.Manager class supports queryset methods.

>> Book.objects.all()           # returns a queryset.
>> Book.object.filter(author='george-orwell' )        # returns all books by author george-orwell.
>> Book.objects.delete()    # deletes all instances.

 Here Book.objects is an instance of Manager class django.db.models.manager.Manager.
Ok, which means all these methods should be defined for this class.  No, all these methods are not defined for manager class but for Queryset class.

Manager class inherits from  django.db.models.manager.BaseManagerFromQuerySet class.
And magic is there is no such class as BaseManagerFromQuerySet.
That class  is dynamically generated by method from_queryset of class BaseManager.
>> BaseManager.from_queryset(QuerySet)

This is where all magic happens.
What it does is it takes all the methods of QuerySet and dynamically attach all these methods to a newly created class BaseManagerFromQuerySet. And this class is used as a base class for Manager class.

And that' s how filter(), delete() etc. can be invoked on manager.



Hope you guys find it useful.

Reference:
manager.py

python: new-style and old-style class

A new-style class inherits from object (or type) where as old-style class does not. In python 3.0 old-style class is removed.

In this post I am going to explore differences between the two.

1: types

Here we can seen that type of obj is not of class but of built-in type instance.  Attribute __class__ of obj gives information about class to which it belongs.
So type and __class__ attributes differ for old-style class.

Lets see what happens with New-style class (python 3.6): 

type and __class__ attributes does not differ for new-style class.


2: MRO (Method Resolution Order)

In addition to above differences, old-style class does not support mro.

whereas new-style class does have that:

Also one can see that new-style class inherits from a object.


3: super
super() does not works with old-style classes.

Works with new-style

the reason being is that super() expect type object for class In case of old-style type(ABC) is classobj not type But in case of new-style type(NewABC) is type.

4: descriptor protocol
New-style classes have implemented descriptor protocol whereas old-style does not.


5: raising an exception
In new style objects cannot be raised as exception unless derived from Exception.

Thanks for reading.

repr and str in python

The purpose of repr() and str() is to display information about object. But there is slight distinction between the two. In this post we will explore that.

By definition repr() method is official representation of object. That means if i want to create an object repr should be descriptive enough to give that information.
Lets take an example of datetime object.

>> from datetime import datetime
>> now_datetime = datetime.now()
>> repr(now_datetime)

prints

>> 'datetime.datetime(2017, 10, 2, 11, 0, 7, 382330)'

so if a want to create a new datetime object repr should be referred for that matter.

new_datetime = datetime.datetime(2017, 11, 12, 15, 56, 9, 343831)

Thats exactly the purpose of repr.

As per doc str() is a 'informal' representation of object. It has be string and you can put whatever you feel like.

>> str(now_datetime)
>> '2017-10-02 11:00:07.382330'


Now the difference
str() it should only return valid string object whereas repr() should either return string or a python expression. 
In case of python expression which can be used to create object.

>> make_this_repr = "x+1"
>> repr(make_this_repr)
>> "'x+1'"
>> x =10
>> eval(make_this_repr)
>> 11


 
Hope you find it useful, thanks for reading.

Full-Text Search in MySQL

In this post i am going to explore basic of this feature. Following is the table i am going to use for FULL-TEXT search explanation.

SELECT * FROM articles;
+—-+———————–+——————————————+
| id | title | body |
+—-+———————–+——————————————+
| 1 | MySQL Tutorial | DBMS stands for DataBase … |
| 2 | How To Use MySQL Well | After you went through a … |
| 3 | Optimizing MySQL | In this tutorial we will show … |
| 4 | 1001 MySQL Tricks | 1. Never run mysqld as root. 2. … |
| 5 | MySQL vs. YourSQL | In the following database comparison … |
| 6 | MySQL Security | When configured properly, MySQL … |
| 7 | Database Theory | I am going to teach you database theory |
+—-+———————–+——————————————+

Whle creating table i have used FULLTEXT (title,body) to allow full text search on these columns. Behind the scene it creates a index on these columns.

Lets do full-text search for string ‘database’.

SELECT * FROM articles WHERE MATCH (title,body) AGAINST ('database' IN NATURAL LANGUAGE MODE);
+—-+——————-+——————————————+
| id | title | body |
+—-+——————-+——————————————+
| 7 | Database Theory | I am going to teach you database theory |
| 1 | MySQL Tutorial | DBMS stands for DataBase … |
| 5 | MySQL vs. YourSQL | In the following database comparison … |
+—-+——————-+——————————————+

FULL TEXT allows different mode to run search query, here it is IN NATURAL LANGUAGE MODE.

• It means search string is a natural human language.

• By default full-text search is case-insensitive.

• Minimum length of word to be found is 3, which can be manipulated with innodb_ft_min_token_size .
• The returned rows are sorted with highest relevance first. relevance is kind of score is calculated by MYSQL internally.

Above query search search string in title and description. Now let search in title only.
SELECT * FROM articles WHERE MATCH (title) AGAINST ('database' IN NATURAL LANGUAGE MODE);
since we have defined the full-text search on title and body, To search only in column body we need to define FULL-TEXT index on that column.

Lets now do full-text search for string ‘to’.

SELECT * FROM articles WHERE MATCH (title,body) AGAINST ('to' IN NATURAL LANGUAGE MODE);

No result, but ‘to’ sting exist in body with id=7 ?
This is because full-text search will skip certain words, which are refereed as stop-words.
You can find out list for stop-words like:
SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_DEFAULT_STOPWORD;
+——-+
| value |
+——-+
| a |
| about |
| an |
| are |
| as |
| at |
| be |
| by |
| com |
| de |
| en |
| for |
| from |
| how |
| i |
| in |
| is |
| it |
| la |
| of |
| on |
| or |
| that |
| the |
| this |
| to |
| was |
| what |
| when |
| where |
| who |
| will |
| with |
| und |
| the |
| www |
+——-+

What if you want specific case of search, like get all result having string ‘database’ and not having ‘MySQL’. . That is where BOOLEAN mode of full-text search helps.

SELECT * FROM articles WHERE MATCH (title,body) AGAINST ('+database -MySQL' IN BOOLEAN MODE);
+—-+—————–+—————————————–+
| id | title | body |
+—-+—————–+—————————————–+
| 7 | Database Theory | I am going to teach you database theory |
+—-+—————–+—————————————–+

more boolean modes:

• find all rows which have atleast one of ‘MySQL’ and ‘YourSQL’
  SELECT * FROM articles WHERE MATCH (title,body) AGAINST (‘MySQL YourSQL’ IN BOOLEAN MODE);
  +—-+———————–+——————————————+
  | id | title | body |
  +—-+———————–+——————————————+
  | 5 | MySQL vs. YourSQL | In the following database comparison … |
  | 6 | MySQL Security | When configured properly, MySQL … |
  | 1 | MySQL Tutorial | DBMS stands for DataBase … |
  | 2 | How To Use MySQL Well | After you went through a … |
  | 3 | Optimizing MySQL | In this tutorial we will show … |
  | 4 | 1001 MySQL Tricks | 1. Never run mysqld as root. 2. … |
  +—-+———————–+——————————————+

• find rows that contains both words
  SELECT * FROM articles WHERE MATCH (title,body) AGAINST (‘+MySQL +YourSQL’ IN BOOLEAN MODE);
  +—-+——————-+——————————————+
  | id | title | body |
  +—-+——————-+——————————————+
  | 5 | MySQL vs. YourSQL | In the following database comparison … |
  +—-+——————-+——————————————+

Similar kind of search can be made using Boolean mode.

So this was very basic of full-text search, hope you find it helpful.

hello world in open source

More recently i have contributed to an open source project. There is a little story behind that.

Contributing accidentally
One late night i was going through the one of the article , quite interesting one. The article explain how there are social platform bots behave as human and what are the cons of such. The article was informative , i learned about the possible use case of machine learning.

Once done with article, my curiosity led me to find if there is any library or something which can determine if post by an social account is a bot or not. Being Back-end developer you think that way, can't help it.

I found this botornot-python library. I installed the library and tried it out. And then i went through the source code of that, i found one little stuff which can be done in better way. Next step was to fix that and open a pull request keeping finger crossed. :)
The owner of the library gladly accepted contribution, thanks to him.

And it was my first open source contribution, hurray !!!.  I felt so exited and satisfied since from long time i have been looking for that moment.

Motivation
Its about contributing to make project better with best of my ability and to learn from open source community.

Keep it going
As they say best place to look for project to contribute are the ones which you are already using.
faker is a interesting library that i have been using from long time for writing unit tests. After about a week or so i contributed to the same project,  joy was more this time.

So this how i started my open source contribution, thanks for reading.

Fun with Python

Today is April Fool and on python mailing list following kept me laughing.

 

Why the version 8? It’s just to be greater than Perl 6 and PHP 7, but 

it’s also a mnemonic for PEP 8. By the way, each minor release will now 

multiply the version by 2. With Python 8 released in 2016 and one 

release every two years, we will beat Firefox 44 in 2022 (Python 64) and 

Windows 2003 in 2032 (Python 2048).

and this too

 

DON’T PANIC! You are still able to import your legacy code into 

Python 8, you just have to rename all your modules to add a “_noqa” suffix 

to the filename. For example, rename utils.py to utils_noqa.py. A side 

effect is that you have to update all imports. For example, replace 

“import django” with “import django_noqa”. After a study of the PSF, 

it’s a best option to split again the Python community and make sure 

that all users are angry.

Seriously we can have fun with python, read full thread

Optimizing Django ORM queries

I’m have been working on a project using Django and Django-REST-Framework. A particular endpoint of the application executing number of database queries happening . Once you know its happening , then need you need to figure out why is it happening.

Findings:

• select_related and prefetch_related
  Most of the time application models have reference to some other model referred by ForeignKey or ManyToManyor OnetoOne.
  When a query is made for base relation and when you use referred relation it will make extra query, this extra query can be avoided.
  using select_related and prefetch_related removed lot of unnecessary queries. A big boost.

• Structure queries well

for person in Person.objects.all():
    if person.create_date >=today_date and person.status not in [X, Y]:
    # do something with person

Above loop can be avoided by constructing the simple query.

for person in Person.objects.filter(created_date__gte=today_date).exclude(status__in=[X, Y]):
    # do something with person

• Beware of the response data (django-rest-framework)
  A serializer may have all the models fields defined, but for specific endpoint you may not need all the fields to be serialized especially the related fields or the model properties which results in database queries. You can write some custom serializers.

Above three gave quite a boost to database performance, to one API end-point i managed to get the number of queries from 469 to 91.

Apart from this stuff one go for database level optimization. In my case it is PostgreSQL database and once can consider to set the following database settings:

shared_buffers          25% of RAM
effective_cache_size    75% of RAM
work_mem                5MB
wal_buffers             16MB
checkpoint_segments     10
maintenance_work_mem    50MB for each GB of RAM

Hope you find this article helpful.

Generic function in python

There times in python’s land when you needs to have different kind of behaviour for a function depending upon the type of arguments.

Decorator singledispatch from functools module from python 3.4 can help us out to create a different definitions of a with common interface.

we can define a generic definition of a function and can call same function with different set of arguments type:

fun("python land") 
fun(23, True)
fun([11, 22, 33])
fun(object())

This is how a generic function will look:

from functools import singledispatch
@singledispatch
def fun(arg, verbose=False):
    if verbose:
        print("verbosing....")
    print(arg)

Now different behaviour can be defined for this generic function

@fun.register(int)
def fun_int(arg, verbose=False):
    if verbose:
        print("numbers: ", end=" ")
    print(arg)

@fun.register(list)
def fun_list(arg, verbose=False):
    if verbose:
        print("Enumerate this:")
    print([ele+1 for ele in arg])

@fun.register(object)
def fun_int(arg):
    print(arg.__class__)

source code can be found here

Hope you liked the article, keep reading.