Load tested my Django site

Benchmark HTTP Latency with wrk

 

Recently I started reading through Actix-web, Django,tokio etc, so thought of bench marking the latency. Started with Django as it is pretty easy get started and its in python.

 

looks like wrk ( https://github.com/wg/wrk )  is a good tool to start with. 

"wrk is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue."

Installation also pretty easy in Ubuntu:

1. sudo apt-get install build-essential libssl-dev git -y

2. git clone https://github.com/wg/wrk.git wrk

3. cd wrk/

4. sudo make

5. sudo cp wrk /usr/local/bin

Run your Django server using 

python manage.py runserver

 

then another terminal you can run your wrk as below

 

wrk -t12 -c400 -d30s --latency http://127.0.0.1:8000

 I only had single page and not much custom files in the Django and not data base request. Not sure this is a good response .. Please share your comments.

 Also got something like this Django logs as well.

[27/Mar/2021 17:39:14] "GET / HTTP/1.1" 200 196
[27/Mar/2021 17:39:14] "GET / HTTP/1.1" 200 196
Exception happened during processing of request from ('127.0.0.1', 33856)
[27/Mar/2021 17:39:14] "GET / HTTP/1.1" 200 196
[27/Mar/2021 17:39:14] "GET / HTTP/1.1" 200 196

I am planning to try this same experiment with actix-web and Rust , will share once I have something..

 

Callbacks in Rust and Python



Callbacks

Following is the definition of callbacks from wiki https://en.wikipedia.org/wiki/Callback_(computer_programming)

callback in Rust

Whenever I read any web code, callbacks are important part of the code.
So tried to build a basic callback in rust to understand what is it .


I believe , callbacks are basically storing functions/closures in array and calling them whenever required. Actually it is a little difficult to build in Rust because of the syntax and type checks.
But it’s work , type of the functions will be verified in compile time.
For example , if I give x+2 , instead of x +=2 in the below code, rust comiplier gives compilation error.
below code


Rust callback

//create a callbacks list and make some basic operations like
//register a function and calling it etc.

//callbacks struct to store the vector( list of functions)
struct CallbackV1{
    callbacks:Vec<Box<FnMut(i32)>>
}

//implemenation of the CallbackV1
impl CallbackV1{

    fn new() ->Self{

        CallbackV1 {
            callbacks : Vec::new()
        }
    }
	
	//method to register functions in the callbacks vector
    fn register(&mut self,f:Box<FnMut(i32)>){
        self.callbacks.push(f)
    }

	//method to call the functions in the callbacks
    fn call(&mut self,p:i32){

		//iterate over the callbacks vector and take a mutable reference
        for callback in self.callbacks.iter_mut(){
			
			//dereference the callback function and call with passed argument.
            (&mut *callback)(p);
        }
    }
}

fn add(x:i32){

    println!("{}",x+1);
}

fn main() {
//   let mut x = 2;
   let mut  mycallbacks = CallbackV1::new();
   let f1 = |mut x:i32| { 
                            x += 2
                            // println!("{}",x) ;
            };
   let f2 = add;

   mycallbacks.register(Box::new(f1));
   mycallbacks.register(Box::new(f2));

   mycallbacks.call(4);


}

Callback in Python

Built the same code with Python as well. Its quite easy to build the same in python. But the type checks are not there . Its simply call the functions registered. I believe this can lead to vulnerabilities .

Python Callback code

#create a callbacks list and make some basic operations like
#register a function and calling it etc.

class CallbacksV1:
    
    def __init__(self):
        self.callbacks = []
    
    def register(self,f):
        self.callbacks.append(f)
    
    def call(self,value):
        
        callback =  self.callbacks.pop()
            
        callback(value)

    
def add(x):
    print(x+1)

def add_two(x):
    print(x+2)
    return (x+2) 

c = CallbacksV1()
c.register(add)
c.register(add_two)

c.call(3)
c.call(5)

python async programming

Python async example1

Python recursion,memorization,decorator memorization

Python recursion example1

Python recursion with memorization

Python recursion with memorization with python decorator

Python data structures list,dic,set,counter etc.. explained in malayalam

I tried to explain some of the python built in data structures in Malayalam here.

Python example for list

Python example for dictionary

Python example for Set

Python example for Counter

Python example for defaultdictonary

Traverse through 2 dimensional list

A list is a data structure that holds an ordered collection of items i.e. you can store a sequence of items in a list
Slicing the list and manipulating the list is easy in python. Since its a sequence is iterable, we can use for loop on list. "Sequence in Python, sequence is the generic term for an ordered set.Lists are the most versatile sequence type. The elements of a list can be any object, and lists are mutable - they can be changed. Elements can be reassigned or removed, and new elements can be inserted" python two dimensional list

Python TRIE

Python implementation for TRIE


TRIE is very interesting data structure and its very much used in word suggestion algorithms. Python implementation for this would easiest way to learn it.



Python Trie example 1

Python Generator

Have you ever  think through how to build an function that generate infinite series ?
You might be  thinking why can't we use an infinite loop! the problem is you can't come of the loop and again resume from the position/state where you stopped .

But generators helps in this case, they are just like functions but the difference is they have yield keyword in the definition which is like a return wrapped with generator-iterator object

"When you call either a function or a generator, a stackframe is created. It has the local variables
(including the arguments passed into the function),a code pointer to the active opcode, and a stack for pending try-blocks,with-blocks, or loops.

In a regular function, execution begins immediately.When return is encountered, the final result is kept and the stackframe is freed along with everything it referenced.

In a generator function, the stackframe is wrapped in a generator-iterator object and returned immediately. The code in the generator function only runs when called by next(g) or g.send(v). Execution is suspended when yield is encountered. "



Python generator infinite sequence example

Python generator things to be known ( exhausted generator )

Python inheritance - Examples

Python inheritance explained with Visualization

Python inheritance example 1

Python inheritance ( with child methods) example 2

Python inheritance and overriding example3

How to deploy python programs to Docker

Running python programs in Docker

It is quite easy to run your python programs in docker. For getting started with python in docker, we need to install docker in our system.

I am using Ubuntu and followed the instruction details beginners commands to start with Docker.

Once you have installed docker, we can start building our docker. We can create a separate folder/directory for this in our system.

FROM tells Docker which image you base your image on (in the example, Python 3).

RUN tells Docker which additional commands to execute.

CMD tells Docker to execute the command when the image loads.

Dockerfile:

#base image
FROM python:3

#adding our first program/application to docker image
ADD app.py /
ADD app2.py /

#this time we are using a script to run our applications.
ADD script.sh /
#make the script executable
RUN ["chmod", "+x", "./script.sh"]
#changed the command to run the script
CMD ./script.sh 
#you can read more about commands in docker at https://docs.docker.com
#add the command instruction
#CMD ["python","./app.py"]

script file:

You can specify your python programs in the script.
Note: I have explicitly used python3 to run the programs.

#!bin/bash

#first process
python3 app.py

#second process
python3 app2.py

Python applications

print("this is my first pythong docker program")

python function arguments

Python function arguments 

You might have seen *, ** (asterisks ) in functions definition -arguments in python. Don't confuse them with C pointers.They are actually totally different.

Below examples take you through  defaults ,keyed and VarArgs parameter passing in python.


Reference: https://python.swaroopch.com/functions.html






python function arguments Example 1

python function arguments Example 2

Python function arguments Example 3



Python arguments with tuple and dictionary

6 ways to run python

Python language interpreter

Python interpreter can be used many ways from command line.

Beginners guide: https://wiki.python.org/moin/BeginnersGuide

Developers guide: https://devguide.python.org/

•  Call a Python interactive shell (REPL):

           python

•   Execute script in a given Python file:

           python script.py

• Execute script as part of an interactive shell:

          python -i script.py

•  Execute a Python expression:

          python -c "expression"

•  Run library module as a script (terminates option list):

          python -m module arguments

•  Interactively debug a Python script:

         python -m pdb script.py

     

Python data science libraries that you should know

8 python libraries for data science

1. NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.

In python interpreter

>>> import numpy as np
>>> a = np.arange(15).reshape(3,5)
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

2.scikit-learn - is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities.features various algorithms like support vector machine, random forests, and k-neighbours, and it also supports Python numerical and scientific libraries like NumPy and SciPy.

>>> clf = RandomForestClassifier(random_state=0)
>>> x=[[1,2,3],
... [11,12,13]]
>>> y = [0,1]
>>> clf.fit(x,y)
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='gini', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=100,
                       n_jobs=None, oob_score=False, random_state=0, verbose=0,
                       warm_start=False)
>>> clf.predict(x)
array([0, 1])
>>> clf.predict([[4,5,6],[13,14,15]])
array([0, 1])

3.pandas
When working with tabular data, such as data stored in spreadsheets or databases, Pandas is the right tool for you. Pandas will help you to explore, clean and process your data. In Pandas, a data table is called a DataFrame.

https://pandas.pydata.org/getting_started.html

The primary two components of pandas are the Series and DataFrame

>>> import pandas as pd
>>> data = {
...     'naveen':[50,40,30,20],
...     'John':[23,50,34,22]
... }
>>> marks = pd.DataFrame(data)
>>> marks
   naveen  John
0      50    23
1      40    50
2      30    34
3      20    22
>>> marks = pd.DataFrame(data,index=['English','Maths','Science','History'])
>>> marks
         naveen  John
English      50    23
Maths        40    50
Science      30    34
History      20    22

4.Sympy -
SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible

>>> from sympy import solve,Eq,symbols
>>> x, y, z, d = symbols('x y z d') 
>>> eq1 = Eq(x+y,8) 
>>> eq2 = Eq(x+z,13)  
>>> eq3 = Eq(z+d,6)  
>>> eq3 = Eq(z-d,6)  
>>> eq4 = Eq(y+d,8)
>>> solve(eq1,eq2,eq3,eq4,(x,y,z,d))

5.mathplotlib -Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy

>>> import matplotlib.pyplot as plt
>>> import numpy as nu
>>> x = nu.linspace(0,10,100)
>>> plt.plot(x,x,label='linear')
[<matplotlib.lines.Line2D object at 0x7fe300680910>]
>>> plt.legend()
<matplotlib.legend.Legend object at 0x7fe2f38f6450>
>>> plt.show()

6. Tensorflow
TensorFlow is a free and open-source software library for dataflow and differentiable programming across a range of tasks. It is a symbolic math library, and is also used for machine learning applications such as neural networks.

https://www.tensorflow.org/api_docs/python

https://machinelearningmastery.com/introduction-python-deep-learning-library-tensorflow/

7. Keras: The Python Deep Learning library
Keras is an open-source neural-network library written in Python. It is capable of running on top of TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or PlaidML.
There are two main types of models available in Keras: the Sequential model, and the Model class used with the functional API.

https://keras.io/

8.Scipy -

https://www.scipy.org/getting-started.html

The SciPy ecosystem

Scientific computing in Python builds upon a small core of packages:

• Python, a general purpose programming language. It is interpreted and dynamically typed and is very well suited for interactive work and quick prototyping, while being powerful enough to write large applications in.

• NumPy, the fundamental package for numerical computation. It defines the numerical array and matrix types and basic operations on them.

• The SciPy library, a collection of numerical algorithms and domain-specific toolboxes, including signal processing, optimization, statistics, and much more.

• Matplotlib, a mature and popular plotting package that provides publication-quality 2-D plotting, as well as rudimentary 3-D plotting.

Python Malayalam Book

I started long back to write Python programming helper book in Malayalam.  But unfortunately I got side tracked and haven't touch that for long time.

https://davnav.github.io/MalayalamPythonebook/first%20steps


Now again I am brushing it up .  Yesterday I moved that to github and converted to markdown files.

https://github.com/davnav/MalayalamPythonebook/

I can do lot of  md formatting now.

Hopefully I make some progress in coming days..

Looking for your support as well.

Learn Python ,RUST and C - blog post 2

Understanding Data Types in Python,Rust and C progamming languages.

I am not sure whether I can cover all data types comparison in one blog post. So I might be doing it in multiple blog posts.
Standard Data Types:*

• Numbers.
• String.
• List.
• Tuple.
• Dictionary.

Numbers Datatypes
**Python programming **

Type	format	description
int	a = 10	signed integer
long	356L	Long integers
float	a = 12.34	floating point values
complex	a = 3.14J	(J) Contains integer in the range 0 to 255

The variable types are dynamically typed in python.So when you assign a value to a variable, python compiler determines the type of the variable.
But once the type is defined , the defined operations can only be performed for that particular type.
but you can change the type of the variable by assigning a different value. So in python it is the developer responsibility to maintain and make sure type of the variable intact through out the program.
For example,

>a = 10;
>a = "naveen" //allowed
>a = a + 1 // not allowed
>a = 11  //allowed
>a = a +1 // now this is allowed

**Rust programming **:

Length	signed	unsigned
8bit	i8	u8
16bit	i16	u16
32bit	i32	u32
64bit	i64	u64
128 bit	i128	u128
arch	isize	usize

signed and unsigned means whether a number can be -ve or +ve.
Each signed variant can store numbers from -(2^{n -1}) to 2^n-1
unsigned variants can vary from 0 to 2^n-1
eg: u8 means 0 to 2^8-1 = 128
signed means -2^8-1 to 2^8-1
Rust’s floating-point types are f32 and f64, which are 32 bits and 64 bits in size, respectively.
In the below example we assigned a value 129 to varialble ‘b’ which is of ‘i8’ ,so the program won’t get compiled.

fn main(){
   let a:i8 = 127;
   let b:i8= 129;  //this will cause overflow.
   let c:f32=123.32;
   let d:i8 = -127;
   let e:u8 = 127;

  println!("{}.{},{},{},{}",a,d,c,e,b);
}

when you compile the program,

error: literal out of range for `i8`
 --> datatype.rs:3:13
  |
3 |   let b:i8= 129;
  |             ^^^
  |
  = note: #[deny(overflowing_literals)] on by default

String:
Strings in python can be denoted by single quotes(’), double quotes("), triple quotes(""").

name1    = 'naveen'
name2    = "davis"
fullname = """naveen davis
                       vallooran"""

It possible to access the characters in a string using their index in python.

nam1[0] prints the character 'n'

Learn Python ,Rust and C - blog post 1

Learning Python ,Rust , C programming together

I have seen in different forums that people ask following questions " Why should I learn multiple programming language ?" or “Should I learn C or Python or JavaScript ?”.
My opinion would be , you can concentrate on one language and learn it deep. But as a developer some point of time it is important that to learn multiple programming languages and understand the difference and know where these languages are strong. This would helps you take decision on which language to use for your application.

Python Hello world : you can run this program from a python interpreter or save the program in in file and run with python.
install python - python

//python programming 
print("hello world")

Rust Hello world
The below Rust program needs to be saved in filename.rs and compiled with Rust compiler.t
Please follow the link for Rust installation How to install rust

///Rust programming 
fn main(){
 println!("hello world");
}

or You can create a rust programs using Cargo.
The link will help you to install cargo - Cargo

C Hello world
You preferred to have GCC compiler to generate C program executable.
installation -
Ubuntu GCC installation
Windows GCC

The program can be saved in a helloworld.c and compile and execute as below .

#include<stdio.h>
int main(){
    printf("hello world\n");
}

Python	Rust	C
Interpreting language	compiled language	compiled language
platform independent	cross platform individual building or compilation for each platform that it supports,	platform dependent
Strongly typed dynamically typed	strongly typed and dynamically typed	weekly typed and statically typed
objected oriented	Object-Oriented Programming is a way of modeling programs that originated with Simula in the 1960s and became popular with C++ in the 1990s. There are many competing definitions for what counts as OOP, and under some definitions, Rust is object-oriented; under other definitions, it is not.	structural language
inbuilt garbage collector	no separate garbage collector,but memory management is done automatically in compile time	manual memory management

Plot multiple Coins prices in Single graph using Python Mathplotlib

How to use Mathplotlib for plotting multiple graphs:

https://steemit.com/steem-dev/@naveendavisv/plot-bitcoin-and-ethereum-price-graph-using-python-mathplotlib-program

Python program to read bitcoin price and plot as a graph

How to plot bitcoin price using Mathplotlib + mysql

This program is explained in the below link .https://steemit.com/python/@naveendavisv/bitcoin-price-graph-plot-with-python-program-mysql

How to publish top 10 hot posts from steemit using python program

Its very interedting play with data steemit blockchain provides

https://steemit.com/steem-dev/@naveendavisv/5-how-to-steemit-application-with-pistion-and-flask-python-programming-modules

How to get the Steemit 'followers' for an account using Python Piston module

You can get steemit followers using python program.This is fun and learning experience to understand steemit and blockchain. Read more technical details here. https://steemit.com/steemit-dev/@naveendavisv/4-how-to-build-steemit-applications-with-piston-python-module