Apache spark project in Eclipse with Maven

If you are really looking to develop some spark projects, the easiest and best option would be eclipse with Maven which gives all framework tools to do the development easier.There are some tutorial videos in youtube which explain how to install and start with it.

I just started with the 'wordcount' program, once I configured the eclipse&scala IDE in Ubuntu.

My screenshots:

Spark Eclipse Maven Live video record:






Youtube tutorials:

spark streaming

Spark Streaming from local Network and Counting the words in near to real time. This is awesome!!.



Next I like to find some trending news/feeds using apache spark streaming.

Machine Learning

Machine learning , nice word. I know that I was studying about machines/computers from Plus+2. Then what is new in this learning :-).But this is different.Here we makes Machine to learn from its previous experience or outputs/inputs. If you want to give a try on ML, I think this is a best course to start with Free and Open source materials .

apache spark- a software which gives you great learning experience

The beauty of open source projects is that if you try to learn it , it will take you through lot of subject areas that you never heard about. Apache Spark is one of them for me. I started hearing about it when I started to follow big data&hadoop. Ok,I thought , let me install it in my ubuntu machine . I found that it is quite easy and the interesting part is that it took me through interesting subjects like ML(machine learning), Streaming,functional programming (scala) etc..

Apache Spark Machine Learning Tutorial | MapR

Apache Spark Machine Learning Tutorial | MapR

Introduction to Spark Architecture

After seeing the below video , I understood that the importance learning new programming languages. The new languages are coming with some new ideas. More than learning the new languages in the perspective of Jobs opening, those are quite interesting to know how and where the research happens to create new technologies.

circuit simulators

I was searching for some free circuit simulators. Below are some cool web versions of circuit simulators.

 http://www.falstad.com/circuit/

https://www.partsim.com/simulator

http://www.docircuits.com


Looking for these features can be brought into a Electronics course making tool.

Easy jump to OS building ..

I know the basic components of an Operating system . The basic theories around OS like application layer, kernel, system calls, boot loader,BIOS etc.  But I am scared to write one OS.. But below blog is motivating me to make an attempt to write an Operating system of My own.

http://os.phil-opp.com/multiboot-kernel.html

why to build a Circuitacademy ?

I would like to build a circuit academy - online electronics courses study and practice website. But not sure where to start with :-)

Below link has the more details.

circuitacademy

Most of the btech students loose their electronics interest after btech because there is no opportunity or good practical simulation examples of  circuits. They will be studying lot of theories and basic circuits. But real world circuits are complex, but it's difficult to practice or build these basic circuits. Online electronics courses are very less and there is very less opportunity for a electronics student to practice and master these basic and complex circuits . If you see www.codeacademy.com, there will be lot of courses in computer programming related subjects and we can practice programming languages. So the aim of this web application to help electronics enthusiasts to build and practice electronics courses and share it online. The challenges/difficulty in building electronics courses will be -

There should be EDA tools(given in next slides) available in the course tools bar for teachers to build circuit and play and show during the courses.

 


Please let me know if you are not able to open the link. help me on how to start with this project.

I don't know why I havn't done an FPGA programming in my Btech?

I haven't got any hand on experience in FPGA programming in my Electronics and communication course in Thrissur Engineering College. But there is a subject for HDL and VLSI subjects. I am sure that I have studied HDL language in 5th semester or 7th and got just pass mark 40 out of 100. :-).  I do like that language because its gives clear idea on how the AND or OR or NAND  gates are used to build a small digital system.

But its very sad that haven't got any hand on in FPGA programming in my Btech.

Timing Analysis in Project IceStorm (Open Source iCE40 FPGA Flow)

MYSQL- query for last one month count,last 60days,Feb month report generation

select SUM(CASE WHEN (DATE_FORMAT(order_date,'%m-%d-%y') > DATE_FORMAT(CURRENT_DATE - INTERVAL 30 DAY,'%m-%d-%y')) THEN 1 ELSE 0 END) AS ord,age from order1 group by age ;



select SUM(CASE WHEN (DATE_FORMAT(order_date,'%m-%d-%y') > DATE_FORMAT(CURRENT_DATE - INTERVAL 60 DAY,'%m-%d-%y')) THEN 1 ELSE 0 END) AS ord_from_60_days,SUM(CASE WHEN (DATE_FORMAT(order_date,'%m-%d-%y') > DATE_FORMAT(CURRENT_DATE - INTERVAL 30 DAY,'%m-%d-%y')) THEN 1 ELSE 0 END) AS ord_from_1month,age from order1 group by age ;


+------------------+-----------------+------+
| ord_from_60_days | ord_from_1month | age |
+------------------+-----------------+------+
| 7 | 4 | 23 |
| 0 | 0 | 25 |
+------------------+-----------------+------+
2 rows in set (0.00 sec)

hackerrank

Are you want to program some algorithms or some basic CS concepts ? I would say you can go and try it in https://www.hackerrank.com

Dynamic Programming | Set 10 ( 0-1 Knapsack Problem) | GeeksforGeeks

Its very interesting to watch GeeksofGeeks videos, if you are interested to learn algorithms and some cool solution for problems.

Rust applicatons

Are searching some of the applications written in Rust? below you have some of the interesting applications.

1. http://www.redox-os.org/  - operating system written in Rust
2. http://zinc.rs/ - Zinc is an experimental attempt to write an ARM stack that would be similar to CMSIS or mbed in capabilities
3. http://maidsafe.net/ - Maidsafe, a company that tries to create an encrypted, completely decentralized "successor" to the internet
4.https://github.com/servo/servo - Servo, the new browser engine being developed by Mozilla 
5. Iron is a high level web framework built in and for Rust, built on hyper.
ironframework.io/doc/iron/

6. wtftw, a tiling window manager - https://kintaro.github.io/rust/window-manager-in-rust-01/

debug a python program

How to debug a python program:

It is important to get know the debugging method ,if we really started to learn a new programming  language:

In Python programming language we have a tool similar to gdb which is pdb.

You can start debugging as below :

 python -pdb -m selectsort.py

more details on pdb - https://docs.python.org/2/library/pdb.html

http://infohost.nmt.edu/tcc/help/pubs/python22/pdb-commands.html

 

s(tep)

Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function).

n(ext)

Continue execution until the next line in the current function is reached or it returns. (The difference between next and step is that step stops inside a called function, while next executes called functions at (nearly) full speed, only stopping at the next line in the current function.)

unt(il)

Continue execution until the line with the line number greater than the current one is reached or when returning from current frame.

New in version 2.6.

r(eturn)

Continue execution until the current function returns.

c(ont(inue))

Continue execution, only stop when a breakpoint is encountered.

j(ump) lineno

Set the next line that will be executed. Only available in the bottom-most frame. This lets you jump back and execute code again, or jump forward to skip code that you don’t want to run.

It should be noted that not all jumps are allowed — for instance it is not possible to jump into the middle of a for loop or out of a finally clause.

l(ist) [first[, last]]

List source code for the current file. Without arguments, list 11 lines around the current line or continue the previous listing. With one argument, list 11 lines around at that line. With two arguments, list the given range; if the second argument is less than the first, it is interpreted as a count.

a(rgs)

Print the argument list of the current function.

p expression

Evaluate the expression in the current context and print its value.

Rust pointer implementaiton

https://aml3.github.io/RustTutorial/html/02.html#Starting_to_Corrode:%3Cbr%3EPointers,_Memory,_Strings,_and_I/O

I am really scared of pointers in C programming itself.. Rust have more features than C for pointers .. :-)


But this really good read the above link on Rust memory management. I believe that there are lot of values for learning these new features .

Rust vs C program speed

I was just going through some of the Rust programs and tutorials...

This is one of the site which benchmark programing language speed. Rust programming language doesn't compromise the speed

http://benchmarksgame.alioth.debian.org/u64q/rust.html

http://benchmarksgame.alioth.debian.org/why-measure-toy-benchmark-programs.html


If you are really looking for some Rust applications , here you can browse.

http://pcwalton.github.io/blog/2013/05/20/safe-manual-memory-management/


This is one good link for those who want to start with Rust - https://aml3.github.io/RustTutorial/

Search a Tree - C program

Computer is a robot , our program  should identify the repetitive task and make computer do that repetitive task with less lines of code.... programmer should be able to identify the repetitive task and apply recursive functions..

Search a value in a Tree - C program

the beauty of this code is that searching the tree (function:searchtree ) is going to be the one function.But we will make the computer to do the same task repetitively just passing the next node.


#include
#include

struct node {
    int a;
    struct node* left;
    struct node* right;
};

struct node* insert_node(int a){

    struct node* newnode;
    newnode = (struct node*) malloc (sizeof(struct node));
    newnode->a = a;
    newnode->left = NULL;
    newnode->right = NULL;

    return newnode;
}

int searchtree(struct node* root,int num){
    if(root == NULL)
        return 0;
    if(root->a == num ||searchtree(root->left,num))
        return 1;
    if(root->a == num || searchtree(root->right,num)){       
        return 1;
    }   
   
}
       
   
           

int main(){

    int num;
    struct node* root;
   
    root = insert_node(1);
    root->left = insert_node(2);
    root->right = insert_node(3);
    root->right->right = insert_node(4);   
    root->right->left = insert_node(5);   
    root->left->left = insert_node(6);
    root->left->right = insert_node(7);
    root->left->right->left = insert_node(8);

    printf("type a number:");
    scanf("%d",&num);

    if(searchtree(root,num)){
        printf("number found in tree");
    }
    else{
        printf("number not found in tree");
    };
    return(0);
}

Python linked list

If you are planning to refresh your data structure knowledge, I would recommend this website http://www.geeksforgeeks.org/data-structures/

Tree or Linked list datastructure  programming heavily use the recursion concept ... recursion is something that sometimes cause me brain crash :-)

But once you understand it and learn how to use recursion , then that is where you start mastering datastructure concepts.

If you look at Tree or linked list, it is a repetition of a node or a same structure(http://quiz.geeksforgeeks.org/binary-tree-set-1-introduction/)   .. so when there is repetition of something, then there is always scope for recursion concepts to apply...

Python Linked List:

Python programming language also have OOP concepts. So that means we can define a class http://www.diveintopython.net/object_oriented_framework/defining_classes.html

You can go through the linked list details in the below link
http://quiz.geeksforgeeks.org/linked-list-set-1-introduction/ . You will see two classes they defined for implementing linked list.

I was just trying to build without 2 classes. We can build a linked list, I dont know why they used 2 classes..I am not sure on how python is doing this dynamic memory allocation.



Program:

Strace:

Strace on a tree structure C program

When I tried to access 'unallocated node value' , as usual 'Segmentation fault' came as output :-)

But i am not fully clear on how mmap() system call allocate the memory...

I also came to know that mprotect() system call sets the protection,

If the calling process tries to access memory in a manner that violates the protection, then the kernel generates a SIGSEGV signal for the process.

Program and Strace details:

 

Try Strace on wget command

Its fun to try Strace on different commands to see what actually happens behind the command. There are some very good options to filter out required systems calls from the log.
The following link gives examples on how to use Strace http://www.thegeekstuff.com/2011/11/strace-examples/

one interesting system call is 'select' . I was only heard about 'select' in sql queries,but this is a different place i am seeing select :-)

"select() and pselect() allow a program to monitor multiple file descriptors, waiting until one or more of the file descriptors become "ready" for some class of I/O operation (e.g., input possible). A file descriptor is considered ready if it is possible to perform a corresponding I/O operation"

The sigaction() system call is used to change the action taken by a process on receipt of a specific signal.

 socket system call:

Strace experiments on Python

I was watching the video on 'Type  Python ,Press enter. What happens'. I thought of checking whether 'strace' can get the fork() system call that he explained in the video.

Unfortunately I couldn't get ....



But i came to know about the write and read system calls when we type the arithmetic operatons on python ... strace log pasted below. Still I dont know where the calculation is happening.... is there any tool to list which part is calculating it ? where is the fork() ?? brk(0x19ed000) = 0x19ed000 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 write(2, ">>> ", 4) = 4 read(0, "1+2\n", 1024) = 4 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0206e7b000 write(1, "3\n", 2) = 2 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 write(2, ">>> ", 4) = 4 read(0, "1+4\n", 1024) = 4 write(1, "5\n", 2) = 2 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 write(2, ">>> ", 4) = 4 read(0,

Hackers school

This is something programming students can try to collaborate with self motivated programmers.

https://www.recurse.com/blog/77-hacker-school-is-now-the-recurse-center

How to start with RUST LANG in linux machine

I am started hearing about RUST programming language http://doc.rust-lang.org/book/README.html, so thought of give a try 'helloworld' program. The impressive thing is,the language focus on safety,speed and concurrency.

Rust is a systems programming language focused on three goals: safety, speed, and concurrency. It maintains these goals without having a garbage collector, making it a useful language for a number of use cases other languages aren’t good at: embedding in other languages, programs with specific space and time requirements, and writing low-level code, like device drivers and operating systems

There is very good documentation - http://doc.rust-lang.org

Below is the installation steps, helloworld program . I gave a 'Strace' on the program to see any difference.I couldn't understand any difference :-).. i dont know how to find the difference in speed or concurrency.. need to read more ...

Unconventional way of studying things..

Sometimes I think that I studied lot of things in engineering text books, but don't have any clue on how the concepts/formula derived or how much analysis they put behind on those concepts.That research or hacking experience generally missing in text books. Sorry ..I am not saying school/college text books are bad. We ( as students) always needs to read text books .. but what I am trying to convey is... sometimes  textbooks are missing practical possibilities of  formulas or concepts.

There comes the importance of unconventional way of studying things..  or the hackers(in good sense) way of mastering the interested areas. Our schools and  colleges should give these kind of opportunities for students.

 Okay.....

These things came to my mind when I was watching the below video... 





She (http://jvns.ca )  explains the tool 'strace'

I tried 'strace' against our normal linux command 'ls'. This actually listed me amazing tracing log ...

 This is just an example .All concepts that we are studying/hearing should have unconventional way of mastering it or we should at least try to develop it !!!

None of the computer text books might have taught you on, what actually happens behind the 'ls' command. But the unconventional way is, find out some tool which can list the activities behind 'ls' and mastering what is it doing.... obviously it is very difficult, but  it gives you the full concepts behind the 'ls' upto very low/machine level.

Unconventional way of studying things opens opportunities which are not described in most of the text books That gives you the confidence to question the actual concepts or current implementation.

But it will have some initial difficulties and extra research  out of your text books or tutorials. But I am sure that there will be always some supporting tools/methods which somebody might have implemented already or you can implement such a method if it's not there!! :-)

Below is the screenshot of 'strace' on  'ls' in command line

some research on each system calls behind means ..

LXR Cross Referencer to read linux kernel source code

LXR Cross Referencer

This is the best place to refer one who like to know about linux kernel. http://lxr.free-electrons.com/ LXR Cross Referencer, usually known as LXR, is a general-purpose source code indexer and cross-referencer that provides web-based browsing of source code, with links to the definition and usage of any identifier.

I also found the following site is very helpful to start with linux kernels http://kernelnewbies.org/FirstKernelPatch

Installed wireshark for Networking package analysis

WIRESHARK

Wireshark is a network protocol analyzer. This is actually nice tool which gives opportunity to analyze  and understand different network layers. Each packet  traffic is diplayed when we start browsing a website.Its very easy to install in Ubuntu using 'Ubuntu Software center'. After installing I used the command and changed to root. I am not sure any alternative way for getting my 'wlan0' wifi internet without changing to root.

sudo dpkg-reconfigure wireshark-common

started wireshark with 'sudo wireshark'

Planning to learn networking modules in Linux Kernel

I am just writing this blog to make sure that I will refresh my system programming knowledge. Every time when I start motiviated with some concepts , it used to end in a week :-). Now I am going to try Networking modules in Linux kernel. As you all know , linux is like a ocean and any time we can go do some research and learn .. that what I like in GNU/Linux.

I got some tutorials from Youtube to start with the networking modules in Linux kernel.. don't know the opportunities that I am going to get in this. sharing some of the videos below .

Django with Angular js

This is very nice document http://django-angular.readthedocs.io/en/latest/angular-model-form.html ,one django learner should practice to get confidence on his/er django web development skills. I am sure the document gives you good learning experience in web development .

What is Django ?

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source https://www.djangoproject.com/
What is Angularjs?

AngularJS is a structural framework for dynamic web apps. It lets you use HTML as your template language and lets you extend HTML's syntax to express your application's components clearly and succinctly.
https://docs.angularjs.org/guide/introduction
http://www.w3schools.com/angular/

GeoDjango opens new opportunities for free and low cost geo application

Since google map matrix has pricing plans, web developers are started thinking of an alternative. I guess, you can use https://www.openstreetmap.org/#map=15/10.2708/76.3780 as an alternative since lot of companies and institutions are contributing for it.

Its quite interesting and innovative idea that Django ( python webdevelopment framework) is coming up with module for geographic web application GeoDjango. GeoDjango is an included contrib module for Django that turns it into a world-class geographic Web framework. GeoDjango strives to make it as simple as possible to create geographic Web applications, like location-based services. Its features include:


Django model fields for OGC geometries and raster data.
Extensions to Django’s ORM for querying and manipulating spatial data.
Loosely-coupled, high-level Python interfaces for GIS geometry and raster operations and data manipulation in different formats.
Editing geometry fields from the admin.

how to build Django admin page

It might be a time consuming and headache to build admin page for website. But the case is so cool with Django :-). I started with Django because I like python language. But its so amazing that you can build some pretty cool things in short span of time.It was a little difficult to start with it,but in my experience its pretty easy thing once you used with it. Below is the tutorial I followed to build the admin page.

http://www.djangobook.com/en/2.0/chapter06.html

Models in Django to create tables

How to connect Mysql database to Django

manged to install mysql database in ubuntu and created some tables using Models in Django.

Below online book is really helpful to master Django- http://www.djangobook.com/en/2.0/chapter05.html

models.py:




Mysql:

Apache Spark installed in Ubuntu

I am not sure much uses of Apache Spark. But thought of installing and play around .
https://www.mapr.com/blog/game-changing-real-time-use-cases-apache-spark-on-hadoop

Successfully installed Apachespark in Ubuntu using the below blog instructions.
http://blog.prabeeshk.com/blog/2014/10/31/install-apache-spark-on-ubuntu-14-dot-04/

This is the beauty of Open Source softwares, you can install it and play around it ..and contribute for the project.

For more details - http://spark.apache.org/

examples - http://spark.apache.org/examples.html

sqllite3 commands in Ubuntu Shell

How to install and use Sqllite3 in Ubuntu: 

Its quite easy to install sqllite3 in Ubuntu using 'apt-get install sqlite3'

naveen@naveen-Inspiron-7352:~/mysite$ sqlite3 mydb
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .databases
seq  name             file                                                     
---  ---------------  ----------------------------------------------------------
0    main             /home/naveen/mysite/mydb                                 
sqlite> exit
   ...> exit()
   ...> create table employee(id integer,name TEXT );
Error: near "exit": syntax error
sqlite> create table employee(id integer,name TEXT );
sqlite> insert into employee(1,'naveen');
Error: near "1": syntax error
sqlite> insert into employee values (1,'naveen');
sqlite> ;
sqlite> select * from employee
   ...> ;
1|naveen
sqlite> .exit

Django Login Page

After the struggle of 2 days :-), I succeed to create login page in Django with sqllite3. First we need to set up sqllite .. visit this site for more details http://www.saltycrane.com/blog/2007/11/django-project-2-sqlite-setup/

The other issue was , I was keep on getting the 'CSRF cookie not set'. I managed to exempt the CSRF using below technique, but its better use CSRF. http://stackoverflow.com/questions/22812721/why-do-i-get-csrf-cookie-not-set-when-post-to-django-rest-framework
and finally my login page worked !!!!!!!!!!!!

How to create a collection and document in MongoDB

MongoDb installation in Ubuntu

How to build your First Openshift Python application

How to use python maths library for Pi ,e, sqrt of 2

device file creation kernel module programming Linux -part 4