Python Tutorial for Beginners: Learn in 3 Days

Learn Python : Tutorial for Beginners

1. Getting Started with Python
   
   • Python 2.7 vs. 3.6
   • Python for Data Science
   • How to install Python?
   • Spyder Shortcut keys
   • Basic programs in Python
   • Comparison, Logical and Assignment Operators
2. Data Structures and Conditional Statements
   • Python Data Structures
   • Python Conditional Statements
3. Python Libraries
   • List of popular packages (comparison with R)
   • Popular python commands
   • How to import a package
4. Data Manipulation using Pandas
   • Pandas Data Structures - Series and DataFrame
   • Important Pandas Functions (vs. R functions)
   • Examples - Data analysis with Pandas
5. Data Science with Python
   • Logistic Regression
   • Decision Tree
   • Random Forest
   • Grid Search - Hyper Parameter Tuning
   • Cross Validation
   • Preprocessing Steps

You should go for Python 3.6. In terms of learning Python, there are no major differences in Python 2.7 and 3.6. It is not too difficult to move from Python 3 to Python 2 with a few adjustments. Your focus should go on learning Python as a language.

1. YouTube
2. Instagram
3. Reddit
4. Dropbox
5. Disqus

1. Download Anaconda. It comes with Python software along with preinstalled popular libraries.
2. Download Python from its official website. You have to manually install libraries.

Recommended :Go for first option and download anaconda. It saves a lot of time in learning and coding Python

Spyder - Python Coding Environment

1. Press F5 to run the entire script
2. Press F9 to run selection or line 
3. Press Ctrl + 1 to comment / uncomment
4. Go to front of function and then press Ctrl + I to see documentation of the function
5. Run %reset -f to clean workspace
6. Ctrl + Left click on object to see source code 
7. Ctrl+Enter executes the current cell.
8. Shift+Enter executes the current cell and advances the cursor to the next cell

#Basics
x = 10
y = 3
print("10 divided by 3 is", x/y)
print("remainder after 10 divided by 3 is", x%y)

x = 100
x > 80 and x <=95
x > 35 or x < 60

x > 80 and x <=95
Out[45]: False

x > 35 or x < 60
Out[46]: True

x = 100
y = 10
x += y
print(x)

print(x)
110

Similarly, you can use x-=y, x*=y and x /=y

1. x = [1, 2, 3, 4, 5]
2. y = [‘A’, ‘O’, ‘G’, ‘M’]
3. z = [‘A’, 4, 5.1, ‘M’]

x = [1, 2, 3, 4, 5]
x[0]
x[1]
x[4]
x[-1]
x[-2]

x[0]
Out[68]: 1

x[1]
Out[69]: 2

x[4]
Out[70]: 5

x[-1]
Out[71]: 5

x[-2]
Out[72]: 4

x[:3] returns [1, 2, 3]

1. A tuple cannot be changed once created whereas list can be modified.
2. A tuple is created by placing comma-separated values inside parentheses ( ). Whereas, list is created inside square brackets [ ]

K = (1,2,3)
City = ('Delhi','Mumbai','Bangalore')

for i in City:
    print(i)

Delhi
Mumbai
Bangalore

1. Function starts with def keyword followed by function name and ( )
2. Function body starts with a colon (:) and is indented
3. The keyword return ends a function  and give value of previous expression.

def sum_fun(a, b):
    result = a + b
    return result 

z = sum_fun(10, 15)

def sum_fun(a, b=0):
    result = a + b
    return result
z = sum_fun(10)

k = 27
if k%5 == 0:
  print('Multiple of 5')
else:
  print('Not a Multiple of 5')

1. pandas. For data manipulation and data wrangling. A collections of functions to understand and explore data. It is counterpart of dplyr and reshape2 packages in R.
2. NumPy. For numerical computing. It's a package for efficient array computations. It allows us to do some operations on an entire column or table in one line. It is roughly approximate to Rcpp package in R which eliminates the limitation of slow speed in R.
3. Scipy.  For mathematical and scientific functions such as integration, interpolation, signal processing, linear algebra, statistics, etc. It is built on Numpy.
4. Scikit-learn. A collection of machine learning algorithms. It is built on Numpy and Scipy. It can perform all the techniques that can be done in R using glm, knn, randomForest, rpart, e1071 packages.
5. Matplotlib. For data visualization. It's a leading package for graphics in Python. It is equivalent to ggplot2 package in R.
6. Statsmodels. For statistical and predictive modeling. It includes various functions to explore data and generate descriptive and predictive analytics. It allows users to run descriptive statistics, methods to impute missing values, statistical tests and take table output to HTML format.
7. pandasql.  It allows SQL users to write SQL queries in Python. It is very helpful for people who loves writing SQL queries to manipulate data. It is equivalent to sqldf package in R.

0   -2.412015
1   -0.451752
2    1.174207
3    0.766348
4   -0.361815
dtype: float64

-2.412015

-0.451752

0   -2.412015
1   -0.451752
2    1.174207

import numpy as np
import pandas as pd

mydata = {'productcode': ['AA', 'AA', 'AA', 'BB', 'BB', 'BB'],
        'sales': [1010, 1025.2, 1404.2, 1251.7, 1160, 1604.8],
        'cost' : [1020, 1625.2, 1204, 1003.7, 1020, 1124]}
df = pd.DataFrame(mydata)

Sample DataFrame

mydata= pd.read_csv("C:\\Users\\Deepanshu\\Documents\\file1.csv")

Make sure you use double backslash when specifying path of CSV file. Alternatively, you can use forward slash to mention file path inside read_csv() function.

Detailed Tutorial : Import Data in Python

3. To see number of rows and columns

You can run the command below to find out number of rows and columns.

df.shape

 Result : (6, 3). It means 6 rows and 3 columns.

4. To view first 3 rows

The df.head(N) function can be used to check out first some N rows.

df.head(3)

     cost productcode   sales
0  1020.0          AA  1010.0
1  1625.2          AA  1025.2
2  1204.0          AA  1404.2

5. Select or Drop Variables

To keep a single variable, you can write in any of the following three methods -

df.productcode
df["productcode"]
df.loc[: , "productcode"]

To select variable by column position, you can use df.iloc function. In the example below, we are selecting second column. Column Index starts from 0. Hence, 1 refers to second column.

df.iloc[: , 1]

We can keep multiple variables by specifying desired variables inside [ ]. Also, we can make use of df.loc() function.

df[["productcode", "cost"]]
df.loc[ : , ["productcode", "cost"]]

Drop Variable

We can remove variables by using df.drop() function. See the example below -

df2 = df.drop(['sales'], axis = 1)

6. To summarize data frame

To summarize or explore data, you can submit the command below.

df.describe()

              cost       sales
count     6.000000     6.00000
mean   1166.150000  1242.65000
std     237.926793   230.46669
min    1003.700000  1010.00000
25%    1020.000000  1058.90000
50%    1072.000000  1205.85000
75%    1184.000000  1366.07500
max    1625.200000  1604.80000

To summarise all the character variables, you can use the following script.

df.describe(include=['O'])

Similarly, you can use df.describe(include=['float64']) to view summary of all the numeric variables with decimals.

To select only a particular variable, you can write the following code -

df.productcode.describe()
OR
df["productcode"].describe()

count      6
unique     2
top       BB
freq       3
Name: productcode, dtype: object

7. To calculate summary statistics

We can manually find out summary statistics such as count, mean, median by using commands below

df.sales.mean()
df.sales.median()
df.sales.count()
df.sales.min()
df.sales.max()

8. Filter Data

Suppose you are asked to apply condition - productcode is equal to "AA" and sales greater than or equal to 1250.

df1 = df[(df.productcode == "AA") & (df.sales >= 1250)]

It can also be written like :

df1 = df.query('(productcode == "AA") & (sales >= 1250)')

In the second query, we do not need to specify DataFrame along with variable name.

9. Sort Data

In the code below, we are arrange data in ascending order by sales.

df.sort_values(['sales'])

10.  Group By : Summary by Grouping Variable

Like SQL GROUP BY, you want to summarize continuous variable by classification variable. In this case, we are calculating average sale and cost by product code.

df.groupby(df.productcode).mean()

                    cost        sales
productcode                          
AA           1283.066667  1146.466667
BB           1049.233333  1338.833333

Instead of summarising for multiple variable, you can run it for a single variable i.e. sales. Submit the following script.

df["sales"].groupby(df.productcode).mean()

11. Define Categorical Variable

Let's create a classification variable - id which contains only 3 unique values - 1/2/3.

df0 = pd.DataFrame({'id': [1, 1, 2, 3, 1, 2, 2]})

Let's define as a categorical variable.
We can use astype() function to make id as a categorical variable.

df0.id = df0["id"].astype('category')

Summarize this classification variable to check descriptive statistics.

df0.describe()

       id
count    7
unique   3
top      2
freq     3

Frequency Distribution

You can calculate frequency distribution of a categorical variable. It is one of the method to explore a categorical variable.

df['productcode'].value_counts()

BB    3
AA    3

12. Generate Histogram

Histogram is one of the method to check distribution of a continuous variable. In the figure shown below, there are two values for variable 'sales' in range 1000-1100. In the remaining intervals, there is only a single value. In this case, there are only 5 values. If you have a large dataset, you can plot histogram to identify outliers in a continuous variable.

df['sales'].hist()

Histogram

13. BoxPlot

Boxplot is a method to visualize continuous or numeric variable. It shows minimum, Q1, Q2, Q3, IQR, maximum value in a single graph.

df.boxplot(column='sales')

BoxPlot

Data Science using Python - Examples

In this section, we cover how to perform data mining and machine learning algorithms with Python. sklearn is the most frequently used library for running data mining and machine learning algorithms. We will also cover statsmodels library for regression techniques. statsmodels library generates formattable output which can be used further in project report and presentation.

1. Install the required libraries

Import the following libraries before reading or exploring data

#Import required libraries
import pandas as pd
import statsmodels.api as sm
import numpy as np

2. Download and import data into Python

With the use of python library, we can easily get data from web into python.

# Read data from web
df = pd.read_csv("https://stats.idre.ucla.edu/stat/data/binary.csv")

Variables Type Description
gre Continuous Graduate Record Exam score
gpa Continuous Grade Point Average
rank Categorical Prestige of the undergraduate institution
admit Binary Admission in graduate school

The binary variable admit is a target variable.

3. Explore Data

Let's explore data. We'll answer the following questions -

1. How many rows and columns in the data file?
2. What are the distribution of variables?
3. Check if any outlier(s)
4. If outlier(s), treat them
5. Check if any missing value(s)
6. Impute Missing values (if any)

# See no. of rows and columns
df.shape

Result : 400 rows and 4 columns

In the code below, we rename the variable rank to 'position' as rank is already a function in python.

# rename rank column
df = df.rename(columns={'rank': 'position'}) 

Summarize and plot all the columns.

# Summarize
df.describe()
# plot all of the columns
df.hist()

Categorical variable Analysis

It is important to check the frequency distribution of categorical variable. It helps to answer the question whether data is skewed.

# Summarize
df.position.value_counts(ascending=True)

1     61
4     67
3    121
2    151

Generating Crosstab 

By looking at cross tabulation report, we can check whether we have enough number of events against each unique values of categorical variable.

pd.crosstab(df['admit'], df['position'])

position   1   2   3   4
admit                   
0         28  97  93  55
1         33  54  28  12

Number of Missing Values

We can write a simple loop to figure out the number of blank values in all variables in a dataset.

for i in list(df.columns) :
    k = sum(pd.isnull(df[i]))
    print(i, k)

In this case, there are no missing values in the dataset.

4. Logistic Regression Model

Logistic Regression is a special type of regression where target variable is categorical in nature and independent variables be discrete or continuous. In this post, we will demonstrate only binary logistic regression which takes only binary values in target variable. Unlike linear regression, logistic regression model returns probability of target variable.It assumes binomial distribution of dependent variable. In other words, it belongs to binomial family.

In python, we can write R-style model formula y ~ x1 + x2 + x3 using  patsy and statsmodels libraries. In the formula, we need to define variable 'position' as a categorical variable by mentioning it inside capital C(). You can also define reference category using reference= option.

#Reference Category
from patsy import dmatrices, Treatment
y, X = dmatrices('admit ~ gre + gpa + C(position, Treatment(reference=4))', df, return_type = 'dataframe')

It returns two datasets - X and y. The dataset 'y' contains variable admit which is a target variable. The other dataset 'X' contains Intercept (constant value), dummy variables for Treatment, gre and gpa. Since 4 is set as a reference category, it will be 0 against all the three dummy variables. See sample below -

P  P_1 P_2 P_3
3  0 0 1
3  0 0 1
1  1 0 0
4  0 0 0
4  0 0 0
2  0 1 0

Split Data into two parts

80% of data goes to training dataset which is used for building model and 20% goes to test dataset which would be used for validating the model.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

Build Logistic Regression Model

By default, the regression without formula style does not include intercept. To include it, we already have added intercept in X_train which would be used as a predictor.

#Fit Logit model

logit = sm.Logit(y_train, X_train)

result = logit.fit()

#Summary of Logistic regression model

result.summary()

result.params

                          Logit Regression Results                           
==============================================================================
Dep. Variable:                  admit   No. Observations:                  320
Model:                          Logit   Df Residuals:                      315
Method:                           MLE   Df Model:                            4
Date:                Sat, 20 May 2017   Pseudo R-squ.:                 0.03399
Time:                        19:57:24   Log-Likelihood:                -193.49
converged:                       True   LL-Null:                       -200.30
                                        LLR p-value:                  0.008627
=======================================================================================
                      coef    std err          z       P|z|      [95.0% Conf. Int.]
---------------------------------------------------------------------------------------
C(position)[T.1]     1.4933      0.440      3.392      0.001         0.630     2.356
C(position)[T.2]     0.6771      0.373      1.813      0.070        -0.055     1.409
C(position)[T.3]     0.1071      0.410      0.261      0.794        -0.696     0.910
gre                  0.0005      0.001      0.442      0.659        -0.002     0.003
gpa                  0.4613      0.214     -2.152      0.031        -0.881    -0.041
======================================================================================

#Confusion Matrix
result.pred_table()
#Odd Ratio
np.exp(result.params)

# AUC on test data
false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, y_pred)
auc(false_positive_rate, true_positive_rate)

accuracy_score([ 1 if p > 0.5 else 0 for p in y_pred ], y_test)

#Drop Intercept from predictors for tree algorithms
X_train = X_train.drop(['Intercept'], axis = 1)
X_test = X_test.drop(['Intercept'], axis = 1)

#Decision Tree
from sklearn.tree import DecisionTreeClassifier
model_tree = DecisionTreeClassifier(max_depth=7)

#Fit the model:
model_tree.fit(X_train,y_train)

#Make predictions on test set
predictions_tree = model_tree.predict_proba(X_test)

#AUC
false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions_tree[:,1])
auc(false_positive_rate, true_positive_rate)

Feature engineering plays an important role in building predictive models. In the above case, we have not performed variable selection. We can also select best parameters by using grid search fine tuning technique.

#Random Forest
from sklearn.ensemble import RandomForestClassifier
model_rf = RandomForestClassifier(n_estimators=100, max_depth=7)

#Fit the model:
target = y_train['admit']
model_rf.fit(X_train,target)

#Make predictions on test set
predictions_rf = model_rf.predict_proba(X_test)

#AUC
false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions_rf[:,1])
auc(false_positive_rate, true_positive_rate)

#Variable Importance
importances = pd.Series(model_rf.feature_importances_, index=X_train.columns).sort_values(ascending=False)
print(importances)
importances.plot.bar()

from sklearn.model_selection import GridSearchCV
rf = RandomForestClassifier()
target = y_train['admit']

param_grid = { 
'n_estimators': [100, 200, 300],
'max_features': ['sqrt', 3, 4]
}

CV_rfc = GridSearchCV(estimator=rf , param_grid=param_grid, cv= 5, scoring='roc_auc')
CV_rfc.fit(X_train,target)

#Parameters with Scores
CV_rfc.grid_scores_

#Best Parameters
CV_rfc.best_params_
CV_rfc.best_estimator_

#Make predictions on test set
predictions_rf = CV_rfc.predict_proba(X_test)

#AUC
false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions_rf[:,1])
auc(false_positive_rate, true_positive_rate)

# Cross Validation
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict,cross_val_score
target = y['admit']
prediction_logit = cross_val_predict(LogisticRegression(), X, target, cv=10, method='predict_proba')
#AUC
cross_val_score(LogisticRegression(fit_intercept = False), X, target, cv=10, scoring='roc_auc')

from sklearn.preprocessing import LabelEncoder
def ConverttoNumeric(df):
    cols = list(df.select_dtypes(include=['category','object']))
    le = LabelEncoder()
    for i in cols:
        try:
            df[i] = le.fit_transform(df[i])
        except:
            print('Error in Variable :'+i)
    return df

ConverttoNumeric(mydf)

Encoding