Python

Links

==> The Python Standard Library <==
The Python Tutorial
Python 2.7.6 docs
Python Quick Reference 2.7 — Extremelly complete

Other versions of Python are available [1]

Nice example of generating / testing regex in Python (with nice / small test framework) [2]

Python 3

Built-in Types

PEP

Tools

autopep8 — A tool that automatically formats Python code to conform to the PEP 8 style guide

sudo pip install --upgrade autopep8

Python modules

Python comes with a wide range of libraries, called modules. There are several ways to install these modules.

Using the distribution

For instance, in Debian:

apt-cache search --names-only python-       # View available modules
sudo apt-get install python-pyscard         # Install the pyscard module

Using easy_install

sudo easy_install Pygments

Using pip

sudo pip install Pygments

From source

wget http://sourceforge.net/projects/pyscard/files/pyscard/pyscard%201.6.12/pyscard-1.6.12.tar.gz#md5=908d2530972ea91eb4bb66987e0e1e98
tar -xvzf pyscard-1.6.12.tar.gz
cd pyscard-1.6.12
sudo ./setup.py install

Python variants

iPy

Use iPy (ipython) to get an interactive shell with auto-completion, instant help...

%magic                    # Get help on %magic commands (%run,...)
?run                      # Get help on %run magic
%run script.py            # Run given script
%run -i script.py         # ... with inspect mode on
%run -i -e script.py      # ... ... and ignore sys.exit() call
!cmd                      # Run shell command 'cmd', for instance ...
!ls                       # ... List file in current directory

Pypy

PyPy is a fast, compliant alternative implementation of the Python language, which usually runs python programs faster thanks to its Just-in-Time compiler.

Install: On Lucid 64-bit, the easiest is to download the dedicated tarball:

wget https://bitbucket.org/pypy/pypy/downloads/pypy-2.2.1-linux64.tar.bz2
tar -cvjf pypy-2.2.1-linux64.tar.bz2

Install virtualenv, then install pypy as virtual environment my-pypy-env

sudo apt-get install python-virtualenv
virtualenv -p pypy-2.2.1-linux64/bin/pypy my-pypy-env

Modules must be installed separatedly for this virtual environment. For instance

./my-pypy-env/bin/pip install libnum

Run: Run python programs using python or pypy

./my-pypy-env/bin/pypy

Reference

range

Basic

Statements

try: statement(s) except [expression [,target]]: statement(s) [else: statement(s)]	try: statement(s) finally: statement(s)	try: statement(s) except [expression [,target]]: statement(s) finally: statement(s)
expression is a class or tuple of classes. target is variable that will store exception object. `else` clause is executed if `try` block terminates, i.e. not on exception or if a `break` occurs. `try-except-finally` is Python 2.5.

Basic - Examples

for i in range(10):
    print i                      # carriage return

for i in range(10):
    print i,                     # no carriage return

for key in d:                       # Loop over keys in dictionary d
for key, value in d.iteritems():    # Loop over keys and values in dictionary d

a = 'global'
def afunction():
    global a
    a = 'still using global'
    b = 'local'

import os.path
os.path.isfile(fname)            # True if fname exists and is a file

if not os.path.exists(directory):
    os.makedirs(directory)       # Create directory if does not exists

try:                             # Avoid race condition if directory created by another process
    os.makedirs(path)            # But we could fix solution above as well
except OSError:                  # This one always trigger an exception in nominal case
    if not os.path.isdir(path):  
        raise

s.upper()                             # string s to uppercase
', '.join(set_3)                      # Join a sequence
hex_data = "deadbeef".decode("hex")   # "\xde\xad\xbe\xef"
map(ord, hex_data)                    # [0xDE, 0xAD, 0xBE, 0xEF]

sys.argv, len(sys.argv)          # Argument list, number of arguments ([0] -> exec name)
if ("-h" in sys.argv) or ("--help" in sys.argv):
    printUsage()
for a in range(len(sys.argv)):
    if sys.argv[a] == "-e":
        # handler

# Sort based on object attribute
ut.sort(key=lambda x: x.count, reverse=True)   # To sort the list in place...
newlist = sorted(ut, key=lambda x: x.count, reverse=True)  # To return a new list, use the sorted() built-in function...

(From stackoverflow [3])

for c in list(sha256.digest()):
    key.append(ord(c))

Random

IV = []
for i in range(16):
    IV.append(randint(0, 255))

Miscellaneous conversion

print list("abc")               # ['a', 'b', 'c']

Format operator %

print '%x' % variable            # Print hex

math

print 1//2                       # floor division (PEP-238)

System

sys.exit()

Modules

import datetime
print datetime.datetime.today()  
print datetime.datetime.now()    # similar, but possibly more accurate
print datetime.date.now()        # date only

Advanced

mymodule = __import__('mymodule')          # Import module from string - see http://effbot.org/zone/import-string.htm

Modular inverse [4]

# Using gmpy - FASTEST
import gmpy
gmpy.invert(1234567, p)                      # 1000000 loops, best of 3: 737 ns per loop (p 1024-bit)
gmpy.divm(1, 1234567, p)                     # 1000000 loops, best of 3: 933 ns per loop (p 1024-bit)

# Using egcd function - NO DEPS, BUT SLOWER
def egcd(a, b):
    if a == 0:
        return (b, 0, 1)
    else:
        g, y, x = egcd(b % a, a)
        return (g, x - (b // a) * y, y)

def modinv(a, m):
    g, x, y = egcd(a, m)
    if g != 1:
        raise Exception('modular inverse does not exist')
    else:
        return x % m
timeit modinv(1234567,p)                     # 100000 loops, best of 3: 13.6 us per loop (p 1024-bit)

# Using pow() - SIMPLEST BUT SLOWEST
timeit pow(1234567,p-2,p)                    # 100 loops, best of 3: 4.22 ms per loop

modular exponentiation

from gmpy import mpz
def power_mod(a, b, n):
    return long(pow(mpz(a),b,n))

Python list

See [5]

Tips

Simple HTTP Server

It's very easy to setup an ad-hoc HTTP server with Python. Just open a shell in a folder with some contents to share, and type:

python -m SimpleHTTPServer

More available at http://docs.python.org/2/library/internet.html (see BaseHTTPServer and CGIHTTPServer).

Detect interactive mode

References: [6], [7]

Started with	First method	Second method	Third method	Fourth method
	`import __main__ as main print hasattr(main, '__file__')`	`def in_ipython(): try: __IPYTHON__ except NameError: return False return True`	`import sys print hasattr(sys, 'ps1'):`	`import sys print bool(sys.flags.interactive)`
`python mymod.py`	True	-	-	-
`python -i mymod.py`	True	-	-	True
`python` then `import mymod`	-	-	True	-
`ipython mymod.py`	True	True	-	-
`ipython -i mymod.py`	True	True	-	-
`ipython` then `run mymod.py`	True	True	-	-
`ipython` then `run -i mymod.py`	True	True	-	-
`ipython` then `import mymod`	-	True	-	-
`ipython -i` then `import mymod`	-	True	-	-

Find duplicates in list

From stackoverflow [8]

import collections

def fastest():                         # 134 us - Fastest
    seen = set()
    seen_add = seen.add                                            # To avoid lookup 'add' ever time an item is inserted
    seen_twice = set( x for x in l if x in seen or seen_add(x) )   # adds all elements it doesn't know yet to seen and all other to seen_twice
    return list( seen_twice )                                      # turn the set into a list (as requested)

def compact():                         # 415 us
    return [x for x, y in collections.Counter(l).items() if y > 1]

def slowest():                         # 19.2 ms
    return list(set([x for x in l if l.count(x) > 1]))

Start post-mortem debugger on exception

From stackoverflow [9]

>>> import pdb
>>> pdb.pm()

Miscellaneous

Detect whether a variable is defined

Note it is bad practice to define a variable conditionally [10]. An interesting use case is to run code and define variable conditionally based on interactive status.

# Using try ... except
try: myvar
except NameError: print "variable 'myvar' IS defined"

# Using vars() / globals()
'myvar' in vars() or 'myvar' in globals()
# ...pedantic...
'myvar' in vars(__builtins__)

Analyse memory usage

Dowser

See [11] — seems better suited to find memory leaks, not to analyse usage for memory hungry applications

memory_profiler

See [12]
Install

sudo pip install -U memory_profiler
sudo pip install psutil

Add @profile decorator

@profile
def primes(n): 
    ...

Run the profiler

 python -m memory_profiler primes.py

Do's and don't's

foo = 'abcdef' l = list(foo) # DO	foo = 'abcdef' l = [c for c in foo] # don't
foo = list(...) g = map(blah,foo] # DO	foo = list(...) g = [blah(i) for i in foo] # don't

Traps

Frequent mistakes. Beware the snake can bite you!

Confuse a method and a property in a test: SOLUTION: Stick to a convention. Like always define methods like isxyyz() or hasabc() as methods. Note that defining them as property would raise an exception if used as a function, and hence might be safer.

if A.isdummy():            # This will fail isdummy is a property
if A.isdummy:              # Always True if isdummy is a method

Mix 0 with None in a sequence: Testing whether an element is defined is more difficult.

a = [0,None,None,None]
bool(a[0])           # --> False
bool(a[1])           # --> False !!! How can we tell them apart?
a[1] == None         # --> True      This works, but is unusual and likely bad practice

Mixing property and normal getter: SOLUTION: prefix all getter method with get, like getvalue()

b = a.prop           # Using a property, OR
b = a.getprop()      # Using a getter

Docstrings

Specifications: pep-0257

To write good module docstrings, "think about somebody doing help(yourmodule) at the interactive interpreter's prompt — what do they want to know?" [13].
See pep-0257 for more recommendations

Using doctest

You can include tests, in the form of examples, in your Python modules' docstrings. Properly written, these tests can be executed and verified by the doctest module. [14]

Libraries

Big numbers

gmpy based on GMP
libnum a lighter bignum library, but compatible with pypy.

Unicode

Set source file encoding

Add any of these lines [15]:

# -*- coding: utf-8 -*-
# vim: set fileencoding=utf-8 :

Write the BOM

See [16]

import codecs

file = codecs.open("lol", "w", "utf-8")
file.write(u'\ufeff')                          # or use unicode name: u'\N{ZERO WIDTH NO-BREAK SPACE}'
file.close()

# Using https://docs.python.org/2/library/codecs.html#module-encodings.utf_8_sig
with codecs.open("test_output", "w", "utf-8-sig") as temp:
    temp.write("hi mom\n")

Handling unicode

Some recommends to always process unicode internally, and decode on input and encode on output [17]:

line = line.decode('utf-8')
# ...treat line as unicode...
print line.encode('utf-8')

But this is error prone. So another solution proposed is to redefine sys.stdout:

import sys
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)

An hackish way (not recommended):

# -*- coding: utf-8 -*-
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
print u"åäö"

Python

Contents

Links

Python modules

Python variants

iPy

Pypy

Reference

Basic

Basic - Examples

Modules

Advanced

Tips

Simple HTTP Server

Detect interactive mode

Find duplicates in list

Start post-mortem debugger on exception

Miscellaneous

Analyse memory usage

Do's and don't's

Traps

Docstrings

Libraries

Unicode

Navigation menu

Python

Links

Python modules

Python variants

iPy

Pypy

Reference

Basic

Basic - Examples

Modules

Advanced

Tips

Simple HTTP Server

Detect interactive mode

Find duplicates in list

Start post-mortem debugger on exception

Miscellaneous

Analyse memory usage

Do's and don't's

Traps

Docstrings

Libraries

Unicode

Navigation menu

Search