• LOGIN
  • No products in the cart.

Conventions over Restrictions – Programming the Python Way

Python is a powerful programming language. It supports procedural and object-oriented programming, and provides important features for functional programming. Its dynamic typing and many meta-programming features allow doing nearly everything at run time. While this freedom provides lots of opportunities for elegant solutions, Python might also be perceived as a dangerously unrestricted language. The common solution is to use conventions to solve a problem in the preferred, hence, “pythonic” way. Experienced Python programmers tend to stick to these conventions as much as possible and only diverge when the benefits are substantial compared to that of the conventional solution.

This article gives a short overview of Python features focusing on conventions and their benefits. Since Python syntax is often called executable pseudo code, a reader with solid programming experience does not need to have previous knowledge of Python to follow along.

Python – a quick overview

Python was publicly released in 1991 by its creator Guido van Rossum who was after a language that filled the gap between powerful but low-level system programming languages such as C and high-level but limited languages such as the ones used in shell programming. Since then Python has greatly developed and gained many users.

Python borrows from many different language such as C, Smalltalk, ABC, Java and Haskell. At first, this sounds like a hodgepodge of features, but the opposite is true: Python is very consistent and tries to offer one preferred solution for a problem. That does not mean that there are no other ways to accomplish something but there are conventions that call for the most “pythonic” solution. This “pythonic” solution takes full advantage of what Python offers. This solution will be the best in the vast majority of cases.

Features

Let’s look at some of the features that make Python so attractive that big players such as Google consider it a major tool and start-ups use it at the core of their app. Python compiles to bytecode that runs inside the Python interpreter, which is available for all major and a bunch of not so common operating systems. Writing platform-independent code is easy and Python offers many useful platform abstraction libraries. Typically, Python programs are considerably shorter than programs with comparably functionality written in C, C++, C# or Java. In addition, Python source code is usually very readable, because (1) the language itself was designed to be highly readable and (2) there are several readability-enhancing conventions around that developed over time and are cherished by many experienced Python programmers as best practice.

Data types

Python comes with only a few but really powerful data types. Besides simple data types such as numbers, it has built-in compound data types including strings, lists, dictionaries and sets. These data types can hold not only data, but also come with versatile algorithms that allow manipulation of their data. For example, you will rarely write your own sort algorithm in Python unless you want to demonstrate how an algorithm actually works. The built-in function sorted() or the sort() method of lists offers a very fast sorting mechanism for Python objects. Dictionaries and sets also offer well tested, performant algorithm that often allows implementing complex logic with just a few lines of Python.

Consistency

Python tries to be consistent whenever possible and is successful to enable the programmer to guess how things work once he understands the basic principles. One important principle is “Everything is an object.” I tend to repeat this sentence all the time in my courses. And indeed it helps. For example, numbers are objects as well as classes and functions. This allows storing functions as dictionary values and adds powerful possibilities that can make elegant code. In the end, it is just a consequence of being consistent.

Python users

Python is a general-purpose programming language that can be used for a wide variety of areas by different types of users.

Because of the comparably little effort needed to learn how to write basic programs, Python makes a very good teaching language to introduce people to programming. In fact, it can be used for high school students and as introductory programming courses at universities. For example, Python is the most used language by the top-ten US universities for Computer Science 101 courses.

But Python is also a good choice for professional software developers. It plays well with agile methodologies. For example, testing and test-driven development is an integral part of many larger Open Source Python libraries. Python scales and is suitable for large applications as its heavy use by YouTube suggests.

Scientists and engineers like Python. The language is small. Therefore, the user can understand and remember all important language features. This is especially useful for occasional programmers including many scientists. There are many high-quality libraries such as Jupyter, NumPy, Pandas, SciPy, IPython and matplotlib, and many more for specific domains that help to solve scientific and technical problems with often surprisingly little code. In fact, Python in combination with these libraries can be a useful alternative to MATLAB or other special domain languages for many common tasks in the scientific-technical field.

Many sysadmins that tried Python for their work just loved it. Scripts that are typically written in bash or similar operating system specific languages can often be expressed much clearer in Python. With a little care, these Python scripts can be largely operating system independent. They are often just a bit longer than bash scripts but arguably much easier to read. The longer the scripts, the more this readability shines. After all, small scripts tend to grow.

DevOps loves Python. For example, OpenStack is largely written in Python. Also, many test engineers use Python for their tasks.

Batteries included

Python comes with an extensive standard library that includes modules for all kinds of common programming tasks such as working with the operating system, network programming, logging, working with dates and times, profiling, XML or JSON parsing and much more. Also, the Python Package Index, PyPI, provides tens of thousands Open Source libraries for all kinds of tasks. It is very likely that there is a library there that does exactly or mostly what is needed as part of one’s next programming projects.

Python, CPython, Jython, IronPython, PyPy and Co.

When we talk about Python, we mostly mean CPython, the reference implementation written in C. This can be considered as the official implementation that defines the language. Several other implementations behave just like CPython but are based on different platforms. Jython is implemented in Java and runs on the JVM with full access to Java libraries. IronPython is implemented in C# and runs on the .NET platform with access to all its offerings. PyPy is written in Python and provides, amongst other things, a just-in-time compiler (JIT) that can considerably speedup Python programs for many use cases. There are a few more implementations with other focuses. But all of them should run pure Python source code, provided it was written for the Python version they implement.

Speed

Python is fast enough for many tasks. Server-side web programming is often limited by the database. Therefore, the performance of Python is good enough for many web use cases. Python is fast for text processing. Also for sysadmins, the speed of Python is commonly of little concern. The only real area where pure Python programs are slow is number crunching. So, solving a large system of equations in pure Python or processing large images might be a bit slow. Yet, these things are routinely done with Python.

Python is easily extended with C. Therefore, performance-critical parts can be implemented in C. In fact, NumPy, the Python library for working with numerical arrays, is written in C but can be used from Python by importing it. There are mature tools like SWIG, SIP and Cython used to connect Python and C with little effort.

The PyPy interpreter comes with a JIT (just-in-time compiler) that can speedup especially numerical calculations in loops. There are benchmarks where PyPy approaches C speed.

Python 2 or 3?

Python has been around for well over twenty-five years. Things are changing over time and Python needs to change too. Up to version 2.7, the last version of the 2.x branch, Python is backward compatible with very early versions. The chances are good that a script written in the last millennium with Python 1.5.2 still runs with Python 2.7 without or with only some minor modifications. For example, Python gained some keywords along the way such as yield. If a programmer used yield as an identifier, she would get a syntax error and would need to choose a different word as identifier.

Python 3 is the first intentional backward-incompatible version of Python. A few things needed to be removed and cleaned up. This would not have been possible without syntactic changes. Examples are: old-style classes are gone, all strings are only Unicode, the print statement became a function and several redundant features are gone too. Overall, Python has not changed too much for most users. It has become cleaner and even more consistent.

While Python 2 version is still widely used in production system while Python 3 is the future. Nearly all important Python libraries support Python 3. In fact, Ubuntu uses Python 3 as the default system Python version in one of its next releases. While there are tools that (semi-)automate the conversion from Python 2 to 3 or vice versa, nowadays, the preferred approach is a single code base for both Python 2 and 3.

All examples shown in this article are compatible with both Python 2 and Python 3. This shows that changes for a Python user are rather small, despite  the overhaul behind the scenes.

Get going and get interactive

Installing Python is simple. If you are on Linux, BSD or MaxOS X, you should have Python pre-installed. If not or if you would like to install a different version, you can do so via your package manager. Windows users should go to www.python.org and download the MSI installer for Python 2.7 or 3.5. Just clicking your way through the default settings should be fine.

We can try things interactively. Just typing python at the command line will bring up the interactive interpreter that prompts the user with >>>. If this does not work, you need to add the Python executable to the environmental variable PYTHONPATH. This may be the case on Windows. So, add c:\python35 or the path you specified during installation to PYTHONPATH in the control panel (system, extended, environmental settings, may vary depending on your Windows version). Now, every line of code you type will give an immediate response after the Enter. This is a great way to explore things. Listing 1 shows how to start an interactive session and do sophisticated math. Here, we use Python 3. Typing just python will bring up your default Python, which will probably be Python 2.7. There are much more powerful interactive interpreters around. The most prominent one is IPython that makes a very nice environment for exploratory Python programming.

Listing 1: Start of an interactive session.

python3

Python 3.5.2 (default, Jul  2 2016, 17:52:12)

[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> 1 + 1

2

Python works with a byte code compiler. A user usually has nothing special to do in invoking the compiler. Just saving your Python code in a file with the extension py and running python file_name.py from the command line should work. There are a few variations on this as listing 2 shows. On Windows, the file extension py may be associated with the Python-Interpreter. Therefore, typing the name of your program might be enough. On Unix like systems, you can make your script executable with chmod +x and add a shebang line such as # /usr/bin/env python at the beginning of the file.

Listing 2: Various ways to start a Python script at the command line connected to the Python interpreter.

python3 my_module.py

python my_module.py

my_module_on_windows.py

./my_module_on_unix_with_shebang.py

The Jupyter Notebook (previously known as IPython Notebook) provides a very interesting tool. It combines the interactivity of the Python prompt with storable document. In addition to that, it allows markdown for formatted text, tales and diagrams. It offers many productive tools for working with command line, run time measurements and can be converted into many different formats.

Conventions over rules

Python gives you a lot of flexibility to do many things at runtime. At the same time, there are many recommendations and conventions, what should be done, when and how. Why not enforce these conventions as rules? Conventions are good and useful for the common case. However, Python lets you explore not so common cases that might be the right solution for your very problem. It might not be obvious in the first place, but it is a good style to hide your unconventional programming with conventions. For example, you need to interface with an external library that requires some meta-programming magic to get what you want. The next thing you should do is to create a pythonic interface for the application program even though you are initially the only user of your library. If the language enforces rules that would disallow the magic in the first place, you might need to expose an ugly API or work around the rules anyway. Python allows you to do many things. With this power comes great responsibility and conventions can help you to become aware when you might do something unwise.

Readability

Python is renowned for its readability. This is a real killer feature because programmers read code much more than they  write it. Therefore, if a code is well-readable, the programmer can read and understand it faster, make changes quicker and can be overall more productive. Python is sensitive to indentation. Some people new to Python initially think this is a bad idea but most of them will realize after only a few minutes of playing around with the interactive interpreter or writing a few small scripts that indentation makes programming much easier if the language requires you to use the right indentation. Listing 3 shows a very simple for-loop. The body needs to be indented. As soon as the code is dedented and the loop is finished; no end keyword needed.

Listing 3: A for-loop that requires indentation for its body.

>>> for entry in [1, 2, 3, 4, 5]:

       print(entry)

1

2

3

4

5

Listing 4 shows how conditions can be treated with if-elif-else in Python. The colon at the end of the line is required and in turn requires an intended next line.

Listing 4: An if-elif-else construct that requires indentation after the colon.

>>> a = 10

>>> b= 20

>>> if a > b:

...     print('one')

... elif b > a:

...     print('two')

... else:

...     print('three')

...

two

In addition to this enforced indentation, many recommendations help to improve code readability. The famous Python Enhancement Proposal 8, short PEP8, is a detailed description how Python code should look like to be considered “good-looking.” Most of the larger Python projects adhere to PEP8 with some small variations that are often due to historical reasons. There are tools around that check Python code for PEP8 compliance such as pycodestyle (renamed from pep8), PyLint and PyFlakes. The latter two do static code analyses that go beyond PEP8.

What’s an iterable?

Iteration over some data structure is one of most common activities in procedural programming. Python lets you iterate over all objects that support the iterator protocol. These objects are then called iterables and can be iterated over with for-loops. People who are new to Python often do this similarly to the example in listing 5.

Listing 5: Unpythonic way to iterate over a list.

>>> my_list = [1, 2, 3]

>>> for i in range(len(my_list)):

...     print(my_list[i])

...

1

2

3

This is regarded as highly unpythonic. Listing 6 shows the way it is typically done in Python. No need to use indices. We can access the elements of a list directly.

Listing 6: Pythonic way to iterate over a list.

>>> for entry in my_list:

...     print(entry)

...

1

2

3

The same technique works for other sequences such as tuples, sets and dictionaries as shown in listing 7. In general, all sequences are iterable. We can also define own iterables but we won’t go into details about this here.

Listing 7: Iterating over other sequences / iterables

>>> my_tuple = (4, 5, 6)

>>> my_set = {7, 8, 9}

>>> my_dict = {10: 'a', 11: 'b', 12: 'c'}

>>> for entry in my_tuple:

...     print(entry)

...

4

5

6

>>> for entry in my_set:

...     print(entry)

...

8

9

7

>>> for entry in my_dict:

...     print(entry)

...

10

11

12

Iteration over more than one sequence is a common task. Again, explicit use of indices is not needed. We can use the built-in function zip() to access elements from two sequences at the same time. Listing 8 gives two examples. The second example shows what happens if both sequences are of different length. The additional elements of longer sequence are ignored.

Listing 8: Zipping two sequences.

>>> list(zip('abc', 'xyz'))

[('a', 'x'), ('b', 'y'), ('c', 'z')]

>>> list(zip([1, 2, 3, 4], [5, 6, 7]))

[(1, 5), (2, 6), (3, 7)]

The result is a list of tuples. This can really be useful, because we can iterate over two sequences at once. Listing 9 shows how to do this.

Listing 9: Zipping two sequences.

>>> for entry1, entry2 in (zip('abc', 'xyz')):

...     print(entry1, entry2)

...

a x

b y

c z

We use the zip() function to create the pairs as we did in Listing 8. But instead of converting the result to list, we just iterate over it. The variables entry1 and entry2 will hold the first and second value of each pair in the loop. Again, no explicit index anywhere. (Note: Listing 9 was created with Python 3. If you run this with Python 2, add from __future__ print_function as the first code line to get the same result.)

When it comes to loops, another nice feature of Python is list comprehensions. Adapted from Haskell, they allow very short, yet readable code. Listing 10 shows an example for a list comprehension.

Listing 10: A simple list comprehension.

[x + 10 for x in [1, 2, 3]]

[11, 12, 13]

We create a new list from the existing list [1, 2 ,3] by adding 10 to each element. This may seem a bit different from the common looping, but many users really seem to like it after they get used to it.

We are not limited to two sequences and they can be of different type as long as they implement the iterator protocol. Listing 11 shows an example for four different sequences.

Listing 11: Zipping more sequences.

>>> [(x1, x2, x3, x4) for x1, x2, x3, x4 in zip(my_list, my_tuple, my_set, my_dict)]

[(1, 4, 8, 10), (2, 5, 9, 11), (3, 6, 7, 12)]

Now, we combine a list comprehension with the looping over more than one sequence. The result is a total rearrangement of our original data structures; all in one line.

A class example

Let’s look at a code example to see some of the language features and conventions at work. Listing 12 shows a sample module with a sample class. A module is a file with Python source code in it and has the extension py.

Listing 12: A sample module with a sample class, simple_class.py

#! /usr/bin/env python

# coding: utf-8

"""A sample module with one (silly) class.

"""

class MyClass(object):

   """Just a sample class.

   """

   def __init__(self, value):

       self.value = value

   def add(self, increment):

       """

       Add something to instance attribute `value`.

       """

       self.value += increment

   def show_value(self):

       """

       Print the value to the screen.

       """

       print(self.value)

   def __repr__(self):

       return 'MyClass({})'.format(self.value)

   def __str__(self):

       return 'instance of MyClass with value {0}'.format(self.value)

if__name__=='__main__':

   def test():

       """

       Test our class.

       """

       my_instance = MyClass(10)

       my_instance.show_value()

       my_instance.add(5.5)

       my_instance.show_value()

       print(my_instance)

   test()

10

15.5

instance of MyClass with value 15.5

The so called shebang (#!) points to the Python interpreter. On Unix-based operating systems such as Linux or Mac OS X, making the file executable allows executing it directly without typing python simple_class.py. We encode our file with UTF-8. While this is the default in Python 3, we need to make this explicit in Python 2 with # coding: utf-8. Now comes the first convention. We add a docstring to our module that tells us what it is doing. The text A sample module with one (silly) class. is written in triple quotes that preserve all new lines. This dosctrings serves as documentation when we read our source. At the same time, it is available at run time and there are plenty of tools such as Sphinx that use it to generate neat looking and searchable HTML or PDF documentation.

A class is defined with the keyword class. We inherit it from an object. While this is not necessary in Python 3, we should always do it in Python 2 to get new-style classes. Otherwise, in Python 2, we would get a long-depreciated old-style class. Of course our class also has a docstring. The special method __init__() allows to give our instance, here represented by self, initial attributes. We just assign value to self.value. There are three more methods in our class. The method add() just adds increment to our value, and show_value() prints it to the screen. The special method __repr__() returns a representation of the instance that, when evaluated as code, would create e a new instance. The special method __str__() returns a human-readable description of the instance. All methods except the special methods have docstrings. Again, these strings will be around at run time.

We would like to use our program standalone as well as a library in another program. To achieve this, we use the line if __name__ == '__main__’: Each module has a name that is stored in __name__. When the program runs from the command line, the name is __main__. Otherwise, if  we import the module, its name will be the file name without the extension py, hence simple_class. Thus,  if we only run our script from the command line, our function test() will be executed. The function test() has, of course, a docstring. It makes an instance of MyClass and calls some of its methods and eventually prints the instance, which triggers our method __str__().

More

These examples  provide only a small insight into common programming practices in Python. However, there are many more. The best way to get know them is to read good code or take a course for that matter (shameless plug).

Conclusion

Python is a powerful language that gives the user great flexibility to solve problems. At the same time, there is a culture of conventions that strongly recommends how to write Python code.

While they might be seen as unnecessary restrictions, these conventions turn out to be upmost helpful in the vast majority of cases. It makes sense to limit oneself to the solution that is deemed most pythonic, unless there are very good reasons not to do so. But these situations are really rare and will become rarer, the longer you program in Python.

 

About the author

mikemulMike Müller is a Python enthusiast and Python user since 1999. He has been teaching Python courses since 2004. He is the founder and CEO of Python Academy, a training and consulting company specializing in Python. He is a frequent speaker and tutorial instructor at Python conferences and actively involves in organizing Python conferences. For example, he was the chairman of the PyCon DE in 2011 and 2012 conferences as well as EuroPython 2014. He is the chairman of the German Python Software Foundation. Contact the author on www.python-academy.com

Resources

Python Academy – http://www.python-academy.com

Python – http://python.org

Jython – http://jython.org

IronPython – http://ironpython.net

PyPy – http://pypy.org

Cython – http://www.cython.org

SWIG – http://www.swig.org

SIP – http://wiki.python.org/moin/SIP

Jupyter – http://jupyter.org

NumPy – http://numpy.scipy.org

pandas – http://pandas.pydata.org

SciPy – http://scipy.org

IPython – http://ipython.org

matplotlib – http://matplotlib.sourceforge.net

Sphinx – http://www.sphinx-doc.org

October 16, 2017

0 responses on "Conventions over Restrictions - Programming the Python Way"

Leave a Message

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© HAKIN9 MEDIA SP. Z O.O. SP. K. 2013