The Development Process
of Python

Ezio Melotti

Something about me

Outline

Community

Community / Developers

Community / Mailing Lists (MLs)

python-dev
the "heart of Python's development"
core-mentorship
help for contributors
python-ideas
where new ideas are proposed
python-committers
discussions for committers only
python-checkins
where all the commits are reported
python-bugs-list & new-bugs-announce
where the changes in the issue tracker are reported

Community / IRC channels

#python-dev
  • the official IRC channel about Python development
  • some of the core developers hang around there
  • CIA-NN reports updates in the tracker and commits
  • py-bb reports buildbots status
#python-docs
  • for documentation-related issues

Both on irc.freenode.net

Community / Conferences

Community / Others

Python insider
The official Python blog
http://blog.python.org/
Twitter
http://twitter.com/#!/PythonInsider
Planet Python
A collection of articles about Python
http://planet.python.org/
PSF (Python Software Foundation)
a non-profit organization devoted to the Python programming language
...

Tools

Tools / Issue Tracker

Tools / Issue Tracker

Stats for the last 10 months:

Tools / Buildbots & bbreport

buildbots
  • recompile Python
  • execute the whole test suite
bbreport
  • a command-line tool to analyze the buildbots results
  • useful to find quickly what is failing and where
  • written by me, Florent Xicluna, Victor Stinner, R. David Murray
  • http://code.google.com/p/bbreport/

Tools / Mercurial (HG)

HG

HG / Terms and concepts

Repository (repo)
The dir named .hg in the repository root dir that contains the history of the project.
Clone
A copy of a repository.
Committing
Saving local changes to the repository.
Updating
Applying changes from the repo to the local copy.
Changeset (cset)
An atomic collection of changes to files in a repository.
Pushing/Pulling
Exchanging changesets from a repo to another.

HG / Differences with SVN

HG / Differences with SVN

HG / Differences with SVN

HG / Repo structure

6 named branches in the repo:

"default" will be the next 3.x

HG / Status of the branches

developmentbug-fix onlysecurity-fix only

ReleaseStatus
3.3 (default)development
3.2 / 2.7bug-fix only
3.1 / 2.6 / 2.5security-fix only

3.3 will be out in August 2012, 3.2 released in February 2011.

2.6.7, 2.7.2, and 3.1.4 released this June.

HG / Status of the branches

pastpastpresentfuture
Python 3.4development
Python 3.3developmentbug-fix
Python 3.2developmentbug-fixsecurity-fix
Python 2.7developmentbug-fixsecurity-fix
Python 3.1developmentbug-fixsecurity-fix
Python 2.6developmentbug-fixsecurity-fix

HG / Merging order

(3.1 →) 3.2default
(2.6 →) 2.7

HG / Basic usage

HG / Basic usage

HG / Clones organization

Single clone:

(dotted line = cloning; solid line = pushing/pulling.
h.p.o = hg.python.org repo; cpy = local pristine clone;)

HG / Clones organization

A clone per branch:

HG / Clones organization

HG / hgrc

HG config file

Workflow

  1. Find something to fix/enhance
  2. Get a clone of Python
  3. Update and compile
  4. Fix/enhance it and check that it works
  5. Run the tests with regrtest
  6. Create or apply a patch
  7. Get someone else to review it

Workflow / Find something to fix/enhance

Workflow / Get a clone of Python

The main Python repo is at http://hg.python.org/cpython

read-only
hg clone http://hg.python.org/cpython
read-write (for committers)
hg clone ssh://hg@hg.python.org/cpython

Open http://hg.python.org/cpython with a browser to navigate through the source.

Workflow / Update and compile

Update:

Compile:

You need to run make only if C files change.

Python supports incremental compilation.

Workflow / Fix/enhance it and check it

Workflow / Fix/enhance the documentation

Workflow / Run the tests with regrtest

Runner of the Python test suite. Lives in Lib/test/regrtest.py.

Examples:

Workflow / Create or apply a patch

Create a patch:
Run make patchcheck before creating it.
hg diff > issue1234.diff
Apply it:
patch -p1 < issue4321.diff, or
hg import --no-commit < issue4321.diff

You can also try patches uploaded on the tracker and:

  1. see if they apply cleanly and on what branches
  2. see if they solve the problem on your machine
  3. possibly fix and update them
  4. report your findings on the tracker

Workflow / Get someone else to review it

Most likely someone will ask you to fix something in the patch

Workflow / Guidelines and tips

Remember to:

  1. Follow the PEP 8 for Python code and the PEP 7 for C code
  2. Follow the conventions that already exist in the file(s)
  3. Add/update tests and run them with regrtest
  4. Check for refleaks with a debug build and regrtest -R 3:2
  5. Add/update documentation, docstrings and comments
  6. Fix one issue at time
  7. Make patches againts the oldest branch (usually 2.7/3.2)
  8. Make the patches easy to review

Developers will usually take care of merging and updating Misc/NEWS.

Structure of CPython

Structure of CPython / Python tree

trunk$ ls --group-directories-first
build   Misc    RISCOS        libpython2.7.a  python
Demo    Modules Tools         LICENSE         python-gdb.py
Doc     Objects config.log    Makefile        README
Grammar Parser  config.status Makefile.pre    setup.py
Include PC      configure     Makefile.pre.in
Lib     PCbuild configure.in  pyconfig.h
Mac     Python  install-sh    pyconfig.h.in

Structure of CPython / Lib

Structure of CPython / Objects

Contains the C implementation of the Python objects:

Structure of CPython / Modules

Contains the C implementation of some Python modules:

Structure of CPython / Python, Include, Misc

Python
Contains the core of CPython.
Include
Contains the headers (*.h) of the C files in Objects, Modules, and Python.
Misc
Other files, e.g.:
  • Misc/NEWS: where most of the changes are listed
  • Misc/ACKS: acknowledgments to contributors

Structure of CPython / Doc

Contains the ReST sources of the documentation.

Questions

Questions?

Read the new devguide!

Saturday and Sunday, join us at the CPython sprint!
We got core devs, bugs, and something to eat!