The Development Process
of Python
Ezio Melotti
Something about me
- 26 years old
- Python Core Developer since June 2009
- Information Technology student and
Python Programming teacher
at the Turku University of Applied Sciences (Finland)
- Presented the talk "The development process of Python"
at EuroPython 2010 in July 2010 and
at PyCon IT 4 in May 2010
- Presented the talk "Unicode and Python 3"
at PyCon IT 3 in May 2009
Outline
- Community
- Tools
- HG
- Workflow
- CPython
Community
- Community
- Tools
- HG
- Workflow
- CPython
- Developers
- Mailing lists
- IRC channels
- Conferences
- Others
Community / Developers
- 154 core developers
- thousands of contributors
Community / Mailing Lists (MLs)
- python-dev
- the "heart of Python's development"
- core-mentorship
- help for contributors
- python-ideas
- where new ideas are proposed
- python-committers
- discussions for committers only
- python-checkins
- where all the commits are reported
- python-bugs-list & new-bugs-announce
- where the changes in the issue tracker are reported
Community / IRC channels
- #python-dev
- the official IRC channel about Python development
- some of the core developers hang around there
- CIA-NN reports updates in the tracker and commits
- py-bb reports buildbots status
- #python-docs
- for documentation-related issues
Both on irc.freenode.net
Community / Conferences
- EuroPython (Europe)
- PyCon (US)
- Other Python conferences around the world: Argentina, Asia Pacific,
Australia, Brasil, Finland, France, India, Ireland, Italy,
Japan, New Zealand, Poland, UK, Ukraine, ...
- Other Python-related conferences, e.g.: ScyPy conferences
(Euro, US, India), DjangoCon (Euro, US)
Tools
- Community
- Tools
- HG
- Workflow
- CPython
- Issue Tracker
- buildbot & bbreport
- Mercurial (HG)
Tools / Issue Tracker
-
http://bugs.python.org/
- Used to track bugs and feature requests
- Based on Roundup
- Supports Rietveld
- Supports OpenID
- Weekly issues reports sent to python-dev
- There's also a meta-tracker
Tools / Issue Tracker
Stats for the last 10 months:
- 2800+ open
(2700 last year)
- 1200+ open with patches
(1100 last year)
- 21200+ closed
(18200 last year)
Tools / Buildbots & bbreport
- buildbots
- recompile Python
- execute the whole test suite
- bbreport
- a command-line tool to analyze the buildbots results
- useful to find quickly what is failing and where
- written by me, Florent Xicluna, Victor Stinner, R. David Murray
- http://code.google.com/p/bbreport/
HG
- Community
- Tools
- HG
- Workflow
- CPython
- Terms and concepts
- Differences with SVN
- Repo structure
- Status of the branches
- Merging order
- Basic usage
- Clones organization
- hgrc
HG / Terms and concepts
- Repository (repo)
- The dir named
.hg
in the repository root dir
that contains the history of the project.
- Clone
- A copy of a repository.
- Committing
- Saving local changes to the repository.
- Updating
- Applying changes from the repo to the local copy.
- Changeset (cset)
- An atomic collection of changes to files in a repository.
- Pushing/Pulling
- Exchanging changesets from a repo to another.
HG / Differences with SVN
- HG is a DVCS
- distributed
- peer-to-peer
- doesn't require a network for most of the operation
- SVN is a VCS
- centralized
- client-server
- requires a network for most of the operation
- Two different tools, two different approaches
HG / Differences with SVN
HG / Differences with SVN
HG / Repo structure
6 named branches in the repo:
- default
- 3.2
- 3.1
- 2.7
- 2.6
- 2.5
"default" will be the next 3.x
HG / Status of the branches
development →
bug-fix only →
security-fix only
Release | Status |
3.3 (default) | development |
3.2 / 2.7 | bug-fix only |
3.1 / 2.6 / 2.5 | security-fix only |
3.3 will be out in August 2012, 3.2 released in February 2011.
2.6.7, 2.7.2, and 3.1.4 released this June.
HG / Status of the branches
| past | past | present | future |
Python 3.4 | | | | development |
Python 3.3 | | | development | bug-fix |
Python 3.2 | | development | bug-fix | security-fix |
Python 2.7 | | development | bug-fix | security-fix |
Python 3.1 | development | bug-fix | security-fix | |
Python 2.6 | development | bug-fix | security-fix | |
HG / Merging order
- Two parallel lines of development
- Changes applied to the oldest applicable branch first
- merged with newer branches
- copied from 2.x to 3.x or vice versa
(3.1 →) 3.2 → default
(2.6 →) 2.7
HG / Basic usage
hg init
creates a new repository
- ...but you won't need this
hg clone
creates a copy of an existing repo:
hg clone http://hg.python.org/cpython
or
hg clone ssh://hg@hg.python.org/cpython
(for committers)
hg branches
to see the list of branches
hg branch
to see the current branch
hg up <branchname>
to change branch
HG / Basic usage
hg pull
to get the latest
changesets in the local repo
hg up
to apply them to the working copy
hg pull -u
to do both at once
hg stat
to see the changed file in the working copy
hg diff
to see the changed code in the working copy
hg ci -m 'message'
to save local changes in the repo
hg push
to send the committed cset to another repo
HG / Clones organization
Single clone:
(dotted line = cloning; solid line = pushing/pulling.
h.p.o = hg.python.org repo; cpy = local pristine clone;)
- Pros
- Simple structure
- Less disk space used
- Cons
- Frequent branch changes and recompilations
- Easier to get the wrong branch
HG / Clones organization
A clone per branch:
- Pros
- No branch changes, no recompilations
- Easier to know the current branch
- Cons
- More complex structure
- More disk space used
HG / Clones organization
- You can work on several different bugs/features at once
- Several ways to do it:
- a clone per issue
- a branch per issue
- mq (Mercurial Queues) extension
- shelve extension
- plain old patches
- ...
Workflow
- Community
- Tools
- HG
- Workflow
- CPython
- Find something to fix/enhance
- Get a clone of Python
- Update and compile
- Fix/enhance it and check that it works
- Run the tests with regrtest
- Create or apply a patch
- Get someone else to review it
Workflow / Find something to fix/enhance
- A problem that you encountered
- An issue in the tracker
- A feature request
- A broken test
- A red buildbot
- A new test to add
- An "XXX" in the code
- A problem in the documentation
Workflow / Get a clone of Python
The main Python repo is at
http://hg.python.org/cpython
- read-only
hg clone http://hg.python.org/cpython
- read-write (for committers)
hg clone ssh://hg@hg.python.org/cpython
Open
http://hg.python.org/cpython with a browser to navigate through
the source.
Workflow / Update and compile
Update:
hg pull -u
to update the clone and the working copy
Compile:
./configure --with-pydebug && make -j2
to compile Python
--with-pydebug
to compile a debug build
-j2
to use 2 (or more) cores
--with-wide-unicode
for wide builds ...
You need to run make
only if C files change.
Python supports incremental compilation.
Workflow / Fix/enhance it and check it
- For bugs:
- write the tests and check that they fail
- fix the bug (and possibly run
make
)
- run the tests and check that they pass
- For new features the procedure is the same, but:
- you might have to ask on python-ideas/python-dev first
- you might have to write a PEP for major features/changes
Workflow / Fix/enhance the documentation
- The documentation is in
Doc/
- Written in ReST
- easy to learn
- easy to read/write
- Built with Sphinx
- use
make html
(in Doc/
) to build the HTML doc
and check the result in Doc/build/html/
make latex
(in Doc/
) and then
make all-pdf
(in Doc/build/latex/
)
to build the PDFs
- supports incremental building
Workflow / Run the tests with regrtest
Runner of the Python test suite. Lives in Lib/test/regrtest.py
.
Examples:
./python -m test.regrtest
to run the test suite
./python -m test.regrtest -uall
to run all the tests
./python -m test.regrtest -j4
to run the tests in 4 processes
./python -m test.regrtest test_foo
to run test_foo
./python -m test.regrtest -R 3:2
to check for refleaks
./python -m test.regrtest --help
to see all the options
Workflow / Create or apply a patch
- Create a patch:
- Run
make patchcheck
before creating it.
hg diff > issue1234.diff
- Apply it:
patch -p1 < issue4321.diff
, or
hg import --no-commit < issue4321.diff
You can also try patches uploaded on the tracker and:
- see if they apply cleanly and on what branches
- see if they solve the problem on your machine
- possibly fix and update them
- report your findings on the tracker
Workflow / Get someone else to review it
- Upload the patch or add a link to a remote hg repo on the tracker
- Get reviews on the issue or on the integrated Rietveld instance
- Ask someone on #python-dev to take a look
- Ask on the core-mentorship ML
- Ask on the python-dev ML
- When it's committed someone will review the commit
Most likely someone will ask you to fix something in the patch
Workflow / Guidelines and tips
Remember to:
- Follow the PEP 8 for Python code and the PEP 7 for C code
- Follow the conventions that already exist in the file(s)
- Add/update tests and run them with regrtest
- Check for refleaks with a debug build and
regrtest -R 3:2
- Add/update documentation, docstrings and comments
- Fix one issue at time
- Make patches againts the oldest branch (usually 2.7/3.2)
- Make the patches easy to review
Developers will usually take care of merging
and updating Misc/NEWS
.
Structure of CPython
- Community
- Tools
- HG
- Workflow
- CPython
- Python tree
- Lib
- Objects
- Modules
- Python, Include, Misc
- Doc
Structure of CPython / Python tree
trunk$ ls --group-directories-first
build Misc RISCOS libpython2.7.a python
Demo Modules Tools LICENSE python-gdb.py
Doc Objects config.log Makefile README
Grammar Parser config.status Makefile.pre setup.py
Include PC configure Makefile.pre.in
Lib PCbuild configure.in pyconfig.h
Mac Python install-sh pyconfig.h.in
Structure of CPython / Lib
- Contains most of the stdlib
- Python modules and packages
Lib/test
contains most of the Python test suite
- some packages have their own test suite
(e.g.
Lib/ctypes/test
)
Structure of CPython / Objects
Contains the C implementation of the Python objects:
- lists (listobjects.c)
- dicts (dictobject.c)
- tuples (tupleobject.c)
- strings (stringobject.c and unicodeobject.c)
- sets (setobject.c)
- files (fileobject.c)
- etc...
Structure of CPython / Modules
Contains the C implementation of some Python modules:
- math (mathmodule.c)
- cmath (cmathmodule.c)
- datetime (datetimemodule.c)
- itertools (itertoolsmodule.c)
- socket (socketmodule.c)
- etc...
Structure of CPython / Python, Include, Misc
- Python
- Contains the core of CPython.
- Include
- Contains the headers (*.h) of the C files in Objects, Modules, and Python.
- Misc
- Other files, e.g.:
Misc/NEWS
: where most of the changes are listed
Misc/ACKS
: acknowledgments to contributors
Structure of CPython / Doc
Contains the ReST sources of the documentation.
Doc/library
contains the ReST doc of the modules
Doc/c-api
contains the ReST doc of the C API
Doc/build/html
contains the HTML pages
generated with make html
Questions
Questions?
Read the new devguide!
Saturday and Sunday, join us at the CPython sprint!
We got core devs, bugs, and something to eat!