8000 DOC : move MEP19 · matplotlib/matplotlib@f5f0c2c · GitHub
[go: up one dir, main page]
Skip to content

Commit f5f0c2c

Browse files
tacaswelltacaswell
committed
DOC : move MEP19
 - updated section on auto-building docs - updated section on S3 uploads - correctly credit Thomas Kluyver (takluyver), not me, for launchpad builds
1 parent 7884afc commit f5f0c2c

File tree

2 files changed

+194
-0
lines changed

2 files changed

+194
-0
lines changed

‎doc/devel/MEP/MEP19.rst

Copy file name to clipboard
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
===============================
2+
MEP19: Continuous Integration
3+
===============================
4+
5+
Status
6+
======
7+
8+
**Discussion**
9+
10+
Branches and Pull requests
11+
==========================
12+
13+
Abstract
14+
========
15+
16+
matplotlib could benefit from better and more reliable continuous
17+
integration, both for testing and building installers and
18+
documentation.
19+
20+
Detailed description
21+
====================
22+
23+
Current state-of-the-art
24+
------------------------
25+
26+
**Testing**
27+
28+
matplotlib currently uses Travis-CI for automated tests. While
29+
Travis-CI should be praised for how much it does as a free service, it
30+
has a number of shortcomings:
31+
32+
- It often fails due to network timeouts when installing dependencies.
33+
34+
- It often fails for inexplicable reasons.
35+
36+
- build or test products can only be saved from build off of branches
37+
on the main repo, not pull requsts, so it is often difficult to
38+
"post mortem" analyse what went wrong. This is particularly
39+
frustrating when the failure can not be subsequently reproduced
40+
locally.
41+
42+
- It is not extremely fast. matplotlib's cpu and memory requirements
43+
for testing are much higher than the average Python project.
44+
45+
- It only tests on Ubuntu Linux, and we have only minimal control over
46+
the specifics of the platform. It can be upgraded at any time
47+
outside of our control, causing unexpected delays at times that may
48+
not be convenient in our release schedule.
49+
50+
On the plus side, Travis-CI's integration with github -- automatically
51+
testing all pending pull requests -- is exceptional.
52+
53+
**Builds**
54+
55+
There is no centralized effort for automated binary builds for
56+
matplotlib. However, the following disparate things are being done
57+
[If the authors mentioned here could fill in detail, that would be
58+
great!]:
59+
60+
- @sandrotosi: builds Debian packages
61+
62+
- @takluyver: Has automated Ubuntu builds on Launchpad
63+
64+
- @cgohlke: Makes Windows builds (don't know how automated that is)
65+
66+
- @r-owen: Makes OS-X builds (don't know how automated that is)
67+
68+
**Documentation**
69+
70+
Documentation of master is now built by travis and uploaded to http://matplotlib.org/devdocs/index.html
71+
72+
@NelleV, I believe, generates the docs automatically and posts them on
73+
the web to chart MEP10 progress.
74+
75+
Peculiarities of matplotlib
76+
---------------------------
77+
78+
matplotlib has complex requirements that make testing and building
79+
more taxing than many other Python projects.
80+
81+
- The CPU time to run the tests is quite high. It puts us beyond the
82+
free accounts of many CI services (e.g. ShiningPanda)
83+
84+
- It has a large number of dependencies, and testing the full matrix
85+
of all combinations is impractical. We need to be clever about what
86+
space we test and guarantee to support.
87+
88+
Requirements
89+
------------
90+
91+
This section outlines the requirements that we would like to have.
92+
93+
#. Testing all pull requests by hooking into the Github API, as
94+
Travis-CI does
95+
96+
#. Testing on all major platforms: Linux, Mac OS-X, MS Windows (in
97+
that order of priority, based on user survey)
98+
99+
#. Retain the last n days worth of build and test products, to aid in
100+
post-mortem debugging.
101+
102+
#. Automated nightly binary builds, so that users can test the
103+
bleeding edge without installing a complete compilation
104+
environment.
105+
106+
#. Automated benchmarking. It would be nice to have a standard
107+
benchmark suite (separate from the tests) whose performance could
108+
be tracked over time, in different backends and platforms. While
109+
this is separate from building and testing, ideally it would run on
110+
the same infrastructure.
111+
112+
#. Automated nightly building and publishing of documentation (or as
113+
part of testing, to ensure PRs don't introduce documentation bugs).
114+
(This would not replace the static documentation for stable
115+
releases as a default).
116+
117+
#. The test systems should be managable by multiple developers, so
118+
that no single person becomes a bottleneck. (Travis-CI's design
119+
does this well -- storing build configuration in the git
120+
repository, rather than elsewhere, is a very good design.)
121+
122+
#. Make it easy to test a large but sparse matrix of different
123+
versions of matplotlib's dependencies. The matplotlib user survey
124+
provides some good data as to where to focus our efforts:
125+
https://docs.google.com/spreadsheet/ccc?key=0AjrPjlTMRTwTdHpQS25pcTZIRWdqX0pNckNSU01sMHc#gid=0
126+
127+
#. Nice to have: A decentralized design so that those with more
128+
obscure platforms can publish build results to a central dashboard.
129+
130+
Implementation
131+
==============
132+
133+
This part is yet-to-be-written.
134+
135+
However, ideally, the implementation would be a third-party service,
136+
to avoid adding system administration to our already stretched time.
137+
As we have some donated funds, this service may be a paid one if it
138+
offers significant time-saving advantages over free offerings.
139+
140+
Backward compatibility
141+
======================
142+
143+
Backward compatibility is not a major concern for this MEP. We will
144+
replace current tools and procedures with something better and throw
145+
out the old.
146+
147+
Alternatives
148+
============
149+
150+
151+
Hangout Notes
152+
=============
153+
154+
CI Infrastructure
155+
-----------------
156+
157+
- We like Travis and it will probably remain part of our arsenal in
158+
any event. The reliability issues are being looked into.
159+
160+
- Enable Amazon S3 uploads of testing products on Travis. This will
161+
help with post-mortem of failures (@mdboom is looking into this
162+
now).
163+
164+
- We want Mac coverage. The best bet is probably to push Travis to
165+
enable it for our project by paying them for a Pro account (since
166+
they don't otherwise allow testing on both Linux and Mac).
167+
168+
- We want Windows coverage. Shining Panda is an option there.
169+
170+
- Investigate finding or building a tool that would collect and
171+
synthesize test results from a number of sources and post it to
172+
Github using the Github API. This may be of general use to the
173+
Scipy community.
174+
175+
- For both Windows and Mac, we should document (or better yet, script)
176+
the process of setting up the machine for a build, and how to build
177+
binaries and installers. This may require getting information from
178+
Russel Owen and Christoph Gohlke. This is a necessary step for
179+
doing automated builds, but would also be valuable for a number of
180+
other reasons.
181+
182+
The test framework itself
183+
-------------------------
184+
185+
- We should investigate ways to make it take less time
186+
187+
- Eliminating redundant tests, if possible
188+
189+
- General performance improvements to matplotlib will help
190+
191+
- We should be covering more things, particularly more backends
192+
193+
- We should have more unit tests, fewer integration tests, if possible

‎doc/devel/MEP/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,5 @@ Matplotlib Enhancement Proposals
2121
MEP13
2222
MEP14
2323
MEP15
24+
MEP19
2425
MEP25

0 commit comments

Comments
 (0)
0