SciPy 0.7.0 is the culmination of 16 months of hard work. It containsmany new features, numerous bug-fixes, improved test coverage andbetter documentation. There have been a number of deprecations andAPI changes in this release, which are documented below. All usersare encouraged to upgrade to this release, as there are a large numberof bug-fixes and optimizations. Moreover, our development attentionwill now shift to bug-fix releases on the 0.7.x branch, and on addingnew features on the development trunk. This release requires Python2.4 or 2.5 and NumPy 1.2 or greater.
Please note that SciPy is still considered to have “Beta” status, aswe work toward a SciPy 1.0.0 release. The 1.0.0 release will mark amajor milestone in the development of SciPy, after which changing thepackage structure or API will be much more difficult. Whilst thesepre-1.0 releases are considered to have “Beta” status, we arecommitted to making them as bug-free as possible. For example, inaddition to fixing numerous bugs in this release, we have also doubledthe number of unit tests since the last release.
However, until the 1.0 release, we are aggressively reviewing andrefining the functionality, organization, and interface. This is beingdone in an effort to make the package as coherent, intuitive, anduseful as possible. To achieve this, we need help from the communityof users. Specifically, we need feedback regarding all aspects of theproject - everything - from which algorithms we implement, to detailsabout our function’s call signatures.
Over the last year, we have seen a rapid increase in communityinvolvement, and numerous infrastructure improvements to lower thebarrier to contributions (e.g., more explicit coding standards,improved testing infrastructure, better documentation tools). Overthe next year, we hope to see this trend continue and invite everyoneto become more involved.
Python 2.6 and 3.0#
A significant amount of work has gone into making SciPy compatiblewith Python 2.6; however, there are still some issues in this regard.The main issue with 2.6 support is NumPy. On UNIX (including Mac OSX), NumPy 1.2.1 mostly works, with a few caveats. On Windows, thereare problems related to the compilation process. The upcoming NumPy1.3 release will fix these problems. Any remaining issues with 2.6support for SciPy 0.7 will be addressed in a bug-fix release.
Python 3.0 is not supported at all; it requires NumPy to be ported toPython 3.0. This requires immense effort, since a lot of C code hasto be ported. The transition to 3.0 is still under consideration;currently, we don’t have any timeline or roadmap for this transition.
Major documentation improvements#
SciPy documentation is greatly improved; you can view a HTML referencemanual online or download it as a PDFfile. The new reference guide was built using the popular Sphinx tool.
This release also includes an updated tutorial, which hadn’t beenavailable since SciPy was ported to NumPy in 2005. Though notcomprehensive, the tutorial shows how to use several essential partsof Scipy. It also includes the ndimage documentation from thenumarray manual.
Nevertheless, more effort is needed on the documentation front.Luckily, contributing to Scipy documentation is now easier thanbefore: if you find that a part of it requires improvements, and wantto help us out, please register a user name in our web-baseddocumentation editor at https://docs.scipy.org/ and correct the issues.
Running Tests#
NumPy 1.2 introduced a new testing framework based on nose. Starting withthis release, SciPy now uses the new NumPy test framework as well.Taking advantage of the new testing framework requires noseversion 0.10, or later. One major advantage of the new framework isthat it greatly simplifies writing unit tests - which has all readypaid off, given the rapid increase in tests. To run the full testsuite:
>>> import scipy>>> scipy.test('full')
For more information, please see The NumPy/SciPy Testing Guide.
We have also greatly improved our test coverage. There were just over2,000 unit tests in the 0.6.0 release; this release nearly doublesthat number, with just over 4,000 unit tests.
Building SciPy#
Support for NumScons has been added. NumScons is a tentative new buildsystem for NumPy/SciPy, using SCons at itscore.
SCons is a next-generation build system, intended to replace thevenerable Make with the integrated functionality ofautoconf/automake and ccache. Scons is written in Pythonand its configuration files are Python scripts. NumScons is meant toreplace NumPy’s custom version of distutils providing moreadvanced functionality, such as autoconf, improved fortransupport, more tools, and support for numpy.distutils/sconscooperation.
Sandbox Removed#
While porting SciPy to NumPy in 2005, several packages and moduleswere moved into scipy.sandbox. The sandbox was a staging groundfor packages that were undergoing rapid development and whose APIswere in flux. It was also a place where broken code could live. Thesandbox has served its purpose well, but was starting to createconfusion. Thus scipy.sandbox was removed. Most of the code wasmoved into scipy, some code was made into a scikit, and theremaining code was just deleted, as the functionality had beenreplaced by other code.
Sparse Matrices#
Sparse matrices have seen extensive improvements. There is nowsupport for integer dtypes such int8, uint32, etc. Two newsparse formats were added:
new class
dia_matrix: the sparse DIAgonal formatnew class
bsr_matrix: the Block CSR format
Several new sparse matrix construction functions were added:
sparse.kron: sparse Kronecker productsparse.bmat: sparse version ofnumpy.bmatsparse.vstack: sparse version ofnumpy.vstacksparse.hstack: sparse version ofnumpy.hstack
Extraction of submatrices and nonzero values have been added:
sparse.tril: extract lower trianglesparse.triu: extract upper trianglesparse.find: nonzero values and their indices
csr_matrix and csc_matrix now support slicing and fancyindexing (e.g., A[1:3, 4:7] and A[[3,2,6,8],:]). Conversionsamong all sparse formats are now possible:
using member functions such as
.tocsr()and.tolil()using the
.asformat()member function, e.g.A.asformat('csr')using constructors
A = lil_matrix([[1,2]]); B = csr_matrix(A)
All sparse constructors now accept dense matrices and lists of lists.For example:
A = csr_matrix( rand(3,3) )andB = lil_matrix( [[1,2],[3,4]] )
The handling of diagonals in the spdiags function has been changed.It now agrees with the MATLAB(TM) function of the same name.
Numerous efficiency improvements to format conversions and sparsematrix arithmetic have been made. Finally, this release containsnumerous bugfixes.
Statistics package#
Statistical functions for masked arrays have been added, and areaccessible through scipy.stats.mstats. The functions are similarto their counterparts in scipy.stats but they have not yet beenverified for identical interfaces and algorithms.
Several bugs were fixed for statistical functions, of those,kstest and percentileofscore gained new keyword arguments.
Added deprecation warning for mean, median, var, std,cov, and corrcoef. These functions should be replaced by theirnumpy counterparts. Note, however, that some of the default optionsdiffer between the scipy.stats and numpy versions of thesefunctions.
Numerous bug fixes to stats.distributions: all generic methods nowwork correctly, several methods in individual distributions werecorrected. However, a few issues remain with higher moments (skew,kurtosis) and entropy. The maximum likelihood estimator, fit,does not work out-of-the-box for some distributions - in some cases,starting values have to be carefully chosen, in other cases, thegeneric implementation of the maximum likelihood method might not bethe numerically appropriate estimation method.
We expect more bugfixes, increases in numerical precision andenhancements in the next release of scipy.
Reworking of IO package#
The IO code in both NumPy and SciPy is being extensivelyreworked. NumPy will be where basic code for reading and writing NumPyarrays is located, while SciPy will house file readers and writers forvarious data formats (data, audio, video, images, matlab, etc.).
Several functions in scipy.io have been deprecated and will beremoved in the 0.8.0 release including npfile, save, load,create_module, create_shelf, objload, objsave,fopen, read_array, write_array, fread, fwrite,bswap, packbits, unpackbits, and convert_objectarray.Some of these functions have been replaced by NumPy’s raw reading andwriting capabilities, memory-mapping capabilities, or array methods.Others have been moved from SciPy to NumPy, since basic array readingand writing capability is now handled by NumPy.
The Matlab (TM) file readers/writers have a number of improvements:
default version 5
v5 writers for structures, cell arrays, and objects
v5 readers/writers for function handles and 64-bit integers
new struct_as_record keyword argument to
loadmat, which loadsstruct arrays in matlab as record arrays in numpystring arrays have
dtype='U...'instead ofdtype=objectloadmatno longer squeezes singleton dimensions, i.e.squeeze_me=Falseby default
New Hierarchical Clustering module#
This module adds new hierarchical clustering functionality to thescipy.cluster package. The function interfaces are similar to thefunctions provided MATLAB(TM)’s Statistics Toolbox to help facilitateeasier migration to the NumPy/SciPy framework. Linkage methodsimplemented include single, complete, average, weighted, centroid,median, and ward.
In addition, several functions are provided for computinginconsistency statistics, cophenetic distance, and maximum distancebetween descendants. The fcluster and fclusterdata functionstransform a hierarchical clustering into a set of flat clusters. Sincethese flat clusters are generated by cutting the tree into a forest oftrees, the leaders function takes a linkage and a flat clustering,and finds the root of each tree in the forest. The ClusterNodeclass represents a hierarchical clusterings as a field-navigable treeobject. to_tree converts a matrix-encoded hierarchical clusteringto a ClusterNode object. Routines for converting between MATLABand SciPy linkage encodings are provided. Finally, a dendrogramfunction plots hierarchical clusterings as a dendrogram, usingmatplotlib.
New Spatial package#
The new spatial package contains a collection of spatial algorithmsand data structures, useful for spatial statistics and clusteringapplications. It includes rapidly compiled code for computing exactand approximate nearest neighbors, as well as a pure-python kd-treewith the same interface, but that supports annotation and a variety ofother algorithms. The API for both modules may change somewhat, asuser requirements become clearer.
It also includes a distance module, containing a collection ofdistance and dissimilarity functions for computing distances betweenvectors, which is useful for spatial statistics, clustering, andkd-trees. Distance and dissimilarity functions provided includeBray-Curtis, Canberra, Chebyshev, City Block, Cosine, Dice, Euclidean,Hamming, Jaccard, Kulsinski, Mahalanobis, Matching, Minkowski,Rogers-Tanimoto, Russell-Rao, Squared Euclidean, StandardizedEuclidean, Sokal-Michener, Sokal-Sneath, and Yule.
The pdist function computes pairwise distance between allunordered pairs of vectors in a set of vectors. The cdist computesthe distance on all pairs of vectors in the Cartesian product of twosets of vectors. Pairwise distance matrices are stored in condensedform; only the upper triangular is stored. squareform convertsdistance matrices between square and condensed forms.
Reworked fftpack package#
FFTW2, FFTW3, MKL and DJBFFT wrappers have been removed. Only (NETLIB)fftpack remains. By focusing on one backend, we hope to add newfeatures - like float32 support - more easily.
New Constants package#
scipy.constants provides a collection of physical constants andconversion factors. These constants are taken from CODATA RecommendedValues of the Fundamental Physical Constants: 2002. They may be foundat physics.nist.gov/constants. The values are stored in the dictionaryphysical_constants as a tuple containing the value, the units, and therelative precision - in that order. All constants are in SI units,unless otherwise stated. Several helper functions are provided.
New Radial Basis Function module#
scipy.interpolate now contains a Radial Basis Function module.Radial basis functions can be used for smoothing/interpolatingscattered data in n-dimensions, but should be used with caution forextrapolation outside of the observed data range.
New complex ODE integrator#
scipy.integrate.ode now contains a wrapper for the ZVODEcomplex-valued ordinary differential equation solver (by PeterN. Brown, Alan C. Hindmarsh, and George D. Byrne).
New generalized symmetric and hermitian eigenvalue problem solver#
scipy.linalg.eigh now contains wrappers for more LAPACK symmetricand hermitian eigenvalue problem solvers. Users can now solvegeneralized problems, select a range of eigenvalues only, and chooseto use a faster algorithm at the expense of increased memoryusage. The signature of the scipy.linalg.eigh changed accordingly.
Bug fixes in the interpolation package#
The shape of return values from scipy.interpolate.interp1d used tobe incorrect, if interpolated data had more than 2 dimensions and theaxis keyword was set to a non-default value. This has been fixed.Moreover, interp1d returns now a scalar (0D-array) if the inputis a scalar. Users of scipy.interpolate.interp1d may need torevise their code if it relies on the previous behavior.
Weave clean up#
There were numerous improvements to scipy.weave. blitz++ wasrelicensed by the author to be compatible with the SciPy license.wx_spec.py was removed.
Known problems#
Here are known problems with scipy 0.7.0:
weave test failures on windows: those are known, and are being revised.
weave test failure with gcc 4.3 (std::labs): this is a gcc 4.3 bug. Aworkaround is to add #include <cstdlib> inscipy/weave/blitz/blitz/funcs.h (line 27). You can make the change inthe installed scipy (in site-packages).
