| Age | Commit message (Collapse) | Author |
|---|
|
The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several
packages.
Although our manifest file has several issues:
- Its unorganized and hard to read and understand it for an average
human being.
- When a new package needs to be added, the user actually has to modify
the script that creates the manifest, then call the script to create
a new manifest, and then submit a patch for both the script and the
manifest, so its a little convoluted.
- Git complains every single time a patch is submitted to the manifest,
since it violates some of its guidelines.
- It changes or may change with every release of python, its impossible
to know if the required files for a certain package have changed
(it could have more or less dependencies), the only way of doing so
would be to install and test them all one by one on separate individual
images, and even then we wouldnt know if they require less dependencies,
we would just know if an extra dependency is required since it would
complain, lets face it, this isnt feasible.
- The same thing happens for new packages, if someone wants to add a
new package, its dependencies need to be checked manually one by one.
This patch fixes those issues, while adding some additional features.
Features/Fixes:
- A new manifest format is used (JSON), easy to read and understand.
This file is parsed by the python recipe and python packages read
from here are passed directly to bitbake during parsing time.
- It provides an automatic manifest creation task (explained below),
which automagically checks for every package dependencies and adds
them to the new manifest, hence we will have on each package exactly
what that package needs to be run, providing finer granularity.
- Dependencies are also checked automagically for new packages (explained below).
- Fixes the manifest in the following ways:
* python-core should be base and all packages should depend on it,
fixes lang, string, codecs, etc.
* Fixes packages with repeated files (e.g. bssdb and db, or
netclient and mime, and many others).
- Sitecustomize was fixed since encoding was deprecated.
- The JSON manifest file invalidates bitbake's cache, so if it changes
the python package will be rebuilt.
- It creates a solution for users that want precompiled bytecode files
(*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to
include such files, some argument they get faster boot time, even when the
files would be created on their first run?, but they also sometimes give a
magic number error and take up space, so we leave it to the user to
decide if they want them or not.
- Fixes python-core dependencies, e.g.
When python is run on an image, it TRIES to import everything it needs,
but it doesnt necessarily fails when it doesnt find something, so even if
we didnt know, we had errors like (trimmed on purpose):
# trying /usr/lib/python2.7/_locale.so
# trying /usr/lib/python2.7/lib-dynload/_locale.so
# trying /usr/lib/python2.7/_sysconfigdata.so
while it didnt complain about _locale it should have imported it,
after creating a new manifest with the automated script we get:
# trying /usr/lib/python2.7/lib-dynload/_locale.so
dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so
How to use (after a new release of python, or maybe before every OE
release):
- A new task called create_manifest was added to the python package,
which may be invoked via:
$ bitbake python -c create_manifest
This task runs a script on native python on our HOST system, and since
the python and python-native packages come from the same source, we can
use it to know the dependencies of each module as if we were doing it
on an image, this script is called create_manifest.py and in a very
simplistic way it does the following:
1. Reads the JSON manifest file and creates a dictionary data structure
with all of our python packages, their FILES, RDEPENDS and SUMMARY.
2. Loops through all of them and runs every module listed on them
asynchronously, determining every dependency that they have.
3. These module dependencies are then handled, to be able to know which
packages contain those files and which should RDEPEND on one another.
4. The data structure that comes out of this, is then used to create a
new manifest file which is automatically copied onto the user's python
directory replacing the old one.
Create_manifest script features:
- Handles modules which dont exist anymore (new release for example).
- Handles modules that are builtin.
- Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
- Deals with packages which include folders.
- Deals with packages which include FILES with a wildcard.
- The manifest can be constructed on a multilib environment as well.
- This method works for both python modules and shared libraries used
by python.
How to add a new package:
- If a user wants to add a new package all that has to be done is
modify the python2-manifest.json file, and add the required file(s)
to the FILES list, the script should handle all the rest.
Real example:
We want to add a web browser package, including the file webbrowser.py
which at the moment is on python-misc.
"webbrowser": {
"files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
"rdepends": [],
"summary": "Python Web Browser support"}
Run bitbake python -c create_manifest and the resulting manifest
should be completed after a few seconds, showing something like:
"webbrowser": {
"files": ["${libdir}/python2.7/webbrowser.py"],
"rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
"summary": "Python Web Browser support"}
Known errors/issues:
- Some special packages are handled differently: core, misc,
modules,dev, staticdev.
All these should be handled manually, because they either include
binaries, static libraries, include files, etc. (something that we
cant import).
Specifically static libraries are not not supported by this method
and have to be handled by the user.
- The change should be transparent to the user, other than the fact
that now we CANT build python-foo (it was pretty dumb anyway, since
what building python-foo actually did was building the whole python
package anyway), but doing IMAGE_INSTALL_append = " python-foo"
would create an image with the requested package with no issues.
[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* runpy allows running modules/scripts with 'python -m foo'
* python3-setuptools RDEPENDS on plistlib (present in python2)
* pip3 RDEPENDS on _markupbase (add to python3-core)
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The generators create python-*-manifest.inc files with
lines over 2500 characters long which breaks sending
patches via git send-email (because of smtp limitation).
This patchset formats all the long lines into multiple lines.
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Although 'test' is listed in the python module list
(https://docs.python.org/3/py-modindex.html) it is meant only to be
used 'internally' by folks developing python itself. Per the
documentation:
Note The test package is meant for internal use by Python only. It
is documented for the benefit of the core developers of Python. Any
use of this package outside of Python’s standard library is
discouraged as code mentioned here can change or be removed without
notice between releases of Python.
Per the above it is best not to include this module to discourage
folks who might not head the above warnings. Additionally this module
is one of the largest py modules going, by dropping this unneeded
module from the 'modules' package we can reduce overall image size,
something which is important for many embedded deployments.
The generator scripts as well as the manifests have thus been modified
accordingly, providing a generic mechanism to exclude modules from the
'all modules' package.
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Also clean up the logic in the script to be more Pythonic.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Move config/Makefile in libdir from core package to dev package for
python, because it is only needed in development process.
Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reference url is a stale, non existent site that returns a 404, so get rid of it
Change impacts both the manifest files and the scripts that generate the manifests
Run the following from within recipes-devtools/python
../../../scripts/contrib/python/generate-manifest-2.7.py > python-2.7-manifest.inc
../../../scripts/contrib/python/generate-manifest-2.7.py -n > python-native-2.7-manifest.inc
../../../scripts/contrib/python/generate-manifest-3.5.py > python-3.5-manifest.inc
../../../scripts/contrib/python/generate-manifest-3.5.py -n > python-native-3.5-manifest.inc
Signed-off-by: Bob Cochran <openembedded@mindchasers.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A potential flaw was introduced by commit a8dc4bc0:
[ python: fix python-tests rdepends ]
It made python-tests RDEPENDS on python-modules which is correct, but
this should be done in generate-manifest-2.7.py, and re-generate the
manifest.
Fixed by adding this dependency in generate-manifest-2.7.py as well.
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
python-native supposes to RPROVIDE all native packages as added in
generate-manifest-2.7.py, but it does not so far, this leads a problem
that sometimes bitbake cant find a runtime provider for a python-*-native
when a new runtime dependency on it being required, this usualy happens
after a new native python-* recipe is created or the old native python-*
recipes are upgraded.
To give a example, the following commit is trying to address such a issue:
commit 4583cd1bb15306e8f0ab7bcd80732e6f35aa4533:
[ python-native: Make python-native also RPROVIDE python-unittest-native ]
To avoid manually extending RPROVIDE every time when a new runtime
dependency is introduced, an argument '-n/--native' is added to the
manifest generator, allowing it create a native python manifest, with a
RPROVIDE line only, the RPROVIDE should contain all the sub-packages.
The generated python-native-2.7-manifest.inc is also added which is
included by python-native recipe.
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
|
|
file() API doesn't exist in python 3, convert to open(). Also handle
some cases where files aren't closed. Compatible with python 2.7.
[Contributions from Ed and Richard]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The compile() method returns objects that inherit from ast.AST so it's best that
python-core contains this class.
[YOCTO #8684]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
python-xml depends on python-elementtree as the latter just contains a C library
used by the former. However there's no point to this split apart from
increasing the number of packages, so merge -elementtree into python-xml.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Dominique Hunziker <dominique.hunziker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Upgrades python-setuptools to 19.2, easy_install works out of the box
adds the package python-plistlib to the manifest as it is needed by
setuptools now, and also updates runtime dependencies
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The python-io package includes ssl.py module which imports the
contextlib library.
This applied to Python 2.7.9 but not 3.3.
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* SimpleHTTPServer imports shutil so we need
definitly python-shell
* SocketServer can be run without the threading modul,
but I think we should run the full blown stuff, so
let us add python-threading
* this work was sponsored by sysmocom GmbH
Signed-off-by: Henning Heinold <henning@itconsulting-heinold.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
python-cryptography requires fractions module which is currently unpackaged.
Signed-off-by: Tim Orling <TicoTimo@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
- Based on Paul Eggletons work to partially upgrade to Python 2.7.6
Modified:
default-versions.inc: switched to python 2.7.9
generate-manifest-2.7.py: fixed _sysconfigdata
python-2.7-manifest.inc: fixed _sysconfigdata
python.inc: Updated checksums and source, no LICENSE
change just updated some dates
python-native_2.7.3 -> python-native_2.7.9 and updated patches
python_2.7.3 -> python_2.7.9, and added ac_cv_file__dev_ptmx=no
ac_cv_file__dev_ptc=no in EXTRA_OECONF to solve python
issue #3754, only needed when cross compiling, also updated patches
use_sysroot_ncurses_instead_of_host.patch: New patch to use ncursesw
from sysroot instead of hosts, introduced by fix for python issue #15268
Rebased:
01-use-proper-tools-for-cross-build.patch
03-fix-tkinter-detection.patch
05-enable-ctypes-cross-build.patch
06-avoid_usr_lib_termcap_path_in_linking.patch
avoid_warning_about_tkinter.patch
builddir.patch
fix_for_using_different_libdir.patch
host_include_contamination.patch
multilib.patch
nohostlibs.patch
search_db_h_in_inc_dirs_and_avoid_warning.patch
Deleted (fixed on upstream):
06-ctypes-libffi-fix-configure.patch
CVE-2013-4073_py27.patch
gcc-4.8-fix-configure-Wformat.patch
json-flaw-fix.patch
posix_close.patch
pypirc-secure.patch
python-2.7.3-CVE-2012-2135.patch
python-2.7.3-CVE-2013-1752-smtplib-fix.patch
python-2.7.3-CVE-2014-1912.patch
python-2.7.3-CVE-2014-7185.patch
python-2.7.3-berkeley-db-5.3.patch
python-fix-build-error-with-Readline-6.3.patch
remove-BOM-insection-code.patch
remove_sqlite_rpath.patch
python2.7.3-nossl3.patch
[YOCTO #7059]
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
|
|
This is required for python code using 'with' statements.
Signed-off-by: Philip Tricca <flihp@twobit.us>
|
|
Adding argparse module from Python's standard library. The module has replaced
previous python-argparse package that was build using source that are not part
of Python 2.7 release.
Signed-off-by: Maciej Borzecki <maciej.borzecki@open-rnd.pl>
Signed-off-by: Maciek Borzecki <maciek.borzecki@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A piece of JSON initialization code that runs when you "import json"
tries to use the hex-decoder, thus breaks if you do not have
python-codecs installed. Example:
>>> import json
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/json/__init__.py", line 108, in <module>
from .decoder import JSONDecoder
File "/usr/lib/python2.7/json/decoder.py", line 24, in <module>
NaN, PosInf, NegInf = _floatconstants()
File "/usr/lib/python2.7/json/decoder.py", line 18, in _floatconstants
_BYTES = '7FF80000000000007FF0000000000000'.decode('hex')
LookupError: no codec search functions registered: can't find encoding
This patch adds a runtime dependency on python-codecs for python-json and
re-generates the python manifests for Python v2.7. Solves [YOCTO #7020].
Signed-off-by: Magnus Olsson <magnus@minimum.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Fix this ssl import error:
Python 2.7.3 (default, Dec 5 2014, 16:24:17)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ssl.py", line 92, in <module>
import base64 # for DER-to-PEM translation
ImportError: No module named base64
Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
importlib isn't currently included in any of the python packages, so
create a new one for recipes that require it.
Cc: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Matt Fleming <matt.fleming@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some changes were made earlier by me to SUMMARY values in
python-2.7-manifest.inc without changing the manifest script.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Typically what's being set here is a short description, so it makes much
more sense to set SUMMARY.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The multiprocessing module needs the mmap module. We got away with not
having this for the build appliance because python-mmap was added to
packagegroup-self-hosted, but this is the proper place to have it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
After installing python-unittest, the following errors occur when executing
"import unittest" from a python commandline:
ImportError: No module named StringIO
ImportError: No module named pprint
ImportError: No module named difflib
ImportError: No module named pprint
ImportError: No module named fnmatch
Fix this by adding the missing dependencies to the generator script and run
the generator.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python-multiprocessing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As python-multiprocessing requires python-threading and
python-pickle, this commit adds them as runtime dependency.
The observed behavior was:
When typing 'import multiprocessing' in the python shell on a
minimal image with only the python-multiprocessing recipe installed,
python reports at first:
Python 2.7.3 (default, Jun 27 2013, 08:26:25)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing;
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/multiprocessing/__init__.py", line 65, in <module>
from multiprocessing.util import SUBDEBUG, SUBWARNING
File "/usr/lib/python2.7/multiprocessing/util.py", line 38, in <module>
import threading # we want threading to install it's
ImportError: No module named threading
After adding python-threading as runtime dependency and rebuilding
the image, python reports:
Python 2.7.3 (default, Jun 27 2013, 08:26:25)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing;
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/multiprocessing/__init__.py", line 84, in <module>
import _multiprocessing
ImportError: No module named cPickle
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@oss.bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When trying to import python-pprint on a minimal image, it reports that
the cStringIO python module is missing.
This is provided with python-io, so we add python-io as runtime
dependency.
The complete observed trace was:
Python 2.7.3 (default, Apr 4 2013, 07:45:36)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pprint
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/pprint.py", line 40, in <module>
from cStringIO import StringIO as _StringIO
ImportError: No module named cStringIO
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@oss.bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Modify the include file and script to generate a missing RDEPENDS.
Install python on target with python-io. Import ssl:
Python 2.7.3 (default, Feb 9 2013, 16:04:35)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ssl.py", line 58, in <module>
ImportError: No module named textwrap
Installing python-textutils solves the issue.
Signed-off-by: MiLo <milo-software@users.sourceforge.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
random.py imports hashlib, so add this missing dependency.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
The os.popen function would fail (more or less) silently if the executed
program cannot be found, and here what we need is os.system not os.popen
since it doesn't use the return value, use os.unlink() and ignore
exceptions from it would be better as Chris suggested.
[YOCTO #2454]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
Add intercept multilib header for pyconfig.h in python.
This is part of the bug fixing [YOCTO #2216].
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you install the top-level python package only on a minimal
system which has no other python packages installed then python
is not functional at all. Without any extra packages installed
this error is seen:
# python
Traceback (most recent call last):
File "/usr/lib/python2.7/site.py", line 64, in <module>
import traceback
ImportError: No module named traceback
Installing python-lang only partly fixes the problem as this
error still exists:
# python
Traceback (most recent call last):
File "/usr/lib/python2.7/site.py", line 569, in <module>
main()
File "/usr/lib/python2.7/site.py", line 551, in main
known_paths = addusersitepackages(known_paths)
File "/usr/lib/python2.7/site.py", line 278, in addusersitepackages
user_site = getusersitepackages()
File "/usr/lib/python2.7/site.py", line 253, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/usr/lib/python2.7/site.py", line 243, in getuserbase
USER_BASE = get_config_var('userbase')
File "/usr/lib/python2.7/sysconfig.py", line 520, in get_config_var
return get_config_vars().get(name)
File "/usr/lib/python2.7/sysconfig.py", line 400, in get_config_vars
import re
ImportError: No module named re
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* move 2to3 to separate package and include lib2to3 (was in python-misc)
* fix pattern for python-unittest (was in python-misc because it's in subdirectory now)
* add pydoc_data to python-pydoc (was in python-misc)
* add more stuff to smtpd, audio, codecs, ctypes, html, io, json, mime,
pickle, stringold, xmlrpc
* move all FILES_ details from python recipe to manifest generator so it's in one place
* added manual line break in FILES_${PN}-core, because git send-email
doesn't like too long lines
$ git send-email -1 dfaae65839f0ab23e5b2ae2a68df0f370bca84d2
fatal: /tmp/k8zbDajUNP/0001-python-improve-packaging.patch: 64: patch contains a line longer than 998 characters
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
script: add needed files into the python-core package
regenerate python-2.7-manifest.inc file with newer script
Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
|
* it needs to be regenerated to actually package something
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
|
Alejandro Hernandez
Tim Orling
Mark Asselstine
Ross Burton
Li Zhou
Bob Cochran
Ming Liu
Ed Bartosh
Richard Purdie
Dominique Hunziker
Ash Charles
Henning Heinold
Tim Orling
Philip Tricca
Maciej Borzecki
Magnus Olsson
Maxin B. John
Matt Fleming
Paul Eggleton
Mike Looijmans
Lukas Bulwahn
MiLo
Robert Yang
Lianhao Lu
Gary Thomas
Martin Jansa
Nitin A Kamble