Skip to content

Commit 6ea3d19

Browse files
opacamjoergbrech
opacam
authored and
joergbrech
committed
[LIBS - PART I] Initial refactor of library recipes (kivy#1944)
This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also:   - make the imports from the right module   - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also:   - make the imports from the right module   - fix hardcoded arch   - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t
1 parent a5e5897 commit 6ea3d19

File tree

22 files changed

+214
-163
lines changed

22 files changed

+214
-163
lines changed

‎pythonforandroid/build.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,7 @@ def build_recipes(build_order, python_modules, ctx, project_dir,
575575
info_main('Building {} for {}'.format(recipe.name, arch.arch))
576576
if recipe.should_build(arch):
577577
recipe.build_arch(arch)
578+
recipe.install_libraries(arch)
578579
else:
579580
info('{} said it is already built, skipping'
580581
.format(recipe.name))

‎pythonforandroid/recipe.py

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,25 @@ class Recipe(with_metaclass(RecipeMeta)):
106106

107107
archs = ['armeabi'] # Not currently implemented properly
108108

109+
built_libraries = {}
110+
"""Each recipe that builds a system library (e.g.:libffi, openssl, etc...)
111+
should contain a dict holding the relevant information of the library. The
112+
keys should be the generated libraries and the values the relative path of
113+
the library inside his build folder. This dict will be used to perform
114+
different operations:
115+
- copy the library into the right location, depending on if it's shared
116+
or static)
117+
- check if we have to rebuild the library
118+
119+
Here an example of how it would look like for `libffi` recipe:
120+
121+
- `built_libraries = {'libffi.so': '.libs'}`
122+
123+
.. note:: in case that the built library resides in recipe's build
124+
directory, you can set the following values for the relative
125+
path: `'.', None or ''`
126+
"""
127+
109128
@property
110129
def version(self):
111130
key = 'VERSION_' + self.name
@@ -479,9 +498,14 @@ def apply_patches(self, arch, build_dir=None):
479498

480499
def should_build(self, arch):
481500
'''Should perform any necessary test and return True only if it needs
482-
building again.
501+
building again. Per default we implement a library test, in case that
502+
we detect so.
483503
484504
'''
505+
if self.built_libraries:
506+
return not all(
507+
exists(lib) for lib in self.get_libraries(arch.arch)
508+
)
485509
return True
486510

487511
def build_arch(self, arch):
@@ -492,6 +516,19 @@ def build_arch(self, arch):
492516
if hasattr(self, build):
493517
getattr(self, build)()
494518

519+
def install_libraries(self, arch):
520+
'''This method is always called after `build_arch`. In case that we
521+
detect a library recipe, defined by the class attribute
522+
`built_libraries`, we will copy all defined libraries into the
523+
right location.
524+
'''
525+
if not self.built_libraries:
526+
return
527+
shared_libs = [
528+
lib for lib in self.get_libraries(arch) if lib.endswith(".so")
529+
]
530+
self.install_libs(arch, *shared_libs)
531+
495532
def postbuild_arch(self, arch):
496533
'''Run any post-build tasks for the Recipe. By default, this checks if
497534
any postbuild_archname methods exist for the archname of the
@@ -554,6 +591,27 @@ def install_libs(self, arch, *libs):
554591
def has_libs(self, arch, *libs):
555592
return all(map(lambda l: self.ctx.has_lib(arch.arch, l), libs))
556593

594+
def get_libraries(self, arch_name, in_context=False):
595+
"""Return the full path of the library depending on the architecture.
596+
Per default, the build library path it will be returned, unless
597+
`get_libraries` has been called with kwarg `in_context` set to
598+
True.
599+
600+
.. note:: this method should be used for library recipes only
601+
"""
602+
recipe_libs = set()
603+
if not self.built_libraries:
604+
return recipe_libs
605+
for lib, rel_path in self.built_libraries.items():
606+
if not in_context:
607+
abs_path = join(self.get_build_dir(arch_name), rel_path, lib)
608+
if rel_path in {".", "", None}:
609+
abs_path = join(self.get_build_dir(arch_name), lib)
610+
else:
611+
abs_path = join(self.ctx.get_libs_dir(arch_name), lib)
612+
recipe_libs.add(abs_path)
613+
return recipe_libs
614+
557615
@classmethod
558616
def recipe_dirs(cls, ctx):
559617
recipe_dirs = []

‎pythonforandroid/recipes/freetype/__init__.py

Lines changed: 11 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
from pythonforandroid.toolchain import Recipe
1+
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.logger import shprint, info
33
from pythonforandroid.util import current_directory
4-
from os.path import exists, join
4+
from os.path import join, exists
55
from multiprocessing import cpu_count
66
import sh
77

@@ -26,16 +26,7 @@ class FreetypeRecipe(Recipe):
2626

2727
version = '2.5.5'
2828
url = 'http://download.savannah.gnu.org/releases/freetype/freetype-{version}.tar.gz' # noqa
29-
30-
def should_build(self, arch):
31-
return not exists(
32-
join(
33-
self.get_build_dir(arch.arch),
34-
'objs',
35-
'.libs',
36-
'libfreetype.so',
37-
)
38-
)
29+
built_libraries = {'libfreetype.so': 'objs/.libs'}
3930

4031
def get_recipe_env(self, arch=None, with_harfbuzz=False):
4132
env = super(FreetypeRecipe, self).get_recipe_env(arch)
@@ -111,11 +102,14 @@ def build_arch(self, arch, with_harfbuzz=False):
111102
# First build, install the compiled lib, and clean build env
112103
shprint(sh.make, 'install', _env=env)
113104
shprint(sh.make, 'distclean', _env=env)
114-
else:
115-
# Second build (or the first if harfbuzz not enabled), now we
116-
# copy definitive libs to libs collection. Be sure to link your
117-
# recipes to the definitive library, located at: objs/.libs
118-
self.install_libs(arch, 'objs/.libs/libfreetype.so')
105+
106+
def install_libraries(self, arch):
107+
# This library it's special because the first time we built it may not
108+
# generate the expected library, because it can depend on harfbuzz, so
109+
# we will make sure to only install it when the library exists
110+
if not exists(list(self.get_libraries(arch))[0]):
111+
return
112+
self.install_libs(arch, *self.get_libraries(arch))
119113

120114

121115
recipe = FreetypeRecipe()

‎pythonforandroid/recipes/harfbuzz/__init__.py

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
from pythonforandroid.toolchain import Recipe
1+
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.util import current_directory
33
from pythonforandroid.logger import shprint
44
from multiprocessing import cpu_count
5-
from os.path import exists, join
5+
from os.path import join
66
import sh
77

88

@@ -23,13 +23,7 @@ class HarfbuzzRecipe(Recipe):
2323
version = '0.9.40'
2424
url = 'http://www.freedesktop.org/software/harfbuzz/release/harfbuzz-{version}.tar.bz2' # noqa
2525
opt_depends = ['freetype']
26-
27-
def should_build(self, arch):
28-
return not exists(
29-
join(
30-
self.get_build_dir(arch.arch), 'src', '.libs', 'libharfbuzz.so'
31-
)
32-
)
26+
built_libraries = {'libharfbuzz.so': 'src/.libs'}
3327

3428
def get_recipe_env(self, arch=None):
3529
env = super(HarfbuzzRecipe, self).get_recipe_env(arch)
@@ -68,12 +62,12 @@ def build_arch(self, arch):
6862
_env=env,
6963
)
7064
shprint(sh.make, '-j', str(cpu_count()), _env=env)
71-
self.install_libs(arch, join('src', '.libs', 'libharfbuzz.so'))
7265

7366
if 'freetype' in self.ctx.recipe_build_order:
74-
# Rebuild freetype with harfbuzz support
67+
# Rebuild/install freetype with harfbuzz support
7568
freetype = self.get_recipe('freetype', self.ctx)
7669
freetype.build_arch(arch, with_harfbuzz=True)
70+
freetype.install_libraries(arch)
7771

7872

7973
recipe = HarfbuzzRecipe()

‎pythonforandroid/recipes/jpeg/__init__.py

Lines changed: 2 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
from pythonforandroid.recipe import Recipe
22
from pythonforandroid.logger import shprint
33
from pythonforandroid.util import current_directory
4-
from os.path import join, exists
4+
from os.path import join
55
from os import environ, uname
6-
from glob import glob
76
import sh
87

98

@@ -16,15 +15,11 @@ class JpegRecipe(Recipe):
1615
name = 'jpeg'
1716
version = '2.0.1'
1817
url = 'https://github.com/libjpeg-turbo/libjpeg-turbo/archive/{version}.tar.gz' # noqa
18+
built_libraries = {'libjpeg.a': '.', 'libturbojpeg.a': '.'}
1919
# we will require this below patch to build the shared library
2020
# patches = ['remove-version.patch']
2121

22-
def should_build(self, arch):
23-
return not exists(join(self.get_build_dir(arch.arch),
24-
'libturbojpeg.a'))
25-
2622
def build_arch(self, arch):
27-
super(JpegRecipe, self).build_arch(arch)
2823
build_dir = self.get_build_dir(arch.arch)
2924

3025
# TODO: Fix simd/neon
@@ -59,10 +54,6 @@ def build_arch(self, arch):
5954
_env=env)
6055
shprint(sh.make, _env=env)
6156

62-
# copy static libs to libs collection
63-
for lib in glob(join(build_dir, '*.a')):
64-
shprint(sh.cp, '-L', lib, self.ctx.libs_dir)
65-
6657
def get_recipe_env(self, arch=None, with_flags_in_cc=False):
6758
env = environ.copy()
6859

‎pythonforandroid/recipes/libcurl/__init__.py

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,18 @@
11
import sh
2-
from pythonforandroid.toolchain import Recipe, shprint, shutil, current_directory
3-
from os.path import exists, join
2+
from pythonforandroid.recipe import Recipe
3+
from pythonforandroid.util import current_directory
4+
from pythonforandroid.logger import shprint
5+
from os.path import join
46
from multiprocessing import cpu_count
57

68

79
class LibcurlRecipe(Recipe):
810
version = '7.55.1'
911
url = 'https://curl.haxx.se/download/curl-7.55.1.tar.gz'
12+
built_libraries = {'libcurl.so': 'dist/lib'}
1013
depends = ['openssl']
1114

12-
def should_build(self, arch):
13-
super(LibcurlRecipe, self).should_build(arch)
14-
return not exists(join(self.ctx.get_libs_dir(arch.arch), 'libcurl.so'))
15-
1615
def build_arch(self, arch):
17-
super(LibcurlRecipe, self).build_arch(arch)
1816
env = self.get_recipe_env(arch)
1917

2018
r = self.get_recipe('openssl', self.ctx)
@@ -31,10 +29,6 @@ def build_arch(self, arch):
3129
_env=env)
3230
shprint(sh.make, '-j', str(cpu_count()), _env=env)
3331
shprint(sh.make, 'install', _env=env)
34-
shutil.copyfile('{}/lib/libcurl.so'.format(dst_dir),
35-
join(
36-
self.ctx.get_libs_dir(arch.arch),
37-
'libcurl.so'))
3832

3933

4034
recipe = LibcurlRecipe()

‎pythonforandroid/recipes/libexpat/__init__.py

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,32 @@
11

22
import sh
3-
from pythonforandroid.toolchain import Recipe, shprint, shutil, current_directory
4-
from os.path import exists, join
3+
from pythonforandroid.recipe import Recipe
4+
from pythonforandroid.util import current_directory
5+
from pythonforandroid.logger import shprint
6+
from os.path import join
57
from multiprocessing import cpu_count
68

79

810
class LibexpatRecipe(Recipe):
911
version = 'master'
1012
url = 'https://github.com/libexpat/libexpat/archive/{version}.zip'
13+
built_libraries = {'libexpat.so': 'dist/lib'}
1114
depends = []
1215

13-
def should_build(self, arch):
14-
super(LibexpatRecipe, self).should_build(arch)
15-
return not exists(
16-
join(self.ctx.get_libs_dir(arch.arch), 'libexpat.so'))
17-
1816
def build_arch(self, arch):
19-
super(LibexpatRecipe, self).build_arch(arch)
2017
env = self.get_recipe_env(arch)
2118
with current_directory(join(self.get_build_dir(arch.arch), 'expat')):
2219
dst_dir = join(self.get_build_dir(arch.arch), 'dist')
2320
shprint(sh.Command('./buildconf.sh'), _env=env)
2421
shprint(
2522
sh.Command('./configure'),
26-
'--host=arm-linux-androideabi',
23+
'--host={}'.format(arch.command_prefix),
2724
'--enable-shared',
2825
'--without-xmlwf',
2926
'--prefix={}'.format(dst_dir),
3027
_env=env)
3128
shprint(sh.make, '-j', str(cpu_count()), _env=env)
3229
shprint(sh.make, 'install', _env=env)
33-
shutil.copyfile(
34-
'{}/lib/libexpat.so'.format(dst_dir),
35-
join(self.ctx.get_libs_dir(arch.arch), 'libexpat.so'))
3630

3731

3832
recipe = LibexpatRecipe()

‎pythonforandroid/recipes/libffi/__init__.py

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
from multiprocessing import cpu_count
33
from pythonforandroid.recipe import Recipe
44
from pythonforandroid.logger import shprint
5-
from pythonforandroid.util import current_directory, ensure_dir
5+
from pythonforandroid.util import current_directory
66
import sh
77

88

@@ -31,8 +31,7 @@ class LibffiRecipe(Recipe):
3131

3232
patches = ['remove-version-info.patch']
3333

34-
def should_build(self, arch):
35-
return not exists(join(self.ctx.get_libs_dir(arch.arch), 'libffi.so'))
34+
built_libraries = {'libffi.so': '.libs'}
3635

3736
def build_arch(self, arch):
3837
env = self.get_recipe_env(arch)
@@ -46,10 +45,6 @@ def build_arch(self, arch):
4645
'--disable-builddir',
4746
'--enable-shared', _env=env)
4847
shprint(sh.make, '-j', str(cpu_count()), 'libffi.la', _env=env)
49-
ensure_dir(self.ctx.get_libs_dir(arch.arch))
50-
self.install_libs(
51-
arch, join(self.get_build_dir(arch.arch), '.libs', 'libffi.so')
52-
)
5348

5449
def get_include_dirs(self, arch):
5550
return [join(self.get_build_dir(arch.arch), 'include')]

‎pythonforandroid/recipes/libglob/__init__.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
available via '-lglob' LDFLAG
44
"""
55
from os.path import exists, join
6-
from pythonforandroid.recipe import CompiledComponentsPythonRecipe
6+
from pythonforandroid.recipe import Recipe
77
from pythonforandroid.toolchain import current_directory
88
from pythonforandroid.logger import info, shprint
99
import sh
1010

1111

12-
class LibGlobRecipe(CompiledComponentsPythonRecipe):
12+
class LibGlobRecipe(Recipe):
1313
"""Make a glob.h and glob.so for the python_install_dir()"""
1414
version = '0.0.1'
1515
url = None
@@ -20,6 +20,7 @@ class LibGlobRecipe(CompiledComponentsPythonRecipe):
2020
# https://raw.githubusercontent.com/white-gecko/TokyoCabinet/master/glob.c
2121
# and pushed in via patch
2222
name = 'libglob'
23+
built_libraries = {'libglob.so': '.'}
2324

2425
depends = [('hostpython2', 'hostpython3')]
2526
patches = ['glob.patch']
@@ -60,7 +61,6 @@ def build_arch(self, arch):
6061
cflags.extend(['-shared', '-I.', 'glob.o', '-o', 'libglob.so'])
6162
cflags.extend(env['LDFLAGS'].split())
6263
shprint(cc, *cflags, _env=env)
63-
shprint(sh.cp, 'libglob.so', join(self.ctx.libs_dir, arch.arch))
6464

6565

6666
recipe = LibGlobRecipe()

‎pythonforandroid/recipes/libglob/glob.patch

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -911,7 +911,7 @@ diff -Nur /tmp/x/glob.c libglob/glob.c
911911
diff -Nur /tmp/x/glob.h libglob/glob.h
912912
--- /tmp/x/glob.h 1969-12-31 19:00:00.000000000 -0500
913913
+++ libglob/glob.h 2017-08-19 15:22:18.367109399 -0400
914-
@@ -0,0 +1,102 @@
914+
@@ -0,0 +1,104 @@
915915
+/*
916916
+ * Copyright (c) 1989, 1993
917917
+ * The Regents of the University of California. All rights reserved.
@@ -952,10 +952,12 @@ diff -Nur /tmp/x/glob.h libglob/glob.h
952952
+
953953
+#include <sys/cdefs.h>
954954
+#include <sys/types.h>
955+
+#ifndef ARG_MAX
955956
+#define ARG_MAX 6553
957+
+#endif
956958
+
957959
+#ifndef _SIZE_T_DECLARED
958-
+typedef __size_t size_t;
960+
+#include <stddef.h>
959961
+#define _SIZE_T_DECLARED
960962
+#endif
961963
+

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy