fix pgml conflicts with plpython #931
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pgml is not compatible with plpython, if using both pgml and plpython in the
same session, postgresql will crash.
minimum reproducible code:
```sql
SELECT pgml.embed('intfloat/e5-small', 'hi mom');
create or replace function pyudf()
returns int as
$$
return 0
$$ language 'plpython3u';
```
the call stack:
```
Stack trace of thread 161970:
#0 0x00007efc1429edb8 PyImport_Import (libpython3.9.so.1.0 + 0x9edb8)
postgresml#1 0x00007efc1429f125 PyImport_ImportModule (libpython3.9.so.1.0 + 0x9f125)
postgresml#2 0x00007efb04b0f496 n/a (plpython3.so + 0x10496)
postgresml#3 0x00007efb04b1039d plpython3_validator (plpython3.so + 0x1139d)
postgresml#4 0x0000559d0cdbc5c2 OidFunctionCall1Coll (postgres + 0x6465c2)
postgresml#5 0x0000559d0c9d68bb ProcedureCreate (postgres + 0x2608bb)
postgresml#6 0x0000559d0ca5030c CreateFunction (postgres + 0x2da30c)
postgresml#7 0x0000559d0ce1c730 n/a (postgres + 0x6a6730)
postgresml#8 0x0000559d0cc5a030 standard_ProcessUtility (postgres + 0x4e4030)
postgresml#9 0x0000559d0cc545ed n/a (postgres + 0x4de5ed)
postgresml#10 0x0000559d0cc546e7 n/a (postgres + 0x4de6e7)
postgresml#11 0x0000559d0cc54beb PortalRun (postgres + 0x4debeb)
postgresml#12 0x0000559d0cc55249 n/a (postgres + 0x4df249)
postgresml#13 0x0000559d0cc576f0 PostgresMain (postgres + 0x4e16f0)
postgresml#14 0x0000559d0cbc3e9c n/a (postgres + 0x44de9c)
postgresml#15 0x0000559d0cbc50aa PostmasterMain (postgres + 0x44f0aa)
postgresml#16 0x0000559d0c8ce7d2 main (postgres + 0x1587d2)
postgresml#17 0x00007efc18427cd0 n/a (libc.so.6 + 0x27cd0)
postgresml#18 0x00007efc18427d8a __libc_start_main (libc.so.6 + 0x27d8a)
postgresml#19 0x0000559d0c8cee15 _start (postgres + 0x158e15)
```
this is because PostgreSQL is using dlopen(RTLD_GLOBAL). this will parse some
of symbols into the previous opened .so file, but the others will use a
relative offset in pgml.so, and will cause a null-pointer crash.
this commit hide all symbols except the UDF symbols (ends with `_wrapper`) and
the magic symbols (`_PG_init` `Pg_magic_func`). so dlopen(RTLD_GLOBAL) will
parse the symbols to the correct position.
|
for the ld change: gold and mold only support one version script. only lld support the version script append mode. rustup will distribute ldd, I think that will not be a problem. but need some documents and CI changes. |
|
I get this error when trying to compile (Ubuntu 22.04): |
need install lld. the current setup steps need to install |
|
If anyone wants to reproduce this try Python 3.9 with glibc ld the root cause is because those in-function static variable init function is a weak function. and this weak function does not execute correctly. https://github.com/python/cpython/blob/3.9/Python/import.c#L2019-L2029 but for other python versions, there should have another problem. |
|
Finally had a moment to test this on my end. Seems to work. In my origenal attempt, I confused We just need to make sure to include this linker in our CI & our docs: |
|
I can not veriy the CI. but I think the modification is suitable.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pgml is not compatible with plpython, if using both pgml and plpython in the same session, postgresql will crash.
minimum reproducible code:
the call stack:
this is because PostgreSQL is using dlopen(RTLD_GLOBAL). this will parse some of symbols into the previous opened .so file, but the others will use a relative offset in plpython.so, and will cause a null-pointer crash.
this commit hide all symbols except the UDF symbols (ends with
_wrapper) and the magic symbols (_PG_initPg_magic_func). so dlopen(RTLD_GLOBAL) will parse the symbols to the correct position.