PostgresML is spawning too many threads. #1161
Comments
|
Hi @higuoxing, thanks for filing this issue. We are aware that |
Sorry. Not only |
|
But I do see there're paramters for limiting the degree of parallelism from pgml. E.g., |
|
|
Thanks! Many packages are using openmp as their multi-threaded backend for CPU runtime. What do you think of introducing a new GUC |
|
Since this has bitten a few people, I think it's worth having the |
|
Environment variables like |
Ah, you're right. When the program gets started, openmp simply ignores modified environment variables1.
One possible solution is to call diff --git a/pgml-extension/src/lib.rs b/pgml-extension/src/lib.rs
index 6c2884ce..c024ddee 100644
--- a/pgml-extension/src/lib.rs
+++ b/pgml-extension/src/lib.rs
@@ -21,9 +21,16 @@ pg_module_magic!();
extension_sql_file!("../sql/schema.sql", name = "schema");
+extern "C" {
+ fn omp_set_num_threads(num_threads: i32);
+}
+
#[cfg(not(feature = "use_as_lib"))]
#[pg_guard]
pub extern "C" fn _PG_init() {
+ unsafe {
+ omp_set_num_threads(1);
+ }
bindings::python::activate().expect("Error setting python venv");
orm::project::init();
}Footnotes |
|
I've considered this, is there a way to ensure that the function exists before we call it? Nothing wrong with |
Since the OpenMP spec mentions that there's a function called BTW, PostgreSQL is loading libraries with the
IMHO, modifying |
|
I agree. We'll accept a patch using |
|
Hi, I submit a PR to resolve this issue, could you please take a look? #1362 |
Uh oh!
There was an error while loading. Please reload this page.
Recently, we found that PostgresML was spawning too many threads and threads is waiting for each other. No deadlock but low performance.
Root cause
The root cause is that PyTorch spawns too many threads, which cause severe CPU contention.
This can be solved by limiting the degree of parallelism with e.g.,
The root cause and workaround is found and provided by @xuebinsu
Probably we can add some GUC parameter to allow user tunning the performance?
Reproducing steps
Run test without limiting the number of threads. (Finished in 273.66s)
Run test by limiting the number of threads that openmp can spawn with
OMP_NUM_THREADS=1. (Finished in 7.81s)Stack trace
Stack trace
The text was updated successfully, but these errors were encountered: