Skip to content

Defects of deploy mechanism #1259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
frg01 opened this issue Jan 2, 2024 · 4 comments
Closed

Defects of deploy mechanism #1259

frg01 opened this issue Jan 2, 2024 · 4 comments

Comments

@frg01
Copy link

frg01 commented Jan 2, 2024

Defects of the deployment mechanism. I trained four times and did not deploy automatically.
Assume that my fourth model has the highest score, but its predict cannot be used.

At this time, I use deploy to deploy. There are three ways to deploy, most_recent, best_score, and rollback. most_recent will deploy the fourth model, best_score will also deploy the fourth model, and rollback will not work. So this project is an invalid project, right?

I can only delete my project, there is no other way.

-- just change xgboost 、xgboost_random_forest 、lightgbm、catboost
select from pgml.train(
project_name => 'Diabetes Regression',
task => 'regression',
relation_name => 'pgml.diabetes',
y_column_name => 'target',
algorithm=> 'ada_boost'
);

SELECT from pgml.deploy(
project_name => 'Diabetes Regression',
strategy => 'best_score'
);

@montanalow
Copy link
Contributor

montanalow commented Jan 2, 2024

We handle that hypothetical case. Rollback is restricted to models other than the currently deployed one.

https://github.com/postgresml/postgresml/blob/master/pgml-extension/src/api.rs#L356-L362

Feel free to re-open if I've missed what you're getting at, or you have a test case that illustrates misbehavior. The code you've listed runs for me.

@frg01
Copy link
Author

frg01 commented Jan 2, 2024

We handle that hypothetical case. Rollback is restricted to models other than the currently deployed one.

https://github.com/postgresml/postgresml/blob/master/pgml-extension/src/api.rs#L356-L362

Feel free to re-open if I've missed what you're getting at, or you have a test case that illustrates misbehavior. The code you've listed runs for me.

So is the description of rollback in pgml's official documentation not updated or not comprehensive enough?
https://postgresml.org/docs/introduction/apis/sql-extensions/pgml.deploy
”“”
rollback The model that was last deployed for this project is immediately redeployed, overriding the currently deployed model.
“”“
*** The description here means that rollback will deploy the most recently deployed model, but my four trains are not automatically deployed. How to use rollback?
*** Suppose my fourth model prediction fails, then most_recent and best_score will cause the fourth model to be deployed. Because the fourth model meets two conditions: 1. The fourth model has the highest score. 2. The fourth model is the most recent.
*** And I want to ask a question, why can't I specify the model ID for deployment? I'm really confused.

@montanalow
Copy link
Contributor

Our docs are also open source, so you can contribute something that makes the behavior more clear.

https://github.com/postgresml/postgresml/blob/master/pgml-cms/docs/introduction/apis/sql-extensions/pgml.deploy.md?plain=1#L87

  1. It's not clear to me why any of your training runs were not deployed previously.
  2. It's not clear to me why you'd want to rollback if nothing has been deployed.
  3. Suppose a "prediction fails"... what does that mean specifically? Do you mean a bug in PostgresML, or a model that has poor performance in some case. If this is the case, a rollback should work, if you had a previously successful model. Otherwise, we'll need a bug report/patch/fix outside of a rollback cycle.
  4. Specifying a model.id to deploy seems like a nice enhancement. PRs welcome.

@frg01
Copy link
Author

frg01 commented Jan 3, 2024

Our docs are also open source, so you can contribute something that makes the behavior more clear.

https://github.com/postgresml/postgresml/blob/master/pgml-cms/docs/introduction/apis/sql-extensions/pgml.deploy.md?plain=1#L87

  1. It's not clear to me why any of your training runs were not deployed previously.
  2. It's not clear to me why you'd want to rollback if nothing has been deployed.
  3. Suppose a "prediction fails"... what does that mean specifically? Do you mean a bug in PostgresML, or a model that has poor performance in some case. If this is the case, a rollback should work, if you had a previously successful model. Otherwise, we'll need a bug report/patch/fix outside of a rollback cycle.
  4. Specifying a model.id to deploy seems like a nice enhancement. PRs welcome.

If I could use model.id for deployment, then the bugs in my hypothetical testing strategy would be solved.

I will clearly explain my hypothesis here, subject to several conditions:

  1. Bugs may occur anywhere.
  2. Do not use automatic deployment during train.
  3. During the four training sessions, the last train has the highest score.
  4. This highest score, which is the fourth model, can train successfully, but predict cannot. (This is an assumption, it could happen)

Based on the above 4 conditions.
We found that after manually executing deploy, we could not use this project. The specific reasons are as follows:

  1. best_score will only deploy the fourth highest-scoring model, but it cannot predict.
  2. most_recent can only deploy the fourth model, because it is the model after the most recent training.
  3. Rollback can only deploy models that have been deployed. I did not use automatic deployment during train, that is, none of the four models were deployed.

Now, we can say that this project cannot be used and has to be deleted, because we can only deploy a fourth model that cannot be predicted

thanks lot for your reply!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy