-
Notifications
You must be signed in to change notification settings - Fork 5.8k
Job queue time units #452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Job queue time units #452
Conversation
…ing a datetime.time as the interval.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, Wesley. Great job! :)
Thank you, @KamranMackey! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work on this PR!
However, there are some things that would have to be changed before we can merge it.
I really like the idea of being able to specify which days the jobs should run (and several users have actually requested it), but this implementation is simply not up to the standards of the rest of the library.
from threading import Thread, Lock, Event | ||
from queue import PriorityQueue, Empty | ||
|
||
|
||
class Days(Enum): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Weeks unfortunately don't always start on a monday. In some locales it's sundays.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bomjacob Does this actually matter all that much? I know this might come off as rude, but I don't see why this matters. It's a really tiny thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And as far as I know, it would require too much code to be able to do an change to where Sunday is day 0 in certain locales, as we don't even know how many locales have Sunday as the first day of the week.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well I'm not sure, but I'd say that the entirety of North America is a pretty big demographic...
Question is if it even matters... as long as the code below correctly knows that a monday is a monday (and not necesarily a 0) then everything is fine... I'm not entirely sure that python is that smart though :/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If that's the case across all locales (that is, python doesn't automatically change that weekday depending on where it's run) then it shouldn't matter and this should be fine...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to the official Datetime docs that is the case, so I think we're good on the mapping of the days.
@@ -56,6 +68,12 @@ def __init__(self, bot, prevent_autostart=None): | |||
""":type: float""" | |||
self._running = False | |||
|
|||
def dt_time_to_seconds(self, dt_time): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not use .total_seconds()
here?
@@ -126,7 +156,9 @@ def tick(self): | |||
self.logger.debug('Running job %s', job.name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This debug would wrongly say that it's running a job even though it's not, if it's on an excluded day.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've fixed this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm using a separate get_days
function here, because if I put it in the class, it won't be filtered out by the list comprehension, and I won't be able to easily get the values from the attributes defined in the class.
@@ -169,6 +178,26 @@ def test_inUpdater(self): | |||
finally: | |||
u.stop() | |||
|
|||
def test_time_units(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please split this into multiple test that only test one thing each.
E.g. have one that tests if seconds (remember to test floats too btw) and another test which tests if datetime objects work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cannot find a reliable way to test the float
, one time it succeeds, one time it fails. Do you know a way in which I can reliably test floats? I know for certain that floats work, is not testing them an option, because if the test starts on the wrong millisecond and rounds the wrong way, the test will fail.
@@ -21,10 +21,22 @@ | |||
import logging | |||
import time | |||
import warnings | |||
import datetime | |||
from enum import Enum |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Enums are a new feature that appeared in 3.4. Note that this library also supports python 2.7.
While they have been backported and are available on pypi, I would rather not add a whole dependency for something that is used so very little.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bomjacob I don't think one extra dependency will hurt people. And Enums are not used so "very little". And there's no way to track how often Enums are being used either. But I know that they're used more than your "so very little" stat.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Granted, we use enums elsewhere in the project... but a enum in python could just as well be a class with some attributes.... There's a reason it wasn't added until 3.4... it's pretty much unnecessary. And of cause if there was a library that would make our library easier to maintain, easier to read or easier to develop on, we wouldn't hesitate to add it as a dependency - but Enums simply aren't that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bomjacob I'd personally like to keep the enum library, it's supported and a very clean way of doing it, it isn't a very big dependency, and could be used more throughout the project as it grows.
An alternative would look like:
>>> def enum(**named_values):
... return type('Enum', (), named_values)
...
>>> Days = enum(mon=0, tue=1, wed=2, thu=3, fri=4, sat=5, sun=6)
>>> Days.mon
0
>>> class Days(object):
... def __init__(self, day):
... if day not in (Days.mon , Days.tue, Days.wed, Days.thu, Days.fri, Days.sat, Days.sun):
... raise ValueError('gender not valid')
... self.day = day
...
I don't like this pattern because it relies upon the integers too much and has way more overhead.
In my original example the Enum doesn't relie on the integers too much, only to check if one of the values of the Days which were supplied with the Job is today. The Enum subclass returns a special Enumeration object instead of the literal integer value. This is a lot nicer for error handling because people should use the supplied Days Enum subclass. If you don't do this, people could just pass in integers from 0-6 and it'd accept. As far as I know, we can't check if a value comes from a certain class if we do this with a plain new-style class. With an Enum we can check if it comes from the class it should come from and enforce people to use the Days Enum subclass.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you want to prevent users from using integers? It's not really very python like to require inputs to be of a special type if you don't have to. My recommendation would be as follows:
Make the Days a simple class with the weekdays as class variables (attributes), this is what we use for enums elsewhere in the project and it's worked fine (if you believe that it absolutely wouldn't work here, please do correct me).
Just raise a ValueError if 0 < day > 7 inside ``Job's
init` or `JobQueue`'s `put`.
In the future we might upgrade to use Enums everywhere, but for now it's simply not necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code is more often read than written. Preventing users from using plain integers makes it clearer what someone's trying to accomplish when looking over the code.
For example:
import datetime
test_job = Job(callback, datetime.time(hour=3), repeat=True, days=(0, 4, 6))
If I make my bot open source and someone reads this, it's very unclear what the original developer tries to do without first reading the documentation of this repo. It's also very confusing if you have different locals (like you pointed out) and you assume that 0 is sunday instead of monday. Of course, we would like to make the developer use our Days class. But he/she isn't enforced to do it, and I think we should assume people wouldn't if they don't really have to.
This code:
import datetime
from telegram.ext.jobqueue import Days
test_job = Job(callback, datetime.time(hour=3), repeat=True, days=(Days.mon, Days.fri, Days.sun))
This code is a little longer, but it's entirely self-documenting! A developer who is unknown with the codebase will instantly know how it works. The syntax is clear, the functionality is clear, and we're enforcing them to use the Enum from the project instead of integers like we saw in the first example. I think this is more Pythonic, because "there should be one-- and preferably only one --obvious way to do it", except for string formatting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree to use enum because Python 2 is going to drop its support soon. Python 3 is the new standard.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see @tsnoam comment below. Bottom line is that for now, don't use Enums unless they're part of future
(which I don't think they are).
self.callback = callback | ||
self.interval = interval | ||
self.repeat = repeat | ||
self.context = context | ||
|
||
if not isinstance(days, tuple): | ||
err_msg = "The 'days' argument should be of type 'tuple'" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please just write the msg directly in the exception, it simply looks a bit cleaner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everywhere or only this instance?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everywhere preferably :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the max line length for the project? I use 84 as my personal standard, so when a message is too long, I use a variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@voider1 Max line length is 99 characters.
Thank you! I will address all the changes. |
…days it wouldn't.
…n extra variable.
…e get_seconds() method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last commit message should be: "Adding error handling for the method 'timedelta_to_seconds'."
…n't found a reliable way to test it.
Here's some comments from @tsnoam from our dev team group:
isinstance(x, number)
|
Point 2 doesn't work if it isn't a subclass of Enum. If the bottom-line is not using Enum, I'd have to manually get the ints from the class or implement an iterator in the class. >>> class Days(object):
... mon = 0
... tue = 1
...
...
>>> tuple(Days)
Traceback (most recent call last):
File "<input>", line 1, in <module>
tuple(Days)
TypeError: 'type' object is not iterable Point 3, do you mean that when I set the time for Sunday at 00:00 it'll never execute because then it's Monday? If that's the case, place an heads up in the documentation when this gets added that 00:00 is a day later, and that they should execute at 23:59 instead of 00:00 or just change the day. Point 5 those are a lot of small changes, sadly, not entirely relying on integers, making sure users use the Days Enum. You guys should really think about making Enums part of the project. They're a lot nicer to work with then plain new-style classes, and do not add overhead in the code with implementing iterators or manually getting al the values from the class (which I'll have to do now). I'll make them new-style classes for now. But please, really just consider adding this to the whole project. It's a small dependency which makes these things a lot easier. |
@bomjacob I've addressed all the issues you've mentioned. Could you review it again, please? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the way this is going. I requested some changes, but most of them are small.
One more thing: We had multiple requests to add the possibility of adding Jobs that run at a certain time instead of defining next_t
and interval
as time deltas (not referring to the timedelta
class here).
While I'm not asking you to implement that, I would appreciate if you could give it some thought, and, if required, modify this PR so that it doesn't make the implementation of that any harder than necessary.
sun = 6 | ||
|
||
|
||
def get_days(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know what this is supposed to do, but it's still unreadable. I think the best solution would be to do something like we did in MessageEntity
. Other ideas?
from threading import Thread, Lock, Event | ||
from queue import PriorityQueue, Empty | ||
|
||
|
||
class Days(object): | ||
mon = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we switched to a class with constants, the correct naming convention would be UPPER_CASE
if I'm not mistaken.
@@ -56,6 +72,12 @@ def __init__(self, bot, prevent_autostart=None): | |||
""":type: float""" | |||
self._running = False | |||
|
|||
def timedelta_to_seconds(self, tdelta): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method seems to be redundant, especially since this looks like a purely internal method and you already check isinstance(var, datetime.timedelta)
return tdelta.total_seconds() | ||
except AttributeError: | ||
raise AttributeError("timedelta_to_seconds only takes a datetime.timedelta object") | ||
|
||
def put(self, job, next_t=None): | ||
"""Queue a new job. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the docstring so it mentions that next_t
can also be of type timedelta
self.callback = callback | ||
self.interval = interval | ||
self.repeat = repeat | ||
self.context = context | ||
|
||
print(days) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug print
|
||
""" | ||
job_queue = None | ||
|
||
def __init__(self, callback, interval, repeat=True, context=None): | ||
def __init__(self, callback, interval, repeat=True, context=None, days=get_days()): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to my earlier comment, this would change to days=Days.ALL_DAYS
(or maybe even days=Days.EVERY_DAY
? Your choice ^^)
@@ -208,22 +242,36 @@ class Job(object): | |||
callback (function): The callback function that should be executed by the Job. It should | |||
take two parameters ``bot`` and ``job``, where ``job`` is the ``Job`` instance. It | |||
can be used to terminate the job or modify its interval. | |||
interval (float): The interval in which this job should execute its callback function in | |||
seconds. | |||
interval (float or datetime.time): The interval in which this job should execute its |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe you switched to timedelta
instead of time
, please update this docstring to reflect that change.
try: | ||
job.run(self.bot) | ||
for day in job.days: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If someone would, for whatever reason, put a day twice (like Job(..., days=(Days.mon, Days.mon))
) this would run the job twice on those days. I'm not sure this is a bug or a feature, but it could be easily changed by changing the type of Job.days
from tuple
to set
. Your choice.
… favour of a tuple.
@jh0ker I adressed all the changes and fixed them. |
@jh0ker About the idea of running them at a certain time, I'll maybe do that after this PR gets merged. |
LGTM :) |
This enables users to add intervals in seconds (default), minutes and hours for their Jobs.
It also enables users to define on which days the Job should be run (all by default).
It's backwards compatible!