-
Notifications
You must be signed in to change notification settings - Fork 10.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crawler logs are cut by day #3628
Crawler logs are cut by day #3628
Comments
Scrapy seems to use logging inside. You can have a look: https://docs.python.org/3/library/logging.handlers.html#timedrotatingfilehandler |
Also, you could use an external unix tool to directly rotate the logs files without having to make changes on how Scrapy works with logs: https://www.tutorialspoint.com/unix_commands/logrotate.htm |
If that's possible to do using stdlib logging module, it would be nice to document how. Pull requests are welcome! |
I am working on this, will raise a pull request |
Is it worked on? If not, I'm going to work on it. |
@Roman519 Go for it! 😃 |
I've shoe horned a TimedRotatingFileHandler into root logger by way of log.configure_logging function. Do I need to add unit tests for something as simple as this. I'm asking because the default root filehandler doesn't have a unit test that I can see. |
We are trying for new patches to provide complete coverage of their changes, so that we eventually reach that sweet 100% test coverage. So unless the tests are really difficult to write, it would be great to have them. |
Is this being worked on? Looks interesting, would like to contribute if theres not a pr I missed :D |
There is no pull request at the moment, I believe. |
@Gallaecio in my scrapy projects i configure log handlers in from logging.handlers import TimedRotatingFileHandler
from scrapy.utils.log import configure_logging
logHandler = TimedRotatingFileHandler('crawl.log', when='midnight', interval=1)
configure_logging(install_root_handler=False)
logging.basicConfig(
format='%(asctime)s [%(name)s] %(levelname)s: %(message)s',
level=logging.INFO,
handlers=[logHandler]) Is this right approach? |
It does look like the recommended approach in the documentation. |
@Gallaecio is there pull request for this enhancement? I am interested to contribute. |
No pull request at the moment. |
- Modified _get_handler method to return TimedRotatingFileHandler if LOG_ROTATION variable is set to True in settings. - This enhancement is regards to issue : Crawler logs are cut by day scrapy#3628 Changes to be committed: modified: scrapy/utils/log.py
@kmike @Gallaecio, could you check the modification proposed in #4465? I think this will provide the requested enhancement. |
Hi! |
Still relevant, with 3 stale, open pull requests. |
Could I go ahead and take a stab at this and complete the stale pull requests? I'm a new contributor and would love to help out. |
I hope scrapy's log can provide more convenience, such as crawlers running for a long time, log can be cut by day, so that the log file is not too large.
The text was updated successfully, but these errors were encountered: