Fix: Add timeout to partition html #2416
Conversation
pratiksinghchauhan
changed the title
|
Hi @pratiksinghchauhan thanks for the contribution. Could you please add a line to |
|
@badGarnet! I will make the changes and update the PR. Thanks! |
|
@pratiksinghchauhan - Is this something you'd still like to work on? |
|
@MthwRobinson Apologies for the delay! I am still interested in getting this done. I have already completed the feature set just need to get the documentation done. I will get it done this weekend for sure. |
|
Sounds good - thanks! |
|
@MthwRobinson I have updated the PR. Do let me know. |
| @@ -39,6 +39,7 @@ def partition_html( | |||
| include_metadata: bool = True, | |||
| headers: dict[str, str] = {}, | |||
| ssl_verify: bool = True, | |||
| request_timeout: Optional[int] = None, | |||
There was a problem hiding this comment.
Small tweak, but since we already have **kwargs in partition_html, could we pass this through with that instead of another explicit parameter?
| @@ -131,7 +132,7 @@ def partition_html( | |||
| ) | |||
|
|
|||
| elif url is not None: | |||
| response = requests.get(url, headers=headers, verify=ssl_verify) | |||
| response = requests.get(url, headers=headers, verify=ssl_verify, timeout=request_timeout) | |||
There was a problem hiding this comment.
And then this can be kwargs.get("request_timeout") here
This PR adds a timeout parameter to requests in partition_html to stop the code from blocking indefinitely if the URL fails to respond withing the given timeout,