[bug]: fails to get video links

[1bn3mar]

*IDKy but the script just refused to work for me
I'm using macOS and mpv as my media player i didn't do anything special.

For me what happened was i just followed the installation process then tried to watch a show
Steps to reproduce the behavior:

1. $ lobster
2. specify which show exactly season & episode etc.
3. then it just outputs this: gsed: -e expression When continuing from the history, it would be great to be able to navigate episodes back and forward #1, char 0: no previous regular expression
   No subtitles found

P.S. this happens for anything i try to watch.
lobster.log

[1bn3mar]

This is the output
gsed: -e expression #1, char 0: no previous regular expression
No subtitles found

[masoniis]

For what it's worth, I am new to the script as well and I run into the same issue on NixOS and MacOS. For NixOS I am using the flake, and here is what it looks like running the command:

> lobster
Search Movie/TV Show: rick and morty
sed: -e expression #1, char 0: no previous regular expression
No subtitles found

[justchokingaround]

curl -s "https://keys4.fun" | tr -d '\n ' | sed -nE "s@.*\"rabbitstream\":\{\"keys\":\{\"v\":\"([^\"]*)\",\"h\":\"([^\"]*)\",\"b\":([0-9]*),\"agent\":\"([^\"]*)\",\"key\":\"([^\"]*)\".*@\1\t\2\t\3\t\4\t\5@p" | read -r v h b agent key

https://rabbitstream.net/ajax/v2/embed-4/getSources?id={}&v={}&h={}&b={}

rabbitstream changed their encryption
above is the fix in question, which i'll do soon ™️

[LuisOrg26]

rabbitstream changed their encryption above is the fix in question, which I'll do soon ™️

Hi does this works with windows too? I'm new in this and the command doesn't worked for me

$ https://rabbitstream.net/ajax/v2/embed-4/getSources?id={}&v={}&h={}&b={}
[1] 750
bash: https://rabbitstream.net/ajax/v2/embed-4/getSources?id={}: No such file or directory
[2] 751
[3] 752
[1]   Exit 127                https://rabbitstream.net/ajax/v2/embed-4/getSources?id={}
[2]   Done                    v={}
[3]+  Done                    h={}

This is the output i got

[justchokingaround]

that was just some pseudo code, ure supposed to replace {} with the actual values

[justchokingaround]

yes, that is normal. also the encryption has changed again since my previous comment, so i need to wait for keys4.fun to update their stuff or look for another fix

[pitsi]

Honest and semi-offtopic question... because I do not remember how the keys variable/output looked like before the change.

Assuming that you want key "b" from the output. Wouldn't it be easier to add jq as a dependency for the script and get it like so

$ curl -s https://keys4.fun/ | jq .rabbitstream.keys.b
1878522368

[justchokingaround]

i prefer avoiding additional unneeded deps

[pitsi]

During the weekend, I tried patching the script using jq to get the key parameters and add them to the fake url above. It failed with the same error as before.

And today I saw how a kodi addon was patched so as to work again with the new decryption method. Oh my....

[1bn3mar]

Hi, sorry but what r the values that i should replace {} with??

[justchokingaround]

corresponding values u can find on https://keys4.fun , tho its broken rn

[1bn3mar]

So, ur saying there's noway to use lobster rn?

[pitsi]

@1bn3mar
I made 3 new variables, as you can see below, which get keys v, h and b like so (replace the letters as needed)
keyb=$(curl -s https://keys4.fun | jq .rabbitstream.keys.b)
and I changed the json_data variable to this
json_data=$(curl -s "${provider_link}/ajax/embed-${embed_type}/getSources?id=${source_id}&v=${keyv}&h=${keyh}&b=${keyb}" -H "X-Requested-With: XMLHttpRequest")
but it fails with the same error, so it needs something else to be changed too.

@justchokingaround
Do we need all those keys? A while ago, the script was grabbing 2 16-character long strings only as keys. If you do undestand python, I can show you what the kodi addon dev added to make it work again.

[justchokingaround]

u also need to pass -A $agent as the user agent, without it it wont work. sure show me the python commit @pitsi

[pitsi]

I see... I will try it again when I have some time.

As for the kodi addon, there is no commit for it, because the "change" is always a zip file that gets uploaded. But if you want, I can download both (old and new) and create a .patch file from them.
If not, I can just mention the addon and the dev and you get them from his repo.

[justchokingaround]

the patch would be enough thx

[pitsi]

https://paste.debian.net/hidden/39332f7a/

Paste will be available for 7 days.

[justchokingaround]

thx i'll check it today

[justchokingaround]

@pitsi can u actually mention the repo pls?

[pitsi]

https://github.com/mbebe/blomqvist/tree/master/zips/plugin.video.fmoviesto

Version 1.30 is the new one that works.

[pitsi]

I just tried it with agent parameter, e.g.
curl -s longurl?${source_id}&v={keyv}&h={keyh}&b={keyb}" -A "${agent}" -H "X-Requested-With: XMLHttpRequest"
but the error remains.

The key parameter is probably needed, because somewhere the entire html code from vidcloud is parsed, but my brain cells are too few to figure out where. Here is a log in case it helps.

https://paste.debian.net/hidden/6dd4c3ba/

[1bn3mar]

@pitsi Yeah, to me this is a real brain teaser I'm no programmer by any means; I just don't know how to get the keys.

[pitsi]

What do you want me to explain?
Tbh, it has been so many days since the real issue and new keys were found, but there is still no fix for it, so maybe rabbitstream is doing a great work protecting its links. I hope this won't mean the end of lobster :(

[justchokingaround]

it's not the end. i'm just a bit busy at work atm, so i don't have time to reverse engineer everything and can only port an existing solution or use keys that would be updated regularly by someone. i looked at the python code and the problem is all of the decryption is done in it, it doesnt rely on any key provider. porting that to shell would take me more time than i have atm

[pitsi]

But there is a key (provider?) in that python code, at the variable named klucze.

[1bn3mar]

In conclusion lobster does not work rn unless the user tinkers with it; if yes I'll just close the issue if not can someone please give me a oneliner to run or a preconfigured file to replace mine cuz if not I'll keep the question's coming.

PS: i don't get notified that's why i take too long.

[justchokingaround]

it's broken on my side not the user's. i should be able to come up with a fix soon tm

[pitsi]

What do you mean "tinkers with it"? What can be changed?

[1bn3mar]

@pitsi I mean the whole conversation between you and @justchokingaround i did not get one bit of it.

[1bn3mar]

Anyway's thank you all.

[Ciarands]

@Dragon-Batch
A fair few questions lol but ill answer some for you

1. Chomsky is talking about something I was working on, I haven't turned it into an API yet just because busy with life, there is no API that I am aware of currently though.
2. Saves them money I'd guess, take us @ movie-web for instance, we probably cost them a bit of money just with our users, and we're not serving ads for them, there's certainly reason to want to quickly break stuff, however they're not super creative, best src I've seen so far is flixhq, but even they're kinda trivial to scrape due to some major oversights on their behalf.
3. Not that I'm aware of?
4. It depends on the provider, many host on DMCA ignored VPS's, its really not that expensive and they're making a fair bit of money.
5. No, we don't scrape sites that we know the owners of usually lol.
6. Its not that they're shit at coding, its that its very difficult to come up with a creative sln to stop us due to the nature of the web, I can think of a few methods that would stop 99% of scrapers, but all it takes is 1 person to break it.

[Dragon-Batch]

@Ciarands if you want, i guess i consider my self experienced at making apis with flask in python so i think i could help you make one if you want :33

[triorr]

The key is generated inside a web assembly module disguised as a PNG image in "loading.png". Reverse engineering skills is needed to fix this script.
For more info check this article https://www.pudn.com/Download/item/id/1711319547629731.html

[luvcie]

https://github.com/stillmanpeeer/rabbitstream

Could you be more specific on what is this? @stillmanpeeer

[Dragon-Batch]

its a advertisement to some shity api that costs 160$/month

…

On Fri, 26 Apr 2024, 12:45 pm luvcie, ***@***.***> wrote: https://github.com/stillmanpeeer/rabbitstream Could you be more specific on what is this? — Reply to this email directly, view it on GitHub <#193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARIM46NTSNB4CZX5PXVHFRTY7G5VFAVCNFSM6AAAAABEARSO6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGUZTQNZWGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pitsi]

Although I disagree with the price asked for accessing the api (because its a lot for a single user who will use it) and with its lack of documentation (especially about the requests per day limit), I have to admit that the api works.
When lobster was first patched so as to use it, it did return streams and subtitles for them, so yea, it does work as intended.

[justchokingaround]

so i'm currently implementing a vidsrc scraper to switch over to it from rabbitstream. if anyone has ever implemented rc4 decryption using openssl pls hmu, bc porting my bash code to posix shell, without using arrays is gonna be a bit painful

[pitsi]

All this is what must be done for getting the keys? If so, I may know a github repo that generates them.
How can I contact you though? I am not on matrix and neither on most known and unknown social media.

---edit
Also, can someone please change the title accordingly and pin the issue so that every new issue reporter can see it?

[justchokingaround]

@chomkerman:matrix.org

[pitsi]

Yes, I know that, it's on your profile's info.
I am not on matrix. I also do not make accounts here and there.

[justchokingaround]

oh sorry i thought i said u are on matrix, my bad

[justchokingaround]

i have the keys actually, but its hard to implement the decryption using those keys

[pitsi]

So the problem is that gibberish on the url, like %3A and %2F, we see on the right terminal?

[justchokingaround]

no, the problem is that the decryption code i wrote is bash specific and not posix compliant, since i use an array for S

[pitsi]

Sorry, I can't help you with that :(

[triorr]

@justchokingaround Try something like this

echo -n "yB5_gMJfMGtHJCTb6P4aLZpbJu9hxhicEj==" | sed 's/_/\//g' | sed 's/-/+/g' | base64 -d | openssl rc4 -K $(echo -n "WXrUARXb1aDLaZjI" | xxd -p | tr -d '\n') -nopad

[justchokingaround]

did you generate this using AI ? did this work when u ran it on ur system?

i've tried getting rc4 to work using openssl but couldn't, even with my experience with it. i need someone who has actually done it before and got it working.

also echo -n isn't posix compliant, you should use printf instead, it doesnt output a newline by default

[triorr]

@justchokingaround Sorry. I'm not very familiar with POSIX compliance.
I have tested this on bash, sh, zsh and even fish they all work for me with "echo -n" and "printf".
Try adding the commands one by one to check whats wrong.
First your base64 has a problem with that input try the full base64 url from vidsrc. Because I cut it down for readability and it worked on my machine.
About openssl I think you have the legacy algorithms disabled for security reasons check with this:

$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.1
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.2.1
    status: active

to activate the legacy algorithms you need to add/uncomment these lines

legacy = legacy_sect

[legacy_sect]
activate = 1

[default_sect]
activate = 1

in your /etc/ssl/openssl.cnf

[justchokingaround]

okay thanks a lot, i'll check it later

[justchokingaround]

that worked just fine, thank you a lot @triorr for helping me figure out that the problem was the openssl package i was using (i just changed from openssl to openssl_legacy on nix)

printf "%s" "$url" | sed "s/_/\//g;s/-/+/g" | base64 -d | openssl rc4 -K "$(printf "%s" "$key" | xxd -p)" -nopad

[Kat299]

Guess what happened...

After 1+ week of not touching it at all (because I have nothing to watch these days), I launched lobster today to test it. And it simply fails, because, as you can see in the log below, the api it now uses has blocked access due to too many requests! https://paste.debian.net/hidden/7fc3cc0a/

If the api's dev (@stillmanpeeer) is reading this, please explain how I managed to pass the 5000 requests/day limit that you have set (assuming that the limit is indeed at 5000 requests and not lower or zero), when lobster called it ONCE (or maybe 4 times as you can see in the log) in order to work. I am asking it as an angry user of youtube's api, which "charges" 1 api call per 1 video view, but a single search "costs" 100 api calls. And all that with its daily limit being at 10000 calls!

In other news, keys4fun (the domain) shut down last week and its dev's account here on github gets to a 404 today, so things are now pretty bleak for lobster :(

Yep, that's happening for me too.

[justchokingaround]

dd205d9