[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mastodon generating massive simultaneous hits to outside websites #27266

Closed
naknomum opened this issue Oct 3, 2023 · 8 comments · Fixed by #27351
Closed

mastodon generating massive simultaneous hits to outside websites #27266

naknomum opened this issue Oct 3, 2023 · 8 comments · Fixed by #27351
Labels
bug Something isn't working status/to triage This issue needs to be triaged

Comments

@naknomum
Copy link
naknomum commented Oct 3, 2023

Steps to reproduce the problem

problem is automatic. something -- i am guessing search/index related -- in mastodon codebase is hitting urls from numerous multiple mastodon instances and (crucially) at nearly the exact same time. this is basically creating a distributed denial-of-service type effect on the site which is being indexed.

Expected behaviour

my server not being bombarded by mastodon

Actual behaviour

my server is being bombarded by mastodon

Detailed description

web log is full of hundreds of hits to exact same url, but from hundreds of mastodon instances. all within seconds of each other. a very small sampling is given below, with some information obscured.

site.url - 57.XXX.XXX.15 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0; +https://XXXXXXXX.social/)" 10.118
site.url - 57.XXX.XXX.175 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0; +https://XXXXXXX.com/)" 10.112
site.url - 65.XXX.XXX.220 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0-rc2+glitch; +https://XXXXXXXX.space/)" 10.130
site.url - 95.XXX.XXX.160 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0+glitch; +https://XXXX.social/)" 10.134
site.url - 5.XXX.XXX.139 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0; +https://XXXXXX.lol/)" 10.036
site.url - 128.XXX.XXX.242 - - [02/Oct/2023:02:28:12 +0000] "GET /xxxx/ HTTP/1.1" 499 0 "-" "http.rb/5.1.1 (Mastodon/4.2.0; +https://XXXXX.social/)" 10.126

the behavior is synchronized so close in time such that it creates a very similar effect to a coordinated/distributed denial-of-service attack on the site. this seems undesirable or unfriendly at best.

what i have observed so far is that:

  1. the urls being hit on the site (/xxxx/ above) are urls used in mastodon user profiles
  2. probably 90%+ are mastodon v4.2.0 but there are some lower versions than this, so unsure if it this is fullsearch-related

Mastodon instance

many, hundreds. all?

Mastodon version

4.2.0

Technical details

No response
@naknomum naknomum added bug Something isn't working status/to triage This issue needs to be triaged labels Oct 3, 2023
@naknomum
Copy link
Author
naknomum commented Oct 3, 2023

after a little more analysis of weblogs, i would say peak traffic was approx 30 hits/sec with approx 10-30 hits/sec about average for 7 or 8 seconds.

this is pretty far from real DDOS traffic, but still is hitting my site pretty hard. it also doesnt seem very scalable. if mastodon instances grow x10, so would this traffic. and, arguably, so would the number of users of mastodon linking to outside sites in their profiles.

@naknomum naknomum changed the title mastodon (search?) generating massive simultaneous hits to outside websites mastodon generating massive simultaneous hits to outside websites Oct 4, 2023
@naknomum
Copy link
Author
naknomum commented Oct 4, 2023

i am fairly certain this has to do with verified links, when mastodon fetches the page to look for the rel="me" attribute.

@Cassolotl
Copy link
Cassolotl commented Oct 4, 2023 •
edited
Loading

This reminds me of Mastodon can be used as a DDOS tool #4486 (Aug 2017, 6 years old, oof) and Access is made many times by toot including url #3518. It was fixed with Spread out crawling randomly to avoid DDoSing the link #8445, apparently. Maybe a similar solution is possible here?

Edit: Despite the fix added several years ago, in February this year someone still added this issue to the issue list: Reduce load of preview fetching on third-party servers #23662

@naknomum
Copy link
Author
naknomum commented Oct 4, 2023

This reminds me of Mastodon can be used as a DDOS tool #4486 (Aug 2017, 6 years old, oof) and Access is made many times by toot including url #3518. It was fixed with Spread out crawling randomly to avoid DDoSing the link #8445, apparently. Maybe a similar solution is possible here?

Edit: Despite the fix added several years ago, in February this year someone still added this issue to the issue list: Reduce load of preview fetching on third-party servers #23662

very interesting - sounds related to me. thanks for posting. i do not know the reason -- or even actual coding logic -- behind the decision to do (what i am guessing is) the rel="me" verified links the way they are done. it seems like, mastodon is being overly ambitious in verifying many times. but i could be wrong; i am basing this off forensic evidence on a third party server because i have zero ruby confidence. if nothing else, maybe verification could be done more "on-demand" rather than immediately, which results in this highly synchronized verification flood.

i am fairly certain that if mastodon is successful in growth to 10x or 100x of now, my server would be toast each time a user posted. sure, i was able to "remedy" this with a limit_req in nginx, but that was after i discovered it. so it just seems more "third-party friendly" to not be doing this in the first place.

@renchap
Copy link
Sponsor Member
renchap commented Oct 8, 2023

Closing as duplicate of #23662

We have this on our roadmap and will try to have a solution for it in 4.3.0.

@renchap renchap closed this as not planned Won't fix, can't repro, duplicate, stale Oct 8, 2023
@naknomum
Copy link
Author
naknomum commented Oct 8, 2023

Closing as duplicate of #23662

We have this on our roadmap and will try to have a solution for it in 4.3.0.

thanks for checking it out and addressing it on the roadmap. as you probably noticed, this is (i think) related to verified links, not previews. however, i am guessing it is likely to be an all-in-one kind of solution you will find for the back end. again, thank you, merci.

@renchap
Copy link
Sponsor Member
renchap commented Oct 8, 2023

Sorry, I missed this was about verified links. Re-opening then, and I will see what we can do on this :)

@renchap renchap reopened this Oct 8, 2023
@renchap renchap linked a pull request Oct 10, 2023 that will close this issue
Add variable delay before link verification of remote account links #27351
Merged
@Neustradamus
Copy link

It will be nice to have a solution for this problem...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working status/to triage This issue needs to be triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add variable delay before link verification of remote account links mastodon/mastodon
4 participants
@renchap @Neustradamus @naknomum @Cassolotl