Commit Graph

28 Commits

Author SHA1 Message Date
Eugen Rochko 34ccc058fa Limit total subscribe retries to 10, but space them out more ()
Since there is little point in retrying so often when a service is down
or does not exist anymore. Subscriptions are renewed 1 day before they
should expire, so retrying in 30 minutes, then 2 hours, then 12 hours
is fine. If even after that, the remote server does not work, there is
little sense in retrying more often than once a day

Also, uniqueness of the job should ensure that failed retries will
not result in multiple retries for the same endpoint when the next
resubscription cycle comes
2017-07-10 18:04:23 +02:00
Akihiko Odaki (@fn_aki@pawoo.net) e27f792c24 Some minor change and spec for Account ()
* Introduce domains method to Account relation

Account had followers_domains method, which was excessively specific.
Let relation of Account have domains method instead.

* Move follow_mapping in Account to AccountInteractions

* Introduce shared examples for AccountAvatar inclusion

* Cover Account more
2017-06-22 18:34:27 +02:00
Eugen Rochko e17c2e5da5 Batched remove status service ()
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries

* Add BatchedRemoveStatusService, make SuspendAccountService use it

* Improve method names

* Add test

* Add more tests

* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads

It was nice while it lasted
2017-06-14 18:01:35 +02:00
Eugen Rochko 657496b5a9 Do not cancel PuSH subscriptions after encountering "permanent" error… ()
* Do not cancel PuSH subscriptions after encountering "permanent" error response

After talking with MMN about it, turns out some servers/php setups do
return 4xx errors while rebooting, so this anti-feature that was meant
to take load off of the hub is doing more harm than good in terms of
breaking subscriptions

* Update delivery_worker.rb
2017-05-14 03:22:48 +02:00
Matt Jankowski 0d70fe2659 Spec coverage and refactor for pubsub/delivery worker ()
* Framework for delivery worker spec

* Refactor of pubsub delivery worker
2017-05-12 20:35:36 +02:00
Matt Jankowski 08949cca41 Pubsub confirmation worker spec ()
* Add specs for valid challenge response from pubsub confirmation worker

* Refactor the pubsub confirmation worker
2017-05-11 00:19:37 +02:00
Eugen Rochko 3a38322a54 Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items ()
to all subscribers. IDN-normalize callback URLs for subscriptions on insert.
2017-05-10 02:55:43 +02:00
haoyayoi 2fba94b36e add content-type in pubsubhubhub request header ()
* add content-type in pubsubhubhub request header

* fix type
2017-05-09 16:34:47 +02:00
Eugen Rochko b8e166894b Fix Scheduler::SubscriptionsScheduler ()
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Eugen Rochko 81584779cb More robust PuSH subscription refreshes ()
* Fix  - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also ). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix  - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Eugen 17c591ffba Punycode URI normalization ()
* Fix  - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
Eugen 501514960a Followers-only post federation ()
* Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers

* Authorized followers controller, stub for bulk action

* Soft block in the background

* Add simple test for new controller

* Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results,
rename "private" post setting to "followers-only", fix pagination style, improve post privacy
preferences style, improve warning style

* Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account
2017-04-24 00:38:37 +02:00
Matt Jankowski c44a700252 Quick best practice cleanup of views/helpers ()
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
2017-04-12 18:24:18 +02:00
Eugen 6d6a429af8 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri ()
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
2017-04-07 05:56:56 +02:00
Eugen Rochko bda37489ac Remove PuSH subscriptions when delivery is answered with a 4xx error 2017-04-05 02:34:33 +02:00
Eugen Rochko 40a4053732 Disable PuSH for blocked domains 2017-02-14 04:01:37 +01:00
Eugen Rochko db814543c0 Take out private toot distribution out of PuSH again 2017-02-12 01:31:04 +01:00
Eugen Rochko 446aad4ce2 Make PuSH only distribute to subscriptions that match follower domains
Allow PuSH to distribute private toots with that condition
2017-02-11 15:41:39 +01:00
Eugen Rochko 6c28886317 Improve background jobs params and error handling 2017-01-05 03:28:21 +01:00
Eugen Rochko 86be6d48c9 Fix accidental distribution of hidden stream entries to PuSH subscribers 2017-01-03 00:44:23 +01:00
Eugen Rochko 4c6809f6ab Adding some rescues to workers, e.g. don't fail removal job when status can't be found in the first place (already removed) 2016-12-19 09:31:12 +01:00
Eugen Rochko 668013265c Restoring old async behaviour of thread resolving as it proved to be more robust 2016-12-12 21:12:19 +01:00
Eugen Rochko 4a2347da41 Track successful PuSH deliveries 2016-11-30 15:24:57 +01:00
Eugen Rochko cc70f28f19 Adding rack timeout of 30sec, PuSH jobs moved to push queue so they
can be processed separately
2016-11-29 02:07:14 +01:00
Eugen Rochko 4e351baf88 Fix URLs in inline-rendered XML 2016-11-29 00:26:01 +01:00
Eugen Rochko 3ba6531611 Fix setting of confirmed=true on successful confirmation 2016-11-28 19:24:49 +01:00
Eugen Rochko 50638174c8 More logging to ConfirmationWorker 2016-11-28 19:15:28 +01:00
Eugen Rochko 2d2c81765b Adding embedded PuSH server 2016-11-28 13:36:47 +01:00