monorail
/
mastodon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix idempotency when database writes are slow (#21840) 

There is an idempotency key generated by clients when authoring a post,
and stored in Redis, to ensure that if a user or client retries posting
the same status, we don't get a duplicate.

Hachyderm.io has been experiencing some filesystem and database
performance issues, causing database writes to be slow. This can mean
that there are successful posts, but the reverse proxy returns 504
Gateway Timeout before the idempotency status has been updated; users or
clients who retry (such as Tusky which retries automatically, see
tuskyapp/Tusky#2951) can re-try the same post with the same idempotency
key before it has actually been recorded in Redis, leading to duplicate
posts.

To address this issue, move all of the database updates after the
initial transaction that creates the status into the
`postprocess_status!` method, so we can insert the idempotency key
immediately after the status has been created, significantly reducing
the window in which the status could be created but the idempotency key
not yet stored.

Note: this has not yet been tested; I'm submitting this PR for
discussion and to offer to the Hachyderm.io admins to try out to fix the
multiple posting problem.

Co-authored-by: Brian Campbell <brcampbell@beta.team>
This commit is contained in:
Brian Campbell 2022-12-15 12:08:40 -05:00 committed by GitHub
parent ebf1d74e40
commit 2d12948220
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
app/services/post_status_service.rb
View File

@ -37,12 +37,15 @@ class PostStatusService < BaseService
schedule_status! schedule_status!
else else
process_status! process_status!
postprocess_status!
bump_potential_friendship!
end end
redis.setex(idempotency_key, 3_600, @status.id) if idempotency_given? redis.setex(idempotency_key, 3_600, @status.id) if idempotency_given?
unless scheduled?
postprocess_status!
bump_potential_friendship!
end
@status @status
end end
@ -66,9 +69,6 @@ class PostStatusService < BaseService
ApplicationRecord.transaction do ApplicationRecord.transaction do
@status = @account.statuses.create!(status_attributes) @status = @account.statuses.create!(status_attributes)
end end
process_hashtags_service.call(@status)
process_mentions_service.call(@status)
end end
def schedule_status! def schedule_status!
@ -92,6 +92,8 @@ class PostStatusService < BaseService
end end
def postprocess_status! def postprocess_status!
process_hashtags_service.call(@status)
process_mentions_service.call(@status)
Trends.tags.register(@status) Trends.tags.register(@status)
LinkCrawlWorker.perform_async(@status.id) LinkCrawlWorker.perform_async(@status.id)
DistributionWorker.perform_async(@status.id) DistributionWorker.perform_async(@status.id)