Commit Graph

1106 Commits

Author SHA1 Message Date
Claire 60abcb3c4c Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `config/i18n-tasks.yml`:
  Upstream added new ignored strings, glitch-soc has extra ignored strings
  because of the theming system.
  Added upstream's changes.
2023-01-18 17:38:11 +01:00
Claire 41517a4845
Fix spurious admin dashboard warning when using ElasticSearch 7.x (#23064)
Some 7.x ElasticSearch versions support some 6.x nodes, thus the version check
is inadequate. I am not sure there is a good way to check if a server
implements all the 7.x APIs, so check server version and minimum wire version
instead.
2023-01-18 16:21:48 +01:00
Claire 3e63fcd4f0 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/status.rb`:
  Minor upstream refactor moved hook definitions around,
  and glitch-soc has an extra `before_create`.
  Moved the `before_create` accordingly.
- `app/services/batched_remove_status_service.rb`:
  Minor upstream refactor changed a block in which glitch-soc
  had one extra call to handle direct timelines.
  Adapted changes to keep glitch-soc's extra call.
2023-01-12 10:15:46 +01:00
Markus Unterwaditzer 0c689b9d01
fix: allow verification when page size exceeds 1MB (using HTML5 parser) (#22879)
* fix: allow verification when page size exceeds 1MB
Truncates the page after 1MB instead

Closes #15316

* switch to HTML5 parser, fix rubocop errors

* undo rubocop fixes

Co-authored-by: Chris Zubak-Skees <chriszs@gmail.com>
2023-01-11 21:59:13 +01:00
Kaspar V ae62e5fa53
Fix/remove calling private method with send in model (#22951)
* fix(status): remove send usage for private unlink_from_conversations

- make unlink_from_conversations public method
- rename unlink_from_conversations to unlink_from_conversations!
- fix send call on private method in statuses_vacuum and batched_remove_status_service

* fix(feeds_vacuum): replace find_in_batches with in_batches

because active record query results should be a little more efficient than
itterating with map and each. Postgres can grasp such lists of ids much quicker
than ruby can.
Will probably make allmost no difference, but cannot hurt either.
2023-01-11 21:57:24 +01:00
Claire 6c20dad201 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream updated its README, while we have a completely different one.
  Kept our README.
- `app/controllers/concerns/web_app_controller_concern.rb`:
  Conflict because of glitch-soc's theming system.
  Additionally, glitch-soc has different behavior regarding moved accounts.
  Ported some of the changes, but kept our overall behavior.
- `app/javascript/packs/admin.js`:
  Code changes actually applied to `app/javascript/core/admin.js`
2023-01-05 14:16:25 +01:00
Vyr Cossont 06f979098c
Check OpenSearch compatibility version instead of regular version (#22422)
Fixes #18535.
2023-01-05 13:44:31 +01:00
Jeong Arm fdd1facba1
Fix home TL could contain post from who blocked me (#22849)
* Fix home tl contains post from who blocked me

* Add test

* Fix feed_manager's build_crutches

blocked_by was not includes status' owner

* Add test for status from I blocked

* Fix typo
2023-01-05 13:30:38 +01:00
Claire f3a4d57be1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Discarded upstream changes: we have our own README
- `app/controllers/follower_accounts_controller.rb`:
  Port upstream's minor refactoring
2022-12-15 20:25:25 +01:00
Claire d13702ac06
Fix status cache hydration discrepancy (#19879) 2022-12-15 18:55:55 +01:00
Claire 673c54f114
Fix inability to use local LibreTranslate without setting ALLOWED_PRIVATE_ADDRESSES (#21926)
Fixes #20029
2022-12-15 17:04:38 +01:00
s0 52540771b0
Fix crash in elasticsearch_check.rb (#21006)
Nil unwrap causes the admin dashboard to crash/500 when the Chewy client info version number value is nil.
This occurs when running another ES-compatible backend such as MeiliSearch.
Obviously it would be good for chewy to recognise upstream but at least avoiding the crash would be fine.
2022-12-15 15:56:48 +01:00
Jeong Arm e8cc63105f
Don't delivery a reply to domains which are blocked by author (#22117)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-12-15 15:52:06 +01:00
Claire c8849d6cee
Fix unbounded recursion in account discovery (#22025)
* Fix trying to fetch posts from other users when fetching featured posts

* Rate-limit discovery of new subdomains

* Put a limit on recursively discovering new accounts
2022-12-07 00:15:24 +01:00
Claire fe523a3045
Fix unbounded recursion in account discovery (#1994)
* Fix trying to fetch posts from other users when fetching featured posts

* Rate-limit discovery of new subdomains

* Put a limit on recursively discovering new accounts
2022-12-04 21:23:19 +01:00
kibigo! 2cabc5d188
Use a tree‐based approach for advanced text formatting (#1907)
* Use a tree‐based approach for adv. text formatting

Sanitizing HTML/Markdown means parsing the content into an HTML tree
under‐the‐hood anyway, and it is more accurate to do mention/hashtag
replacement on the text nodes in that tree than it is to try to hack it
in with regexes et cetera.

This undoes the overrides of `#entities` and `#rewrite` on
`AdvancedTextFormatter` but also stops using them, instead keeping
track of the parsed Nokogiri tree itself and using that in the `#to_s`
method.

Internally, this tree uses `<mastodon-entity>` nodes to keep track of
hashtags, links, and mentions. Sanitization is moved to the beginning,
so it should be known that these do not appear in the input.

* Also disallow entities inside of `<code>`

I think this is generally expected behaviour, and people are annoyed
when their code gets turned into links/hashtags/mentions.

* Minor cleanup to AdvancedTextFormatter

* Change AdvancedTextFormatter to rewrite entities in one pass and sanitize at the end

Also, minor refactoring to better match how other formatters are organized.

* Add some tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-12-02 10:29:42 +01:00
Claire fd6f9bf7a7 Merge branch 'main' into glitch-soc/merge-upstream 2022-12-01 11:08:52 +01:00
Claire 68d1df8bc3
Fix some performance issues with /admin/instances (#21907)
/admin/instances?availability=failing remains wholly unefficient
2022-12-01 10:32:10 +01:00
Claire a2e7997592 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Our README is completely different.
  Discarded upstream changes.
2022-11-28 11:33:34 +01:00
Kaspar V 47f0d7021e
refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487)
* refactor(statuses_vacuum): remove dead code - unused

Method is not called inside class and private.
Clean up dead code.

* refactor(statuses_vacuum): make retention_period present test explicit

This private method only hides functionality.
It is best practice to be as explicit as possible.

* refactor(statuses_vacuum): improve query performance

- fix statuses_scope having sub-select for Account.remote scope by
  `joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
  because it is inexplicit, bad practice and even slower than normal
  `.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
  active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
  because in_batches delivers a full blown active record query result,
  in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
  reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
  the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
  another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send

* refactor(status): simplify unlink_from_conversations

- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?

* refactor(status): searchable_by way less sub selects

These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.

This is how it was:

```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
  Status Load (1.6ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (1.5ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```

This is how it is with this change:

```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
  Status Load (1.7ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (0.7ms)  SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```
2022-11-27 20:41:18 +01:00
Claire 767630be24 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile`:
  Upstream removed blank lines.
2022-11-14 20:27:31 +01:00
Claire 71c92d3f56
Fix emoji substitution not applying only to text nodes in backend code (#20641)
Signed-off-by: Claire <claire.github-309c@sitedethib.com>

Signed-off-by: Claire <claire.github-309c@sitedethib.com>
2022-11-14 20:26:21 +01:00
Claire 93912f0498 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/account.rb`:
  Conflict because we (glitch-soc) have disabled trending of posts without
  review.
  Discarded that upstream change.
- `app/views/admin/settings/discovery/show.html.haml`:
  Just an extra setting in glitch-soc.
  Kept that extra setting.
2022-11-12 13:28:03 +01:00
Bearice Ren 28cda42af5
fixes ArgumentError when proxy is used (#20420)
* fixes ArgumentError when proxy is used

* Update app/lib/request.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-11 21:31:03 +01:00
Eugen Rochko f8e8e622e5
Change incoming activity processing to happen in `ingress` queue (#20264) 2022-11-10 14:21:51 +01:00
Claire c118918520 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/custom_emoji.rb`:
  Not a real conflict, just upstream changing a line too close to
  a glitch-soc-specific validation.
  Applied upstream changes.
- `app/models/public_feed.rb`:
  Not a real conflict, just upstream changing a line too close to
  a glitch-soc-specific parameter documentation.
  Applied upstream changes.
2022-11-10 09:36:47 +01:00
James Tucker 78a6b871fe
Improve performance by avoiding regex construction (#20215)
```ruby
10.times { p /#{FOO}/.object_id }
10.times { p FOO_RE.object_id }
```
2022-11-10 05:49:30 +01:00
Claire c989faaa62
Change Request connection logic to try both IPv6 and IPv4 when available (#20108)
Fixes #19751
2022-11-08 16:36:26 +01:00
Claire a8c854b3ea Merge branch 'main' into glitch-soc/merge-upstream 2022-11-08 15:44:57 +01:00
James Tucker 833d9c2f1c
Improve performance by avoiding method cache busts (#19957)
Switch to monkey-patching http.rb rather than a runtime extend of each
response, so as to avoid busting the global method cache. A guard is
included that will provide developer feedback in development and test
environments should the monkey patch ever collide.
2022-11-08 04:00:27 +01:00
Claire be7032b9cf Merge branch 'main' into glitch-soc/merge-upstream 2022-11-07 19:53:30 +01:00
Claire 3114c826a7
Fix filter handling in status cache hydration (#19963) 2022-11-07 19:47:48 +01:00
Claire 21d6bc1689 Fix status cache hydration discrepancy 2022-11-06 17:29:04 +01:00
Claire 0ad919b192 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`:
  glitch-soc change because of having changed the default number of
  available poll options.
  Applied upstream's changes while keeping glitch-soc's default number of
  poll options.
- `public/oops.png`:
  We had a minor graphics change, probably not worth diverging from upstream.
  Took upstream version.
2022-11-06 09:50:41 +01:00
Claire 2f8fb49d13
Fix users not being able to change their hide_followers_count setting (#1889) 2022-11-05 11:55:25 +01:00
Claire bb89f83cc0
Fix additional issues with status cache hydration (#19747)
* Spare one SQL query when hydrating polls

* Improve tests

* Fix more discrepancies

* Fix possible crash when the status has no application set
2022-11-04 20:01:33 +01:00
Claire 03b991de6c
Fix various issues with store hydration (#19746)
- Improve tests
- Fix possible crash when application of a reblogged post isn't set
- Fix discrepancies around favourited and reblogged attributes
- Fix discrepancies around pinned attribute
- Fix polls not being hydrated
2022-11-04 19:33:16 +01:00
Claire c2170991c7
Fix reblogs being discarded after the reblogged status (#19731) 2022-11-04 16:31:44 +01:00
Eugen Rochko 5f9e47be34
Add caching for payload serialization during fan-out (#19642) 2022-11-04 13:21:06 +01:00
Claire cb19be67d1 Merge branch 'main' into glitch-soc/merge-upstream 2022-10-28 19:23:58 +02:00
Eugen Rochko 1546538de9
Fix improperly checking for blocked domain on followed hashtags (#19472)
Fix #19469
2022-10-26 20:40:56 +02:00
Eugen Rochko bf0ab3e0fa
Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
Eugen Rochko 8f07381856
Revert "Remove preference to aggregate reblogs in home/list feeds (#18112)" (#19463)
This reverts commit af396fa35f.
2022-10-26 09:10:18 +02:00
Claire 30453fab80
Add mention of the translation provider when translating a post (#19433) 2022-10-24 18:37:57 +02:00
Claire 8046cf34d6
Change “Translate” button to only show up when a translation backend is configured (#19434)
* Change “Translate” button to only show up when a translation backend is configured

Fixes #19346

* Add `translation` attribute to /api/v2/instance to expose whether the translation feature is enabled

Fixes #19328
2022-10-24 18:30:58 +02:00
Takeshi Umeda 1d34eff63f
Add featured tag add/remove activity handler (#19408) 2022-10-22 11:49:41 +02:00
Eugen Rochko 839f893168
Change public accounts pages to mount the web UI (#19319)
* Change public accounts pages to mount the web UI

* Fix handling of remote usernames in routes

- When logged in, serve web app
- When logged out, redirect to permalink
- Fix `app-body` class not being set sometimes due to name conflict

* Fix missing `multiColumn` prop

* Fix failing test

* Use `discoverable` attribute to control indexing directives

* Fix `<ColumnLoading />` not using `multiColumn`

* Add `noindex` to accounts in REST API

* Change noindex directive to not be rendered by default before a route is mounted

* Add loading indicator for detailed status in web UI

* Fix missing indicator appearing while account is loading in web UI
2022-10-20 14:35:29 +02:00
Eugen Rochko d86cb4cab8 [Glitch] Allow non-logged users to access /web
Port 43b5d5e38d to glitch-soc

Signed-off-by: Claire <claire.github-309c@sitedethib.com>
2022-10-09 18:23:15 +02:00
Claire e7ab476db2 Revert server-side part of "Add logged-out access to the web UI" 2022-10-02 19:36:28 +02:00
Claire 221580a3af Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/home_controller.rb`:
  Upstream made it so `/web` is available to non-logged-in users
  and `/` redirects to `/web` instead of `/about`.
  Kept our version since glitch-soc's WebUI doesn't have what's
  needed yet and I think /about is still a much better landing
  page anyway.
- `app/models/form/admin_settings.rb`:
  Upstream added new settings, and glitch-soc had an extra setting.
  Not really a conflict.
  Added upstream's new settings.
- `app/serializers/initial_state_serializer.rb`:
  Upstream added a new `server` initial state object.
  Not really a conflict.
  Merged upstream's changes.
- `app/views/admin/settings/edit.html.haml`:
  Upstream added new settings.
  Not really a conflict.
  Merged upstream's changes.
- `app/workers/scheduler/feed_cleanup_scheduler.rb`:
  Upstream refactored that part and removed the file.
  Ported our relevant changes into `app/lib/vacuum/feeds_vacuum.rb`
- `config/settings.yml`:
  Upstream added new settings.
  Not a real conflict.
  Added upstream's new settings.
2022-10-02 18:46:22 +02:00