Commit Graph

731 Commits

Author SHA1 Message Date
Alison Wheeler 2f7c3cb628
Update redis_config.rb to remove warning message (#24352) 2023-04-02 06:49:37 +02:00
Eugen Rochko a9b5598c97
Change user settings to be stored in a more optimal way (#23630)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-03-30 14:44:00 +02:00
Claire 0d70deee53
Add migration tests for user mail notification settings (#24277) 2023-03-27 17:54:42 +02:00
Matt Jankowski b9e34ef098
Migration warning helper, and only run in production env (#24253) 2023-03-26 00:39:24 +01:00
Claire 7f8e1bede4
Bump version to v4.1.1 (#24201) 2023-03-21 15:04:21 +01:00
Claire bdeb6ff180
Fix crash in `tootctl` commands making use of parallelization when Elasticsearch is enabled (#24182) 2023-03-20 20:02:58 +01:00
Nick Schonning aa947a143b
Regen rubocop-todo without Max shadowing (#24076) 2023-03-16 12:31:08 +09:00
Eugen Rochko f0e727f958
Add cache headers to static files served through Rails (#24120) 2023-03-16 02:55:54 +01:00
Nick Schonning 25d36b6edd
Autofix Rubocop Style/RedundantArgument (#23798) 2023-03-16 10:34:00 +09:00
Claire 1d0ad558ff
Change sidekiq-bulk's batch size from 10,000 to 1,000 jobs in one Redis call (#24034) 2023-03-15 03:45:15 +01:00
Nick Schonning e762a14c0a
Enable Rubocop Performance/DeleteSuffix (#24077) 2023-03-13 00:03:07 +01:00
Claire f432db7b9f
Fix sidekiq jobs not triggering Elasticsearch index updates (#24046) 2023-03-12 23:47:55 +01:00
9p4 b715bd8e53
Add refreshing many accounts at once with "tootctl accounts refresh" (#23304) 2023-03-08 17:06:53 +01:00
Jean byroot Boussier 922837dc96
Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
Claire 8784498ebf
Fix tootctl accounts migrate error due to typo (#23567) 2023-03-03 20:45:12 +01:00
Claire 3a6451c867
Add support for incoming rich text (#23913) 2023-03-03 20:19:29 +01:00
Nick Schonning 8fd3fc404d
Autofix Rubocop Rails/RootPathnameMethods (#23760) 2023-02-22 09:57:15 +09:00
Nick Schonning 0cfdd1a401
Enable Rubocop Style/StringConcatenation defaults (#23792) 2023-02-22 09:54:36 +09:00
Nick Schonning 59c8d43d94
Autofix Rubocop Style/RescueStandardError (#23745) 2023-02-20 11:01:20 +01:00
Nick Schonning af4c95100c
Autofix Rubocop Style/FormatString (#23743) 2023-02-20 07:58:33 +01:00
Nick Schonning 717683d1c3
Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
Nick Schonning d2dcb6c45a
Autofix Rubocop Style/UnpackFirst (#23741) 2023-02-20 06:51:43 +01:00
Nick Schonning bf785df9fe
Audofix Rubocop Style/WordArray (#23739) 2023-02-20 06:14:10 +01:00
Nick Schonning 81ad6c2e39
Autofix Rubocop Style/StringLiterals (#23695) 2023-02-19 07:38:14 +09:00
Nick Schonning 2177daeae9
Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
Nick Schonning ab7816a414
Autofix Rubocop Style/Lambda (#23696) 2023-02-18 12:39:00 +01:00
Nick Schonning e2a3ebb271
Autofix Rubocop Style/IfUnlessModifier (#23697) 2023-02-18 12:37:47 +01:00
Nick Schonning a6f77aa28a
Autofix Rubocop Lint/AmbiguousOperatorPrecedence (#23681) 2023-02-18 04:30:23 +01:00
Nick Schonning e2567df860
Enable Lint/RedundantCopDisableDirective (#23687) 2023-02-18 04:30:14 +01:00
Nick Schonning d65b2c1924
Apply Rubocop Style/RedundantConstantBase (#23463) 2023-02-18 04:30:03 +01:00
Nick Schonning ac59d6f19f
Enable Rubocop Style/NumericLiterals (#23647) 2023-02-18 11:05:57 +09:00
Nick Schonning 669f6d2c0a
Run rubocop formatting except line length (#23632) 2023-02-18 06:56:20 +09:00
Nick Schonning 0c9d455ea5
Upgrade to Stylelint 15 with Prettier (#23558) 2023-02-13 04:57:03 +01:00
Claire 70c0d754a6
Bump version to 4.1.0 (#23471)
* Bump version to 4.1.0

* Editorialize changelog some more and highlight API changes

* Update changelog
2023-02-10 22:21:23 +01:00
Nick Schonning 11557d1c5a
Apply Rubocop Rails/RootPublicPath (#23447) 2023-02-08 10:38:07 +01:00
Nick Schonning f68bb52556
Apply Rubocop Style/NegatedIfElseCondition (#23451) 2023-02-08 07:07:36 +01:00
Nick Schonning 203739dd3a
Apply Rubocop Performance/StringIdentifierArgument (#23444) 2023-02-08 02:36:20 +01:00
Nick Schonning c92e033cdd
Apply Rubocop Performance/BindCall (#23437) 2023-02-08 09:10:25 +09:00
Claire 79ca19e9b2
Bump version to 4.1.0rc3 (#23384) 2023-02-03 16:39:38 +01:00
Claire 2f112432e6
Bump version to 4.1.0rc2 (#23220) 2023-01-25 16:20:54 +01:00
Claire 8180f7ba19
Bump version to 4.1.0rc1 (#23112) 2023-01-20 14:19:12 +01:00
JT Olio a5fd2fe1cb
Add Storj DCS to cloud object storage options (#21929)
* Add Storj DCS to cloud object storage options

More explanation here: https://forum.storj.io/t/object-storage-provider-for-mastodon-instance/11464/37

* more help for which command to use
2023-01-18 17:47:49 +01:00
Claire cb4e28f405
Add `tootctl domains purge` options to select subdomains and keep domain blocks (#22063)
* Add --include-subdomains option to tootctl domains purge

* Add support for '*.' subdomain wildcard patterns in `tootctl domains purge`

* Fix custom emojis deletion not following subdomain and URI options

* Change `tootctl domains purge` to not purge domain blocks unless --purge-domain-blocks is passed

* Refactor `tootctl domains purge`

* Add feedback on deleted domain blocks
2023-01-18 16:50:50 +01:00
Jeong Arm 0e8f8a1a1c
Implement tootctl accounts prune (#18397)
* Implement tootctl accounts prune

* Optimise query

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-01-13 22:34:16 +01:00
Claire 745bdb11a0
Add `tootctl accounts migrate` (#22330)
* Add tootctl accounts replay-migration

Fixes #22281

* Change `tootctl accounts replay-migration` to `tootctl accounts migrate`
2023-01-13 17:00:23 +01:00
Claire a3a5aa1597
Fix incorrect env file generation in mastodon:setup (#23072)
Regression from #23012
2023-01-13 10:17:07 +01:00
Claire 15b88a83ab
Fix sanitizer parsing link text as HTML when stripping unsupported links (#22558) 2023-01-11 22:21:10 +01:00
Claire a65f86ae55
Fix `$` not being escaped in `.env.production` file generated by `mastodon:setup` (#23012)
* Fix `$` not being escaped in `.env.production` file generated by `mastodon:setup`

* Improve robustness of dotenv escaping
2023-01-11 21:53:11 +01:00
Nick Schonning 558ac411c4
Expand Stylelint glob to include CSS files (#22469) 2023-01-05 13:42:13 +01:00
Dan Peterson 3d3429243f
Fix default S3_HOSTNAME used in mastodon:setup (#19932)
s3-us-east-1.amazonaws.com does not exist.

Co-authored-by: Effy Elden <effy@effy.space>
2022-12-15 16:38:51 +01:00
Claire f239d31f23
Add --email and --dry-run options to `tootctl accounts delete` (#22328) 2022-12-15 14:52:50 +01:00
Evan 78ef635980
Add command to remove avatar and header images of inactive remote accounts from the local database (#22149)
* Add tootctl subcommand media remove-profile-media

* Trigger workflows

* Correcting external linting

* External linting error

* External linting fix

* Merging with remove command

* Linting

* Correct long option names

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Correct long option names

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Correct long option names

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Remove saving a list of purged accounts

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-12-14 19:50:07 +01:00
Claire 55b210b3e5
Fix crash and incorrect behavior in tootctl domains crawl (#19004) 2022-12-13 20:02:32 +01:00
Claire c52263f6f8
Fix deprecation warning in `tootctl accounts rotate` (#22120) 2022-12-07 14:13:10 +01:00
Claire 66a70ebb6e
Fix pre-4.0 admin action logs (#22091)
* Fix BackfillAdminActionLogs post-deployment migration

* Improve migration tests

* Backfill admin action logs again
2022-12-06 23:38:03 +01:00
Claire 098ced7420
Remove support for Ruby 2.6 (#21477)
As pointed out by https://github.com/mastodon/mastodon/pull/21297#discussion_r1028372193
at least one of our dependencies already dropped support for Ruby 2.6, and we
had removed Ruby 2.6 tests from the CI over a year ago (#16861).

So stop advertising Ruby 2.6 support, bump targeted version, and drop some
compatibility code.
2022-11-27 20:41:39 +01:00
Claire d587a268fd
Add logging for Rails cache timeouts (#21667)
* Reduce redis cache store connect timeout from default 20 seconds to 5 seconds

* Log cache store errors
2022-11-27 20:37:37 +01:00
Eugen Rochko 03b0f3ac83
Bump version to 4.0.2 (#20725) 2022-11-15 03:57:18 +01:00
Eugen Rochko 4415dd6036
Bump version to 4.0.1 (#20696) 2022-11-14 22:21:14 +01:00
Eugen Rochko fb389bd73c
Bump version to 4.0.0 (#20636) 2022-11-14 20:27:12 +01:00
Eugen Rochko 75299a042c
Bump version to 4.0.0rc4 (#20634) 2022-11-14 08:50:14 +01:00
Claire 457c37e47a
Fix index name in fix-duplicates task (#20632) 2022-11-14 08:33:48 +01:00
Eugen Rochko b31afc6294
Fix error when passing unknown filter param in REST API (#20626)
Fix #19156
2022-11-14 08:06:06 +01:00
Claire bd806a3090
Update fix-duplicates (#20502)
Fixes #19133
2022-11-13 21:01:38 +01:00
Arthur Isac 1af482659d
Copied Spaces support from packer .rake (#20573) 2022-11-13 20:58:40 +01:00
Eugen Rochko 53028af10e
Bump version to 4.0.0rc3 (#20378) 2022-11-11 08:39:38 +01:00
Pierre Bourdon 36bc90e8aa
blurhash_transcoder: prevent out-of-bound reads with <8bpp images (#20388)
The Blurhash library used by Mastodon requires an input encoded as 24
bits raw RGB data. The conversion to raw RGB using Imagemagick did not
previously specify the desired bit depth. In some situations, this leads
Imagemagick to output in a pixel format using less bpp than expected.
This then manifested as segfaults of the Sidekiq process due to
out-of-bounds read, or potentially a (highly noisy) memory infoleak.

Fixes #19235.
2022-11-11 07:45:16 +01:00
Yamagishi Kazutoshi 19a8563905
Fix `ENV` (#20377) 2022-11-11 01:33:32 +01:00
F 9feba112a7
Make enable_starttls configurable by envvars (#20321)
ENABLE_STARTTLS is designed to replace ENABLE_STARTTLS_AUTO by accepting
three values: 'auto' (the default), 'always', and 'never'. If
ENABLE_STARTTLS isn't provided, we fall back to ENABLE_STARTTLS_AUTO. In
this way, this change should be fully backwards compatible.

Resolves #20311
2022-11-10 21:06:21 +01:00
Eugen Rochko 5187e4e758
Bump version to 4.0.0rc2 (#19831) 2022-11-06 06:59:56 +01:00
Eugen Rochko e02812d5b6
Add assets from Twemoji 14.0 (#19733) 2022-11-04 16:08:41 +01:00
Claire 1dca08b76f
Fix admin action logs page (#19649)
* Add tests

* Fix crash when trying to display orphaned action logs

* Add migration for older admin action logs
2022-11-03 16:06:42 +01:00
Claire e91418436a
Fix mastodon:setup not setting the admin's role properly (#19670)
* Fix mastodon:setup not setting the admin's role properly

* Set contact username when creating admin account in mastodon:setup
2022-11-02 16:35:21 +01:00
pea-sys c68e6b52d9
png optimization(loss less) (#19630) 2022-11-01 15:06:52 +01:00
Eugen Rochko 8ae0936ddd
Bump version to 4.0.0rc1 (#19473) 2022-10-28 00:26:02 +02:00
Eugen Rochko d7595adbf4
Add `--remove-role` option to `tootctl accounts modify` (#19477)
Fix #19152
2022-10-27 14:31:10 +02:00
Jeong Arm 882e54c786
Fix Ambiguous SQL error on tootctl media refresh (#19206) 2022-09-20 23:50:19 +02:00
Claire 1145dbd327
Improve error reporting and logging when processing remote accounts (#15605)
* Add a more descriptive PrivateNetworkAddressError exception class

* Remove unnecessary exception class to rescue clause

* Remove unnecessary include to JsonLdHelper

* Give more neutral error message when too many webfinger redirects

* Remove unnecessary guard condition

* Rework how “ActivityPub::FetchRemoteAccountService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteAccountService#call (default/previous behavior).

* Rework how “ActivityPub::FetchRemoteKeyService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteKeyService#call (default/previous behavior).

* Fix Webfinger::RedirectError not being a subclass of Webfinger::Error

* Add suppress_errors option to ResolveAccountService

Defaults to true (to preserve previous behavior). If set to false,
errors will be raised instead of caught, allowing the caller to be
informed of what went wrong.

* Return more precise error when failing to fetch account signing AP payloads

* Add tests

* Fixes

* Refactor error handling a bit

* Fix various issues

* Add specific error when provided Digest is not 256 bits of base64-encoded data

* Please CodeClimate

* Improve webfinger error reporting
2022-09-20 23:30:26 +02:00
luzpaz 4aa3b9bd01
Fix typos (#18604)
* Fix typos

Found via `codespell -q 3 -S ./CHANGELOG.md,./AUTHORS.md,./config/locales,./app/javascript/mastodon/locales -L ba,keypair,medias,pixelx,ro`

* Follow-up typo fix
2022-08-28 17:44:34 +02:00
Eugen Rochko c556c3a0d1
Add admin API for managing canonical e-mail blocks (#19067) 2022-08-28 03:31:54 +02:00
Jeong Arm e682975afd
Add '--days' option to tootctl media refresh (#18425)
* Add '--days' option to tootctl media refresh

* Fix undefined scope
2022-08-25 04:40:17 +02:00
Brayd fc46fa8f99
Minimal adjustments to the short description (#18001)
Minimal adjustments have been made to the short description so that it logically follows the long description
2022-08-13 15:41:12 +02:00
Eugen Rochko 44b2ee3485
Add customizable user roles (#18641)
* Add customizable user roles

* Various fixes and improvements

* Add migration for old settings and fix tootctl role management
2022-07-05 02:41:40 +02:00
Claire 02851848e9
Revamp post filtering system (#18058)
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
2022-06-28 09:42:13 +02:00
Eugen Rochko fe2d6fe105
Fix wrong aspect ratio of logo in icons (#18639) 2022-06-11 20:32:02 +02:00
Eugen Rochko 45aa5781ce
Change brand color and logotypes (#18592)
- Add rake task for generating Apple/Android icons and favicons from SVG
- Add rake task for generating PNG icons and logos for e-mails from SVG
- Remove obsolete Microsoft icons and configuration
- Remove PWA shortcut icons
2022-06-09 22:25:23 +02:00
Claire 9d4861b498
Remove dependency on running Redis server for db:setup (#18560) 2022-06-01 19:23:31 +02:00
Eugen Rochko fbcbf7898f
Bump version to 3.5.3 (#18530) 2022-05-26 23:26:15 +02:00
Eugen Rochko a9b64b24d6
Change algorithm of `tootctl search deploy` to improve performance (#18463) 2022-05-22 22:16:43 +02:00
Eugen Rochko 679b7158e3
Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Claire f714e24ff1
Fix redis configuration not being changed by mastodon:setup (#18383)
Fixes #18342
2022-05-09 23:19:11 +02:00
Claire 014065913c
Bump version to 3.5.2 (#18295)
* Bump version to 3.5.2

* Change some entries to be more clear

* Add some extra notes

* Fix line wrap

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2022-05-04 00:57:42 +02:00
Claire bc19c083ce
Add ability to set approval-based registration through tootctl (#18248)
Fixes #18235

Add `tootctl settings registrations approved` with
optional `--require-reason` switch.
2022-05-02 17:41:34 +02:00
Eugen Rochko 7b0fe4aef9
Fix opening and closing Redis connections instead of using a pool (#18171)
* Fix opening and closing Redis connections instead of using a pool

* Fix Redis connections not being returned to the pool in CLI commands
2022-04-29 22:43:07 +02:00
Gaelan Steele 74e20f22cd
Fix light-mode emoji borders. (#18131) 2022-04-29 19:23:03 +02:00
Eugen Rochko 3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Claire 33cd80d69c
Fix instance actor being incorrectly created when running migrations (#18109)
* Add migration test about instance actor key

* Fix old migration

* Work around incorrect database state
2022-04-26 21:22:09 +02:00
Eugen Rochko ed5491e5de
Bump version to 3.5.1 (#18000) 2022-04-08 21:57:24 +02:00
0x2019 012537452a
Fix error resposes for `from` search prefix (#17963)
* Fix error responses in `from` search prefix (addresses mastodon/mastodon#17941)

Using unsupported prefixes now reports a 422; searching for posts from an
account the instance is not aware of reports a 404. TODO: The UI for this
on the front end is abysmal.

Searching `from:username@domain` now succeeds when `domain` is the local
domain; searching `from:@username(@domain)?` now works as expected.

* Remove unused methods on new Error classes as they are not being used

Currently when `raise`d there are error messages being supplied, but
this is not actually being used. The associated `raise`s have been
edited accordingly.

* Remove needless comments

* Satisfy rubocop

* Try fixing tests being unable to find AccountFindingConcern methods

* Satisfy rubocop

* Simplify `from` prefix logic

This incorporates @ClearlyClaire's suggestion (see
https://github.com/mastodon/mastodon/pull/17963#pullrequestreview-933986737).

Accepctable account strings in `from:` clauses are more lenient than
before this commit; for example, `from:@user@example.org@asnteo +cat`
will not error, and return posts by @user@example.org containing the
word "cat". This is more consistent with how Mastodon matches mentions
in statuses. In addition, `from` clauses will not be checked for
syntatically invalid usernames or domain names, simply 404ing when
`Account.find_remote!` raises ActiveRecord::NotFound.

New code for this PR that is no longer used has been removed.
2022-04-08 21:21:49 +02:00
Eugen Rochko 6e418bf346
Fix cookies secure flag being set when served over Tor (#17992) 2022-04-08 12:47:18 +02:00
Claire cb45c04d26
Fix migration error handling (#17991) 2022-04-07 20:46:30 +02:00
Claire 5f0fc639da
Fix error re-running some migrations if they get interrupted at the wrong moment (#17989) 2022-04-07 20:17:49 +02:00
Eugen Rochko 6221b36b27
Remove sign-in token authentication, instead send e-mail about new sign-in (#17970) 2022-04-06 20:58:12 +02:00
Holger 39b489ba4c
fix: `s3_force_single_request` not parsed (#17922) 2022-04-01 23:56:23 +02:00
Eugen Rochko 8c7223f4ea
Bump version to 3.5.0 (#17911) 2022-03-30 14:52:37 +02:00
Eugen Rochko d7d049aab7
Bump version to 3.5.0rc3 (#17876) 2022-03-26 04:29:36 +01:00
Eugen Rochko 07f8b4d1b1
Bump version to 3.5.0rc2 (#17855) 2022-03-26 02:54:11 +01:00
Claire 3afd59df0f
Fix tootctl email_domain_blocks add (#17842)
Fixes #17831
2022-03-21 19:10:09 +01:00
Claire b07906bdb0
Fix wrong language code for Kurdish languages (#17812) 2022-03-17 01:37:03 +01:00
Eugen Rochko 4bdce2c513
Bump version to 3.5.0rc1 (#17618)
* Bump version to 3.5.0rc1

* Various fixes and improvements

* Update AUTHORS.md

* Various fixes and improvements

* Update README.md
2022-03-15 08:16:45 +01:00
Claire 642528f455
Update fix-duplicates maintenance task (#17731)
* Update fix-duplicates task to 2022_02_10_153119

Also add support for Appeal to AccountMerging#merge_with!

* Update fix-duplicates task to 2022_03_07_094650

* Update fix-duplicates task to 2022_03_09_213005

* Update fix-duplicates task to 2022_03_07_083603

* Update fix-duplicates task to 2022_03_10_060626

* Update fix-duplicates script to 2022_03_07_083603

* Update fix-duplicates task to 2022_03_10_060706

* Update fix-duplicates task to 2022_03_10_060959

* Silence CodeClimate
2022-03-12 08:33:11 +01:00
Eugen Rochko 75e33fd08f
Fix null values being included in some indexes (#17711)
* Fix null values being included in some indexes

* Update lib/mastodon/migration_helpers.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add documentation link to corruption error message

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-12 08:12:57 +01:00
Claire 61ae6b3535
Add more migration tests (#17710)
* Add migration tests for hide_network settings migration

* Add tests about suspended/suspended_at

* Add more tests regarding the results of migrations

* Fix migration test regarding stale conflicting remote account

* Add migration tests about AccountConversation
2022-03-07 23:40:55 +01:00
Rens Groothuijsen c439e13e12
Enable importing GIF emojis in CLI (#17706) 2022-03-06 23:41:44 +01:00
Josh Soref b5329e0035
Spelling (#17705)
* spelling: account

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: affiliated

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: appearance

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autosuggest

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: cacheable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: component

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: conversations

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: domain.example

Clarify what's distinct and use RFC friendly domain space.

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: environment

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: exceeds

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: functional

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: inefficiency

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: not

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: notifications

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: occurring

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: position

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: progress

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: promotable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reblogging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repetitive

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: resolve

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: saturated

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: similar

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: strategies

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: success

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: targeting

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: thumbnails

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unauthorized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unsensitizes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: validations

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: various

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

Co-authored-by: Josh Soref <jsoref@users.noreply.github.com>
2022-03-06 22:51:40 +01:00
Jeong Arm 3e12abc1fe
Calculate max_id without random vector (#17623) 2022-02-23 16:44:59 +01:00
Claire 166f6e4b50
Fix some media attachments being converted with too high framerates (#17619)
Video files with variable framerates are converted to constant framerate videos
and the output framerate picked by ffmpeg is based on the original file's
container framerate (which can be different from the average framerate).

This means that an input video with variable framerate with about 30 frames per
second on average, but a maximum of 120 fps will be converted to a constant 120
fps file, which won't be processed by other Mastodon servers.

This commit changes it so that input files with VFR and a maximum framerate
above the framerate threshold are converted to VFR files with the maximum frame
rate enforced.
2022-02-22 17:11:22 +01:00
Eugen Rochko b6d7726ecb
Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Eugen Rochko 3413f1c44b
Forward-port version bump to 3.4.6 (#17434) 2022-02-03 14:21:38 +01:00
Claire 54581d43e7
Bump version to 3.4.5 (#17402) 2022-01-31 21:27:40 +01:00
Claire a0e06c3c3e
Add more advanced migration tests (#17393)
- populate the database with some data when testing migrations
- try both one-step and two-step migrations (`SKIP_POST_DEPLOYMENT_MIGRATIONS`)
2022-01-30 23:50:08 +01:00
Claire c6b291afc3
Change index corruption warning to be a little less scary (#17395) 2022-01-30 23:49:52 +01:00
Claire a99adeaad3
Fix edge case in migration helpers that caused crash because of PostgreSQL quirks (#17398) 2022-01-30 22:34:54 +01:00
Claire 8a07ecd377
Remove leftover database columns from Devise::Models::Rememberable (#17191)
* Remove leftover database columns from Devise::Models::Rememberable

* Update fix-duplication maintenance script

* Improve errors/warnings in the fix-duplicates maintenance script
2022-01-23 15:46:30 +01:00
Claire 1e8c885e5a
Change mastodon:webpush:generate_vapid_key task to not require functional env (#17338)
Fixes #17297
2022-01-20 14:51:23 +01:00
Eugen Rochko fe71548844
Fix warnings on Rails boot (#16946) 2021-12-27 00:47:20 +01:00
Claire 76761d5fc0
Add ability for admins to delete canonical email blocks (#16644)
* Add admin option to remove canonical email blocks from a deleted account

* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
2021-12-17 23:02:14 +01:00
Takeshi Umeda 6c8c031bcd
Add remove orphans to tootctl statuses remove (#17067)
* Add remove orphans to tootctl statuses remove

* Add REINDEX and change option from vacuum to compression-database

* Changed to extract the deletion target of conversations to a temporary table

* Support progress bar and exceptions when media remove

* Add continue option

* Fix compression to compress

* Remove skip_remove_orphans
2021-12-12 06:09:14 +01:00
Takeshi Umeda 0ac7efdc50
Fix performance of tootctl statuses remove (#17052)
* Fix performance of tootctl statuses remove

* Fix model class
2021-11-26 22:08:47 +01:00
Takeshi Umeda 06631fdc53
Fix ElasticSearch to Elasticsearch (#17050) 2021-11-26 08:30:02 +01:00
OSAMU SATO 53aca8aecf
Add batch_size option to bin/tootctl search deploy (#17049) 2021-11-26 08:29:53 +01:00
Eugen Rochko 6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Takeshi Umeda 3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire 48f8658d34
Fix upload of remote media with OpenStack Swift sometimes failing (#16998)
Under certain conditions, files fetched from remotes trigger an error when
being uploaded using OpenStack Swift. This is because in some cases, the
remote server will not return a content-length, so our ResponseWithLimitAdapter
will hold a `nil` value for `#size`, which will lead to an invalid value
for the Content-Length header of the Swift API call.

This commit fixes that by taking the size from the actually-downloaded file
size rather than the upstream-provided Content-Length header value.
2021-11-16 21:36:28 +01:00
Claire 18b885ee3a
Fix "bundle exec rails mastodon:setup" crashing in some circumstances (#16976)
Fix regression from #16896
2021-11-11 14:00:30 +01:00
Eugen Rochko 2251db42ec
Forward port version bumps to 3.4.2 and 3.4.3 (#16945)
* Bump version to 3.4.2

* Bump version to 3.4.3
2021-11-06 05:32:14 +01:00
Jeong Arm 884c60002e
Skip blocked domains media on tootctl media refresh (#16914) 2021-10-28 19:30:44 +02:00
Claire 5ba46952af
Fix mastodon:setup to take dotenv/docker-compose differences into account (#16896)
In order to work around https://github.com/mastodon/mastodon/issues/16895,
add a warning to .env.production.sample, and change the mastodon:setup rake
task to:
- output a warning if a variable will be interpreted differently by dotenv
  and docker-compose
- ensure the printed config is compatible with docker-compose
2021-10-25 16:34:15 +02:00
Claire 6ba8bc45cb
Add S3_FORCE_SINGLE_REQUEST env var to work around S3 compatibility issues (#16866)
Fixes #16822
2021-10-18 18:29:04 +02:00
Jeong Arm 3f5f4273b3
Add optional domain restrict to tootctl accounts cull (#16511)
* Add optional domain restrict to accounts cull

* Use "unless" - codeclimate
2021-10-14 21:09:56 +02:00
Claire a8ef6d24d8
Fix `tootctl accounts cull` not excluding domains on timeouts and certificate issues (#16433)
Fixes #16410
2021-10-14 21:08:37 +02:00
Claire 959f7fc580
Fix tootctl self-destruct not sending Delete activities for recently-suspended accounts (#16688)
* Do not block existing users' emails on self-destruct

That is wasteful and unintuitive

* Do not close registrations when running tootctl self-destruct with --dry-run

* Close registrations on self-destruct regardless of known remote accounts

* Fix tootctl self-destruct not sending Deletes for recently-suspended accounts

* Suspend local users even if no remote account is known

* Do not show scary confirmation text if ran with --dry-run
2021-10-14 19:59:28 +02:00
Claire fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Claire 4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Takeshi Umeda 818e0b314f
Fix unsupported video error message handling (#16581) 2021-08-08 15:28:57 +02:00
Claire 5a1e072517
Change references to tootsuite/mastodon to mastodon/mastodon (#16491)
* Change references to tootsuite/mastodon to mastodon/mastodon

* Remove obsolete test fixture

* Replace occurrences of tootsuite/mastodon with mastodon/mastodon in CHANGELOG

And a few other places
2021-07-13 15:46:20 +02:00
Eugen Rochko 771c9d4ba8
Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire 2e0eac71dd
Add --by-uri option to `tootctl domains purge` (#16434)
Fixes #16410
2021-07-07 21:17:00 +02:00
Claire f6088922c0
Update emoji codepoint mappings to v13.1 (#16352) 2021-06-03 16:08:07 +02:00
Eugen Rochko d6486c969f
Bump version to 3.4.1 (#16350) 2021-06-03 04:26:02 +02:00