We have changed how we store reblogs in the redis for bigint IDs. This process is done by 1) scan all entries in users feed, and 2) re-store reblogs by 3 write commands.
However, this operation is really slow for large instances. e.g. 1hrs on friends.nico (w/ 50k users). So I have tried below tweaks.
* It checked non-reblogs by `entry[0] == entry[1]`, but this condition won't work because `entry[0]` is String while `entry[1]` is Float. Changing `entry[0].to_i == entry[1]` seems work.
-> about 4-20x faster (feed with less reblogs will be faster)
* Write operations can be batched by pipeline
-> about 6x faster
* Wrap operation by Lua script and execute by EVALSHA command. This really reduces packets between Ruby and Redis.
-> about 3x faster
I've taken Lua script way, though doing other optimizations may be enough.
Glitch::KeywordMute's name is inferred as glitch_keyword_mutes, and in
templates this turns into e.g. settings/glitch/keyword_mutes. Going
along with this convention means a lot of file movement, though, and for
a UI that's as temporary and awkward as this one I think it's less
effort to slap a bunch of as: options everywhere.
We'll do the Right Thing when we build out the API and frontend UI.
Also make the keyword-building methods private: they always probably
should have been private, but now I have encoded enough fun and games
into them that it now seems wrong for them to *not* be private.
It is possible to cache a Regexp object, but I'm not sure what happens
if e.g. that object remains in cache across two different Ruby versions.
Caching a string seems to raise fewer questions.
Ditto for ending with \b.
Consider muting the phrase "(hot take)". I stipulate it is reasonable
to enter this with the default "match whole word" behavior. Under the
old behavior, this would be encoded as
\b\(hot\ take\)\b
However, if \b is before the first character in the string and the first
character in the string is not a word character, then the match will
fail. Ditto for after. In our example, "(" is not a word character, so
this will not match statuses containing "(hot take)", and that's a very
surprising behavior.
To address this, we only add leading and trailing \b to keywords that
start or end with word characters.
There are two motivations for this:
1. It looks like we're going to add other features that require
server-side storage (e.g. user notes).
2. Namespacing glitchsoc modifications is a good idea anyway: even if we
do not end up doing (1), if upstream introduces a keyword-mute feature
that also uses a "KeywordMute" model, we can avoid some merge
conflicts this way and work on the more interesting task of
choosing which implementation to use.
Word-boundary matching only works as intended in English and languages
that use similar word-breaking characters; it doesn't work so well in
(say) Japanese, Chinese, or Thai. It's unacceptable to have a feature
that doesn't work as intended for some languages. (Moreso especially
considering that it's likely that the largest contingent on the Mastodon
bit of the fediverse speaks Japanese.)
There are rules specified in Unicode TR29[1] for word-breaking across
all languages supported by Unicode, but the rules deliberately do not
cover all cases. In fact, TR29 states
For example, reliable detection of word boundaries in languages such
as Thai, Lao, Chinese, or Japanese requires the use of dictionary
lookup, analogous to English hyphenation.
So we aren't going to be able to make word detection work with regexes
within Mastodon (or glitchsoc). However, for a first pass (even if it's
kind of punting) we can allow the user to choose whether they want word
or substring detection and warn about the limitations of this
implementation in, say, docs.
[1]: https://unicode.org/reports/tr29/https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/
This should eventually be accessible via the API and the web frontend,
but I find it easier to set up an editing interface using Rails
templates and the like. We can always take it out if it turns out we
don't need it.
David Yip