* Fix language filter codes
CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)
* Fix zh name
* Add handling of Linked Data Signatures in payloads
* Add a way to sign JSON, fix canonicalization of signature options
* Fix signatureValue encoding, send out signed JSON when distributing
* Add missing security context
* Update twitter-text to version 1.14.7
* Update tilt to version 2.0.8
* Update statsd-instrument to version 2.1.4
* Update sidekiq to version 5.0.4
* Update sidekiq-scheduler to version 2.1.8
* Update sidekiq-unique-jobs to version 5.0.9
* Update redis-activesupport to version 5.0.3
* Update rails-settings-cached to version v0.6.6
* Update pkg-config to version 1.2.4
* Update parallel_tests to version 2.14.2
* Update jsonapi-renderer to version 0.1.3
* Update i18n-tasks to version 0.9.16
* Update httplog to version 0.99.7
* Update fabrication to version 2.16.2
* Update bootsnap to version 1.1.2
* Update aws-sigv4 to version 1.0.1
* Update aws-sdk-core to version 2.10.21
* Update hashdiff to version 0.3.5
* Update rails to version 5.1.3
* Refactor ResolveRemoteAccountService
* Remove trailing whitespace
* Use redis locks around critical ResolveRemoteAccountService code
* Add test for race condition of lock
* Add dependency on idn-ruby to speed up URI normalization
* Use normalized_host instead of normalize.host when applicable
When we are only interested in the normalized host, calling normalized_host
avoids normalizing the other components of the URI as well as creating a
new object
- Use unicode when selecting emoji through picker
- Convert shortcodes to unicode when storing text input server-side
- Do not convert shortcodes in JS anymore
* Specs for language detection
* Use CharlockHolmes instead of NKF
* Correct mistakes
* Correct style
* Set hint_enc instead of falling back and strip_tags
* Improve specs
* Add dependencies
* Update aws-sdk to version 2.10.4
* Update bootsnap to version 1.1.1
* Update capistrano to version 3.8.2
* Update capybara to version 2.14.4
* Update cld3 to version 3.1.3
* Update http_accept_language to version 2.1.1
* Update sidekiq to version 5.0.3
* Update rspec-sidekiq to version 3.0.3
* Update sidekiq-scheduler to version 2.1.7
* Update oj to version 3.2.0
* Update openssl to version 2.0.4
* Update pg to version 0.21.0
* Update twitter-text to version 1.14.6
* Update unicode-display_width to version 1.3.0
* Update scss_lint to version 0.54.0
* Update hamlit to version 2.8.4
* Update erubi to version 1.6.1
* Update httplog to version 0.99.4
* Update aws-sdk to version 2.10.6
* Add overview of active sessions
* Better display of browser/platform name
* Improve how browser information is stored and displayed for sessions overview
* Fix test
* Update mail to version 2.6.6
* Update aws-sdk to version 2.9.37
* Update capybara to version 2.14.2
* Update oj to version 3.1.0
* Update sidekiq to version 5.0.2
* Update puma to version 3.9.1
* Update sanitize to version 4.5.0
* Update capistrano-rails to version 1.3.0
* Update annotate to version 2.7.2
* Update puma to version 3.9.0
* Update aws-sdk to version 2.9.28
* Update bootsnap to version 1.0.0
* Update nio4r to version 2.1.0
* Update nokogumbo to version 1.4.12
* Update oj to version 3.0.11
* Update pkg-config to version 1.2.3
* Update rubocop to version 0.49.1
* Update sidekiq-scheduler to version 2.1.5
* Update rails to version 5.1.1
* Run `rails app:update`
* Remove the override of polymorphic activity relationship
* Silence warning about otp_secret attribute being unknown to rails
* We will only introduce form_with where we want to use remote data
* Update xpath to version 2.1.0
* Update aws-sdk to version 2.9.25
* Update oj to version 3.0.10
* Update brakeman to version 3.6.2
* Update rufus-scheduler to version 3.4.2
* Update doorkeeper to version 4.2.6
* Update rubocop to version 0.49.0
* Update terminal-table to version 1.8.0
* Update aws-sdk to version 2.9.20
* Update cld3 to version 3.1.2
* Update microformats2 to version 3.1.0
* Update rails-controller-testing to version 1.0.2
* Update oj to version 3.0.8
* Update http-form_data to version 1.0.3
* Update nokogumbo to version 1.4.11
* Update simplecov-html to version 0.10.1
* Update aws-sdk to version 2.9.21
* Update oj to version 3.0.9
* Update sass to version 3.4.24
* Update simple_form to version 3.5.0
* Update rails to version 5.0.3
* Update aws-sdk to version 2.9.17
* Update capistrano to version 3.8.1
* Update climate_control to version 0.2.0
* Update debug_inspector to version 0.0.3
* Update et-orbi to version 1.0.4
* Update oj to version 3.0.7
* Update microformats2 to version 2.9.0
* Update lograge to version 0.5.1
* Clean up usage of Microformats2 object to remove deprecation warnings
* Update microformats2 to version 3.0.1
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.
Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.
CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.
On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.
CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.
The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
Eugen Rochko