Go to file
Daniel Kinzler 81218db057
Make parsoid transform follow disable_storage setting. (#1337)
If disable_storage is set, don't try to load the original HTML from
cache for transform operations.

Bug: T350219
Change-Id: I33c3baae4f0579121b89f6d913852d6c118ebd56
2023-12-14 17:49:23 +01:00
.github/workflows pcs sunset: Emit metrics for PCS latency 2022-10-04 13:41:28 +02:00
.pipeline ci: Provide basic `.pipeline/config.yaml` 2022-05-05 09:25:46 -07:00
doc Remove public endpoints for editing pages (#1326) 2023-09-19 10:08:15 +02:00
lib Drop rebase artifact lib/parsoid.js 2020-11-17 11:49:07 -07:00
maintenance Fix linting errors for new version of eslint config 2019-08-05 14:40:22 -07:00
projects Disable storage for parsoid requests (#1330) 2023-10-11 16:52:06 +03:00
sys Make parsoid transform follow disable_storage setting. (#1337) 2023-12-14 17:49:23 +01:00
test Make parsoid transform follow disable_storage setting. (#1337) 2023-12-14 17:49:23 +01:00
v1 Use GET when fetching related pages (#1336) 2023-12-05 12:55:58 +01:00
.dockerignore Replace istanbul with nyc 2019-02-15 12:37:14 -08:00
.eslintignore Removed mocha-eslint and eslint-config-node-services dependencies. 2019-01-23 14:50:15 -05:00
.eslintrc.yml Remove no-multi-spaces exception 2020-04-13 11:01:54 -07:00
.gitignore ci: Provide basic `.pipeline/config.yaml` 2022-05-05 09:25:46 -07:00
.mocharc.yaml Improve test runner and use service-runner provided tests runner. 2019-03-27 15:54:23 -03:00
DUMMY Use beta for tests (#1339) 2023-12-01 12:48:59 +01:00
LICENSE Switch to Apache2 & link to phabricator / wikimedia repo 2015-02-14 14:18:25 -08:00
README.md Update README.md 2022-10-04 13:41:28 +02:00
config.backend.test.yaml Expose fronted RB on 7231 and backend RB on 7233 2019-08-01 17:32:38 +02:00
config.example.yaml Remove public endpoints for editing pages (#1326) 2023-09-19 10:08:15 +02:00
config.frontend.test.yaml Use beta for tests (#1339) 2023-12-01 12:48:59 +01:00
config.fullstack.test.yaml Use beta for tests (#1339) 2023-12-01 12:48:59 +01:00
index.js Removed mocha-eslint and eslint-config-node-services dependencies. 2019-01-23 14:50:15 -05:00
package.json Update deploy target images 2023-05-11 15:07:45 +03:00
server.js Removed mocha-eslint and eslint-config-node-services dependencies. 2019-01-23 14:50:15 -05:00

README.md

RESTBase Build Status Coverage Status

RESTBase was built to provide a low-latency & high-throughput API for Wikipedia / Wikimedia content. It is basically a storage proxy, which presents a coherent API powered by Swagger specs to the outside, and backs up many of these entry points with storage. The default table storage backend is based on Cassandra, which helps it to perform well at Wikimedia's scale without placing undue burden on operations.

As a proxy, RESTBase does not perform any significant content processing itself. Instead, it requests content transformations from backend services when needed, and typically (depending on configuration) stores it back for later retrieval. For high-volume static end points, most requests will be satisfied directly from storage.

The table storage backends conform to a RESTful table storage API similar to Amazon DynamoDB and Google DataStore. The primary implementation uses Apache Cassandra. Notable features include automatically maintained secondary indexes and some lightweight transaction support. A SQLite backend is under development.

RESTBase systematically emits statsd metrics about storage and backend requests. Specifically, the systematic metric production for backend services provides a good baseline level of instrumentation for tracking performance and errors in a micro-service architecture.

Issue tracking

We use Phabricator to track issues. See the list of current issues in RESTBase.

Installation

Make sure that you have node 6+:

sudo apt-get install nodejs nodejs-legacy nodejs-dev npm

Note: if your distribution does not have a recent version of Node, you can install one via nvm.

From the restbase project directory, install the Node dependencies:

npm install

Start RESTBase:

node server

The defaults without a config file should work for a local Cassandra installation with the default passwords. Restbase has been tested with Cassandra 2.2.6. To customize RESTBase's behaviour, copy the example config to its default location:

cp config.example.yaml config.yaml

You can also pass in the path to another file with the -c commandline option to server.js. If you're running a single Cassandra instance (e.g. a local development environment), set defaultConsistency to one in config.yaml.

Usage

See the Wikimedia REST content API sandbox for a fine example of what RESTBase can do.

Development

Testing

To run all the tests from a clean slate, first make sure Cassandra is running locally, then fire up the tests with npm:

npm test

To run tests from a single file, e.g. test/features/pagecontent/rerendering.js, run mocha with the file as an argument:

mocha test/features/pagecontent/rerendering.js

Note that this might require some setup (e.g. creating the necessary domain and buckets), which is currently done by test/buckets.js.

This also works for a directory, e.g. test/features/pagecontent/:

mocha test/features/pagecontent

Coverage

To check the test coverage, use npm, then browse the report:

npm run-script coverage

The coverage report can now be found in <project>/coverage/lcov-report/index.html.

Design docs