html endpoints in MW core
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	• Pchelolo
	Nov 16 2020, 11:52 PM

Description

Task description unfinished.

In order for VE and transformations to work correctly, we need to stash original HTML which the transform is based on. The requirement for the stash is fixed TTL (8 hours in RESTBase currently), simple key-value access. Loosing a stash will result in either a lost or a dirty edit, so we need some rather durable backend for this store. The size of the store is anticipated to be TBD (TODO: check how large it is in RESTBase)

Stashing should be exposed via ether ?stash=true parameter, or we could expose it as a separate /page/{title}/html_for_edit endpoint??

Details

Subject	Repo	Branch	Lines +/-
[WIP] Rest: Add support for annotated HTML for editing	mediawiki/core	master	+166 -45
[WIP] Rest: Add support for annotated HTML for editing	mediawiki/core	master	+74 -5
ParsoidOutputStash: make storage backend configurable.	mediawiki/core	master	+84 -6
Rest: Migrate parsoid stashing logic from RESTbase	mediawiki/core	master	+638 -167
PageHTMLHandler: add stashing option	mediawiki/core	master	+377 -39

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T324931 Clean up open RESTBase related tickets
In Progress	None	T262315 <CORE TECHNOLOGY> API Migration & RESTBase Sunset
Open	None	T264669 Move VE API from RESTBase into core.
Resolved	None	T267990 Support stashing in page/html and revision/html endpoints in MW core
Resolved	None	T267991 Simple per-IP ratelimit for HTML stashing
Duplicate	None	T308684 Determine directory to place ParsoidStash logic
Resolved	daniel	T308743 PageHTMLHandler: add support for getting fully annotated parsoid HTML.
Resolved	daniel	T308744 PageHTMLHandler should return different eTag for different output modes
Resolved	daniel	T309016 Determine storage requirements for stashing parsoid output for VE edits
Resolved	DAlangi_WMF	T309017 Collect stats on stash and cache usage in the page/html and revision/html endpoints
Resolved	daniel	T359426 REST: endpoints that return rendered page content should support all content models
Resolved	daniel	T309549 Create end-to-end tests for the page/html and revision/html endpoints.
Resolved	daniel	T268205 Store data-parsoid in parsoid ParserCache
Open	DAlangi_WMF	T308424 Determine http cache control and active purging for REST endpoints serving parsoid output

Event Timeline

• Pchelolo created this task.Nov 16 2020, 11:52 PM

• Pchelolo moved this task from Backlog to Ready on the Platform Team Workboards (Green) board.Dec 2 2020, 6:07 PM

daniel moved this task from Ready to Backlog on the Platform Team Workboards (Green) board.Dec 16 2020, 5:19 PM

Adding Platform Engineering as Platform Team Workboards (Green) was archived and as open tasks should have an active project tag

WDoranWMF removed projects: Platform Engineering, Platform Team Workboards (Green).Nov 15 2021, 1:27 PM

• Pchelolo mentioned this in T264669: Move VE API from RESTBase into core..Feb 9 2022, 3:53 PM

• nnikkhoui subscribed.Apr 11 2022, 11:53 PM

DAlangi_WMF subscribed.Apr 26 2022, 3:52 PM

Change 779144 had a related patch set uploaded (by Daniel Kinzler; author: Nikki Nikkhoui):

[mediawiki/core@master] PageHTMLHandler: add stashing option

https://gerrit.wikimedia.org/r/779144

gerritbot added a project: Patch-For-Review.Apr 27 2022, 11:56 AM

WDoranWMF added a project: Platform Team Workboards (MW Expedition).May 5 2022, 4:06 PM

WDoranWMF moved this task from Unsorted pile to Doing on the Platform Team Workboards (MW Expedition) board.

Change 789686 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] Rest: Add support for annotated HTML for editing

https://gerrit.wikimedia.org/r/789686

Change 791416 had a related patch set uploaded (by Daniel Kinzler; author: Derick Alangi):

[mediawiki/core@master] [WIP] Rest: Add support for annotated HTML for editing

https://gerrit.wikimedia.org/r/791416

DAlangi_WMF changed the status of subtask T268205: Store data-parsoid in parsoid ParserCache from Open to In Progress.May 16 2022, 3:35 PM

daniel closed subtask T268205: Store data-parsoid in parsoid ParserCache as Resolved.May 16 2022, 4:31 PM

Change 792264 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] Rest: Migrate stashing logic from RESTBase to core

https://gerrit.wikimedia.org/r/792264

Tagging Perf team for awareness as original author and defacto maintainer of Edit stash in core, specifically the PageEditStash backend service and the ApiStashEdit (Action API). These predate the RESTBase version of this, and it might make sense to re-use or combine this in some way.

It would also make sense I think to verify at this point that the RESTBase port indeed satisfies the requirements and expectations of the PageEditStash/ApiStashEdit model, as in the past for VE/RESTBase we've often taken shortcuts or experimented with something that doesn't account for the rest of the ecosystem.

Krinkle added a subtask: T308684: Determine directory to place ParsoidStash logic.May 18 2022, 9:51 PM

Krinkle merged a task: T308684: Determine directory to place ParsoidStash logic.

Krinkle added subscribers: Jdforrester-WMF, ppelberg, ArielGlenn and 6 others.

Krinkle mentioned this in T308684: Determine directory to place ParsoidStash logic.May 18 2022, 9:54 PM

Change 779144 abandoned by Daniel Kinzler:

[mediawiki/core@master] PageHTMLHandler: add stashing option

Reason:

Superceded by Id35f1423a69e3ff63e4f9883b3f7e3f9521d81d5

https://gerrit.wikimedia.org/r/779144

daniel mentioned this in T308511: [SPIKE] Determine necessity of edit session continuity during data center switchovers .May 19 2022, 8:26 PM

Change 792264 merged by jenkins-bot:

[mediawiki/core@master] Rest: Migrate parsoid stashing logic from RESTbase

https://gerrit.wikimedia.org/r/792264

daniel triaged this task as High priority.May 24 2022, 8:03 AM

daniel added a project: Epic.May 24 2022, 8:14 AM

daniel removed a project: Epic.

aaron moved this task from Inbox, needs triage to Radar on the Performance-Team board.May 24 2022, 6:54 PM

aaron edited projects, added Performance-Team (Radar); removed Performance-Team.

DAlangi_WMF changed the task status from Open to In Progress.May 26 2022, 11:51 AM

DAlangi_WMF changed the status of subtask T308744: PageHTMLHandler should return different eTag for different output modes from Open to In Progress.

DAlangi_WMF changed the status of subtask T309017: Collect stats on stash and cache usage in the page/html and revision/html endpoints from Open to In Progress.

daniel closed subtask T308744: PageHTMLHandler should return different eTag for different output modes as Resolved.May 30 2022, 3:07 PM

Blocked on subtasks, in particular storage backend for deployment on wmf cluster.

In T267990#7939991, @Krinkle wrote:

Tagging Perf team for awareness as original author and defacto maintainer of Edit stash in core, specifically the PageEditStash backend service and the ApiStashEdit (Action API). These predate the RESTBase version of this, and it might make sense to re-use or combine this in some way.

The two do fundamentally different things: PageEditStash stashes the HTML for the page's content after the edit, while the parsoid stash stores the HTML of the page before the edit. They serve different purposes as well: PageEditStash speeds up saving by optimistically parsing before the user hits save, while the parsoid stash keeps part of the state on the server side to optimize load time when opening the editor.

The persistence requirements are different as well: losing an entry from PageEditStash just means the edit is a little slower, while losing data from the parsoid stash means the edit will fail.

DAlangi_WMF closed subtask T309017: Collect stats on stash and cache usage in the page/html and revision/html endpoints as Resolved.May 30 2022, 3:59 PM

DAlangi_WMF changed the status of subtask T267991: Simple per-IP ratelimit for HTML stashing from Open to In Progress.Jun 1 2022, 11:59 AM

DAlangi_WMF changed the status of subtask T309549: Create end-to-end tests for the page/html and revision/html endpoints. from Open to In Progress.

DAlangi_WMF closed subtask T309549: Create end-to-end tests for the page/html and revision/html endpoints. as Resolved.Jun 2 2022, 6:16 PM

Change 802584 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] ParsoidOutputStash: make storage backend configurable.

https://gerrit.wikimedia.org/r/802584

Change 802584 merged by jenkins-bot:

[mediawiki/core@master] ParsoidOutputStash: make storage backend configurable.

https://gerrit.wikimedia.org/r/802584

daniel closed this task as Resolved.Jun 29 2022, 10:54 AM

daniel closed subtask T267991: Simple per-IP ratelimit for HTML stashing as Resolved.