Page MenuHomePhabricator

Allow accessing Wikibase entities from multiple (Wikibase) databases
Closed, ResolvedPublic

Description

Following the discussion in T211800, assuming for now a single entity type can only be provided by single Wikibase instance, make it possible for wikibase to access entities provided by multiple sources.
For now, only direct database database access is considered.

The general idea would be to define "sources" that are providing entities of particular type. Services providing access to entity data will be defined per source (i,e. a particular service is configured to access the database of given source/instance/repository). Wikibase will determine which source-specific service to use to access data of particular entity based on the entity-type vs source configuration.

Entity ID prefixes are not considered for now, as long as there is no technical need (state January 2019) to have a single entity type provided by multiple Wikibase instances/sources./databases.


Update January 2020:

The implementation of "entity-source-based federation" is almost completed (remaining changes have related patches on gerrit) Wikibase Repo "component" (enabled instead of "repository-prefix-based" one using configuration setting), and also close to complete in the Wikibase Client "component".
After finishing the implementation, the remaining step to complete this work would be to remove the "use entity source based federation" setting and disable the possibility to use "repository-prefix-based federation". Consequently all code paths related to "repository-prefix-based federation" would be dropped.

The known remaining steps are listed below (list not complete as of 2020-01-15, being continuously updated) - note the list is broken in the ordered "steps", order of items in the "step" might be arbitrary

...

A follow up to this task removing all of the stuff that this task replaces can be found at T241975: entitysources: Remove old MultiRepository & PerRepository Service containers and config

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+5 -0
operations/mediawiki-configmaster+8 -0
operations/mediawiki-configmaster+24 -0
mediawiki/extensions/Wikibasemaster+14 -136
mediawiki/extensions/Wikibasemaster+197 -0
mediawiki/extensions/Wikibasemaster+10 -436
mediawiki/extensions/Wikibasemaster+0 -37
mediawiki/extensions/Wikibasemaster+5 -291
mediawiki/extensions/Wikibasemaster+26 -12
mediawiki/extensions/Wikibasemaster+55 -90
mediawiki/extensions/Wikibasemaster+330 -14
mediawiki/extensions/Wikibasemaster+42 -0
mediawiki/extensions/Wikibasemaster+28 -10
mediawiki/extensions/Wikibasemaster+1 -1
mediawiki/extensions/Wikibasemaster+68 -4
mediawiki/extensions/Wikibasewmf/1.35.0-wmf.16+6 -0
mediawiki/extensions/Wikibasemaster+26 -4
mediawiki/extensions/Wikibasemaster+6 -0
mediawiki/extensions/Wikibasemaster+118 -46
mediawiki/extensions/Wikibasemaster+4 -0
mediawiki/extensions/Wikibasemaster+22 -4
mediawiki/extensions/Wikibasemaster+91 -1
mediawiki/extensions/Wikibasemaster+12 -7
mediawiki/extensions/Wikibasemaster+66 -0
mediawiki/extensions/Wikibasemaster+747 -183
mediawiki/extensions/WikibaseQualityConstraintsmaster+3 -26
mediawiki/extensions/WikibaseQualityConstraintsmaster+24 -0
mediawiki/extensions/WikibaseLexememaster+41 -7
mediawiki/extensions/Wikibasemaster+45 -20
mediawiki/extensions/Wikibasemaster+141 -51
mediawiki/extensions/WikibaseQualityConstraintsmaster+1 -3
mediawiki/extensions/Wikibasemaster+30 -9
mediawiki/extensions/Wikibasemaster+149 -56
mediawiki/extensions/WikibaseQualityConstraintsmaster+2 -0
mediawiki/extensions/Wikibasemaster+98 -0
mediawiki/extensions/WikibaseQualityConstraintsmaster+175 -57
mediawiki/extensions/Wikibasemaster+629 -151
mediawiki/extensions/WikibaseQualityConstraintsmaster+26 -1
mediawiki/extensions/Wikibasemaster+316 -36
mediawiki/extensions/WikibaseMediaInfomaster+9 -5
mediawiki/extensions/WikibaseQualityConstraintsmaster+6 -20
mediawiki/extensions/Wikibasemaster+3 K -4
mediawiki/extensions/Wikibasemaster+526 -59
mediawiki/extensions/WikibaseQualityConstraintsmaster+22 -0
mediawiki/extensions/Wikibasemaster+10 -0
mediawiki/extensions/Wikibasemaster+413 -3
mediawiki/extensions/Wikibasemaster+1 -1
operations/mediawiki-configmaster+6 -0
operations/mediawiki-configmaster+5 -0
mediawiki/extensions/Wikibasemaster+123 -7
mediawiki/extensions/Wikibasemaster+680 -14
operations/mediawiki-configmaster+3 -0
mediawiki/extensions/Wikibasemaster+2 -2
mediawiki/extensions/Wikibasemaster+129 -2
operations/mediawiki-configmaster+8 -0
mediawiki/extensions/Wikibasemaster+15 -3
mediawiki/extensions/Wikibasemaster+15 -5
mediawiki/extensions/Wikibasemaster+108 -39
mediawiki/extensions/Wikibasemaster+124 -3
mediawiki/extensions/Wikibasemaster+196 -14
mediawiki/extensions/Wikibasemaster+136 -57
mediawiki/extensions/Wikibasemaster+46 -1
mediawiki/extensions/Wikibasemaster+400 -48
mediawiki/extensions/Wikibasemaster+55 -2
mediawiki/extensions/Wikibasemaster+85 -1
mediawiki/extensions/Wikibasemaster+350 -22
mediawiki/extensions/Wikibasemaster+71 -0
mediawiki/extensions/WikibaseMediaInfomaster+12 -2
mediawiki/extensions/Wikibasemaster+419 -19
mediawiki/extensions/Wikibasemaster+283 -1
mediawiki/extensions/Wikibasemaster+90 -2
mediawiki/extensions/WikibaseMediaInfomaster+2 -0
mediawiki/extensions/Wikibasemaster+103 -0
mediawiki/extensions/Wikibasemaster+384 -1
mediawiki/extensions/Wikibasemaster+876 -24
mediawiki/extensions/Wikibasemaster+86 -3
mediawiki/extensions/Wikibasemaster+218 -6
mediawiki/extensions/Wikibasemaster+99 -0
mediawiki/extensions/Wikibasemaster+36 -13
mediawiki/extensions/Wikibasemaster+151 -0
mediawiki/extensions/Wikibasemaster+29 -0
mediawiki/extensions/Wikibasemaster+88 -0
mediawiki/extensions/Wikibasemaster+165 -1
mediawiki/extensions/Wikibasemaster+268 -0
mediawiki/extensions/Wikibasemaster+216 -0
mediawiki/extensions/Wikibasemaster+364 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
DuplicateNone
OpenFeatureNone
OpenFeatureNone
DuplicateNone
ResolvedNone
ResolvedNone
ResolvedNone
Resolved Ramsey-WMF
ResolvedCparle
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedNone
InvalidNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedNone
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
ResolvedWMDE-leszek
DuplicateNone
ResolvedAddshore
ResolvedAddshore
ResolvedAddshore

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@WMDE-leszek which bits of this should the campsite be looking at? / what is ready for review vs doing?

Sorry for the late reply. It took me a bit to review the current state of thing there. I've updated the task description with what I believe is the current state, and tried to outline the remaining steps.
It might make sense to break down certain "steps" from this already long task for better prioritization etc.

Change 494051 had a related patch set uploaded (by Addshore; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Change 490145 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Moved out logic on creating entity source definitions from backwards compatible settings out of WikibaseRepo/Client

https://gerrit.wikimedia.org/r/490145

Change 494051 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getLocalEntityTypes

https://gerrit.wikimedia.org/r/494051

Change 494052 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo::newValidatorBuilders: do not rely on RepositoryDefinitions when the "entity source based federation" is in use.

https://gerrit.wikimedia.org/r/494052

Change 494650 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Constructing GenericServices: use entity source defintions when applicable

https://gerrit.wikimedia.org/r/494650

Change 494651 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use (conditionally) EntitySourceDefinitions in WikibaseRepo::getEnabledEntityTypes

https://gerrit.wikimedia.org/r/494651

Change 564976 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] WikibaseRepo: (conditionally) use entity source definitions instead of conceptBaseUri setting

https://gerrit.wikimedia.org/r/564976

Change 533202 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Prefix the value namespace in the RDF output

https://gerrit.wikimedia.org/r/533202

Change 494055 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] WIP Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

Change 566720 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566721 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Change 566732 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566720 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566720

Change 566732 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Test EntitySourceDefitions::getEntityTypeToSourceMapping with subs

https://gerrit.wikimedia.org/r/566732

Change 566721 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.16] EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities

https://gerrit.wikimedia.org/r/566721

Mentioned in SAL (#wikimedia-operations) [2020-01-23T16:02:40Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/extensions/Wikibase/data-access/src/EntitySourceDefinitions.php: [[gerrit:566721|EntitySourceDefitions::getEntityTypeToSourceMapping fix for sub entities (T242415 T214557)]] (duration: 01m 08s)

Change 494055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Made WikibaseClient::getRepoItemUriParser use data from entity source definitions, when "entity source based federation" in use

https://gerrit.wikimedia.org/r/494055

Change 569255 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569325 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569326 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

Change 569327 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569328 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569329 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndix: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569330 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330

Change 569331 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 569332 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 569329 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] rebuildTermSqlIndex: Adjusted constructor to use right "entity source"

https://gerrit.wikimedia.org/r/569329

Change 569325 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted RepoLinker to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569325

Change 569332 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Make it impossible to write to non local database in PropertyInfoTable

https://gerrit.wikimedia.org/r/569332

Change 570879 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Entity source based federation (implemented in this ticket) is now deployed and enabled everywhere.
Now just to remove the old stuff.

Change 570879 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Do not prefix IDs in HtmlPageLinkRendererBeginHookHandler when not needed

https://gerrit.wikimedia.org/r/570879

Change 569328 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Reorganized slightly ValidatorBuildersTest

https://gerrit.wikimedia.org/r/569328

Change 569326 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Adjusted SearchEntities API class to use concept uri data from entity sources

https://gerrit.wikimedia.org/r/569326

Change 569255 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed useEntitySourceBasedFederation config option

https://gerrit.wikimedia.org/r/569255

Change 569331 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in DataAccessSnakFormatterOutputFormatTest

https://gerrit.wikimedia.org/r/569331

Change 573607 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 573607 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in TermStoreWriter tests

https://gerrit.wikimedia.org/r/573607

Change 569327 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Added RdfDumpGenerator test cases for entity source based access mode

https://gerrit.wikimedia.org/r/569327

Change 569330 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Removed now-redundant tests cases in RepoHooksTest

https://gerrit.wikimedia.org/r/569330

Change 490108 abandoned by Addshore:

[operations/mediawiki-config@master] DNM Define Wikibase "entity sources" on beta commons

Reason:

Looks done to me somewhere else https://codesearch.wmcloud.org/operations/?q=wmgWikibaseRepoLocalEntitySourceName&i=nope&files=&excludeFiles=&repos=

https://gerrit.wikimedia.org/r/490108

Change 490104 abandoned by Addshore:

[operations/mediawiki-config@master] Added wmgWikibaseEntitySources setting for defining Wikibase "entity sources"

Reason:

Also looks done to me https://codesearch.wmcloud.org/operations/?q=wmgWikibaseEntitySources&i=nope&files=&excludeFiles=&repos=

https://gerrit.wikimedia.org/r/490104

Change 490633 abandoned by Addshore:

[operations/mediawiki-config@master] Added wmgWikibaseRepoLocalEntitySourceName to define the "local" source of Wikibase Repo

Reason:

Also looks done to me https://codesearch.wmcloud.org/operations/?q=wmgWikibaseRepoLocalEntitySourceName&i=nope&files=&excludeFiles=&repos=

https://gerrit.wikimedia.org/r/490633

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy