Kanashimi

Bot problems

As explained in User:ASarabadani (WMF)/Growth of databases of Wikidata, the database is growing too quickly. In particular, the number of previous revisions of items, and the number of terms (labels/descriptions/aliases).

Your bot is currently making far too many edits per item. For example, on Special:History/Q33757339, it made 134 edits in just 2 minutes. You should group those into fewer edits, ideally a single edit.

Your bot is also adding redundant descriptions. Wikidata uses language fallback, which means we don't need to copy descriptions to other languages/variants if one of the fallback languages has the same description. For example, in Special:Diff/2243197903:

• sr and sr-ec are the same. Ideally we would use sr-cyrl but that isn't available yet, so I suggest only adding sr.
• zh-hans, zh-cn, zh-my, zh-sg and wuu are the same. You should only add zh-hans.
• zh-hant, zh-hk, zh-mo, zh-tw and zh are the same. You should only add zh-hant.
• pt and pt-br are the same. You should only add pt.
• en-gb is a slight variation of en. The en description is fine in en-gb (we never use the short mm/dd/yyyy form, but "d month year" and "month d, year" are used interchangeably), so you should only add en.

It would be helpful if you could also remove the redundant descriptions when they are the same (or equivalent, for things like en-gb).

Please stop the bot until you have made these changes.

- Nikki (talk) 09:29, 4 September 2024 (UTC)

I've paused the bot. Need time to modify the program code. Kanashimi (talk) 23:21, 4 September 2024 (UTC)

@Nikki Could we add the ability to add and remove qualifiers and references, with clear examples of how to use them at https://www.wikidata.org/w/api.php?action=help&modules=wbeditentity ? I find it very difficult to just use wbeditentity for these functions. Since there are no good methods, I have a feeling it might take a lot of time to rewrite the code... If wbeditentity's API was a bit smarter, I think API users would have much less work to do. Otherwise I might probably still have to use individual functions, as mentioned by https://lists.wikimedia.org/hyperkitty/list/wikibaseug@lists.wikimedia.org/thread/KDG2LDMZSGUZ5OW34LDSWAUOV4JWVRVS/ ... Kanashimi (talk) 08:14, 7 September 2024 (UTC)

@Nikki I've been trying for a few days but still can't figure out the data format that passes the API. Please let me know the data format that can pass the API and I will follow your instructions to modify the program. Kanashimi (talk) 07:11, 11 September 2024 (UTC)

Here is a small hint: You can add a statement with qualifiers and references to a Wikidata:Sandbox item and trace the call to the API in the developer tools of your browser. This should give you the right format. You can also look at the json entity of any item to see how qualifiers and references are structured in the json. The data variable of wbeditentity takes that identical json as a parameter. If you need further help, you could also post what you tried so far. --Sotho Tal Ker (talk) 16:30, 13 September 2024 (UTC)

@Sotho Tal Ker Thank you for your concern. For example, for Q73503190, I edited it this way:

{"labels":{"en":{"language":"en","value":"Toxocara canis infection and granulomatous hepatitis"}},"claims":{"P2093":[{"snaktype":"value","property":"P2093","datavalue":{"value":"S. P. Kaushik","type":"string"},"datatype":"string","exists_index":4,"qualifiers":
::::{"P1545":[{"snaktype":"value","property":"P1545","datavalue":{"value":"1","type":"string"},"datatype":"string"}]},"references":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":5412157,"id":"Q5412157"},"type":"wikibase
::::-entityid"},"datatype":"wikibase-item"}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"9219807","type":"string"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","datavalue":{"value":"https://www.ebi.ac.uk/europepmc/webservic
::::es/rest/search?resulttype=core&format=json&query=SRC%3AMED%20AND%20EXT_ID%3A9219807","type":"string"},"datatype":"url"}],"P813":[{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"c
::::alendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time"}]}},{"snaktype":"value","property":"P2093","datavalue":{"value":"M. Hurwitz","type":"string"},"datatype":"string","exists_index":5,"qualifiers":{"P1545":[{"snaktype":"value","proper
::::ty":"P1545","datavalue":{"value":"2","type":"string"},"datatype":"string"}]},"references":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":5412157,"id":"Q5412157"},"type":"wikibase-entityid"},"datatype":"wikibase-item
::::"}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"9219807","type":"string"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","datavalue":{"value":"https://www.ebi.ac.uk/europepmc/webservices/rest/search?resulttype=core&format
::::=json&query=SRC%3AMED%20AND%20EXT_ID%3A9219807","type":"string"},"datatype":"url"}],"P813":[{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http://www.wikidata.or
::::g/entity/Q1985727"},"type":"time"},"datatype":"time"}]}},{"snaktype":"value","property":"P2093","datavalue":{"value":"C. McDonald","type":"string"},"datatype":"string","exists_index":6,"qualifiers":{"P1545":[{"snaktype":"value","property":"P1545","datavalue":{"value":"3"
::::,"type":"string"},"datatype":"string"}]},"references":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":5412157,"id":"Q5412157"},"type":"wikibase-entityid"},"datatype":"wikibase-item"}],"P698":[{"snaktype":"value","pro
::::perty":"P698","datavalue":{"value":"9219807","type":"string"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","datavalue":{"value":"https://www.ebi.ac.uk/europepmc/webservices/rest/search?resulttype=core&format=json&query=SRC%3AMED%20AND%20EXT_ID
::::%3A9219807","type":"string"},"datatype":"url"}],"P813":[{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"
::::datatype":"time"}]}},{"snaktype":"value","property":"P2093","datavalue":{"value":"P. Pavli","type":"string"},"datatype":"string","exists_index":7,"qualifiers":{"P1545":[{"snaktype":"value","property":"P1545","datavalue":{"value":"4","type":"string"},"datatype":"string"}]
::::},"references":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":5412157,"id":"Q5412157"},"type":"wikibase-entityid"},"datatype":"wikibase-item"}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"921
::::9807","type":"string"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","datavalue":{"value":"https://www.ebi.ac.uk/europepmc/webservices/rest/search?resulttype=core&format=json&query=SRC%3AMED%20AND%20EXT_ID%3A9219807","type":"string"},"datatype"
:::::"url"}],"P813":[{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time"}]}},{"property":"P2093
::::","value":"Kaushik SP","remove":0},{"property":"P2093","value":"Hurwitz M","remove":1},{"property":"P2093","value":"McDonald C","remove":2},{"property":"P2093","value":"Pavli P","remove":3}],"P577":[{"snaktype":"value","property":"P577","datavalue":{"value":{"time":"+199
::::7-07-01T00:00:00Z","timezone":0,"before":0,"after":0,"precision":10,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time","exists_index":0,"references":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type
::::":"item","numeric-id":82494,"id":"Q82494"},"type":"wikibase-entityid"},"datatype":"wikibase-item"}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"9219807","type":"string"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","dat
::::avalue":{"value":"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&retmode=json&id=9219807","type":"string"},"datatype":"url"}],"P813":[{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0
::::,"after":0,"precision":11,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time"}]}}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"9219807","type":"string"},"datatype":"external-id","exists_index":0,"references"
:::::{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":82494,"id":"Q82494"},"type":"wikibase-entityid"},"datatype":"wikibase-item"}],"P698":[{"snaktype":"value","property":"P698","datavalue":{"value":"9219807","type":"stri
::::ng"},"datatype":"external-id"}],"P854":[{"snaktype":"value","property":"P854","datavalue":{"value":"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&retmode=json&id=9219807","type":"string"},"datatype":"url"}],"P813":[{"snaktype":"value","property":"
::::P813","datavalue":{"value":{"time":"+2024-09-14T00:00:00Z","timezone":0,"before":0,"after":0,"precision":11,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time"}]}}]}}

and got error: [invalid-claim] [wikibase-api-invalid-claim]. I think the error message could be more specific so that people know how to modify the data structure. Kanashimi (talk) 23:12, 14 September 2024 (UTC)

You do not have the "mainsnak" anywhere. These are mandatory for adding a claim. Your final output should look something like this (for a single claim):

{"claims":[{"mainsnak":{"snaktype":"value","property":"P56","datavalue":{"value":"ExampleString","type":"string"}},"type":"statement","rank":"normal","qualifiers":{"P1545":[{"snaktype":"value","property":"P1545","datavalue":{"type":"string","value":"1"}}],"P1932":[{"snaktype":"value","property":"P1932","datavalue":{"type":"string","value":"ExampleQualifier"}}]},"qualifiers-order":["P1545","P1932"],"references":[{"snaks-order":["P248","P214","P813"],"snaks":{"P248":[{"snaktype":"value","property":"P248","datavalue":{"value":{"entity-type":"item","numeric-id":"54919"},"type":"wikibase-entityid"}}],"P214":[{"snaktype":"value","property":"P214","datavalue":{"value":"0","type":"string"},"datatype":"external-id"}],"P813":[{"snaktype":"value","property":"P813","datatype":"time","datavalue":{"value":{"after":0,"before":0,"calendarmodel":"http://www.wikidata.org/entity/Q1985727","precision":11,"time":"+2024-09-13T00:00:00Z","timezone":0},"type":"time"}}]}}]}]} Sotho Tal Ker (talk) 23:13, 15 September 2024 (UTC)

@Sotho Tal Ker I tried to reduce the data first, dropping qualifiers, references but still can't get the data structure to pass the API. Still get [invalid-claim] [wikibase-api-invalid-claim].

{"labels":{"en":{"language":"en","value":"Plexiform neurofibromatosis and angiosarcoma of the liver in von Recklinghausen disease"}},"claims":[{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P407","datavalue":{"value":{"entity-type":"item","numeric-id":1860,"id":"Q1860"},"type":"wikibase-entityid"},"datatype":"wikibase-item"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"V. Andreu","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"I. Elizalde","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"C. Mallafré","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"J. Caballería","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"J. M. Salmerón","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"M. Sans","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"A. Mas","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"M. Bruguera","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P2093","datavalue":{"value":"J. Rodś","type":"string"},"datatype":"string"}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P577","datavalue":{"value":{"time":"+1997-07-01T00:00:00Z","timezone":0,"before":0,"after":0,"precision":10,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time","exists_index":0}},{"type":"statement","rank":"normal","mainsnak":{"snaktype":"value","property":"P698","datavalue":{"value":"9219811","type":"string"},"datatype":"external-id","exists_index":0}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Andreu V","remove":0}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Elizalde I","remove":1}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Mallafré C","remove":2}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Caballería J","remove":3}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Salmerón JM","remove":4}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Sans M","remove":5}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Mas A","remove":6}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Bruguera M","remove":7}},{"type":"statement","rank":"normal","mainsnak":{"property":"P2093","value":"Rodś J","remove":8}}]}

Maybe you can help me? Kanashimi (talk) 00:51, 17 September 2024 (UTC)

I suspect you get the error because you try to remove claims without providing the GUID of the claim. To remove a label you do not need to provide a GUID. But if you want to remove a claim you have to provide its GUID as seen in the examples on the wbeditentity API help page. To get the GUID of the claim in question, you need to parse the json entity of the item first. For example, for Q73503190 you can access the json directly via https://www.wikidata.org/wiki/Special:EntityData/Q73503190.json. If you do not want to handle all this by yourself, you can take a look at https://github.com/maxlath/wikibase-sdk for reading data and https://github.com/maxlath/wikibase-edit for writing data. --Sotho Tal Ker (talk) 04:28, 17 September 2024 (UTC)

Fixed --Kanashimi (talk) 09:22, 6 November 2024 (UTC)

While on it, please also prevent your bot from adding adding an author string name (P2093) if that author is already present. Example edits: https://www.wikidata.org/w/index.php?title=Q82053131&diff=2226557155&oldid=2226557145&diffmode=source. Even if the name string in itself is different, the series ordinal (P1545) makes it obvious that they are the same. Thank you. --Sotho Tal Ker (talk) 22:38, 9 September 2024 (UTC)

Fixed Kanashimi (talk) 04:17, 10 September 2024 (UTC)