Johnuniq

mw.loadData tests

The following compares the use of mw.loadData with the use of Lua's require in order to read data from a large module. The Lua time usage values are from the NewPP reports in the HTML source.

Using convert

• Template:Convert • Template which invokes Module:Convert which calls mw.loadData("Module:Convertdata") to read the large Module:Convertdata.
• Template:Convertrequire • Template which invokes Module:Convert which calls require("Module:Convertdata").
• User:Johnuniq/Stress using loadData • Call Template:Convert 1689 times; Lua time usage 2.717 s (1.6 ms/call).
• User:Johnuniq/Stress using require • Call Template:Convertrequire 1689 times (identical, except require is used); Lua time usage 6.139 s (3.6 ms/call).

mw.loadData/require = 2.717/6.139 = 44%.

The following does the same, but only calls the template 10 times.

• User:Johnuniq/Moderate using loadData • Lua time usage 0.051 s (5.1 ms/call).
• User:Johnuniq/Moderate using require • Lua time usage 0.033 s (3.3 ms/call).

mw.loadData/require = 0.051/0.033 = 155%. The overhead of mw.loadData apparently means that it is slower when only a small number of calls are used.

The following does the same, but only calls the template twice.

• User:Johnuniq/Small using loadData • Lua time usage 0.027 s (13.5 ms/call).
• User:Johnuniq/Small using require • Lua time usage 0.019 s (9.5 ms/call).

mw.loadData/require = 0.027/0.019 = 142%. Johnuniq (talk) 09:28, 5 March 2013 (UTC)

Using t-simple

This uses parts of a template and module from Wiktionary (see #Big tables below).

• Module:Languages • Large table copied from wikt:Module:languages; see wikt:Module talk:languages
• Template:Johnuniq/t-simple • Template invoking Module:Johnuniq/t-simple using require.
• User:Johnuniq/water loadData • Call template 400 times using mw.loadData: Lua time usage 0.636 s (1.6 ms/call).
• User:Johnuniq/water • Call template 400 times using require: Lua time usage 7.968 s (19.9 ms/call).

mw.loadData/require = 636/7968 = 8.0% Johnuniq (talk) 09:28, 5 March 2013 (UTC)

Reminder to self

• Module talk:Convert#Overview links to convert template/module pages
• Template:Johnuniq/Investigate investigate intermittent problem using mw
• Module:Johnuniq/Investigate
• User:Johnuniq/test
• Template:Johnuniq/Testargs test how arguments are passed
• Module:Johnuniq/Testargs
• User:Johnuniq/Testargs
• Template:Johnuniq/Testnum check number precision on server
• Module:Johnuniq/Testnum
• User:Johnuniq/Numbers
• User:Johnuniq/sandbox investigating mw.text.tag() failure due to a Scribunto change
• Template:Johnuniq/Simple test contribution of system overhead to Lua time usage using a very simple module
• Module:Johnuniq/Simple
• User:Johnuniq/Simple

Big tables

The following tests a module that defines a large table of data, and compares the Lua time usage when using an alternative implementation that holds all the data packed into a single string. The data is unpacked on demand using a metatable.

• Module:Languages copy of wikt:Module:languages; see wikt:Module talk:languages
• Module:Languagesx emulate Module:Languages with all data packed in a single string
• Template:Johnuniq/t-simple template invoking Module:Johnuniq/t-simple
• Template:Johnuniq/t-simplex template invoking Module:Johnuniq/t-simplex
• Module:Johnuniq/t-simple module requiring Module:Languages
• Module:Johnuniq/t-simplex module requiring Module:Languagesx
• User:Johnuniq/water emulation of some of wikt:water#Translations using t-simple
• User:Johnuniq/waterx as above using t-simplex

The NewPP report in the HTML source for User:Johnuniq/water now shows a "Lua Profile" suggesting that Tim may be implementing per-page caching so a large table is created only once per page. However, currently the module which unpacks the data on demand runs 20 times faster and uses much less memory (if I'm interpreting the situation correctly). Using 400 template instances, information from the NewPP report in the HTML source is:

• User:Johnuniq/water Lua time usage: 8.570s, Lua memory usage: 16.43 MB
• User:Johnuniq/waterx Lua time usage: 0.424s, Lua memory usage: 2.17 MB

Johnuniq (talk) 09:22, 22 February 2013 (UTC)

If Tim is already working on a better solution, wouldn't that make this redundant? CodeCat (talk) 14:32, 5 March 2013 (UTC)

The section here that I was thinking you might want to see is at #Using t-simple above. The "t-simplex" stuff is the ugly trick I mentioned a couple of weeks ago; mw.loadData is the new thing. Johnuniq (talk) 20:13, 5 March 2013 (UTC)

interesting, skip defaults

This is great work, and I hope you're able to generalize it. Could you save memory and time with the notion of default values? You'd store zzj|Zuojiang Zhuang!|||qfa-und| and unpack_item would detect the empty strings for type and script and set them to defaults of "regular" and {"None"}. -- S Page (WMF) (talk) 23:18, 22 February 2013 (UTC)

Yes, good idea and easy. In fact I would have to do something like that for what I'm really working on, which is Module:Convert that requires Module:Convertdata. However, Brad has announced (see wikitech-l link below) that a new mw.loadData() will be available, which means the unpacking approach is probably not useful. I will experiment with that (I just tried loadData here but it is not yet deployed), and will see if there is any significant benefit from combining loadData and packed data. Johnuniq (talk) 23:26, 22 February 2013 (UTC)

mw.loadData

A new mw.loadData() function provides read-only access to a table of data.

• mw:Extension:Scribunto/Lua reference manual#mw.loadData docs for new mw.loadData()
• wikitech-l announcement

I will try this soon. Johnuniq (talk) 22:01, 22 February 2013 (UTC)

Do you know when this will be available on en.Wiktionary? Is there a way we can check, other than to write code that uses it and see if it gives an error? CodeCat (talk) 22:27, 5 March 2013 (UTC)

Using Special:Version here shows "MediaWiki 1.21wmf11" near the top. It's the "wmf11" that implements mw.loadData (it's currently "wmf10" at wikt:Special:Version). I think it will be deployed very soon (should be available in 12 or 24 hours). Johnuniq (talk) 07:35, 6 March 2013 (UTC)

Results

The following are from the HTML source of the indicated pages.

• User:Johnuniq/water standard require() of a large table

<!--
NewPP limit report
Preprocessor visited node count: 2403/1000000
Preprocessor generated node count: 11261/1500000
Post-expand include size: 17459/2048000 bytes
Template argument size: 0/2048000 bytes
Highest expansion depth: 2/40
Expensive parser function count: 0/500
Lua time usage: 8.570s
Lua memory usage: 16.43 MB
Lua Profile:
    init <Module:Languages>                            8020 ms  93.5%
    recursiveClone <mw.lua:103>                         200 ms   2.3%
    Scribunto_LuaSandboxCallback::getExpandedArgument   180 ms   2.1%
    newFrame <mw.lua:228>                                60 ms   0.7%
    (for generator)                                      20 ms   0.2%
    [others]                                            100 ms   1.2%
-->
<!-- Served by mw1172 in 0.211 secs. -->

• User:Johnuniq/waterx standard require(), but data packed in a string

<!--
NewPP limit report
Preprocessor visited node count: 2403/1000000
Preprocessor generated node count: 11261/1500000
Post-expand include size: 17459/2048000 bytes
Template argument size: 0/2048000 bytes
Highest expansion depth: 2/40
Expensive parser function count: 0/500
Lua time usage: 0.424s
Lua memory usage: 2.17 MB
-->
<!-- Served by mw1095 in 0.270 secs. -->

Johnuniq (talk) 22:01, 22 February 2013 (UTC)

Hello, World

I am just stumbling around, having started to learn Lua a few days ago. I will notice activity here for a while, but am more reliably found at w:en:User talk:Johnuniq. Johnuniq (talk) 21:47, 4 September 2012 (UTC)

Module:Convert

You had separate tables for mass, length & temperature in Module:Convert. I've now merged them info one table. -- WOSlinker (talk) 22:11, 5 September 2012 (UTC)

---

Good result. My original thought (having separate tables) was that some tables would have knowledge (code and magic numbers) that related only to the type of unit held in the particular table. However, if everything goes into one table, so much the better.

I guess you know that instead of:

    utype = 1,
    scale = 2,
    offset = 3,
    defaultunit = 4,
    ['kg'] =  {'mass',        1,          0,      'lb'},

the table could have (using a shorter name):

    ['kg'] =  {utype='mass',  scale=1,  offset=0,  udefault='lb'},

That would make the table ugly to read/edit, but if I proceed with my plan to write a script that generates the table from a human-friendly text file, it might be ok. That can be thought about later.

I've been wondering about a couple of issues, and there should be a place to discuss stuff (Template talk:Convert?). One query concerns how everything will be moved to enwiki (when Scribunto is available there and the module is ready). When copied to enwiki, it won't have any history of edits to the module, or any discussions (if we have some discussions—I imagine everyone wants to spend a lot more time coding than talking). Johnuniq (talk) 01:21, 6 September 2012 (UTC)

There is a Special:Import on en.wiki but the choices are currently just to import from meta, de, es, fr, it & pl. I'm guessing that test2 could be added and that import used. -- WOSlinker (talk) 06:30, 6 September 2012 (UTC)

Transwiki import

Select a wiki and page title to import. Revision dates and editors' names will be preserved. All transwiki import actions are logged at the import log.

parms.in_unit_table[units.defaultunit] is 5 table lookups whereas parms.in_unit_table.defaultunit is 3. Uncle G (talk) 13:58, 8 September 2012 (UTC)

SI prefixes

The conversion factor calculations are off. But it's only intended to be a pointer for you to work with. Uncle G (talk) 13:58, 8 September 2012 (UTC)

units table

Hi, just wondering if you could add a "link" option which would be the aricle for that unit to the units table so that in the future when the |lk=on paramter is added back into convert it can use the data from the units table. -- WOSlinker (talk) 11:37, 16 September 2012 (UTC)

e.g.

    ["foot"] = {
	name1    = "foot",
	name1_us = "foot",
	name2    = "feet",
	name2_us = "feet",
	symbol   = "ft",
	sym_us   = "ft",
	utype    = "length",
	scale    = 0.3048,
	offset   = 0,
	default  = "m",
        link     = "Foot (unit)",
    },

I've added the links to the page at en:User:Johnuniq/Conversion data but haven't updated your script at en:User:Johnuniq/Making the units table to output them. -- WOSlinker (talk) 12:13, 16 September 2012 (UTC)

Good idea (I wonder what other complications are needed). I'll fix the script and insert the results soonish. Johnuniq (talk) 03:02, 17 September 2012 (UTC)

I've done that. Module:Convert now has a lot more functionality, although there are lots of problems (see comment at top of Module:Convert). Johnuniq (talk) 10:46, 17 September 2012 (UTC)