GWToolset uses GWToolset\Handlers:UploadHandler:getFileExtension() to attempt to determine the media file’s extension. it uses php’s pathinfo and mediawiki’s MimeMagic to determine a valid file extension. if either of those methods fails, the file extension is assumed to be invalid.

sorry, didn’t know what the acronymn meant: Server Admin Log (SAL). thanks for your help.

SAL?

i just checked both beta and production and the whitelist config change appears in both environments. was the merged patch also deployed to production?

In T101513#1353176, @Krenair wrote:

wat? config changes get automatically sent to beta once they're merged in Gerrit, sure, but production requires a deployer to log into a deployment server, pull the change, and push it to the servers. Approving mediawiki-config patches in Gerrit but not deploying them to production can cause issues and make you unpopular with the next person that does want to deploy a config change.

based on past experience, once these config` patches are merged, the changes are available on beta and production in about 15 minutes. there was no need for a SWAT or regular deploy.

@MarkTraceur,
it looks like there was a problem with the original metadata that fæ cleaned-up; see this revision comparison. but i’d double-check with fæ to make sure. not sure how to add @fæ to this task. will @Fae do it?

Would a 14 million entry XML file be too big?

yes, as @Reedy already mentioned. i believe the largest xml file i used during testing was a 10,000 record file that was 58.7mb in size, which is a good size to limit each xml file to - 10,000 records.

@Artturimatias,
might be good to open a new task for the issue. in any case:

• i tested the url by running curl -i http://www.doria.fi/bitstream/10024/92051/1/1201003266_1.jpg and got a response that seems like it should work.
  • is the error message you mention the complete error message?
• can you upload a small sample of the xml file you’re using?
  • and a link to the metadata mapping you created?
• i don’t see any GWToolset uploads with your username in the beta commons Special Log, so i assume that means that you’re getting the error message as you setup the batch job - correct?

i suggest you try uploading a few maps on commons beta ahead of the hackathon so that any issues you might run into could be addressed as soon as possible.

thanks for taking care of this @yuvipanda!

i see. thanks, but how is MW-Vagrant determining where to get the apt-get package for hhvm?

i agree, that ideally this would be fixed in the hhvm package, but it seems that this issue has existed for some time. i checked out a commit hash from 25 september 2014, 7447c9804b947b4b7eba9af1d0dc93828d59bed9, and still saw this issue when creating a new vm.

installed hhvm version: 3.5.0-trusty in an unpuppetized vagrant instance and it installs php.ini in /etc/hhvm/php.ini. it seems that the issue is with the pkg version 3.3.1+dfsg1-1+wm3, which installs php.ini in /etc/hhvm/php.ini/php.ini. i don’t know how or where this needs to be fixed or reported.

this commit based on this patch seems to have created the issue.

@Bawolff,
would you mind setting that cherry picked merge up?

@Tounoki,
you should be able to run a small test on beta first. if your test works on beta, then “in theory” it should work on production once GWToolset 0.3.8 (1fd5c4d) 16:55, 22 January 2015 is deployed there. you can check this page in production and look for GWToolset to see which version is on production.

@Kelson,
once https://gerrit.wikimedia.org/r/#/c/186241/ is deployed to production you can try the upload again. i suggest using a small subset to start and then the rest once you're able to verify that all is well.

fyi: just added @hashar to the gerrit patch. hopefully he or @Reedy can get to it soon.

@Krenair,
thanks for checking it.

is the constant PHP_SAPI defined in production?
when a job is run by the job runner, is that constant set to 'cli'?

@Steinsplitter,

In T86699#975207, @Glaisher wrote:

In T86699#974587, @Krenair wrote:

Does this need to be discussed somewhere, e.g. on commons?

not always.

the fix has been deployed to production. are you okay with closing this task or is there something i may have missed?

thanks hoo,

this seems to be a job runner/job queue issue.

• is there someone on the ops team who could investigate it?
• is there a way to tell when the job runner is having a problem?
• is there a way to see which job's are in the job queue?