I used GRPC to make the import more distributed. Also, switched from API to Wikimedia Commons replica database with tunneling via developer account. Also, the descriptions are now fetching in background and not blocking the import. The import time is now reduced to 1 minute.
After a lot of effort, I managed to
- Create another tool named
campwiz-backendfor backend hosting - Added build service for golang project using
1.24.1version (Thanks to @dcaro and @fnegri) - Access all the configuration file by adding
/data/project/campwiz-backendinto viper path (Thanks to @dcaro) - Get response by binding to
0.0.0.0instead oflocalhost(Thanks to @jeremy_b) My new issue is - My code is built for nodejs version 22, but toolforge supports upto 18. That's why some functions are not available like
toSorted
Last night, I accidentally remove the flag for port, so it was running the default port. But since two servers were running on the same default port, one of them errored. Now nginx was trying to balance the load between them. But since one of them errored, nginx was switching to the read-only mode. Now I added the port flag.
So, David Caro helped me setup the architecture I wanted. What I wanted was to run the golang server behind nginx reverse proxy.Also, another read-only server would be used as failover server. So, if the main server goes down, the nginx will switch to the read-only server. I also started running a GRPC Server on port 50051. The GRPC server would be used for asynchronous tasks like importing images and descriptions. The GRPC server is not behind the nginx reverse proxy. So, it can be accessed directly. Our solution was (actually, david caror's solution):
- Build a image with standard golang buildpack.
- It built two binaries:
campwizandtaskmanager
- It built two binaries:
- Build another image with
heroku-php-nginxbuildpack from the branchfake-nginx- Actually, it just contains a
composer.jsonfile which tricks the build pack into choosing theheroku-php-nginxbuildpack. - Also contains a
nginx.conffile which is used to configure the nginx server. - This
nginx.conffile is given in thewebentry of theProcfile.
- Actually, it just contains a
- Now, when both of the images are built,
- we first run three continuous jobs using the golang built image:
campwizwhich runs the main server on port 8080taskmanagerwhich runs the GRPC server on port 50051campwiz -readonlywhich runs the read-only server on port 8081 that is used for failover
- Then we run the webservice using our hackish
heroku-php-nginximage.- Now the nginx is forwarding the requests to the main server on port 8080 and the read-only server on port 8081. The GRPC server is running on port 50051. But my current issue is that, I see some performance hit after this change. I am not sure if it is because of the nginx reverse proxy or something else.
- we first run three continuous jobs using the golang built image:
- [] How many new articles were created during this campaign
- [] How many articles were expanded during this campaign
- [] What are the points of those articles
- [] How many users created their since the previous month of the campaign
- [] How many articles were created by the users who created their account during the campaign (New Comers articles Newly Created)
- [] How many articles were expanded by the users who created their account during the campaign (New Comers articles Expanded)
- [] What is the retention rate of the new users
- [] How many articles did the user create during the campaign(s)
- [] How many articles did the user expand during the campaign(s)
- [] What are the points of those articles
- [] How many campaigns did the user participate in
- [] If the user is a new user, how many articles did the user create during the campaign(s) which were not submitted in any of the campaign (Offline contributions)
- [] As year basis, what is the growth rate of that user (The increase of the number of articles created by the user in the current year compared to the previous year)