Hey everyone, Nico here (the guy currently improving/rebuilding WineDirect’s Reports).
In line with our team’s renewed commitment towards making our development process more transparent, I’d like to give some background details related to an issue we encountered last week.
We’ve recently completed a major update on our web application framework that caused some outages specifically around the background processes that generate export files. Australia, in particular, was more adversely affected because it has a smaller pool of background processors compared to our US servers.
To help prevent these issues in the future we will be:
Implementing uptime monitoring on all of our background Reports processors (including Australia) - which comes with incident notifications so that our team is notified as soon as a critical service is down or under heavier load.
Installing backup systems that will restart background service runners when they go down.
Looking into adding more resources for our Australian applications, including more focused queues, so that day-to-day Reports operations are not hampered.
TLDR: We upgraded our application engine and that caused a couple of issues around Reports especially in Australia. In response, we’re implementing early-warning systems, uptime monitoring and automatic service restarts.
Nico, perfect timing. I was just about to send a support ticket saying that sometimes reports are very slow or the screen just seems to freeze for a few moments. I’m glad to hear you are aware and working on the issue. I wish there was some indication when something is being processed on reports so I would know to just wait instead of thinking that the system is frozen.
Sorry to hear about that @vicky ! Do you mind giving me the following details too to help narrow down bottlenecks in our system (Feel free to reach out to me at nico.suria@winedirect.com).
If the problematic report is saved, a link to it.
Otherwise, can I get:
the type of report
the reporting period
the filters and columns applied
What time of the day do you guys usually generate this export?
We are having trouble with our end of day report in Financials. We generate it, and the spinning wheel just goes around and around. During the week it gets generated at about 5-5.30pm, but on weekends around 4pm. Weekend staff are not having the problem so much.
Hi, We are having the same problem in Australia - it seems the end of day reports don’t like running at the end of the day only first thing in the morning. Staff’s patients wearing very thin. Please help!
