The validator is blocket (over 21k wu pending) even if the server status is green.
Maybe a restart of the service...

Looks like the validator needs another nudge from the admin. 25k in validation queue and rising.

Hello everyone,

I have also noticed the alarming number of WUs still pending validation, and although we don't really know what is happening, and I don't have direct access to the server (only Jesus has access), we believe that it is because of something that systems have put on the server.

I tell you briefly, the other day in a meeting, Jesus told me that for security directives they had put a service that analyzes the incoming and outgoing traffic of the server. And what does this have to do with the problem, because we believe that this process is analysing all the incoming and outgoing files to the server, so it is consuming a lot of available resources and making the validation go to very large extremes (as you can see, now it is almost 33k), making it much more difficult to validate the incoming results from your computers to the server.

A while ago I got in touch with Jesús, although now it's more difficult for him to check the mail, since it's a public holiday in Spain, and therefore it's also more difficult to find a quick solution. I'm going to pass the thread to him urgently so that he can see it.

We will keep you informed.

Best regards,

Iván.

---

Validation backlog is creeping up again, almost 7200 WUs as of 9:57:24 UTC

Hello everyone,

I have also noticed the alarming number of WUs still pending validation, and although we don't really know what is happening, and I don't have direct access to the server (only Jesus has access), we believe that it is because of something that systems have put on the server.

I tell you briefly, the other day in a meeting, Jesus told me that for security directives they had put a service that analyzes the incoming and outgoing traffic of the server. And what does this have to do with the problem, because we believe that this process is analysing all the incoming and outgoing files to the server, so it is consuming a lot of available resources and making the validation go to very large extremes (as you can see, now it is almost 33k), making it much more difficult to validate the incoming results from your computers to the server.

A while ago I got in touch with Jesús, although now it's more difficult for him to check the mail, since it's a public holiday in Spain, and therefore it's also more difficult to find a quick solution. I'm going to pass the thread to him urgently so that he can see it.

We will keep you informed.

Best regards,

Iván.

If you have another Server you can send all the files thru it for checking and then send them to the regular server for validating.

I paused the crunching, not to overload the server.
Hope that validator will restart....

Looks like we are going to run out of work, but since the validator is jammed up as is the transitioner this will give time for the system to unclog and maybe Jesus can figure out the problem.

Tasks ready to send 8
Tasks in progress 81280
Workunits waiting for validation 31230
Transitioner backlog (hours) 25.45

i have couple hundred task "ready to report" with 2.5 days deadline, and few hundred on validation...

---

--
I crunch for Ukraine

Just under 2700 pending validation in my queue

It seems that the behavior has changed as follows starting from April 28, 2025.

https://denis.usj.es/denisathome/forum_thread.php?id=313&postid=3015

---

Well, what certainly keeps changing is the number of WUs waiting to be validated... :(
Was 48,000 a few hours ago, now already up to 57,000.
So far, not a single one of the latest Beta batch has been validated, as if the validators aren't running, despite what the server status page says...

Ralf

Validators are busy with the first 2 batches. They are running but very slowly compared to the massive influx of tasks that we are reporting.

Most WUs of the last batch(2025.05.21 00:25 UTC) haven't even been sent out although one of my rigs doesn't seem to be getting any new tasks. This might suggest that the team actively stopped sending out work which is good. Now the question is whether the server will run out of disk space again risking missed deadlines for tasks still in progress (almost 70k as of this moment).

Edit: I received my last fresh tasks at around 00:40 UTC (May 22nd). Since then all my rigs are getting WCG tasks when needed.

---

Whatever is going on, the server also can not show the tasks for our hosts.


Database Error

Fatal error: Uncaught TypeError: count(): Argument #1 ($value) must be of type Countable|array, null given in /home/boincadm/projects/denisathome/html/user/results.php:86 Stack trace: #0 {main} thrown in /home/boincadm/projects/denisathome/html/user/results.php on line 86

It usually takes a long time to load tasks. I can see tasks for my hosts (a couple hundred) but the server indeed has trouble loading some anonymous cruncher's threadripper with 4k+ tasks. If took about a minute, but it did eventually load.

On the other hand, reporting tasks is taking almost a full minute now. Probably hundreds of rigs shoving thousands of tasks onto the server.

---

Even though the SERVER STATUS shows zero task available, I have no problem getting more tasks, 2 to 3 days worth per host (32 threads, 44 threads, and 256 threads). I doubt that helps any, but the more info the better, perhaps?

I hate to do it, but I'm setting no new tasks, and moving on to a different project for now.

Uploads have started failing now, server finally ran out of disk space.

---

Confirmed, Uploads are stuck. Wonderful.

Let us pray.

"Dear Jesus of DENIS, what's up, dude? Amen"

( I sincerely apologize, no offense meant to anyone. Especially not to the ONE/ The Trinity. )

Seems like they started trickling now. Managed to upload all my backlog. Will continue to crunch what I have left, hopefully we can all upload and report before the first deadlines tomorrow.

---

Things aren't looking too bad right now. Validation queue is flattening while tasks in progress are dropping fast. WUs waiting should start dropping in a few hours when enough rigs run out of work and the validator can clear more WUs than people are reporting.

---

Good Job !

I was alsotrying to create this graph, thinking with my aging brain (I'm over 60yrs) about making a shell script to get the source data, but I got stuck dealing with launchd (macOS).

~~Would you be able to publish it on GitHub?~~

---