I get again
DENIS@home 23.04.2025 15:37:44 CEST [error] Error reported by file upload server: Server is out of disk space

@Jesús: Thanks for your explanation about the expected Outcome of your actual Beta-App testing.
For me it is OK and needed to get a reliable enviroment, ohterwise I would not participate on Test application.

---

Matthias

@Jesus, Isn't there a way to make zip files for the results. I have seen this in the past with other projects with large data sets. But the zip was both ways, for download and upload.
I don't know if this is possible with your technology or if its outdated these days. But it is something I remember.

After last night's major outage, right now I can *slowly* upload all the tasks that have been waiting, but reporting them is still impossible:
24/04/2025 08:33:23 | DENIS@home | Scheduler request to https://denis.usj.es/denisathome_cgi/cgi failed: Error 524

L.E.: managed to report all tasks on 2/4 laptops after several tries
24/04/2025 08:37:22 | DENIS@home | Scheduler request completed

L.E.2: got a couple of new tasks on one laptop, meanwhile the other 2 are still getting error 524 when trying to report. Seems like the server is struggling on all fronts

I managed to upload all my completed tasks, but I cannot report completed tasks.

Thu 24 Apr 2025 01:33:58 AM EDT | DENIS@home | update requested by user
Thu 24 Apr 2025 01:34:00 AM EDT | DENIS@home | Sending scheduler request: Requested by user.
Thu 24 Apr 2025 01:34:00 AM EDT | DENIS@home | Reporting 151 completed tasks
Thu 24 Apr 2025 01:34:00 AM EDT | DENIS@home | Not requesting tasks: "no new tasks" requested via Manager
Thu 24 Apr 2025 01:36:01 AM EDT | DENIS@home | Scheduler request failed: Error 524
Thu 24 Apr 2025 01:41:49 AM EDT | DENIS@home | Sending scheduler request: To report completed tasks.
Thu 24 Apr 2025 01:41:49 AM EDT | DENIS@home | Reporting 151 completed tasks
Thu 24 Apr 2025 01:41:49 AM EDT | DENIS@home | Not requesting tasks: "no new tasks" requested via Manager
Thu 24 Apr 2025 01:42:17 AM EDT | DENIS@home | work fetch resumed by user

---

I'm thinking the server is slowly freeing up space after validating/deleting and then it allows a few more tasks to be uploaded/reported. Since there are hundreds of rigs waiting to upload and report, it takes time for everyone to get through the bottleneck. Keep requesting updates from time to time until you find an opening.

Last server status update is from 23 Apr 2025, 15:20:21 UTC so we can't really tell what's going on.

As of 720 GMT all systems are running normal.
All my backlog of tasks are uploaded.

Tasks ready to send 22004
Tasks in progress 85556
Workunits waiting for validation 11186
Workunits waiting for assimilation 3
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 5.30

Don't forget that Task data as of 23 Apr 2025, 15:20:21 UTC is from 15 hours ago. Connection to the server is still choppy. Sometimes I get the same Error 524, that appeared during unsuccessful reporting attempts, when refreshing my results page.

Task data as of 24 Apr 2025, 7:13:05 UTC

Tasks ready to send 27406
Tasks in progress 66845
Workunits waiting for validation 13059
Workunits waiting for assimilation 1
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 20.62

Validation and transitioner backlogs are worsening, but the larger drop in tasks in progress hopefully suggests that validation is slowly grinding ahead.
My last validated task was reported over 24 hours ago @ 23 Apr 2025, 6:04:03 UTC

I'm thinking the server is slowly freeing up space after validating/deleting and then it allows a few more tasks to be uploaded/reported. Since there are hundreds of rigs waiting to upload and report, it takes time for everyone to get through the bottleneck. Keep requesting updates from time to time until you find an opening.

So, there is a doubt: maybe the hw of the server is undersized??

The whole work process seems to be very cyclical, each cycle taking roughly a week:
1. Large batches of work gets generated
2. Many rigs jump to crunch whatever they can grab
3. Tens of thousands of WUs get crunched in 1-2 days
4. Validator is overloaded once crunching nears its peak
5. Server runs out of disk space
6. Task upload and reporting barely works for 1-2 days
7. Work generation is stopped
8. Validator catches up in 1-2 days (including tasks that get re-sent because of missed deadlines)
9. All WUs get crunched and validated, project runs out of work.
10. Repeat from 1.

Tasks ready to send 22
Tasks in progress 67160
Workunits waiting for validation 7364
Workunits waiting for assimilation 0
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 7.35

We're already ramping up for the next cycle, moving from phase 2 to 3.

Well, the post-blackout recovery was faster than expected, but the usual issue remains:

Work
Tasks ready to send 22
Tasks in progress 39132
Workunits waiting for validation 8853 and climbing
Workunits waiting for assimilation 2
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 33.30

Now that you mention it, that does seem to be the case.

Tasks ready to send 9
Tasks in progress 56337
Workunits waiting for validation 54164
Transitioner backlog (hours) 45.63

computers:
With recent credit 1321
Registered in past 24 hours 87

The whole work process seems to be very cyclical, each cycle taking roughly a week:
1. Large batches of work gets generated
2. Many rigs jump to crunch whatever they can grab
3. Tens of thousands of WUs get crunched in 1-2 days
4. Validator is overloaded once crunching nears its peak
5. Server runs out of disk space
6. Task upload and reporting barely works for 1-2 days
7. Work generation is stopped
8. Validator catches up in 1-2 days (including tasks that get re-sent because of missed deadlines)
9. All WUs get crunched and validated, project runs out of work.
10. Repeat from 1.

Tasks ready to send 22
Tasks in progress 67160
Workunits waiting for validation 7364
Workunits waiting for assimilation 0
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 7.35

We're already ramping up for the next cycle, moving from phase 2 to 3.

WUs waiting for validation never hit such heights as far as I saw since I recently joined the project. Maybe they managed to increase the server's capacity? I'm grabbing and crunching whatever I can. There seem to be a lot of tasks generated on the 28th that may have been aborted by others.

The whole work process seems to be very cyclical, each cycle taking roughly a week:
1. Large batches of work gets generated
2. Many rigs jump to crunch whatever they can grab
3. Tens of thousands of WUs get crunched in 1-2 days
4. Validator is overloaded once crunching nears its peak
5. Server runs out of disk space
6. Task upload and reporting barely works for 1-2 days
7. Work generation is stopped
8. Validator catches up in 1-2 days (including tasks that get re-sent because of missed deadlines)
9. All WUs get crunched and validated, project runs out of work.
10. Repeat from 1.

A clear and interesting analysis

[error] Error reported by file upload server: Server is out of disk space
since about 13:20 UTC

---

Paul.

Yep, seems like I was too optimistic. Work is still being sent out, tasks generated on the 27th are hitting the deadline and getting recycled. Going to be a massive validation backlog for the weekend.

Backlog still growing.

Tasks ready to send	109
Tasks in progress	69921
Workunits waiting for validation	58760
Workunits waiting for assimilation	3
Workunits waiting for file deletion	0
Tasks waiting for file deletion	0
Transitioner backlog (hours)	49.56

---

Paul.

Uploads & reporting now OK but server backlog still growing.

---

Paul.

It seems that the behavior has changed as follows starting from April 28, 2025.

The whole work process seems to be very cyclical, each cycle taking roughly a week:
1. Large batches of work gets generated
2. Many rigs jump to crunch whatever they can grab
3. Tens of thousands of WUs get crunched in 1-2 days
4. Validator is overloaded once crunching nears its peak
5. ~~Server runs out of disk space~~
6. ~~Task upload and reporting barely works for 1-2 days~~
7. ~~Work generation is stopped~~
8. Validator **overflow** in 1-2 days (including tasks that get re-sent because of missed deadlines)
9. **In every work unit, processing is completed and reported, but all verifications time out and become invalid, causing distribution to resume. As a result, the project's work never runs out.**
10. Repeat from 1.

---

The project does run out of work eventually if they stop generating new tasks. Right now I haven't gotten any tasks on any of my 4 laptops for about 2 hours (got 1 running and almost 800 in PV jail).

Task data as of 1 May 2025, 3:27:11 UTC

Tasks ready to send 0 - seems to be accurate
Tasks in progress 48643 - might be some missed deadlines coming up but I think most of these tasks have been sent out yesterday so the deadline is about 2 days from now
Workunits waiting for validation 72892 - gonna take a while
Workunits waiting for assimilation 1
Workunits waiting for file deletion 0
Tasks waiting for file deletion 0
Transitioner backlog (hours) 0.00 - caught up overnight

---

Task data as of 1 May 2025, 4:27:46 UTC

Tasks in progress 46719 - 1924 tasks seem to have been reported
Workunits waiting for validation 73133 - only 241 new WUs