𝕏

Status of the project // Estado del proyecto

Message boards : News : Status of the project // Estado del proyecto
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2056 - Posted: 11 Apr 2023, 10:01:16 UTC

Dear volunteers,
We update you on the status of the project and how the simulations are going. In recent weeks the search algorithm has been improving the model very little. This indicates us that the optimization is close to be finished. Once this is complete we will pass to the validation phase which we will be done outside of DENIS (DENIS is not yet prepare for the required simulations).

The improvement obtained is very considerable (you can see the graph with the evolution of the error below and we hope that the validation results turn out well). It can be improved, but not all the error can be removed with the optimization (it requires other type of improvements in the model).

On the other hand, looking at the results, last week we started a new optimization by changing an aspect of it. In the initial search, if one of the markers was in the experimentally observed range, we did not allow the algorithm to search where that marker was outside of the experimental range (criterion of keeping at least those markers that were already in the experimental range). In the new search we are allowing it in the hope that even if some marker goes out, the overall behavior will improve. In this case, the uncertainty about the result is higher, but as you can see in the graph below, after the first iterations of the algorithm, the results are quite promising.

Sincerely,
Jesus.





========================================


Estimados voluntarios:
Os actualizamos el estado del proyecto y cómo van las simulaciones. En las últimas semanas el algoritmo de búsqueda ha ido mejorando muy poco el modelo. Esto nos indica que está a punto de terminarse la optimización. Una vez que esto se complete, pasaremos a una fase de validación de la simulación que haremos fuera de DENIS (DENIS no está preparado para ello).

La mejoría obtenida es muy considerable y esperamos que los resultados de la validación salgan bien (podéis ver la gráfica con la evolución del error debajo). Todavía hay margen de mejora, pero no todo el error se puede eliminar en la optimización (requiere otro tipo de mejoras en el modelo).

Por otro lado, viendo los resultados, la semana pasada comenzamos una nueva optimización cambiando un aspecto de la misma. En la búsqueda inicial, si uno de los marcadores estaba en el rango observado experimentalmente, no permitíamos al algoritmo buscar donde ese marcador saliera del rango experimental (criterio de mantener en rango experimental al menos los que ya estaban). En la nueva búsqueda estamos permitiéndolo con la esperanza de que aunque algún marcador salga fuera, el comportamiento global mejore. En este caso la incertidumbre sobre el resultado es algo mayor, pero como podéis ver en la gráfica de debajo, tras las primeras iteraciones del algoritmo, los resultados son bastante prometedores.

Atentamente,
Jesús.


Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2056 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Marcin Szydlowski

Send message
Joined: 24 Sep 22
Posts: 9
Credit: 108,298
RAC: 0
Message 2057 - Posted: 11 Apr 2023, 19:47:26 UTC - in response to Message 2056.  

Good to hear! If I understand correctly, algorithm explores more space than the neighbourhood of local attractors defined by the specific correlation of parameters (correct me if I'm wrong). It looks like the trend will be more smooth and probably will decrease the error. The next weeks will be very exciting!
ID: 2057 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2070 - Posted: 14 Apr 2023, 9:42:35 UTC - in response to Message 2057.  

Hi Marcin,
Yes, the algorithm now explores more space, but based on an polinomical aproximation of the behavior of the model. The polinomical aproximation within the trust region is fixed with all the simulations you send to us.

I am also looking forward to the results of the third iteration.

Best,
Jesús.
Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2070 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Greg_BE

Send message
Joined: 2 Aug 22
Posts: 38
Credit: 1,017,028
RAC: 1
Message 2085 - Posted: 27 Apr 2023, 22:31:19 UTC

Jesus, when will there be new work from your project?
ID: 2085 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2087 - Posted: 28 Apr 2023, 9:34:54 UTC - in response to Message 2085.  

Yesterday... but you are too fast getting the tasks...
Probably, we will send more tasks tomorrow

Best,
Jesús.
Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2087 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
AlterMann

Send message
Joined: 26 Jan 23
Posts: 3
Credit: 289,236
RAC: 0
Message 2089 - Posted: 2 May 2023, 7:55:29 UTC - in response to Message 2056.  

Hi, Jesus

I hope WU to dance on Apple Silicon (macOS Ventura).

----
Chamiu (ΦωΦ)
ID: 2089 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2093 - Posted: 29 May 2023, 16:34:38 UTC

I wish there were a way to supply more work, and more evenly.

When work becomes available, usually once a week, my machine downloads about 100 tasks. I could make it get more, but that would risk failure to complete some on time, so that is not the answer. Alternatively, you could send out work twice a week, perhaps every three or four days. As I understand it, this is not practical either because you need the results from one batch of work before you can send out the next set of work.

Perhaps if you had two related projects, each could send out the same size work once a week, but 3 or 4 days apart, so you could be studying the results of one project while we volunteers could be working on the other. That would probably increase the amount of work you would have to do, of course, and that might be too much to ask for.
ID: 2093 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2094 - Posted: 31 May 2023, 7:58:20 UTC - in response to Message 2093.  

Hi Jean-David,
This is exactly our idea. Little by little, we want to add more optimizations to mantain an stable load of work. Currently, we are running 3 versions of the model optimization (each one with an small variation on the options of the algorithm). One of the limitations is that the postprocessing of your results is done out of the server, and it requieres a manual step, so the three tend to be synchronized because even if one finishes earlier, if that happens at night or during the weekend, the post-processing is done at the same time. Yesterday we sent a lot of work (from the three of them), but I guess they won't last long :-).

I will publish an update of the results the next week (we are finishing the validation process of the first version of the optimization), and I will explain there the new versions we have started (the validation of the first one does not look good, so we will still need your help).

Best,
Jesús.
Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2094 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2095 - Posted: 31 May 2023, 16:07:32 UTC - in response to Message 2094.  

I will publish an update of the results the next week (we are finishing the validation process of the first version of the optimization), and I will explain there the new versions we have started (the validation of the first one does not look good, so we will still need your help).


You will continue getting my help if you continue sending me work units. Currently I have little over 100 DENIS tasks in my machine waiting to go, and I process them four at a time. They seem to take about 65 minutes each on my main machine. And I notice the current batch on your machine is quite large, so I may get some of those too.
ID: 2095 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
10esseeTony

Send message
Joined: 7 Jun 15
Posts: 1
Credit: 21,096,543
RAC: 0
Message 2096 - Posted: 11 Jun 2023, 0:22:19 UTC

Why do you wait until the last moron running DENIS on his IBM 8088 green screen PC returns the final task sent out 9 years ago, before you make another batch of tasks? This is HIGHLY INEFFICIENT.
ID: 2096 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2097 - Posted: 11 Jun 2023, 2:46:14 UTC - in response to Message 2096.  

Why do you wait until the last moron running DENIS on his IBM 8088 green screen PC returns the final task sent out 9 years ago, before you make another batch of tasks? This is HIGHLY INEFFICIENT.


If I have told you once, I have told you a million times, don't exaggerate!

But a bit more seriously, there are more than one way of measuring efficiency, and perhaps the management of this project has a different idea of efficiency than you do. I admit to getting frustrated by running out of work every week, usually after three or four days. So it is not running my machine efficiently (even though I run CPDN, WCG, Rosetta, Einstein, and (grudgingly) MilkyWay and Universe). But maybe management of this project measures efficiency in terms of efficiency of its machines, its people, and other things.
ID: 2097 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2098 - Posted: 12 Jun 2023, 15:06:34 UTC - in response to Message 2097.  

Hello,
It is not only a question of efficiency (although also). To generate a new iteration, we need all the results from the previous iteration, the 45 markers in the 1000 points. We have to wait for all the results. That's why there is a time limit (it varies depending on the simulation but it is between 2 and 3 days). If we give you time, why not wait for the time we have given you? If a user does not send the result in the established time, it is sent to another volunteer.

Are there ways to not expect all the results? Yes, but they imply that part of the simulations that you would do would never be used. That, for us, is a misuse of your resources (efficiency in another sense).

Can some tasks take forever if multiple users don't submit them? Yes, but in those cases we launch the last ones on a server (a few) to finish the iteration. When there are more than 200 left, it is better to wait than to execute them locally. Below that number, we earn time. It's a good balance between how long you/us have to wait and how many simulations are duplicated.

Also remember that between iterations, there is a process that still involves manual actions, so part of the pause is caused by this. We hope to be able to automate as much as possible, but it is again a balance between the technical part and the scientific part of the project. We need to grow in both. We could stop everything and work on the technical part for a long time until it was perfect (without producing scientific results and practically without sending you work beyond betas), or gradually improve the technical part while we use it to already be able to do research.

As already mentioned, luckily there are other projects to which you can give your time. That's why we want to be efficient: If we waste your resources, we don't just take it only from you, we take also from other scientists. This project has had its ups and downs, but we are achieving sustained growth. I think this is the way, although I understand that many of you would like another situation. I also want there to be a more constant flow (and less manual part...)

To all of you, with the bit of your time that you give us, thank you very much.

Best,
Jesús.
Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2098 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2099 - Posted: 12 Jun 2023, 15:55:19 UTC - in response to Message 2098.  

A lot of that falls under what I meant by other definitions of efficiency. But Jesus explained it better that I ever could.

OTOH:
As already mentioned, luckily there are other projects to which you can give your time. That's why we want to be efficient: If we waste your resources, we don't just take it only from you, we take also from other scientists. This project has had its ups and downs, but we are achieving sustained growth. I think this is the way, although I understand that many of you would like another situation. I also want there to be a more constant flow (and less manual part...)


True, there are other projects, but in the last few months, even about a year, important (to me) projects have been having miserable availability of work. The worst has been WCG that was pretty much down for about year, though with brief intervals where work was available. Right now it is sending work on two of their efforts, but nominally they have five. And ClimatePrediction has not been sending out work for quite a while too although now their web site is up reliably; just no work since April except for a few re-runs that timed out from other users.

As a result, I am now running a lot of MilkyWay and Universe tasks that are of little interest to me, but at least they are pretty reliable about keeping their servers up and supplying work.
ID: 2099 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Marcin Szydlowski

Send message
Joined: 24 Sep 22
Posts: 9
Credit: 108,298
RAC: 0
Message 2100 - Posted: 12 Jun 2023, 16:38:26 UTC - in response to Message 2099.  
Last modified: 12 Jun 2023, 16:40:40 UTC

If you're interested, Einstein@Home guarantee constant work for different sub projects. Server status shows that current batches of tasks need at least few months.
ID: 2100 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2101 - Posted: 14 Jun 2023, 14:18:11 UTC - in response to Message 2100.  

If you're interested, Einstein@Home guarantee constant work for different sub projects. Server status shows that current batches of tasks need at least few months.


Maybe an entire batch requires a few months, but individual tasks require only seven to eight hours on my Linux machine.

Computer 224473
CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16
Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.7 (Ootpa) [4.18.0-425.13.1.el8_7.x86_64|libc 2.28]
BOINC version 	7.20.2
Memory 	125.34 GB
Cache 	 16896 KB
Swap space 	15.62 GB
Total disk space 	488.04 GB
Free Disk Space 	473.01 GB
Measured floating point speed 	6.04 billion ops/sec
Measured integer speed 	       25.59 billion ops/sec

ID: 2101 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000
Avatar

Send message
Joined: 19 Jan 17
Posts: 12
Credit: 73,572
RAC: 0
Message 2102 - Posted: 14 Jun 2023, 14:58:27 UTC - in response to Message 2101.  

Why does this project block Russian IPs?
ID: 2102 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Marcin Szydlowski

Send message
Joined: 24 Sep 22
Posts: 9
Credit: 108,298
RAC: 0
Message 2104 - Posted: 14 Jun 2023, 21:07:47 UTC - in response to Message 2101.  

So are you looking for tasks which require 20+ hours? :)
ID: 2104 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jean-David Beyer

Send message
Joined: 6 Mar 23
Posts: 34
Credit: 2,078,354
RAC: 3
Message 2105 - Posted: 15 Jun 2023, 19:40:49 UTC - in response to Message 2104.  

So are you looking for tasks which require 20+ hours? :)


Yes, but I am not looking for them here. I have ClimatePrediction for that. Latest ones have taken about 10 days each. But they lately sending out very little work. Here is my latest one.


Task     Work unit Sent     	               Reported	            Status	Run time (sec)	CPU time (sec)	Credit	  Application
22318648 12138603  30 May 2023, 3:38:46 UTC 9 Jun 2023, 1:20:39 UTC Completed 	852,578.34 	843,274.30 	33,854.34 UK Met Office HadAM4 at N216 resolution v8.52
i686-pc-linux-gnu

ID: 2105 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jesús Carro
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 18 Mar 15
Posts: 281
Credit: 2,748,608
RAC: 9
Message 2106 - Posted: 16 Jun 2023, 7:40:54 UTC - in response to Message 2102.  

Why does this project block Russian IPs?


We are not or not in general. We have a firewall to protect the web page: we received several atacks years ago. This is managed by the IT deparment. If you indicate me the IPs you are having problems with, I can ask them.

Best,
Jesús.
Jesús Carro
Universidad San Jorge
@InSilicoHeart
ID: 2106 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kotenok2000
Avatar

Send message
Joined: 19 Jan 17
Posts: 12
Credit: 73,572
RAC: 0
Message 2108 - Posted: 16 Jun 2023, 9:42:55 UTC - in response to Message 2106.  

Responded in personal message.
ID: 2108 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : News : Status of the project // Estado del proyecto