Unable to download data from web services on compute nodes, only from login on Yggdrasil

HPC Technical
1

Dear HPC Staff,

I’m trying to download data from a seismological data web service server. The server address is http://eida.ethz.ch. I am only able to connect via the login node on Yggdrasil (I know computations aren’t allowed there, it was just for testing :slight_smile: ), and none of the compute nodes I have tried can connect (connection hangs and times out). Is this caused by the way the nodes are configured ?

To reproduce this error, one can try this command:

(obspy) (yggdrasil)-[savardg@login1 ~]$ wget "http://eida.ethz.ch/fdsnws/station/1/query?net=CH&sta=*&format=xml&level=channel&latitude=47.5745&longitude=7.6425&maxradius=.15&nodata=404" -O test.xml
--2022-07-26 18:13:12--  http://eida.ethz.ch/fdsnws/station/1/query?net=CH&sta=*&format=xml&level=channel&latitude=47.5745&longitude=7.6425&maxradius=.15&nodata=404
Resolving eida.ethz.ch (eida.ethz.ch)... 129.132.144.217
Connecting to eida.ethz.ch (eida.ethz.ch)|129.132.144.217|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/xml]
Saving to: ‘test.xml’

    [ <=>                                                                                                                ] 327,959     --.-K/s   in 0.02s

2022-07-26 18:13:12 (15.2 MB/s) - ‘test.xml’ saved [327959]

On the compute nodes, it hangs instead:

(obspy) (yggdrasil)-[savardg@cpu003 ~]$ wget "http://eida.ethz.ch/fdsnws/station/1/query?net=CH&sta=*&format=xml&level=channel&latitude=47.5745&longitude=7.6425&maxradius=.15&nodata=404" -O test.xml
--2022-07-26 18:03:04--  http://eida.ethz.ch/fdsnws/station/1/query?net=CH&sta=*&format=xml&level=channel&latitude=47.5745&longitude=7.6425&maxradius=.15&nodata=404
Resolving eida.ethz.ch (eida.ethz.ch)... 129.132.144.217
Connecting to eida.ethz.ch (eida.ethz.ch)|129.132.144.217|:80... failed: Connection timed out.
Retrying.

--2022-07-26 18:05:12--  (try: 2)  http://eida.ethz.ch/fdsnws/station/1/query?net=CH&sta=*&format=xml&level=channel&latitude=47.5745&longitude=7.6425&maxradius=.15&nodata=404
Connecting to eida.ethz.ch (eida.ethz.ch)|129.132.144.217|:80... failed: Connection timed out.
Retrying.

Kind regards,

Genevieve

2

Perhaps my issue might be related to this one ?

3

In my case, the problem was mainly related to extremely slow access (read and write) to the shared storage (/home and /home/scratch). The behavior was the same for login node and compute nodes.

A test of whether it is a shared storage problem was downloading to a local drive, for example, the /tmp directory. For you, this can be a solution → download to /tmp of compute node (or login node) and then copy to shared storage (will take time but eventually will finish).

EDIT: I just checked, and it looks like compute node can not even establish a connection to eida.ethz.ch. My guess would be it is something firewall-related.

1 Like
4

Hi,

we were contacted by the mainteners of this web service service a couple of days ago because they saw too many requests coming from Yggdrasil.

They said:

Could you please forward this information to the appropriate user and tell him or her to limit the number of connections to 1

As it may happens that you are the user, you are now informed. My guess is that they now limit the number of connections to their service.

1 Like
5

Thank you for your replies @Yann.Sagon and @maciej.falkiewicz. Looks like I’m limited to a serial download with this server.