Services.gradle.org seems blocked on automated build

I cannot figure out why the creation of a snap (geogebra-discovery) fails on the automated build server:

Downloading https://services.gradle.org/distributions/gradle-6.7.1-bin.zip

Exception in thread "main" java.net.UnknownHostException: services.gradle.org
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)

I guess Gradle wants to connect to some servers, maybe services.gradle.org, but this seems blocked. I use my own build machinery to build the application, so I don’t use the gradle plugin.

A similar issue was reported at https://stackoverflow.com/questions/51244696/unable-to-access-gradle-plugin-url-with-snapcraft, but that one used the gradle plugin.

Thanks for any suggestion in advance.

Ancient post I know, but in case it helps anyone else (and as the author of that linked Stackoverflow question which I’ve since updated) - use -Djava.net.useSystemProxies=true as an additional option to Gradle.

Thanks! Finally I learned that if the gradle plugin is used, things just work (connection to outside is allowed). Otherwise, connections to Gradle repositories are blocked.

Just bear in mind that the Gradle plugin is sadly not compatible with the newer corexx bases. I used to use that Gradle plugin, but had to shift away from it (and thus do some more investigating / debugging to work out what was going on.)

Good news though is with the above option you can avoid the plugin entirely if you wish :slight_smile:

1 Like

Unfortunately, this option does not work in core22 at the moment. I had to use Remote build fails to resolve 'services.gradle.org' - #9 by norse-dreki, but it does not work locally, just when the server builds the snap.

I hit the same issue here building on core24 since my build needs to use “./gradlew”.

I tried adding the -Djava.net.useSystemProxies=true but that didn’t help.

This is output from the launchpad/snapcraft build.

Do you know if this is specific to core24?

In other words, was this a snap that previously was a core22 snap where Launchpad was able to use ./gradlew?

I was able to get the same error on core22, yes. But also, I don’t want to run on core22 - so I guess its not really an option either.

Is there a reason downloading

:: Downloading https://services.gradle.org/distributions/gradle-8.7-bin.zip

Is prohibited in the build environment?

It works just fine at my local workstation:

wget https://services.gradle.org/distributions/gradle-8.7-bin.zip
--2024-07-11 20:27:40--  https://services.gradle.org/distributions/gradle-8.7-bin.zip
Resolving services.gradle.org (services.gradle.org)... 104.16.73.101, 104.16.72.101, 2606:4700::6810:4965, ...
Connecting to services.gradle.org (services.gradle.org)|104.16.73.101|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/gradle/gradle-distributions/releases/download/v8.7.0/gradle-8.7-bin.zip [following]
--2024-07-11 20:27:40--  https://github.com/gradle/gradle-distributions/releases/download/v8.7.0/gradle-8.7-bin.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/d6619bcc-7f88-47a8-8f67-28fc5c020703?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240711%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240711T182740Z&X-Amz-Expires=300&X-Amz-Signature=d2fc80895c4b666e54d413f5d7cfc4d3983f2b58f2710a9ca578fa0ab3de2c7f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-8.7-bin.zip&response-content-type=application%2Foctet-stream [following]
--2024-07-11 20:27:40--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/d6619bcc-7f88-47a8-8f67-28fc5c020703?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240711%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240711T182740Z&X-Amz-Expires=300&X-Amz-Signature=d2fc80895c4b666e54d413f5d7cfc4d3983f2b58f2710a9ca578fa0ab3de2c7f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-8.7-bin.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134184980 (128M) [application/octet-stream]
Saving to: ‘gradle-8.7-bin.zip.2’

gradle-8.7-bin.zip.2                                 100%[====================================================================================================================>] 127,97M  51,5MB/s    in 2,5s    

2024-07-11 20:27:43 (51,5 MB/s) - ‘gradle-8.7-bin.zip.2’ saved [134184980/134184980]

The infra uses a HTTP/HTTPS proxy. These variables are set, as $http_proxy and $https_proxy, and many apps respect them.

Some however don’t, often substituting in their own config.

Electron apps are common for this, which is where you’ll see e.g:

override-build: |
      if [ -n "${http_proxy:-}" ]; then # Sorts out network proxy issues on Launchpad builds
        export ELECTRON_GET_USE_PROXY=1
        export GLOBAL_AGENT_HTTP_PROXY="${http_proxy:-}"
        export GLOBAL_AGENT_HTTPS_PROXY="${http_proxy:-}"
        export YARN_HTTP_PROXY="${http_proxy:-}"
        export YARN_HTTPS_PROXY="${http_proxy:-}"
      fi

Check if Gradle has an equivalent.

From what I’m seeing, likely:

gradlew -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=3129

Where you’d substitute with the Launchpad ones (reason I haven’t here is because I don’t have time to check how that’s formatted, you might need to manipulate the string).

1 Like

Right, I’m not sure where I can find those settings but certainly a good lead for me. Thanx alot!

I found some information about gradle and proxy, but I can’t find any information about the system proxy used within launchpad…

https://docs.gradle.org/current/userguide/networking.html

You can see how the old plugin used to do it, in Python, here:

Functionally it’s just translating that into Bash; except rather than appending to the build command, it’s likely easier to just issue as a separate command/config line.

By the looks of that plugin, I expect the format is 1.2.3.4:8080, I.E ip:port, where the cut command could split the two easily, e.g:

cut $http_proxy -d ':' -f 1 for IP

cut $http_proxy -d ':' -f 2 for port

In the end, the solution was very involved.

  1. Figure out that there was a proxy.
  2. Figure out which kind of proxy.
  3. Figure out the settings for the proxy.
  4. Figure out how to use it with gradlew.
  5. Figure out how to use it with the Canonical build environment from within the snap.

It ended up me having to write a supporting script to dig out the settings.

#!/bin/sh

get_proxy_options() {
    proxy_options=""

    for var in http https; do
        proxy=$(printenv "${var}_proxy")

        if [ -n "$proxy" ]; then
            # Remove trailing /
            proxy=$(echo "$proxy" | sed 's|/$||')
            # Extract hostname and port using POSIX-compatible tools
            proto="$(echo $proxy | sed -e's,^\(.*://\).*,\1,g')"
            url=$(echo $proxy | sed -e s,$proto,,g)
            hostname=$(echo $url | cut -d: -f1)
            port=$(echo $url | cut -d: -f2)
            proxy_options="${proxy_options} -D${var}.proxyHost=${hostname}"
            if [ -n "$port" ]; then
                proxy_options="${proxy_options} -D${var}.proxyPort=${port}"
            fi
        fi
    done

    echo "$proxy_options"
}

# Call the function and store the result
proxy_options=$(get_proxy_options)

# Print the proxy options
echo "$proxy_options"

Then finally use it with gradlew as below:



override-build: |
      echo "BEGIN BUILD OF BESU - gradlew in override-build"

      echo "Extracting potential https_proxy and/pr http_proxy as needed when building with Canonical buildfarm"
      PROXY_OPTS=$($SNAPCRAFT_STAGE/utils/getproxy.sh)
      GRADLE_OPTS="-Djava.net.useSystemProxies=true $PROXY_OPTS" ./gradlew --no-daemon installDist
      
      echo "COMPLETED BUILD OF BESU - gradlew"

This was a nightmare and took me about two full days to mess with since it takes forever to test every small change.

1 Like