Problems installing Spark 2.9.4 on Fedora 34

Colin_Riddel · August 30, 2021, 2:32am

I ma hoping someone can give me some guidance here,
I am trying to install Spark on a new Fedora 34 system
-------
First, the RPM

[root@grumpy xmpp]# rpm -ivvv spark-2.9.4.rpm 
ufdio:       1 reads,    17654 total bytes in 0.000003 secs
ufdio:       1 reads,     5684 total bytes in 0.000001 secs
ufdio:       1 reads,    17654 total bytes in 0.000019 secs
D: ============== spark-2.9.4.rpm
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: PRAGMA secure_delete = OFF: 0
D: PRAGMA case_sensitive_like = ON: 0
D:  read h#    1694 
Header SHA256 digest: OK
Header SHA1 digest: OK
D: added key gpg-pubkey-45719a39-5f2c0192 to keyring
D:  read h#    2534 
Header SHA256 digest: OK
Header SHA1 digest: OK
D: added key gpg-pubkey-07d7d182-608af092 to keyring
D:  read h#    2539 
Header SHA256 digest: OK
Header SHA1 digest: OK
D: added key gpg-pubkey-94843c65-5dadbc64 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: spark-2.9.4.rpm: Header SHA256 digest: OK
D: spark-2.9.4.rpm: Header SHA1 digest: OK
error: spark-2.9.4.rpm: invalid signature tag Archivesize (1046)
ufdio:       5 reads,    21556 total bytes in 0.000006 secs
error: spark-2.9.4.rpm cannot be installed
D: 	added source package [0]
D: found 1 source and 0 binary packages
D: Exit status: 1
[root@grumpy xmpp]#

--------------------------
then cloning the source from git
I get the following error

[root@grumpy core]# mvn exec:java
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------< org.igniterealtime.spark:spark-core >-----------------
[INFO] Building Spark Core 3.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:sparkEmoticonSet:3.0.0-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:POPO.adiumemoticonset:3.0.0-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:GTalk.AdiumEmoticonset:3.0.0-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:Default.adiumemoticonset:3.0.0-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.263 s
[INFO] Finished at: 2021-08-30T14:12:19+12:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-core: Could not resolve dependencies for project org.igniterealtime.spark:spark-core:jar:3.0.0-SNAPSHOT: The following artifacts could not be resolved: org.igniterealtime.spark:emoticons:zip:sparkEmoticonSet:3.0.0-SNAPSHOT, org.igniterealtime.spark:emoticons:zip:POPO.adiumemoticonset:3.0.0-SNAPSHOT, org.igniterealtime.spark:emoticons:zip:GTalk.AdiumEmoticonset:3.0.0-SNAPSHOT, org.igniterealtime.spark:emoticons:zip:Default.adiumemoticonset:3.0.0-SNAPSHOT: Failure to find org.igniterealtime.spark:emoticons:zip:sparkEmoticonSet:3.0.0-SNAPSHOT in https://oss.sonatype.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of maven-snapshots-repo has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

guus · September 2, 2021, 7:38am

These are two different problems. The first problem seems to be an issue with the RPM that we distribute. That needs looking into. To avoid any confusion, can you identify the exact file that you’re using please?

The latter problem probably occurs because you’re trying to build the latest version of the code, instead of the code that is tagged as a release. Try switching to the v2.9.4 tag!

guus · September 2, 2021, 7:46am

This post from an unrelated project seems to describe a similar issue: invalid signature tag Archivesize on rpm packages created by install4j · Issue #1635 · rpm-software-management/rpm · GitHub

We have two paths to build RPMs: one through a custom script, and one through Install4j. I’m thinking that the release RPM is build using the custom script (but I’m not sure - @akrherz?).

It is worth trying a nightly build of Spark, which is generated from Install4j. You can find those here: Ignite Realtime: Spark Nightly Builds

akrherz · September 2, 2021, 12:56pm

We only use install4j’s RPM these days.

Colin_Riddel · September 2, 2021, 8:31pm

Good Morning
I have found the following that seems to explain the issue
rpm is not recognising install4j signatures as valid
Awaiting the RPM maintainers to update

github.com/rpm-software-management/rpm

invalid signature tag Archivesize on rpm packages created by install4j

opened 03:52PM - 13 Apr 21 UTC

closed 08:17AM - 26 May 21 UTC

openaudible

REGRESSION

Install4j is a popular installer maker. It appears that some or all rpm's create…d by them are not compatible with the latest rpm 4.16.1.3 installed with fedora. rpm works as expected in versions 4.16.0 and earlier. Example: rpm -qvvpR install4j_linux-x64_9_0_1.rpm returns error: run invalid signature tag Archivesize (1046) Below is a simple Dockerfile that shows how to reproduce this: ``` # Dockerfile to demonstrate rpm bug with latest rpm or install4j # To test, save this as "Dockerfile" and run: # docker build -t testrpm . # to Debug in bash, docker run -it testrpm FROM fedora:latest WORKDIR /test RUN yum -y install wget RUN wget https://download-gcdn.ej-technologies.com/install4j/install4j_linux-x64_9_0_1.rpm RUN echo "rpm -qvvpR install4j_linux-x64_9_0_1.rpm" > test.sh && chmod +x test.sh # Run and ignore the expected error RUN ./test.sh;exit 0 ENTRYPOINT ["/bin/bash"] ``` This was found by a user in issue: https://github.com/openaudible/openaudible/issues/603 A work-around for is to downgrade rpm from 4.16.1.3 to 4.16.0 with the simple command: yum downgrade rpm which you can also test by running the above docker: ``` docker run -it testrpm ./test.sh # fail yum -y downgrade rpm ./test.sh # success ``` These are reproducible using the "latest" versions as of 4/13/2021.

Thanks for your help

Colin_Riddel · September 2, 2021, 9:16pm

I have retried installing from source for versions 3.0.0-beta, 2.9.4, 2.9.3 and 2.9.2 all with the same 4 files missing.

WARNING] The POM for org.igniterealtime.spark:emoticons:zip:sparkEmoticonSet:3.0.0-beta is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:POPO.adiumemoticonset:3.0.0-beta is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:GTalk.AdiumEmoticonset:3.0.0-beta is missing, no dependency information available
[WARNING] The POM for org.igniterealtime.spark:emoticons:zip:Default.adiumemoticonset:3.0.0-beta is missing, no dependency information available

Am I missing something here

Cheers
Colin

guus · September 3, 2021, 7:34am

This problem seems to have been fixed in Install4j 9.0.2 (see last item in their changelog). I suspect that upgrading that (and generating new installers), would resolve the issue. I’m unsure if that means that we need to update install4j configuration in individual projects (to remain compatible), and/or if our license covers the new version. We’ll work on that, but I’d like @akrherz to be at least awake while I’m potentially breaking things.

Our buildserver currently has a lot of different versions of install4j installed (I’m unsure if that’s intended, or an artifact of earlier upgrades), but uses version 8.0.10.

guus · September 3, 2021, 8:00am

I was impatient and tried a thing. @Colin_Riddel would you mind giving the RPM in this build a test run? Install4j-generated-media

@akrherz - what I did:

Obtained TAR.GZ archive of Install4j 9.0.4 installer from ej-technologies - Java APM, Java Profiler, Java Installer Builder
Extracted in directory that contains all other Install4j instances
Added symbolic link from install4j9.0.4/jres to the shared jres directory
executed install4j9.0.4/bin/install4jc --license=<OURLICENSEKEY>

To test a build without affecting others, I’ve changed the argument that points to the to-be-used install4j home directory in the Spark Nightly Build plan from to the new installation directory (rather than reconfiguring the symbolic link that the plan was using before.

Colin_Riddel · September 6, 2021, 2:23am

Good Afternoon
Apologies for the delay in getting back to you, I have been away for a couple of days

I grabbed that RPM and it installed fine, I had a bit of fiddling needed to set the JVM path as it is expecting INSTALL4J_JAVA_HOME not JAVA_HOME.
it fails without any error messages when run from the gnome4 menu

Regards
Colin