Exception in thread "main" java.lang.NullPointerException | Java Community | Help. Code. Learn. | Page 1

astral grove Mar 14, 2023, 11:06 AM

#

Trying to read files from arg[0] location and wordcount summary of all files to arg[1] by using Hadoop in AWS ElasticMapReduce. Working in local window environment but failed on AWS

    at bow.Bow.main(Bow.java:78)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)```

📎 bow.txt

summer dirgeBOT Mar 14, 2023, 11:06 AM

#

⌛ This post has been reserved for your question.

Hey @astral grove! Please use /close or the Close Post button above when you're finished. Please remember to follow the help guidelines. This post will be automatically closed after 300 minutes of inactivity.

TIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here.

astral grove Mar 14, 2023, 11:08 AM

#

📎 controller.txt

#

#

📎 pom.txt

#

untold fractal Mar 14, 2023, 11:21 AM

#

astral grove Trying to read files from arg[0] location and wordcount summary of all files to ...

If you take a look at like 78 of Bow, you can see it's for (int i=0;i<files.length;i++) {

#

files is null

#

probably because the file specified in args[0] doesn't exist

astral grove Mar 14, 2023, 11:23 AM

#

astral grove

The step have argument s3://zacattempt1/testFiles

#

Im afraid .listFiles() dosent works in linux

untold fractal Mar 14, 2023, 11:49 AM

#

astral grove The step have argument `s3://zacattempt1/testFiles`

this is not a valid File

untold fractal Mar 14, 2023, 11:49 AM

#

astral grove Im afraid `.listFiles()` dosent works in linux

it does but not for s3:// URLs

#

it works for files and directories

#

like /home/youruser/somefile

astral grove Mar 14, 2023, 11:52 AM

#

It works for offical hadoopwordcountexample

untold fractal Mar 14, 2023, 12:05 PM

#

That probably doesn't use File

#

but instead URL or something like that I guess

astral grove Mar 14, 2023, 12:33 PM

#

    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
    at java.nio.file.Files.newDirectoryStream(Files.java:457)
    at bow.Bow.main(Bow.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)```
```java
java.nio.file.Path dir = FileSystems.getDefault().getPath( args[0] );
 DirectoryStream<java.nio.file.Path> stream = Files.newDirectoryStream( dir );
  for (java.nio.file.Path path : stream) {}```

#

Trying new stuff and still failed

astral grove Mar 14, 2023, 12:53 PM

#

Turns out wildcard in augment works with hadoop addInputPath

    FileInputFormat.addInputPath(job, new Path(args[0]));

augment s3://zacattempt1/testFiles/file*.txt

#Exception in thread "main" java.lang.NullPointerException