#Segmentation fault when searching in large amount of big csv files

5 messages · Page 1 of 1 (latest)

sour salmon
#

Hello everyone,
I have this program that i have been developing for a lot of time that is supposed to check in multiple CSV like files if some specific type of data is present.
As a example lets say with have multiple csv files one with the columns username,score and one with username,language
If we search for all the users with the language (fr) the program is supposed to check in all the CSV files for this collumn
Basicly it is a fucked SQL. (Before telling me so i have some obligations to do it like this)

My problem is a SIGSEGV (Segmentation fault) when searching in a lot of large files (RAM didn't seem to get overloaded when i check with HTOP)

Thread 1 "program" received signal SIGSEGV, Segmentation fault.
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:262
262    ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.

Specific code triggering the error:

lofty rivetBOT
#

When your question is answered use !solved to mark the question as resolved.

Remember to ask specific questions, provide necessary details, and reduce your question to its simplest form. For tips on how to ask a good question use !howto ask.

sour salmon
# 
for (std::filesystem::recursive_directory_iterator i(_folder), end; i != end; ++i) {
        if (!is_directory(i->path())) {
            filestream.open(i->path(),std::ios::in);
            if (filestream.is_open()){
                std::string tp;
                int collumnNumber = -1;
                std::vector<std::string> header;
                while (std::getline(filestream, tp)){
            tp = StringOperation::strip(tp, '\r');
                    std::vector<std::string> splited = StringOperation::split(tp, ',');
                    if (collumnNumber == -1) {
                        header = splited;
                        for (int i = 0; i != splited.size(); i++) {
                            if (splited[i].compare(collumn) == 0) {
                                collumnNumber = i;
                            }
                        }
                        if (collumnNumber == -1)
                            break;
                    } else {
                        if (splited[collumnNumber].compare(value) == 0) {
                            for (int i = 0; i != splited.size(); i++) {
                                if (i != collumnNumber) {
                                    if (header[i].compare(COLLUMN) == 0) {
                                          return splited[i];
                                    }
                                        
                                }
                            }
                        }

                    }
                }
                filestream.close();
            }
        }
    }
warped falcon
#

There are 3 problems that could cause a segfault here that I can see:

  1. The i being passed in is invalid
  2. Some of the headers are escaped with “” and contain ,s, leading to an incorrect index
    Some lines not containing enough data that they do not contain your index (especially if the files have a trailing new line)
#

Also, ideally you should probably use a debugger for checking segfaults, since some are pretty hard to find