Right now we only warn/error/debug log. It would be nice to introduce some INFO logging as well. We may not particularly want to write the info logs to file (as that log could get quite big) - but it wouldn’t hurt to have it for the purposes of watching the console to determine what is doing what and when.
This could be for a future version of spark of course.
So. You are relly suggesting to have info console? Isn’t debug console doing this already? You can turn on debug mode in Advanced settings and restart Spark. Then debug window will appear on a next start and you can see some stuff there as sent/received packets, etc.
For me slf4j / log4j would be fine. So one could decide what to log and hopefully change the log level while Spark is running. Actually info and debug logs are usually needed only in a few classes if things seem to go wrong.
Yeah, i’m probably in the minority here and I understand that; however, there are definite and different uses for Debug/error/warn/info logging. We’re using all but INFO logging. I know it is probably more of a cosmetic thing to most, but for some it helps a ton to just see what’s going on as it is going on.
A good example (albeit probably not great example for spark) is a server environment. Where you want specific bits of information always pumped out to the console.
Right now spark doesn’t do a lot of logging in the background unless something is blowing up, which is fine for finding out what happens when it blows up - but info logging allows us to see what’s going on prior to the blow up. just a thought for the future