I didn’t see if this had been entered in as an issue but I noticed that my Conversation HIstory is no longer showing more than the 5k message limit, but in the wrong direction!
I can see the really old messages up through to 5k messages, but new messages are of course removed.
Looks like the removing of messages may want to go in the other direction. This is happening on all of the workstations running the latest version of spark (2.6.3).
I can’t find the exact svn update, but i remember Wolf adding some 5k limit, because history was freezing after that limit. Probably he should do it in a reverse.
Well, since there seems to be the need for large history files and history is stored in XML, we arerunning into a problem. Large XML file can not be parsed fast. This caused complains about the loading times. This caused the fix: http://issues.igniterealtime.org/browse/SPARK-1361 and http://bamboo.igniterealtime.org/browse/SPARK-INSTALL4J-389/commit
Either someone comesup with a faster processor for the history XML or we move the history file to plain text and use the system editor (notepad or whatever) to display this text file.
hah well maybe i just chat wayyy to much with my friends at work. The issue is actually with the JEditorPane - as it turns out the performance is horrendously slow when displaying very large blocks of tests. After doing some pretty basic performance testing I found the loading/parsing of the entire XML file takes under a second or so, the huge block of time mostly comes from the displaying. Though I’m on a mac so I’m not sure if maybe the mac is taking a long time to draw.
That said, i have a very dirty patch for SPARK-1363 that can be tested, it should also (in your consoles) display the time it’s taking to do the various parts of the job. Take a look and see, the largest hit is on window.setText which is painfully slow since it must update the drawing after each line of text. vs setPage() which updates the drawing after loading the entire Document.
using setPage requires the transcript be persisted to HTML before loading into the view however, so they both have their ups/downs.
I’ll get the patch attached asap.
I went ahead and created a ticket for fixing the current behavior SPARK-1407 (reversing the limit)
Any ETA on this fix?
Nobody is currently working on this, so no ETA, sorry.
looks like some minor improvements using invokeLater on the text. cut the time by a noticable amount on my system. others may want to test to see how it affects their system.
Well, it takes around 20 secs to load 1.33 MB xml of history for me. With or without the patch. The only difference, and probably for the best, is that with the patch it won’t show anything until it is loaded. Because currently it shows you a bit of history and then you wait for 20 seconds for scrollbar to appear. On the other hand, watching at the white window for 20 seconds may convince that it’s just freezed.
it’s still so strange that 1.33MB of text would cause such a headache for a text area. especially once it’s been invoked into it’s own thread and handled outside of the UI thread (why it doesn’t show up until after it’s been completely loaded).
Something else has to be going on.
It’s probably not fair on my system, on an SSD it pops up in about .7 seconds
EDIT: to add, i noticed (maybe i noted this before) if we go with an unstylized look the text loads near instantaneous.
I don’t think that loading such a small file should be so hard for a HDD. Yes, something else must be going on. Maybe the file itself is causing it, not the size of it. It is a real history (3-4 years of chatting on different spark versions). I don’t see difference while using Windows appearance.
so i went and did quite a bit of testing today. No it’s definitely not the size of the file.
It is the fact that we’re stylizing it that is causing the issue. If we swap to a JTextArea (without any styles) the text loads near instantly.
The HTMLEditorKit that we are using is causing the slow downs. I am experimenting with changing up the HTML to something a bit more modern (since it is using one giant table, and if the JVM parses tables the way the w3 standards expect then it makes complete sense why it would take so long to render a 1MB table).
So far i’ve had some moderate success, but in reading online, it’s actually considered good practice to implement or override the stylededitorkit’s and only implement the tags / features needed to expedite parsing.
Never would have thought that such a simple problem would be that hard to fix…
This is one of the times I curse the fact that I can’t code…
Yes, using table is not a good idea…
How about spliting the entries in pages before loading them into the pane? For example 2k entries per page and buttons to navigate through all available pages?
That would be fine by me.
it might be more beneficial to do it by date and not a message limit. e.g.
messages 1 day old
messages 2days to 1 week
messages 1 week to month
Messages older…
I say this only because there were a lot of times when I needed to get a message that happened at a specific point in time and i can usually remember it happened some time in the last month or so, but not the exact day / time.
Though honestly we can probably fix the stylededitorkit faster than redesign the UI and how it loads the html too
The problem is, if you break through the 5k limit,or how high you set it, you will lost some of your messages in the history. I guess it doesnt matter if the old or the new ones… i would prefer to see my latest messages in the search results.
But you are right, the stylededitorkit should be fixed faster then the gui and to sort by date sounds indeed like a good idea.
I think a thing we could fix fast is to do the search in 5k blocks. If we have 5500 entries in the history and the first and the last is a hit, the current search would give us only the first entry. If we search through the first 5000 to get the first hit and then search through the last 500 entries to get the next hit would be an improvement.
I wrote a first patch for testing the search in 5k block, what do you think?
ChatTranscriptPlugin.java.patch.zip (1799 Bytes)
Sorry for posting twice but i cant edit posts on my iPhone.
If we can do a search on the whole history, is there a reason for displaying a amount of 5000 entries?
Maybe we should just add the function to search by date and display a maximum of, for example 1000, messages with the info that there are some more if there are to much results.
Or is there a benefit by displaying this much entries i dont get?
I have tried with and without that patch and it seems that the order is different, but when i search i think i find the same results. Well, maybe i don’t have 5k messages in that history. Btw, you are talking about the search, but the main topic here was the whole history. Sometimes i want to see all my history, not just search for something. Pagination is ok with me, either by messages number or by date.